JP2005260731A - Camera selecting device and camera selecting method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To select an optimum camera corresponding to the viewpoint of an animal from among a plurality of cameras. <P>SOLUTION: In this image processing device 10 when a viewer's position is specified, a viewer position similarity operating section 14 calculates an area of ▵C<SB>k</SB>PQ formed by the viewer's position P, the position of a camera C<SB>k</SB>and a viewpoint Q. At the same time, the section 14 calculates an angle formed by an optical axis center vector e<SB>k⊥</SB>of the camera Ck and a vector PQ. A camera selector 15 selects two Cks in an order of cameras having a smallest value obtained by substituting the area and the angle into a predetermined evaluation formula E(C<SB>k</SB>). The two selected cameras Ck<SB>min1</SB>and Ck<SB>min2</SB>photograph a field A to be photographed centered at the viewpoint Q, and output photographed images to an arbitrary viewpoint image generator 16. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a technique for photographing a moving object using a plurality of cameras arranged at different positions.

2. Description of the Related Art Conventionally, a system for photographing a moving object existing in an object scene using cameras installed at a plurality of places has been put into practical use. For example, Patent Document 1 discloses a lens selection device that can easily select a combination of a camera and a lens according to a user's request. In this apparatus, when selecting a lens, shooting conditions such as the camera screen size, shooting distance, focal length, subject size, and aperture method are taken into consideration. When a lens that matches the photographing condition is searched, this is displayed on the display device, and the lens used for photographing is determined after waiting for selection by the user.
JP-A-10-257361

  However, the above-mentioned prior art is mainly intended for photographing a moving body including surveillance, and is not intended to provide an image viewed from an arbitrary viewpoint, and therefore has the following problems. It was. In other words, the characteristics of individual cameras are taken into account as the camera selection criteria, but the degree of similarity between the camera viewpoint and the viewpoint of the moving object or the resolution characteristics of the camera according to the position is not correlated. It was not considered enough. For this reason, the optimal camera is not always selected from the viewpoint of providing the user with high-quality imaging.

  Therefore, an object of the present invention is to select an optimal camera corresponding to the viewpoint of a moving object from a plurality of cameras.

  In order to solve the above-described problems, a camera selection device according to the present invention is specified by a specifying unit that specifies the position of a viewer (animal body) in an object field that can be captured by a plurality of cameras, and the specifying unit. The area of the triangle formed by the viewer position, the camera position, and the viewpoint, and the angle formed by the optical axis center vector of the camera and the vector from the viewer position to the viewpoint are related to the plurality of cameras. Calculation means for calculating, and selection means for selecting a camera that outputs a captured image centered on the viewpoint using a calculation result by the calculation means.

  The camera selection method according to the present invention includes a specifying step of specifying a position of a viewer (animal body) in a field that can be photographed by a plurality of cameras, a viewer position specified in the specifying step, and a camera A calculation step of calculating an angle formed by the area of the triangle formed by the position and the viewpoint, the optical axis center vector of the camera, and the vector from the viewer position to the viewpoint with respect to the plurality of cameras; And a selection step of selecting a camera that outputs a captured image centered on the viewpoint using the calculation result in the calculation step.

  Since these inventions use a triangular area formed by the specified viewer position, the camera position, and the viewpoint when selecting a camera, a camera having a viewpoint that is more similar to the viewpoint from the viewer position Can be selected. In addition, since the angle between the optical axis center vector of the camera and the vector from the viewer position to the viewpoint is used in combination, it is possible to simultaneously select the camera considering the resolution characteristics of the camera according to the shooting direction. . The camera selected in this way is a camera that can output a high-quality captured image from the specified viewer position. That is, according to these inventions, an optimal camera can be selected from a plurality of cameras according to the viewpoint of the moving object.

  Preferably, in the camera selection device according to the present invention, the selection unit uses the calculation result to select a plurality of cameras that output a photographed image centered on the viewpoint, and occlusion occurs in photographing with one camera. In such a case, the image processing apparatus further includes a generation unit that generates a presentation image by applying an image captured by another camera to the occlusion portion.

  Depending on the shape of the object scene and the intervention of the shield, there may be a blind spot even in the object scene, and there may be a viewpoint that cannot be taken with one camera. Assuming such a situation, the present invention avoids the occurrence of occlusion by complementing an image portion that has become a blind spot from one camera with an image captured by another camera. Thereby, for example, a captured image regarding an area invisible to the user who is present in the scene can be presented to a user located remotely.

  In the camera selection device according to the present invention, more preferably, when there are image portions having different resolutions as a result of generating the presentation image by the generation unit, smoothing according to the resolution of each pixel is performed. Image processing means is further provided.

  When image smoothing processing (smoothing processing) is uniformly performed using low-resolution pixel values, there is a concern that the image quality of the high-resolution portion deteriorates accordingly. Such a concern is particularly noticeable when a presentation image is generated by supplementing occlusion of a captured image with a high resolution with an image with a lower resolution. According to the present invention, since the image smoothing process is performed according to the resolution of each pixel constituting the presentation image, the image quality of the entire presentation image is improved while maintaining the image quality of the high-resolution image portion. Can do.

  In the camera selection device according to the present invention, the position of the viewer may be a virtual user position designated in the object scene.

  As in the present invention, it does not matter whether the specified viewer position is an existing user position or a virtual user position. Therefore, the user can variably control the position of a virtual viewer by remote control, so that an image centered on a desired viewpoint can be viewed from an arbitrary position even if it is not actually present in the object scene. can do. In addition, since the present invention can present a viewpoint from any position within the object scene, the user desires to view a photographed image from a high position that cannot be reached by a real user or from a narrow place that cannot be entered. This is particularly effective when

  According to the present invention, it is possible to select an optimal camera according to the viewpoint of a moving object from among a plurality of cameras.

  Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings for illustration only. First, the configuration of the image processing system 1 in the present embodiment will be described. As shown in FIG. 1, the image processing system 1 includes an image processing apparatus 10 (corresponding to a camera selection apparatus) and five cameras C1 to C5 having the same object scene. Further, the image processing apparatus 10 includes, as functional components, a viewer information input unit 11, a shooting environment setting unit 12, a viewer position specifying unit 13 (corresponding to a specifying unit), and a viewer position similarity calculation. A unit 14 (corresponding to the calculation unit), a camera selection unit 15 (corresponding to the selection unit), an arbitrary viewpoint image generation unit 16 (corresponding to the generation unit), an image smoothing unit 17 (corresponding to the image processing unit), A calibration unit 18 and a moving object automatic tracking unit 19 are provided. These units are connected via a bus.

Hereinafter, each component of the image processing system 1 will be described in detail.
The cameras C1 to C5 have a shooting direction and a viewing angle unique to each camera, and shoot a scene including a moving object and its background. The captured image of the scene is output to an arbitrary viewpoint image generation unit 16 described later, and then provided to the user as a presentation image that has been subjected to predetermined processing. The cameras C1 to C5 are not limited to their types, such as fisheye cameras, pinhole cameras, and omnidirectional cameras, but use a wide-angle camera equipped with a fisheye lens from the viewpoint of widening the imaging range that can be acquired at once. It is desirable to do.

  The viewer information input unit 11 sets information related to a user existing in the object scene. Information about the user is, for example, the user's position and line-of-sight direction. The viewer information input unit 11 uses the position and moving direction of the moving object followed by the moving object automatic tracking unit 19 as the viewer information as necessary. The user is not limited to an existing user, and may be a virtual user. Further, the boundary surface between the object scene A and the outside world is not limited to a wall surface, and may be a virtual surface. Remote monitoring (telemonitoring) is realized by connecting the viewer information input unit 11 to an external device via a network.

The shooting environment setting unit 12 sets information related to the shooting environment in accordance with an instruction operation by the user. For example, information related to the positions where the cameras C1 to C5 are disposed is registered in the shooting environment setting unit 12. In addition, information in which a cross section of the object scene A at the height H ₀ is defined as a layout map is registered.

In the present embodiment, an object scene A having a shape as shown in FIG. 2 is assumed. FIG. 2 is a cross-sectional view of the object scene A along the xy plane defined by the x-axis and the y-axis. The cameras C1 to C5 are respectively arranged at positions where a part of the object scene A can be photographed, and there are no positions in the object scene A that cannot be photographed by any of the cameras C1 to C5. The position coordinates of the camera C1 are defined as C ₁ (c _1x , c _1y ). Similarly, the position coordinates of the cameras C2 to C5 are defined as C ₂ (c _2x , c _2y ) to C ₅ (c _5x , c _5y ), respectively. To do. In addition, the center vectors of the optical axes of the cameras C1 to C5 are set as vectors e ₁ 〜 to e ₅そ_れぞ_れ , respectively. Further, the position coordinate of the viewer is P (x, y), and the angle with respect to the optical axis center vector when the viewer is looking at the viewpoint Q (X, Y) is α. At this time, the area of ΔC _k PQ is expressed by Expression (1).

The viewer position specifying unit 13 specifies the position P (x, y) of the viewer in the object scene A with reference to the setting content by the viewer information input unit 11.
The viewer position similarity calculation unit 14 includes position coordinates C ₁ (c _1x , c _1y ) to C ₅ (c _5x , c _5y ) of the cameras C _{1 to} C ₅ registered in advance by the shooting environment setting unit 12. From the viewpoint Q (X, Y), two cameras having high similarity to the current viewer position P (x, y) are selected. Specific processing contents will be described later in the description of the operation, but the viewer position similarity calculation unit 14 determines the viewer position P specified by the viewer position specifying unit 13 and the camera position (for example, C ₁ , C _2). ) And the viewpoint Q, and the angle formed by the optical axis center vector (for example, e _1⊥ , e _2⊥ ) of the camera and the vector PQ is calculated.

  The camera selection unit 15 selects two cameras (for example, C1 and C2) having high similarity to the viewer position P with respect to the viewpoint Q from the calculation result by the viewer position similarity calculation unit 14 using the least square method. select. The two selected cameras capture the scene A around the viewpoint Q and output the captured image to the arbitrary viewpoint image generator 16.

  When generating the presentation image, the arbitrary viewpoint image generation unit 16 individually develops the images captured by the two selected cameras, and preferentially uses the images captured by the camera with the higher image quality. The camera with the higher image quality is, for example, a camera with a high resolution of a captured image. However, when occlusion occurs in the viewing angle of one camera, the arbitrary viewpoint image generation unit 16 can also supplement (synthesize) the blind spot portion with a photographed image of a camera with low image quality.

  The image smoothing unit 17 performs smoothing processing (smoothing) on the image generated by the arbitrary viewpoint image generation unit 16. At this time, when the target image is a combination of a plurality of captured images having different resolutions, the image smoothing unit 17 performs smoothing according to the resolution. That is, for pixels with a low resolution, smoothing is performed using peripheral pixels (large pixels) of the pixel, and for pixels with high resolution, smoothing is performed using peripheral pixels (small pixels) of the pixel. Thereby, a smooth presentation image in which the outline of the pixel disappears is obtained without degrading the image quality of the high resolution portion.

  The calibration unit 18 includes a correction conversion matrix calculation unit 181 and a basic matrix calculation unit 182. The correction conversion matrix calculation unit 181 is a boundary position between a region that is not projected onto a fish-eye developed image surface (CCD (Charge-Coupled Device) surface) and a region that is projected among images captured by the cameras C1 to C5. The coordinates of (boundary feature points) are calculated. Thereafter, a correction conversion matrix is calculated from the curve coefficient of the contour of the fisheye developed image area estimated from the coordinates, and the captured image is calibrated using this matrix. The basic matrix calculation unit 182 performs calibration using a plurality of cameras by a similar method by simultaneously capturing a plurality of boundary feature points by changing the line-of-sight directions of the two fisheye cameras.

  The moving object automatic tracking unit 19 includes a moving object detection unit 191 and a moving object photographing data calculation unit 192. The moving object detection unit 191 extracts a connected area from a temporal difference image in a common shooting area between two cameras that form a stereo pair, and recognizes the detection of the moving object when the pixel value of the area exceeds a threshold value. To do. At the same time, the moving object detection unit 191 extracts a connected area from a difference image from a reference parallax image (reference image) in a common shooting area between two cameras that form a stereo pair, and the pixel value of the area exceeds a threshold value Also recognize the detection of moving objects. That is, the moving object detection unit 191 recognizes the detection of the moving object by generating at least one of the temporal difference and the difference from the reference image.

  The moving object photographing data calculation unit 192 estimates the image area where the moving object exists from the intersection of the vector connecting the vertex of the rectangular area corresponding to each stereo pair and the center point of the camera. Further, by calculating the center of gravity of the rectangular area circumscribing this area, it is possible to capture the moving object at the center of the image. The moving object photographing data calculation unit 192 extracts the temporal change in the center of gravity of the rectangular area as a moving vector of the moving object, thereby enabling shooting from the opposite direction, that is, the direction facing the moving object. .

  Next, the operation of the image processing system 1 will be described, and the steps constituting the camera selection method according to the present invention will be described together. First, the camera selection process executed by the camera selection unit 15 will be described with reference to FIGS. 2 and 3.

As a premise for explanation, in the camera selection processing in the present embodiment, the situation shown in FIG. 2 (field A, camera positions C _{1 to} C ₅ , viewer position P, viewpoint Q) is assumed, and camera selection is performed. The unit 15 tries to select a camera closest to the line of sight from the viewer position P to the viewpoint Q.

In S <b> 1 of FIG. 3, camera information is defined by the shooting environment setting unit 12. As camera information, a position C _k (c _kx , c _ky ) and an optical axis center vector _{ek ⊥} are defined. However, in the present embodiment, since it is assumed that there are five cameras in the object scene A, k is an integer of 1 to 5.

Then, at S2, the imaging environment setting unit 12, a plan view of the object scene A at the height H ₀ is defined as the layout map. Here, the height H ₀ is a distance from the ground to the optical axis center vector _ek⊥ , and is a fixed value. As shown in FIG. 2, this layout map is defined so that the boundary line with the outside world is a straight line.

In S3, the viewer's virtual position P (x, y) at the remote site is input by the viewer information input unit 11. The viewpoint horizontal plane angle of the virtual position P is α, and the elevation angle is β.
When the shooting environment and the viewer information are determined, the viewer position similarity calculation unit 14 causes the intersection point Q (X, Y) between the straight line passing through the viewer position P and _having an inclination α relative to _ek⊥ and the boundary of the layout map. ) Is calculated (S4).

In S _< b> 5, the viewer position similarity calculation unit 14 calculates the following evaluation formula E (C _k ) for the position C _{k of} each camera. Note that λ is a constant representing the resolution characteristic of the camera Ck, and in the case of a fisheye camera, for example, θ or sin θ is used.
E (C _k ) = (Area of _{ΔC k} PQ) ² + λ (An angle θ formed by the vector e _{k ⊥} and the vector C _k Q) ²

In S6, the camera selection unit 15 selects two cameras in order from the one that minimizes E (C _k ) based on the calculation result in S5. Under the above assumption, and k _{min1 = 1} and k _{min2 = 2} is calculated, C1 as the most similarity to the viewer position camera, C2 is selected as the high affinity camera second.

  The camera selection unit 15 has a display control function of the layout map defined in S2, and displays the two cameras selected in S6 and viewer information (viewpoint and viewing angle) on the layout map. (S7). Thereby, the user can grasp | ascertain quickly and easily the information which is used for presentation of an image among the five arranged cameras and the viewer.

  The camera selection part 15 will complete | finish a camera selection process, if the termination instruction | indication by a user is detected (S8; YES). The camera and viewpoint used change over time as the viewer position moves. However, the image processing apparatus 10 repeatedly executes the series of camera selection processes described above, so that the latest information is always sent to the user. Can be presented.

  Next, a process in which the arbitrary viewpoint image generation unit 16 generates a presentation image from a captured image input from the camera will be described with reference to FIGS. In this arbitrary viewpoint image generation process, the occurrence of occlusion is assumed, and a method for solving this will also be described.

In the present embodiment, the situation shown in FIG. 4 is assumed as the xy plane of the object scene A at the height H ₀ and the arrangement positions of the cameras C1 to C5. In FIG. 4, the viewer's position P and viewpoint Q are regarded as three-dimensional, and the respective coordinates are P (x, y, 0) and Q (q _x , q _y , q _z ). In the figure, when the x-axis is defined in the horizontal direction and the y-axis is defined in the vertical direction, the horizontal angle that the vector PQ makes with the x-axis is θ _p , and the viewer's gaze horizontal angle (around the z-axis) It is assumed that α, the visual field horizontal angle width is Δα, the viewer's gaze elevation angle (around the y axis) is β, and the visual field elevation angle width is Δβ. In this case, theta _p satisfies Formula (2) shown in FIG. If the distance between PQs is d, the relationship shown in the following formula (3) is established between θ _p and q _z .

Note that Δθ _C1 is the viewing angle of the camera C1 determined according to the viewing angle of the viewer. R _M2 and R _M3 are intersections between the light beam of the camera C2 and the boundary wall. θ _C1 is the camera viewing direction angle when the viewpoint is at point Q. S _{1 to} S ₂ indicate the occurrence range of occlusion. Δθ _C2 is the viewing angle of the camera C2 determined according to the occurrence range of occlusion. S _{1 to} S _{2 to} S _{3 to} S ₄ indicate virtual expanded image planes.

In S <b> 11 and S <b> 12 of FIG. 5, camera information is defined by the shooting environment setting unit 12 by the same method as the camera selection process. As a result, the position C _k (c _kx , c _ky ) and the optical axis center vector _{ek 軸} are used as subsequent camera information, and the plan view of the _{object scene} A at the height H ₀ is used as the layout map. The
In S <b> 13, the viewer's virtual position P (x, y) is input by the viewer information input unit 11.

In S14, first, the arbitrary viewpoint image generation unit 16 sets Q _v (q _vx , q) that is an intersection of a straight line that passes through the viewer position coordinates P and _has an inclination α with respect to _ek α and a boundary line of the layout map. _vy , q _vz ) coordinates are calculated. Similarly, the arbitrary viewpoint image generation unit 16 has R _s (q _rsx , which is an intersection of a straight line that passes through the viewer position coordinates P and _has an inclination with respect to _ek of α ± Δα and the boundary line of the layout map. The _{coordinates of} q _rsy , q _rsz ) and R _e (q _rex , q _rey , q _rez ) are respectively calculated.

In S15, the arbitrary viewpoint image generation unit 16 calculates straight lines C ₁ Q, C ₁ R _s , and C ₁ R _e from the three coordinates calculated in S14 and the camera coordinates C _1, and these straight lines and layout are calculated. The coordinates of all intersections R _tmp (not shown) with the map boundary are calculated.

In S16, the initial value of phi _p is set. φ _p is an elevation angle that the vector PQ makes with the x-axis (see FIG. 6), and between φ _p and Q (q _x , q _y , q _z ), the conditional expression (4) shown in FIG. To establish. Φ _p set here is added within the range of the viewing angle of the viewer (β−Δβ ≦ φ _p ≦ β + Δβ) with a step size of Δφ _p with S24 in FIG. 7 as a loop end. Similarly, in S17, after the initial value of theta _p satisfying the conditions (2) and relational expression (3) is set, it is added in [Delta] [theta] _p units S25 in FIG. 7 as a loop end. θ _p is updated within a range of a fixed viewing angle (α−Δα ≦ θ _p ≦ α + Δα).

In S18, the arbitrary viewpoint image generation unit 16 becomes a target on the layout map and a straight line passing through the viewer position coordinates P and _having an inclination of θ _p with respect to the vector _ek⊥ (on the viewpoint side in the present embodiment). The coordinates of Q _{p_tmp} (not shown) that is the intersection with the boundary line are calculated. Further, the minimum value Q _{p of} | C ₁ Q _{p_tmp} | is calculated from the calculated Q _{p_tmp} and the camera position C ₁ (S19).

Moving to FIG. 7, in S20, the arbitrary viewpoint image generation unit 16 calculates Q _{c1_tmp} that is the intersection of the straight line C ₁ Q _p and the target boundary line on the layout map from Q _p calculated in S19. To do. By similar processing, Q _{c2_tmp} that is the intersection of the straight line C ₂ Q _p and the boundary line is calculated. Subsequently, arbitrary viewpoint image generation unit 16, the calculated _{Q C1_tmp} and camera position _{C 1} Tokyo _| C _{1 Q c1_tmp |} calculates the minimum value _{Q c1 of} the _{Q C2_tmp} and camera position _{C 2} Metropolitan | _{C 2} A minimum value Q _c2 of Q _c2 — _tmp | is calculated (S21).

In S22, the arbitrary viewpoint image generation unit 16 determines whether or not Q _c1 calculated in S21 matches Q _p calculated in S19. As a result of the determination, if Q _c1 = Q _p (S22; YES), the arbitrary viewpoint image generation unit 16 sets the pixel value corresponding to Q _c1 on the fisheye expanded image to the pixel of the arbitrary viewpoint image. Extracted as a value (S23). Series of processes S17~S23 are, theta _p are sequentially performed for each theta _p to reach the upper limit value (α + Δα), until further series of processes S16~S24 is, phi _p reaches the upper limit value (β + Δβ) It is sequentially performed for each phi _p. As a result, an image with respect to the viewpoint Q from the viewer position P is obtained.

On the other hand, if the result of determination in S22 is not Q _c1 = Q _p (S22; NO), since it can be determined that occlusion has occurred, the image captured by camera C1 is complemented by the image captured by camera C2. It is desirable. Therefore, the arbitrary viewpoint image generation unit 16 determines whether or not the captured image can be complemented with the pixel value corresponding to Q _c2 by determining the identity between Q _c2 and Q _p (S25). ). As a result of the determination, if Q _c2 = Q _p (S26; YES), the arbitrary viewpoint image generation unit 16 sets the pixel value corresponding to Q _c2 on the fisheye expanded image as the pixel value of the arbitrary viewpoint image. (S27). The extracted pixel value is used to generate an image of a part of the scene that has become a blind spot from the camera C1. After that, the processes after S24 described above are executed.

Here, FIG. 8 is a diagram for explaining the processing executed in S23 and S27, that is, a technique for calculating a point on the fisheye expanded image plane corresponding to the point on the real space. As shown in FIG. 8, the fish-eye developed image plane is defined on the xy plane, and the z-axis is defined on the center line of the fish-eye developed image plane. It should be noted that calibration is required for the coordinate system and the fish-eye developed image, but the method will be described later. In FIG. 8, an arbitrary point on the fisheye lens is q (X, Y, Z), and an arbitrary point on the fisheye developed image plane after projection is p ( _ximage , _yimage , 0). Here, if the angle (incident angle) that the vector q makes with the z-axis is θ, and the angle that the vector p makes with the x-axis on the fisheye developed image plane is φ, the angle between θ and X, Y, and Z Equation (5) holds. Further, the equation (6) is established between φ and X, Y.

また、等距離射影方式によっては、ｐ（ｘ_image，ｙ_image，０）は以下の式（８）により特定される。

When the focal length f is used under such conditions, p (x _image , y _image , 0) is specified by the following equation (7) depending on the orthogonal projection method. As the focal length f, for example, the vertical resolution of the fisheye developed image can be used.

Further, depending on the equidistant projection method, p (x _image , y _image , 0) is specified by the following equation (8).

Returning to FIG. 7, if Q _c2 = Q _p is not the result of the determination in S26 (S26; NO), the arbitrary viewpoint image generation unit 16 determines that the occlusion cannot be avoided even if the camera C2 is used. Substitute an inevitable value for occlusion into the pixel value of the arbitrary viewpoint video. Thereafter, the processing after S24 is executed.

When the series of processing of S16 to S25 is finished, smoothing processing (smoothing) by the image smoothing unit 17 is executed (S29). The image smoothing unit 17 smoothes the obtained image using a Gaussian filter which is a well-known and commonly used image processing technique. The obtained image has a resolution lower than that of other image portions as a result of avoiding occlusion in some portions. For this reason, if a uniform filtering process that does not depend on the synthesis portion is executed for all pixels, the image quality of the other low-resolution portions is the same as the high-resolution portions. Therefore, the image smoothing unit 17 performs filtering according to the peripheral pixels of each pixel constituting the image as a feature of the processing. In other words, the filter shape is smoothed in consideration of the resolution characteristics when the image captured by the wide-angle camera is developed on a plane. This makes it possible to generate a smooth presentation image between pixels while utilizing the image quality of the high resolution portion.
The arbitrary viewpoint image generation processing ends when an instruction is given by the user (S30).

As described above, the image processing apparatus 10 according to the present invention has an optimal camera selection function as one of main functions. According to the optimum camera selection function, the image processing apparatus 10 uses a camera in which the area of the triangle formed by the camera position C _k (k is a natural number of 1 to 5), the viewer position P, and the viewpoint Q is minimized. Select to generate an image. △ By taking C _k PQ reduce the area of the distance PC _k is short and, ∠C k _PQ is narrowed, the camera close to the line of sight of the viewer position and viewer are selected. At the same time, the image processing apparatus 10 performs image generation by selecting a camera that minimizes the angle formed by the optical axis center vector _{ek ベ ク ト ル} of the camera and the vector PQ. That is, a camera having an optical axis direction close to the visual line direction of the viewer is selected. For this reason, an image with less distortion can be obtained.

  That is, in the conventional camera selection technology, the closeness of the viewpoint with the generated image is considered, but the resolution characteristics based on the optical characteristics according to the viewpoint of the camera are not considered. By adopting the camera selection criteria as described above, the resolution characteristics based on such characteristics can be reflected in the presented image, so that it is possible to select a camera compatible with a wide-angle camera such as a fish-eye camera. Similarly, when selecting a camera that compensates for occlusion, resolution characteristics based on optical characteristics according to the viewpoint can be taken into consideration, and it is possible to cope with a wide-angle camera.

  The arbitrary viewpoint image generation unit 16 can use the captured image of the fisheye camera corrected by the calibration unit 18 when generating the arbitrary viewpoint image. Hereinafter, the calibration process in the fisheye camera will be described with reference to FIGS.

  When captured images of the plurality of cameras C1 to C5 are input to the calibration unit 18 (S31 in FIG. 9), the correction conversion matrix calculation process and the basic matrix calculation process are executed in parallel. First, correction conversion matrix calculation processing will be described. The correction conversion matrix calculation unit 181 extracts a point (boundary feature point) on the boundary line between the black region that is not projected on the fisheye-expanded image plane and the other region (boundary feature point) as edge position coordinates (S32). For example, the lens shift angle is φ, and the intersection coordinates of the optical axis center and the xy plane are O ′ (x ′, y ′). In this case, as shown in FIG. 10, a plurality of points E that form the boundary lines between the black areas B1 to B4 and the elliptical fish-eye developed image area D1 are extracted. Since this edge extraction process is a well-known and commonly used image analysis technique, a detailed description and illustrations (including mathematical expressions) are omitted, and a preferred method will be briefly described. The correction conversion matrix calculation unit 181 includes, for example, a Sobel filter, and converts the horizontal and vertical directions to 9 (= 3 × 3) pixel values centered on an arbitrary pixel in the acquired image. Are multiplied by the two coefficient matrices, respectively. Then, the change amount of each pixel value is calculated based on the multiplication result, and a portion where the change amount of the pixel value is large (corresponding to a boundary feature point) is detected as an edge. The pixel value is, for example, luminance.

In S33, the arbitrary viewpoint image generation unit 16 estimates a conic coefficient (secondary curve coefficient) from the edge position coordinates extracted in S32 by the least square method.
The arbitrary viewpoint image generation unit 16 calculates a correction conversion matrix R (= RθT) for each fisheye camera based on the estimated conic coefficient (S34). In the calculation, first, a rotation angle is calculated such that the estimated conic coefficient is an elliptical conic coefficient, and a rotation matrix Rθ is obtained from the rotation angle. Further, a parallel movement amount is calculated such that the center of the ellipse and the center of the fish-eye developed image surface (CCD surface) coincide with each other, and a parallel movement matrix T is obtained from this movement amount. Then, a correction conversion matrix R is obtained by multiplying the rotation matrix Rθ and the translation matrix T. This correction conversion matrix R is calculated for each fisheye camera (cameras C1 to C5 in the present embodiment) (S35).

  In S36, it is determined whether or not another fisheye camera is to be used for image synthesis for avoiding occlusion or detection of a moving object. When no other fisheye camera is used (S36; NO), the arbitrary viewpoint image generation unit 16 develops (projects) the image captured by the fisheye camera, and the correction conversion matrix calculated in S34. P ′ is calculated from R (= RθT). P ′ is calculated by multiplying the coordinates P of the fisheye developed image plane by the correction conversion matrix R (S37). Thereby, the fish-eye expansion | deployment image surface where the calibration was performed is produced | generated.

  Next, the basic matrix calculation process will be described. First, the basic matrix calculation unit 182 selects any two fisheye cameras (for example, cameras C1 and C2) from a plurality of fisheye cameras (cameras C1 to C5 in the present embodiment) (S38). The basic matrix calculation unit 182 captures a plurality of feature points simultaneously by changing the line-of-sight directions of the two selected fisheye cameras (S39), and the feature points in the images captured by each fisheye camera. Are extracted (S40). Extraction of feature points can be performed by manual input by operating a mouse or the like.

  In S41, a basic matrix F based on the method of least squares is calculated using the position coordinates of the feature points as input data in the same procedure as in S33. The basic matrix calculation unit 182 normalizes the color space (RGB space) so that the color distributions of the images matched in the projection matrix are the same (S42). A series of processes of S39 to S42 is repeatedly executed within the movable range in the line-of-sight direction with S43 as a loop end. Further, the series of processing is repeatedly executed for all combinations of fisheye cameras with S44 as a loop end. As a result, a basic matrix F for each fisheye camera is calculated for each viewpoint (S45).

  S46 is a process executed when it is determined in S36 that another fisheye camera is used (S36; YES). The arbitrary viewpoint image generation unit 16 calculates P ′ from the correction conversion matrix R (= RθT) calculated in S34 and the basic matrix F calculated in S45 when developing an image captured by the fisheye camera. calculate. P ′ is calculated by multiplying the coordinates P of the fisheye developed image plane by the product of the basic matrix F and the correction conversion matrix R (S46). This completes the calibration process in the fisheye camera.

  FIG. 11 shows an example of the fisheye expanded image area D2 corrected as a result of executing the fisheye camera calibration process. Assuming that the fish-eye lens is a perfect hemisphere and its refractive index is point (hemisphere center) symmetrical, the fish-eye developed image region D1 shown in FIG. The long axis and the short axis of the region are corrected so as to be parallel to the x axis and the y axis, respectively. As a result, a fish-eye developed image area D2 is obtained. In this way, the image processing apparatus 10 can apply appropriate calibration even when a fisheye camera is used for the cameras C1 to C5, so that an arbitrary viewpoint image generated from a captured image, and thus a presentation image can be displayed. The quality can be improved.

  Next, the moving object automatic tracking process will be described with reference to FIGS. FIG. 12 is a diagram illustrating an overview of a system environment that realizes the moving object automatic tracking function. As shown in FIG. 12, the image processing system forms a remote site, a video processing service site, and a user site so as to function as a telemonitoring system that automatically follows a moving object. The devices that are constituent elements of these sites are connected so that signals can be transmitted bidirectionally from a wired or wireless local area network (LAN).

  At the remote site, the cameras C1 to C5 capture the scene A, and the captured images are collected by the image collection device 20. The collected captured images are transmitted to the image processing apparatus 10 via the network N, and then used for processing such as autonomous camera work, arbitrary viewpoint image generation, or moving object detection. In the video processing service site, the image processing apparatus 10 generates a presentation image from a plurality of captured images, and causes the image distribution apparatus 30 to distribute it to the communication terminal 40 via the network N. At the user site, the communication terminal 40 receives and displays the presented image, thereby enabling monitoring by the user U. The communication terminal 40 is, for example, a personal computer, a mobile phone, or a PDA (Personal Digital Assistant) having a communication function.

Next, the moving object automatic tracking process executed by the image processing apparatus 10 by the system will be described.
In S <b> 51 of FIG. 13, camera information is defined by the shooting environment setting unit 12. As camera information, a position C _k (c _kx , c _ky ) and an optical axis center vector _{ek ⊥} are defined. In the present embodiment, since it is assumed that there are five cameras in the object scene A, k is an integer of 1 to 5. Further, a projective transformation matrix and a common shooting area are defined as inter-camera information (S52). Further, a plan view of the object scene A at the optical axis ground height H ₀ is defined as a layout map (S53). Here, the height H ₀ is a distance from the ground to the optical axis center vector _ek⊥ , and is a fixed value.

  When images taken by the cameras C1 to C5 are input to the arbitrary viewpoint image generation unit 16 (S54), the moving object detection unit 191 sets the time t as an initial value (S55), and the five cameras C1 to C5 are set. Two cameras to be a stereo pair are selected from the inside (S56). Subsequently, the moving object detection unit 191 multiplies one image A (x, y, t) of the two selected cameras by the projection transformation matrix HAB, thereby obtaining the other image B (x, y , T), an image B ′ (x, y, t) viewed from the same viewpoint is generated (S57).

In S58, a parallax image CAB (x, y, t) between the image B (x, y, t) and the image B ′ (x, y, t) generated in S57 is calculated. Further, in S59, by calculating CAB (x, y, t) −CAB (x, y, t−1), the difference image TimeDiff (x, y) of the parallax image in unit time is calculated. The moving object detection unit 191 extracts the connection area ST _i from the temporal difference image TimeDiff (x, y) in the imaging area common to the two cameras (S60), and the connection area ST exceeds the threshold Th _T. By determining the presence or absence of _i, the presence or absence of a moving object in the object scene A is confirmed (S61 in FIG. 14).

As a result of the determination, if there is a connected region that satisfies ST _i > threshold Th _T (S61; YES), the moving object detection unit 191 uses a projection histogram on both the x and y axes at this time, and extracts a rectangular region _{R i} circumscribing the consolidated region (S62). On the other hand, when there is no connected region satisfying ST _i > threshold Th _T (S61; NO), the process proceeds to S63, and the moving object detection unit 191 determines the difference image RefDiff at time t from the reference parallax image (reference image). (X, y) is calculated by CAB (x, y, t) -Cref (x, y).

Further, the moving object detection unit 191 performs binarization processing and expansion / reduction processing on the difference image RefDiff (x, y) with respect to the reference parallax image in the common imaging region between the two cameras, and connects The region SR _i is extracted (S64). Thereafter, the presence or absence of the moving object in the scene A is confirmed by determining the presence or absence of the connected region SR _i exceeding the threshold Th _R in the same procedure as S61 (S65). As a result of the determination, if there is a connected region satisfying SR _i > threshold Th _R (S65; YES), the moving object detection unit 191 extracts a rectangular region R _i circumscribing the connected region SR _i ( S62).

  As described above, the moving object automatic tracking unit 19 of the image processing apparatus 10 does not depend on the illumination condition by capturing the change in the parallax information in real time without using the background difference information (S57 to S62). Realize detection of moving objects. In addition, the moving object automatic tracking unit 19 extracts moving objects while monitoring the difference between the disparity information unique to the environment and the disparity information detected in real time (S63 to S65, S62). This makes it possible to detect the moving object without depending on the state of the moving object, in other words, whether the moving object is stationary or moving.

The processing in S56 to S66 is executed for all stereo pairs (two cameras each) among the cameras C1 to C5 that photograph the object A, and then the processing after S67 in FIG. Move to the data calculation module. First, in S67, the moving object photographing data calculation unit 192 connects each vertex of the rectangular region R _i and the center point of the camera for each connected region ST _i or SR _i extracted in the stereo pair selected in S56. Four direction vectors are extracted. Subsequently, after calculating the intersection of the four extracted direction vectors, the region surrounded by these vectors is recognized as the region where the moving object exists (S68).

The moving object photographing data calculation unit 192 calculates the center of gravity G _i (t) of the rectangular area circumscribing the area recognized in S68 (S69), and then extracts the value as the position of the moving object. Furthermore, DiffG _i (t), which is a difference vector between G _i (t) at time t and G _i (t−1) at time t−1 before unit time, is extracted as a moving vector of the moving object (S70). ). The above-described series of processing from S55 to S70 is repeatedly executed until an end instruction is given with S71 as a loop end. Subsequently, the moving object photographing data calculation unit 192 outputs the position G _i (t) of the moving object and the opposite direction vector of the difference vector DiffG _i (t) to the arbitrary viewpoint image generation unit 16 (S72). ). Since the moving object usually moves with the moving direction as the front, a camera (camera facing the moving object) whose imaging direction is the opposite direction of the moving direction is used for image generation. This makes it possible to capture the entire body from the front.

  As described above, the image processing apparatus 10 according to the present invention has a moving object tracking function as one of main functions. The image processing apparatus 10 extracts a temporal difference between frames and also extracts an image portion (difference) that does not return to the initial parallax information (reference image) even when projective transformation is performed. Then, the moving object is detected by adding at least one of the differences between the parallax fluctuations obtained from the images captured by a pair of cameras at the same time. For this reason, accurate detection of moving objects is less affected by changes in lighting conditions such as shadows or changes in the state of moving objects such as stationary, compared to conventional moving object detection methods that use difference information from an updated background image. There is an effect that becomes possible. In addition, the conventional moving object detection method based on the parallax information of the stereo camera has an effect that it can be applied not only to a plane such as a pedestrian crossing and a license plate but also to a general place having a three-dimensional shape.

  Furthermore, the image processing apparatus 10 calculates the position information and the movement vector of the moving object detected in the object scene A, and realizes autonomous camera work by applying the above-mentioned camera selection criteria to this. To do. Thereby, since a moving body can always be image | photographed from the front, the effective monitoring from a remote place is attained.

In addition, this invention is not limited to this Embodiment, In the range which does not deviate from the meaning, a deformation | transformation aspect can also be taken suitably.
For example, in the above embodiment, the user's virtual viewpoint is one point, but by connecting a plurality of communication terminals to the image processing apparatus 10, a plurality of users can simultaneously perform remote monitoring from a free viewpoint. It becomes. In this aspect, it is particularly effective that the communication terminal is equipped with a 360-degree visual sensor or an orientation sensor (for example, a gyro sensor) so that the direction in which the user tilts the communication terminal can be detected as the viewpoint direction. As a result, for example, it is possible to monitor the environment where the other party is located from an arbitrary viewpoint while communicating via a videophone.

  Moreover, in the said embodiment, although the number of the cameras which the camera selection part 15 selects was two, of course, you may be three or more. If it is clear in the field that no occlusion occurs in the viewpoint Q from the viewer position P, one camera may be used.

It is a figure which shows the functional structure of the image processing apparatus which concerns on this invention. It is an xy top view of a scene showing an example of a positional relationship between a camera, a viewer, and a viewpoint. It is a flowchart for demonstrating a camera selection process. It is an xy plan view of an object scene showing an example of a positional relationship between a viewer, a viewpoint, and a camera in which occlusion occurs. It is a flowchart for demonstrating the first half part of arbitrary viewpoint image generation processing. It is xz top view of a subject field which shows an example of the height of a viewer position, and a look direction. It is a flowchart for demonstrating the latter half part of an arbitrary viewpoint image generation process. It is a figure for demonstrating an example of the method of fish-eye-expanding the point on real space. It is a flowchart for demonstrating a fisheye camera calibration process. It is a figure which shows an example of the fisheye expansion | deployment image area | region before correction | amendment by calibration. It is a figure which shows an example of the fisheye expansion | deployment image area | region after correction | amendment by calibration. It is the schematic which shows an example of the system environment which implement | achieves a moving body automatic tracking type | mold telemonitoring. It is a flowchart for demonstrating the first half part of the moving body detection module which comprises a moving body automatic tracking process. It is a flowchart for demonstrating the second half part of the moving object detection module which comprises a moving object automatic tracking process. It is a flowchart for demonstrating the data calculation module for moving body imaging | photography which comprises a moving body automatic tracking process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image processing system, C1, C2, C3-Cn ... Camera, 10 ... Image processing apparatus, 11 ... Viewer information input part, 12 ... Shooting environment setting part, 13 ... Viewer position specific | specification part, 14 ... Viewer position Similarity calculation unit, 15 ... camera selection unit, 16 ... arbitrary viewpoint image generation unit, 17 ... image smoothing unit, 18 ... calibration unit, 181 ... correction conversion matrix calculation unit, 182 ... basic matrix calculation unit, 19 ... Automatic body tracking unit, 191 ... Animal body detection unit, 192 ... Animal body imaging data calculation unit, 20 ... Image collection device, 30 ... Image distribution device, 40 ... Communication terminal, A ... Object, M ... Animal body , N ... Network

Claims (5)

A specifying means for specifying the position of the viewer in a scene that can be photographed by a plurality of cameras;
The triangular area formed by the viewer position specified by the specifying means, the camera position and the viewpoint, and the angle formed by the optical axis center vector of the camera and the vector from the viewer position to the viewpoint. Calculating means for calculating the plurality of cameras;
A camera selection apparatus comprising: a selection unit that selects a camera that outputs a captured image centered on the viewpoint using a calculation result obtained by the calculation unit. The selection means uses the calculation result to select a plurality of cameras that output a captured image centered on the viewpoint,
The apparatus according to claim 1, further comprising: a generation unit configured to generate a presentation image by applying an image captured by another camera to the occlusion portion when an occlusion occurs in the image captured by one camera. Camera selection device.   The image processing means for performing smoothing according to the resolution of each pixel when there is an image portion having a different resolution as a result of generating the presentation image by the generating means. 3. The camera selection device according to 2.   The camera selection apparatus according to claim 1, wherein the position of the viewer is a position of a virtual user designated in the object scene. A specific step of identifying the position of the viewer in a scene that can be photographed by a plurality of cameras;
The triangular area formed by the viewer position, the camera position, and the viewpoint specified in the specifying step, and the angle formed by the optical axis center vector of the camera and the vector from the viewer position to the viewpoint Calculating with respect to the plurality of cameras;
And a selection step of selecting a camera that outputs a photographed image centered on the viewpoint using the calculation result in the calculation step.

Images