JP5213883B2 - Composite display device - Google Patents

Description

  The present invention relates to a composite display device that combines images taken by a plurality of cameras and displays a combined image.

When monitoring a place where there are many objects to be shielded, a plurality of cameras may be installed so that a blind spot due to shielding of the object can be photographed.
That is, when a certain surveillance camera takes an image of an indoor area, a road surface, or a site surrounding a building, a blind spot shielded by an object installed on the building, road structure, or ground is generated. In such a case, another surveillance camera is installed at a position where an area that is a blind spot can be imaged.
However, when two monitoring cameras are installed, since the shooting directions of the two monitoring cameras are different, it is necessary to combine the images shot by the two monitoring cameras into one image.

For example, Patent Document 1 below discloses an example of a synthesis process.
In the information distribution device (composite display device) disclosed in Patent Document 1 below, another vehicle (for example, a building or the like that is not visible from a vehicle entering the intersection (hereinafter referred to as “target vehicle”) is used. Vehicle that enters the intersection from the camera installed in the intersection so that the image of the vehicle in the left turn direction at the intersection) can be displayed on the image display device mounted on the target vehicle. An image of an area that cannot be seen from (a blind spot area) is acquired, and the image of the blind spot area is converted into an image viewed from a preset virtual viewpoint.
Further, the information distribution apparatus acquires a front image representing a front view of the target vehicle entering the intersection from another camera, converts the front image to an image viewed from the virtual viewpoint, and converts the blind spot image after conversion. And the front image are combined, and the combined image is transmitted to the image display device in the target vehicle.

The viewpoint conversion method is disclosed in Non-Patent Document 1 below, for example.
However, in the method disclosed in Non-Patent Document 1 below, a restriction that cannot be correctly converted unless matching of feature points in which the same part of the same object is captured is made between images taken by a plurality of cameras. It is.
This means that if the other camera cannot capture the same surface as the surface of the object captured by a certain camera, the same feature point will not appear in both captured images, so the viewpoint of the image can be converted. It means you can't.
Also, when two cameras include a subject wearing camouflage clothes in an image of the same region taken from approximately the same direction, or when a plant with many leaves is included, images are taken with two cameras. This means that there are a large number of feature points obtained from shading in the obtained image, and it is difficult to accurately associate a large amount of feature points, so that the viewpoint of the image cannot be accurately converted.

JP 2009-70243 A (paragraph numbers [0007] to [0008])

Koichiro Deguchi "Basics of Robot Vision" 2000, published by Corona, pages 27-31

  Since the conventional composite display device is configured as described above, a large number of feature points can be accurately associated with a complicated pattern or shape of a moving object (for example, a human or a vehicle) in the monitoring target area. In other words, it is impossible to accurately combine images taken by a plurality of cameras. Even if the pattern or shape of the moving object in the monitoring target area is not complicated, even if the moving object is reflected in one camera by the shielding object (for example, a bookshelf installed in the monitoring target area), the other When a moving object is not shown in this camera, there is a problem that the same feature point in the moving object cannot be associated, and images taken by a plurality of cameras cannot be combined.

  The present invention has been made to solve the above problems, and is photographed by a plurality of cameras regardless of the complexity of the pattern and shape of a moving object in the monitoring target area and the presence or absence of a shield in the monitoring target area. It is an object of the present invention to obtain a composite display device capable of compositing synthesized images.

  The composite display device according to the present invention includes a plurality of floor area image extracting means for extracting an image of a floor area where a floor surface is imaged from an image of a monitoring target area imaged by the camera, and an image captured by the camera. The floor area image extracting means uses a plurality of moving object area image extracting means for extracting an image of the moving object area where the moving object is photographed from the monitored area image and camera data indicating the installation status of the camera. Projecting the extracted floor area image to the plane position in the orthogonal coordinate system of the monitoring target area to generate a planar projection image, and the moving object exists on the orthogonal coordinate system of the monitoring target area A plurality of moving body image pasting means for pasting the moving body area image extracted by the moving body area image extracting means to the surface of the board model. It is obtained by an image plate model moving object area image pasted by the generated planar projective images and a plurality of moving object image pasting unit as synthesized by the plane projection means.

  According to the present invention, the plurality of floor area image extracting means for extracting the image of the floor area where the floor surface is imaged from the image of the monitoring area captured by the camera, and the monitoring object imaged by the camera Extracted by the floor area image extraction means using a plurality of moving object area image extracting means for extracting the moving object area image in which the moving object is photographed and the camera data indicating the installation status of the camera from the area image. A plurality of plane projection means for projecting a floor area image to a planar position in the orthogonal coordinate system of the monitoring target area to generate a planar projection image, and an area in which the moving object exists on the orthogonal coordinate system of the monitoring target area A plurality of moving object image pasting means for attaching the moving body region image extracted by the moving body region image extracting means to the surface of the plate model, and the image composition means comprises a plurality of plane projections. Since the planar projection image generated by the stage and the image of the plate model to which the moving body region image is pasted by the plurality of moving body image pasting means are combined, the complexity of the pattern / shape of the moving body in the monitoring target region There is an effect that images taken by a plurality of cameras can be combined regardless of the presence or absence of a shield in the monitoring target area.

It is a block diagram which shows the synthetic | combination display apparatus by Embodiment 1 of this invention. It is a top view which shows the room which is a monitoring object area | region. It is a bird's-eye view which shows the inside of a monitoring object field. Images of the monitoring target area captured by the cameras A, C, and D, floor area images extracted by the floor area image extraction units 2-1, 2-3, and 2-4, and a moving body area image extraction unit 4-1. It is explanatory drawing which shows the image of the moving body area | region extracted by 4-3 and 4-4. (A) is explanatory drawing which shows the monitoring object area | region at the time of seeing the optical axis (center) direction of a camera from right side in parallel with the floor surface of the monitoring object area | region, (b) is description which shows the example of a display screen. FIG. It is explanatory drawing which shows to which point in the real space the pixel in the image of the monitoring object area | region corresponds. It is explanatory drawing which shows the search process of the area | region where a moving body exists. It is explanatory drawing for determining the height of a board model. It is explanatory drawing which shows sticking of the moving body area | region image with respect to a board model. FIG. 6 is an explanatory diagram showing a composite image of a planar projection image and a plate model generated from an image captured by a camera C and a planar projection image and a plate model generated from an image captured by a camera D. It is explanatory drawing which shows the example of a display of the synthesized image by the image display part. It is explanatory drawing which shows the movable direction of a PTZ camera. It is explanatory drawing which shows the mark attached | subjected to the housing | casing of a mobile camera. It is explanatory drawing which shows the example which the moving camera is moving to the position where a fixed camera is not caught. It is explanatory drawing which shows an example which displays only the specific shelf transparently in response to the zoom function of a camera.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing a composite display device according to Embodiment 1 of the present invention.
In FIG. 1, cameras 1-1 to 1-N are fixed cameras installed at different positions, and images of monitoring target areas are taken from different directions and images of the monitoring target areas are shown. Output data.

The floor area image extraction units 2-1 to 2-N are configured by, for example, a GPU (Graphics Processing Unit) that performs image processing. The floor area image extraction units 2-1 to 2-N include, among the images of the monitoring target area captured by the cameras 1-1 to 1-N. Then, a process of extracting an image of the floor area where the floor surface is photographed and storing the image data indicating the floor area image in the floor area image memories 3-1 to 3-N is performed.
However, the floor area image extraction units 2-1 to 2-N are photographed by the cameras 1-1 to 1-N when the cameras 1-1 to 1-N are cameras that perform NTSC (National Television Standards Committee) output. An image capture board that captures captured images in real time is implemented.
On the other hand, when the cameras 1-1 to 1-N are network connection type cameras, a network interface that captures images taken by the cameras 1-1 to 1-N from the network is installed.
The floor area image extracting units 2-1 to 2-N constitute floor area image extracting means.
The floor area image memories 3-1 to 3-N are recording media such as a RAM and a hard disk that store image data indicating floor area images extracted by the floor area image extracting units 2-1 to 2-N.

The moving body area image extraction units 4-1 to 4-N are configured by, for example, a GPU that performs image processing, and moving bodies are captured from images of the monitoring target area captured by the cameras 1-1 to 1-N. An image of the moving object region is extracted, and image data indicating the moving object region image is stored in the moving object region image memories 5-1 to 5-N.
Similarly to the floor area image extraction units 2-1 to 2-N, the moving body area image extraction units 4-1 to 4-N are also mounted with an image capture board or a network interface.
The moving object region image extracting units 4-1 to 4-N constitute moving object region image extracting means.
The moving object area image memories 5-1 to 5-N are recording media such as a RAM and a hard disk that store image data indicating moving object area images extracted by the moving object area image extracting units 4-1 to 4-N.

The camera data acquisition unit 6 receives camera data (for example, the optical axis direction of the camera, the installation position, the installation height, the camera 1-1 to 1-N and the pan / tilt function unit of the cameras 1-1 to 1-N (not shown)) A data input interface for acquiring data indicating focal length and the like is provided, and processing for storing the camera data in the camera data memory 7 is performed.
The camera data memory 7 is a recording medium such as a RAM or a hard disk that stores the camera data acquired by the camera data acquisition unit 6.
The building structure data memory 8 stores building structure data (for example, the dimensions of the building where the cameras 1-1 to 1-N are installed, and the width, length, and height of the bookshelf (shield) in the monitoring target area. The data is a recording medium such as a RAM or a hard disk that stores data (represented by orthogonal coordinates having a constant origin).

The planar projection units 9-1 to 9-N are constituted by, for example, a semiconductor integrated circuit on which a CPU is mounted or a one-chip microcomputer, and the camera data and building structure data stored in the camera data memory 7 The floor area image stored in the floor area image memories 3-1 to 3-N is projected on the plane position in the orthogonal coordinate system of the monitoring target area with reference to the building structure data stored in the memory 8 Then, a plane projection image facing a predetermined direction is generated, and image data indicating the plane projection image is stored in the plane projection image memories 10-1 to 10-N. The plane projection units 9-1 to 9-N constitute plane projection means.
The plane projection image memories 10-1 to 10-N are recording media such as a RAM and a hard disk that store image data indicating the plane projection images generated by the plane projection units 9-1 to 9-N.

The moving body image pasting units 11-1 to 11-N are composed of, for example, a semiconductor integrated circuit on which a CPU is mounted or a one-chip microcomputer, and the camera data and building structure stored in the camera data memory 7 With reference to the building structure data stored in the data memory 8, a plate model (for example, a plate model has a width and a height of a moving object) in an area where a moving object exists on the orthogonal coordinate system of the monitoring target area. And a moving object region stored in the memory of the moving object region image memory 5-1 to 5-N on the surface of the plate model. Perform the process of pasting the image. The moving object image pasting units 11-1 to 11-N constitute moving object image pasting means.
The board model memories 12-1 to 12-N are recording media such as a RAM and a hard disk that store board data indicating the board model to which the moving body area image is pasted by the moving body image pasting units 11-1 to 11-N. .

The image composition unit 13 is constituted by, for example, a GPU that performs image processing, and the planar projection image and the plate model memory 12-1 indicated by the image data stored in the planar projection image memories 10-1 to 10-N. The image of the board model indicated by the board data stored in .about.12-N is synthesized, and the image data representing the synthesized image is stored in the synthesized image memory 14. The image composition unit constitutes image composition means.
The synthesized image memory 14 is a recording medium such as a RAM or a hard disk that stores image data indicating an image synthesized by the image synthesizing unit 13.

  The image display unit 15 is composed of, for example, a semiconductor integrated circuit on which a CPU is mounted, a one-chip microcomputer, or the like, and displays a composite image indicated by image data stored in the composite image memory 14 on a display (not shown). Perform the process to display. The image display unit 15 constitutes image display means.

  In FIG. 1, floor area image extraction units 2-1 to 2-N, moving object area image extraction units 4-1 to 4-N, a camera data acquisition unit 6, and a planar projection unit 9-1 are constituent elements of the composite display device. ~ 9-N, the moving image pasting units 11-1 to 11-N, the image compositing unit 13 and the image display unit 15 are assumed to be configured by dedicated hardware. When configured by a computer, floor area image extraction units 2-1 to 2-N, moving object area image extraction units 4-1 to 4-N, camera data acquisition unit 6, planar projection units 9-1 to 9-N, A program describing processing contents of the moving object image pasting units 11-1 to 11-N, the image compositing unit 13, and the image display unit 15 is stored in a memory of a computer, and the CPU of the computer is stored in the memory. The program may be executed.

Next, the operation will be described.
In the first embodiment, for convenience of explanation, seven (N = 7) cameras 1-1 to 1-7 (cameras A, B, C, D, E, F, G) are as shown in FIG. It shall be installed.
FIG. 2 is a plan view showing a room which is a monitoring target area.
The room, which is the monitoring target area, is surrounded by walls on both sides, and two bookshelves T1 and T2 are installed.
It is assumed that the height from the floor to the ceiling of this room is 3 meters, and the height of the bookshelves T1 and T2 is 2 meters.

The cameras A to G are installed at the height of the ceiling and are installed horizontally with respect to an axis centered on the optical axis.
Further, it is assumed that the pitch angle of the cameras A to G is a constant angle and faces downward.
In FIG. 2, arrows A to G indicate shooting directions of the cameras A to G, and solid lines on both sides of the arrows indicate ranges that can be shot by the respective cameras.
P, Q, and R indicate persons existing in the monitoring target area.
FIG. 3 is a bird's-eye view showing the inside of the monitoring target area. However, in FIG. 3, the display of the outer wall of the room is omitted.

As shown in FIG. 2, the cameras A to G take images of the inside of the room that is the monitoring target area from different directions, and use the floor area image extracting unit 2-1 to display image data indicating the image inside the room. To 2-7 and the moving object region image extracting units 4-1 to 4-7.
However, the photographing with the cameras A to G may be performed separately or may be performed simultaneously in synchronization. Image data indicating the latest image is output from the cameras A to G at regular time intervals.

When the floor area image extraction units 2-1 to 2-7 receive image data from the cameras A to G, the floor area image in which the floor surface is captured from the images of the monitoring target area indicated by the image data. And the image data indicating the floor area image is stored in the floor area image memories 3-1 to 3-7.
In the image data indicating the floor area image, the pixel values in the pixels in the floor area are the pixel values of the image taken by the camera, and the pixel values in the pixels other than the floor area are preset constant values.
Hereinafter, floor area image extraction processing by the floor area image extraction units 2-1 to 2-7 will be described in detail.

In advance, the cameras A to G capture an image of a monitoring target area when no human is present.
The floor area image extraction units 2-1 to 2-7 capture the floor area where the floor surface is imaged and the areas other than the floor surface among the images of the monitoring target area captured by the cameras A to G. The area is distinguished from each other in pixel units, the pixel values of the pixels in the floor area are recorded with the pixel values of the images taken by the cameras A to G, and the pixel values of the pixels other than the floor area are set in advance. Record a constant value.

When the floor area image extraction units 2-1 to 2-7 receive image data from the cameras A to G after the monitoring is actually started, the floor area image extraction units 2-1 to 2-7 constitute the images of the monitoring target area indicated by the image data. The pixel value of the pixel is compared with each pixel value recorded in advance (the pixel value of each pixel constituting the image of the monitoring target area when no human is present) (the same pixel position in the image) And the pixel value difference within a certain value is determined to be a pixel in the floor area. On the other hand, a pixel having a pixel value difference equal to or greater than a certain value is determined to be a pixel other than the floor region.
When the floor area image extraction units 2-1 to 2-7 determine that the pixels constituting the image of the monitoring target area are pixels in the floor area, the pixel values of the pixels are captured by the cameras A to G. The pixel value of the image is set, and the pixel value is stored in the floor area image memories 3-1 to 3-7.
On the other hand, if it is determined that the pixels constituting the monitoring target area image are pixels other than the floor area, the pixel value of the pixel is set to a predetermined constant value, and the pixel value is set for the floor area image. Store in the memories 3-1 to 3-7.

Note that the floor area image extraction units 2-1 to 2-7 have a shading edge on the images captured by the cameras A to G when the floor area color has a shading but there is no pattern. By performing the extraction process (for example, canny filter process), the edge of the portion that changes depending on the shading is raised.
When the floor region is divided by the light and shade edges, the floor region image extraction units 2-1 to 2-7 regard each closed loop by the light and dark edges as one cluster, and each pixel constituting the cluster for each cluster. This pixel value is compared with each pixel value recorded in advance, and it is determined that a pixel having a pixel value difference within a certain value is a pixel in the floor region, so that the cluster is a floor region. Determine whether or not.

Here, FIG. 4 shows the image of the monitoring target area photographed by the cameras A, C, and D, the floor area image extracted by the floor area image extraction units 2-1, 2-3, and 2-4, and the moving object area image. It is explanatory drawing which shows the image of the moving body area | region extracted by the extraction part 4-1, 4-3, 4-4.
As shown in FIG. 4, the floor area image is an image obtained by removing the bookshelves T1 and T2 and the moving persons P, Q, and R from the images of the monitoring target area taken by the cameras A, C, and D. I understand that there is.

When receiving the image data from the cameras A to G, the moving object area image extracting units 4-1 to 4-7 photograph the persons P, Q, and R that are moving objects from the images of the monitoring target area indicated by the image data. The image of the moving object area is extracted, and image data indicating the moving object area image is stored in the moving object area image memories 5-1 to 5-7.
In the image data indicating the moving object region image, the pixel value in the pixel in the moving object region is the pixel value of the image taken by the camera, and the pixel value in the pixel other than the moving object region is a predetermined constant value.
Hereinafter, the moving body region image extraction processing by the moving body region image extracting units 4-1 to 4-7 will be described in detail.

In advance, the cameras A to G capture an image of a monitoring target area when no human is present.
When the moving object region image extracting units 4-1 to 4-7 receive image data from the cameras A to G after the actual start of monitoring, the moving object region image extracting units 4-1 to 4-7 configure the images of the monitoring target region indicated by the image data. The pixel value of the pixel is compared with each pixel value recorded in advance (the pixel value of each pixel constituting the image of the monitoring target area when no human is present) (the same pixel position in the image) The pixel value difference between the pixel values is determined to be a pixel in the moving object region.
Alternatively, when there is a change in the pixel values of a predetermined number or more pixels within a certain range on the image of the monitoring target area, it is determined that these change points are pixels in the moving object area.

In the above example, the image of the monitoring target area when no human is present is compared with the latest image captured by the cameras A to G. However, the latest image captured by the cameras A to G is shown. In addition to the images, for example, the images taken 1 second, 2 seconds, 3 seconds before are recorded, the latest images taken by the cameras A to G, 1 second, 2 seconds ago, 3 seconds An image of the moving object region may be extracted by comparing the previously captured image (compare pixel values at the same pixel position in the image).
That is, the moving body region image extracting units 4-1 to 4-7 are pixels within the moving body region in which the pixel value difference is equal to or greater than a certain value in all the images one second before, two seconds ago, and three seconds ago. Judge that there is.
On the other hand, if the pixel value difference is less than a certain value in any of the images 1 second, 2 seconds, and 3 seconds ago, it is determined that the pixel is a pixel other than the moving object region.

When the moving object region image extracting units 4-1 to 4-7 determine that the pixels constituting the image of the monitoring target region are pixels in the moving object region, the pixel values of the pixels are captured by the cameras A to G. The pixel value of the image is set, and the pixel value is stored in the moving object region image memories 5-1 to 5-7.
On the other hand, if it is determined that the pixels constituting the image of the monitoring target area are pixels other than the moving body area, the pixel value of the pixel is set to a predetermined constant value, and the pixel value is set for the moving body area image. Store in the memories 5-1 to 5-7.
As shown in FIG. 4, the moving body region image is an image obtained by removing the bookshelves T <b> 1 and T <b> 2 and the floor region from the monitoring target region images captured by the cameras A, B, and C.

Here, the moving body region image extraction units 4-1 to 4-7 extract the moving body region image from the monitoring target region image, and the image data indicating the moving body region image is stored as the moving body region image memory 5-1. Although shown about what is stored in -5-7, you may enable it to track a moving body area | region and to identify a several moving body area | region.
Specifically, when the moving object region image extracting units 4-1 to 4-7 extract the moving object region image from the monitoring target region image, for example, the moving object region image extracting unit 4-1 to 4-7 uses the known Mean-Shift method. Keep track of the area.

That is, when the moving object region image extracting units 4-1 to 4-7 extract the moving object region image from the monitoring target region image, the moving object region image is stored as teacher data.
Next, when the moving body region image extracting units 4-1 to 4-7 receive the next captured image from the cameras A to G, for example, after 0.5 seconds, the moving body region indicated by the teacher data in the captured image is displayed. By identifying a region having the same pattern as the image, the movement destination of the moving object region is specified.

When the moving object region image extracting units 4-1 to 4-7 succeed in tracking the moving object region, the moving object region image extracting units 4-1 to 4-7 store, for example, 0 in the moving object region image memories 5-1 to 5-7. The same management number is assigned to the moving object area in the image five seconds ago and the moving object area in the latest image, and the management number is stored together with the image data in the moving object area image memory 5-1 to 5- Record in 7.
As a result, even when a plurality of persons are photographed in the monitoring target area, if the degree of overlap is equal to or less than a certain pixel value, the areas where the plurality of persons are photographed are extracted as separate moving object areas. It becomes possible.

The camera data acquisition unit 6 receives camera data (for example, data indicating the optical axis direction of the camera, the installation position, the installation height, the focal length, etc.) from the cameras A to G and the pan / tilt function units of the cameras A to G (not shown). ) Is acquired, and the camera data is stored in the camera data memory 7.
However, in the first embodiment, it is assumed that the cameras A to G are fixed cameras, and the camera data is a fixed value.

When the floor area image extracting units 2-1 to 2-7 store the image data indicating the floor area image in the floor area image memories 3-1 to 3-7, With reference to the camera data stored in the camera data memory 7 and the building structure data stored in the building structure data memory 8, the floor area image memories 3-1 to 3-7 store the data. The floor area image indicated by the existing image data is projected onto the plane position in the Cartesian coordinate system of the monitored area to generate a plane projection image facing the predetermined direction, and the image data indicating the plane projection image is converted into the plane projection image. Stored in the memory 10-1 to 10-7.
Hereinafter, the process of generating a planar projection image in the planar projection units 9-1 to 9-7 will be specifically described.

  The plane projection units 9-1 to 9-7 project the floor area image to the plane position in the orthogonal coordinate system (the coordinate system of the building structure data) of the monitoring target area in order to generate a plane projection image. The “projection” is obtained by obtaining the plane distance (distance in the photographing direction and distance in the direction perpendicular to the distance in the photographing direction) from the cameras A to G to each pixel in the floor region, thereby orthogonal to the monitoring target region. This means that the pixel value of each pixel in the floor area is set with respect to the plane position in the coordinate system (the coordinate system of the building structure data).

Here, FIG. 5A is an explanatory diagram showing the monitoring target region when the optical axis (center) direction of the camera is viewed from the side in parallel with the floor surface of the monitoring target region. FIG. 5B is an explanatory diagram illustrating a display screen example.
In FIG. 5, “hc” is the camera installation height (distance from the floor surface to the camera installation position), “θp” is the optical axis direction of the camera (the angle at which the camera is directed downward from the horizontal plane), “f “Is the focal length of the camera, and“ Sh ”is the vertical length of the light receiving surface of the camera.
For example, as shown in FIG. 5 (a), assuming that the range in the vertical direction of the image taken by the camera is point M to point J, up to the position where point J, point K, point L, and point M appear. The distance of the light receiving surface is represented by “yJ”, “yK”, “yL”, “yM” as shown in FIG.

In order to project the floor area image to a planar position in the orthogonal coordinate system of the monitoring target area, the distance (shooting) from the point P to each point in the floor area (for example, point J, point K, point L, point M, etc.) It is necessary to obtain the direction distance).
For example, the distance LP from the point P to the point L can be obtained as follows.
First, since ∠LOC is expressed by the following formula (1), LP / hc is expressed by the following formula (2).
∠LOC = arctan (yL / f) (1)
LP / hc = tan ((π / 2) − (θp-arctan (yL / f))) (2)
From the equation (2), the distance LP can be obtained as follows.
LP = hc × tan ((π / 2) − (θp−arctan (yL / f))) (3)
Here, π represents the circumference ratio.

FIG. 6 is an explanatory diagram showing which point in the real space the pixel in the image of the monitoring target area corresponds to.
6A shows a photographed image of the camera C in FIG. 2, and FIG. 6B is a plan view of the positions of the camera C and the bookshelves T1 and T2 as viewed from above.
In FIG. 6, O is the installation position of the camera C, and the camera C is installed at a height of hc from the floor surface.
Point A is a corner point between the bookshelf T1 and the floor, and point B is a corner point between the bookshelf T2 and the floor. Point C is a point where a line obtained by projecting the optical axis onto the floor and an auxiliary line intersect. The auxiliary line in FIG. 6 is parallel to the light receiving surface.
Point D is a point in contact with the floor at the left end of the image captured by camera C, and point E is a point in contact with the floor at the right end of the image captured by camera C. Point F is a point where a line connecting the point DE and a line obtained by projecting the optical axis onto the floor intersects.

In the figure, “Sh” and “f” have the same meaning as “Sh” and “f” in FIG.
As shown in FIG. 6B, the distances of the light receiving surfaces from the left and right centers of the image to the positions where the points A, B, D, and E appear are “xA”, “xB”, “xD”, “ xE ".
In order to project the floor area image to a planar position in the orthogonal coordinate system of the monitoring target area, each point (for example, point A, point B, etc.) in the floor area from the left and right center (for example, point C, point F, etc.) of the image. It is necessary to obtain a distance to a point D, a point E, etc. (a distance in a direction orthogonal to the distance in the shooting direction).

For example, the distance AC from the point C to the point A that is the center of the left and right of the image can be obtained as follows.
First, since ∠AOC is expressed by the following formula (4), AC / CO is expressed by the following formula (5).
∠AOC = arctan (xA / f) (4)
AC / CO = tan (∠AOC) = xA / f (5)
However, since the height of the point O is known, the distance CO between the point C and the point O can be obtained by the method explained in FIG.
From the equation (5), the distance AC can be obtained as follows.
AC = CO × xA / f (6)

As described above, the planar projection units 9-1 to 9-7 are distances (in the shooting direction) from the point P to each point (for example, the point J, the point K, the point L, the point M, etc.) in the floor area. Distance) and the distance (for example, the distance in the photographing direction) from the center of the image (for example, point C, point F, etc.) to each point (for example, point A, point B, point D, point E, etc.) in the floor area. (Distance in the orthogonal direction) and the pixel of each pixel in the floor area with respect to the plane position indicated by those distances (the plane position in the orthogonal coordinate system of the monitoring target area (coordinate system of building structure data)) By setting the value, a planar projection image is generated.
For example, if the point J in FIG. 5 and the point D in FIG. 6 are the same pixel point, by setting the pixel value of the pixel with respect to the plane position indicated by the distance (JP, DF), The pixel is projected onto a planar position.

However, the plane projection units 9-1 to 9-7 generate the plane projection image so that the plane projection image faces a predetermined direction.
For example, if the predetermined direction is set to a direction that coincides with the shooting direction of the camera C, in the case of the camera D, since the shooting direction is opposite to the camera C, it is generated from the image shot by the camera D. As for the planar projection image, the viewpoint direction is rotated by 180 degrees so as to face the same direction as the planar projection image generated from the image captured by the camera C.
Rotate the viewpoint direction so that the planar projection image generated from the images captured by other cameras A, B, E, F, and G also faces the same direction as the planar projection image generated from the image captured by the camera C To generate.
Since the processing itself for generating the planar projection images so as to face the same direction by rotating the viewpoint direction of the planar projection images generated from the captured images of the cameras A to G is a known technique, here Then, detailed description is abbreviate | omitted.

In the moving object image pasting units 11-1 to 11-7, the moving object region image extracting units 4-1 to 4-7 extract images of the moving object region, and image data indicating the moving object region images is stored in the moving object region image memory 5- When stored in 1-5-7, the camera data stored in the camera data memory 7 and the building structure data stored in the building structure data memory 8 are referred to on the orthogonal coordinate system of the monitoring target area. A plate model (a plate model is, for example, a rectangular parallelepiped having the width and height of the moving object, and the length of the depth of the rectangular parallelepiped is a predetermined value) in the area where the moving object exists, Processing for pasting the moving body region images stored in the moving body region image memories 5-1 to 5-7 on the surface of the board model is performed.
Hereinafter, the processing content of the moving body image sticking part 11-1 to 11-7 is demonstrated concretely.

First, the processing contents when a plate model is installed in a region where a moving object exists on the orthogonal coordinate system of the monitoring target region will be described.
FIG. 7 is an explanatory diagram showing a search process for a region where a moving object exists.
7A shows a planar projection image generated from an image taken by the camera C, and FIG. 7B shows a moving object region image extracted from the image taken by the camera C.

First, the moving object image pasting units 11-1 to 11-7 search for an area where a moving object exists on the orthogonal coordinate system of the monitoring target area.
That is, the moving body image pasting units 11-1 to 11-7 search for pixels in the planar projection image with a certain angle difference radially from the camera position O, as shown in FIG.
As a search result, a pixel position that once becomes a pixel on the floor surface and is no longer a pixel on the floor surface is recorded as a candidate point for installing the plate model.
In the example of FIG. 7A, point A, point B, and point C are recorded as candidate points.

Next, as shown in FIG. 7B, the moving object image pasting units 11-1 to 11-7 search the moving object region image from the bottom to the top for every predetermined pixel to obtain the pixel position that hits the moving object. .
Then, the pixel position that hits the moving object is projected to the position on the plane in the same manner as the plane projection units 9-1 to 9-7, and the candidate points (point A, point obtained from the position on the plane and the plane projection image) B, find the distance of point C).
The moving body image pasting units 11-1 to 11-7 determine candidate points within a certain distance as places where the board model is installed.
In the example of FIG. 7, since the points B and C are candidate points within a certain distance, the place where the plate model is installed is determined. However, since the point A is not a candidate point within a certain distance, It is not decided where to install.

When the moving object image pasting units 11-1 to 11-7 determine candidate points on which the plate model is to be installed, a plate model that gives a height to a line segment connecting these candidate points with a broken line (or an approximate straight line) ( 3D model) is generated.
In addition, when a moving body is tracked by the moving body region image extraction units 4-1 to 4-7 and a plurality of moving body region images are extracted, processing for searching each moving body region image from the bottom to the top for each predetermined pixel. I do.
Thereby, the correspondence between the candidate points on the planar projection image and each moving object region image is achieved.
However, in this case, the moving object image pasting units 11-1 to 11-7 associate the plate model with the management number of the moving object tracked by the moving object region image extracting units 4-1 to 4-7.

Next, the moving body image pasting units 11-1 to 11-7 determine the height of the plate model.
Specifically, when the moving object image pasting units 11-1 to 11-7 determine the place where the plate model is installed on the moving object region image, as shown in FIG. 7B, on the moving object region image, Search from top to bottom to find the top point of the detected moving object.
FIG. 8 is an explanatory diagram for determining the height of the plate model. In the example of FIG. 8, the point L ₁ is the highest point, and the point K is the position where the plate model is installed.
The symbols in FIG. 8A have the same meaning as the symbols in FIG.

Next, the moving object image pasting units 11-1 to 11-7 are arranged in the same manner as the plane projecting units 9-1 to 9-7 to install the position P on the plane where the camera is installed and the plate model. A distance PK of K is calculated.
However, regarded as the depth of a monitored object body is a thin object, assumed point of the upper end of the plate model for obtaining and L _2. Point L ₂ is an intersection of a line extending directly above point K and a line connecting camera position O and point L ₁ .
That is, since ∠POC is expressed by the following formula (7), the distance PK is expressed by the following formula (8).
∠POC = π / 2−θp (7)
PK = OL ₂ × sin (∠POC) (8)

Next, the moving object image pasting unit 11-1 to 11-7, the distance is a calculation formula of PK from equation (8) obtains distances OL _2, obtains the height KL ₂ from the plane position to the point L _2.
OL ₂ = PK / sin (∠POC) (9)
KL ₂ = hc−OL ₂ × cos (∠POC) (10)

When the moving object image pasting units 11-1 to 11-7 determine the location where the plate model is installed and the height of the plate model as described above, the moving body image pasting units 11-1 to 11-7 install the plate model at the determined location. The moving object region image indicated by the image data stored in the moving object region image memories 5-1 to 5-7 is pasted on the surface of the plate, and the plate data indicating the plate model on which the moving object region image is attached is used for the plate model. Store in the memories 12-1 to 12-N.
Note that the board data indicates the coordinates of the three-dimensional position of the board model (VRML (Virtual Reality Modeling Language) format, or coordinate data that can be expressed using OpenGL), and how to attach the textured moving object area image. Data consisting of values.

Here, FIG. 9 is explanatory drawing which shows sticking of the moving body area | region image with respect to a board model.
As shown in FIG. 9, when viewed from a certain direction, a plate model is installed so that a moving body can be seen standing at a position where the floor surface is interrupted by another color, and a moving object region image is displayed on the surface of the plate model. It is pasted.
In addition, when several moving body area | region images are extracted by tracking by the moving body area | region image extraction parts 4-1 to 4-7, board data are produced | generated about each moving body area | region image.
Thereby, even when a plurality of persons are shown, the plurality of persons can be displayed separately.

When the moving image pasting units 11-1 to 11-7 store the board data indicating the board model to which the moving body area image is pasted in the board model memories 12-1 to 12-N, the image composition unit 13 The plane projection image indicated by the image data stored in the plane projection image memories 10-1 to 10-7 and the moving object area image indicated by the plate data stored in the board model memories 12-1 to 12-7 are as follows. The pasted image of the board model is synthesized, and image data indicating the synthesized image is stored in the synthesized image memory 14.
Hereinafter, the image composition processing in the image composition unit 13 will be described in detail.

FIG. 10 is an explanatory diagram illustrating a composite image of a planar projection image and a plate model generated from an image captured by the camera C, and a planar projection image and a plate model generated from an image captured by the camera D.
Since the shooting directions of the camera C and the camera D are opposite directions, as described above, the planar projection image generated from the image shot by the camera D is rotated by the viewpoint direction by 180 degrees. The image is directed in the same direction as the planar projection image generated from the photographed image.
As a result, in the example of FIG. 10, the viewpoint directions of the planar projection image generated from the image captured by the camera C and the planar projection image generated from the image captured by the camera D are both the same. The installation position is considered as a viewpoint.

The image compositing unit 13 uses a planar projection image generated from images captured by the cameras A to G and a plate model to which a moving object region image generated from the images captured by the cameras A to G is attached. Since it is described using the coordinate system and the coordinate origin of the building structure data (the viewpoint directions are the same), as shown in FIG. 10, the image data indicating the planar projection image and the moving object region image are pasted. A composite image is generated by superimposing the plate data indicating the plate model that is present as one piece of data.
However, regarding pixel values, the points where two or more planar projection images overlap are averaged.
The board data may overlap. Moreover, you may remove about the board data within a fixed distance among several board data.

Further, textures indicated by plate data within a certain distance may be compared using the Mean-Shift method, and if the textures match, one texture may be deleted. Thereby, the overlapping display can be reduced.
In addition, even when there is plate data in the floor area that can be observed from two cameras at the same time, the textures indicated by the plate data are compared using the Mean-Shift method, and when the textures match, One texture may be erased.

  Note that the image data indicating the synthesized image generated by the image synthesizing unit 13 is obtained by combining a texture corresponding to a pixel value with a floor having a single square and a VRML format in which plate data is combined with this. Become. Alternatively, it may be a coordinate value and texture data group that can be described in OpenGL.

When the image combining unit 13 stores the image data indicating the combined image in the combined image memory 14, the image display unit 15 displays the combined image indicated by the image data on a display (not shown).
That is, the image display unit 15 displays the composite image on the display by using VRML viewer software, for example, so that the viewpoint direction of the composite image indicated by the image data matches the arbitrary viewpoint position indicated by the operator. .
FIG. 11 is an explanatory diagram showing a display example of a composite image by the image display unit 15.
In FIG. 11, an image in which persons P, Q, and R are on the floor is displayed, and bookshelves T1 and T2 that are shielding objects are not displayed. In FIG. 11, the display of the outer wall of the building is omitted for convenience of explanation.

Here, although the image display unit 15 has been described as displaying the composite image on the display, any one of the images taken by the cameras A to G may be selected and displayed.
In this case, on the screen display of the camera, there is provided means for reading the VRML format textured board data and compositing and displaying it.
The VRML display viewpoint position and direction are matched with the installation position and direction of the camera, and the textured board data is displayed on the image captured by the camera.
Further, not only the board data but also floor data or a part thereof may be displayed in a synthesized manner. The floor data and board data may be displayed semi-transparently.

Further, when displaying the composite image on the display, the image display unit 15 may display the management number assigned to the moving object region by the moving object region image extracting units 4-1 to 4-7.
In this case, for example, when the moving object that is reflected in the image taken by the camera A enters the angle of view of the camera B within a certain time from the time when the object moves out of the angle of view of the camera A, the moving object is newly added. When it is detected, the same management number is displayed as the same moving object.
However, if the position of the plate data on the floor of the moving object detected by the camera A and the position of the moving object on the floor of the moving object detected by the camera B are within a certain distance, they are recognized as the same moving object. .
Alternatively, for confirmation, the textures of these plate data may be compared by the Mean-Shift method. As a result, it is possible to track and display the action of a person who has been photographed by any camera.

Further, the management number added to the board data and the position of the board data may be recorded.
As a result, it is possible to record information indicating where a person was and where he / she went.

When there is a moving body region extracted as a lump of moving body by the moving body region image extraction units 4-1 to 4-7, the lump of moving body may then be separated and two moving body regions may be extracted.
In such a case, the moving body region image extraction units 4-1 to 4-7 assign management numbers to the two moving body regions, respectively.
As for a lump of plate data, two pieces of management data are given after the fact as a moving body in which two moving bodies overlap.
On the other hand, when two moving object areas are extracted as a lump of moving objects after the two moving object areas are detected, the management numbers of the two moving objects are assigned to the lump of plate data.
As a result, the image display unit 15 can display two management numbers when two persons are solidified, and thus can also represent an overlap of persons.

  As is apparent from the above, according to the first embodiment, the floor area image that extracts the image of the floor area in which the floor surface is photographed from the images of the monitoring target areas photographed by the cameras A to G. The moving object region image extracting units 4-1 to 2-1 that extract the moving object region image in which the moving object is photographed from the images of the monitoring target regions photographed by the extraction units 2-1 to 2-7 and the cameras A to G. 4-7 and the plane data in the orthogonal coordinate system of the monitoring target area using the floor area image extracted by the floor area image extracting units 2-1 to 2-7 using the camera data indicating the installation status of the cameras A to G Plane projection units 9-1 to 9-7 that project to positions and generate plane projection images, and a plate model is installed in a region where the moving object exists on the orthogonal coordinate system of the monitoring target region, and the plate Extracted by the moving object region image extracting units 4-1 to 4-7 on the surface of the model The moving object image pasting units 11-1 to 11-7 for pasting the moving object region images are provided, and the image synthesizing unit 13 generates the planar projection images and moving object image pasting units generated by the plane projecting units 9-1 to 9-7. Since the image of the plate model to which the moving body region image is pasted by 11-1 to 11-7 is synthesized, the complexity of the pattern / shape of the moving body in the monitoring target region and the shielding object in the monitoring target region Regardless of the presence or absence, there is an effect that images taken by a plurality of cameras can be combined.

Embodiment 2. FIG.
In the first embodiment, the cameras A to G that are capturing the monitoring target area are fixed cameras. However, in addition to the cameras A to G, the monitoring target area is captured using a moving camera. You may do it.
However, in this case, it is necessary to provide camera detection means for detecting the installation status of the mobile camera and store the camera data indicating the installation status of the mobile camera in the camera data memory 7.
Note that when the floor area detection unit 2 extracts a floor area from an image captured by the moving camera, the floor area from the image captured by the fixed camera is determined by regarding a point on the floor surface immediately below the moving camera as an origin. Can be extracted by the same processing as in the case of extracting.

In the second embodiment, the processing contents of the camera data acquisition unit 6 will be specifically described below, assuming that the camera data acquisition unit 6 constitutes a camera detection unit.
For example, when the moving camera or the fixed camera is a pan / tilt / zoom camera (hereinafter referred to as “PTZ camera”), as shown in FIG. 12, rotation and zooming with two axes (pan axis and tilt axis) are possible. is there.
In this case, when the operator operates the PTZ camera, the angle of the pan axis and the tilt axis and the focal length changed by the zoom are output from the PTZ camera as camera data, and the camera data acquisition unit 6 acquires the camera data. And stored in the camera data memory 7.

The camera data acquisition unit 6 can acquire the camera data indicating the installation position of the moving camera as follows.
For example, when a mark for a mark as shown in FIG. 13 is attached to the casing of the mobile camera (in the example of FIG. 13, a mark of ▽ is attached, so that the mark can be easily detected. The camera data acquisition unit 6 is photographed by a built-in camera (not shown) or a moving camera monitoring camera (not shown). The movement position and direction are specified by detecting a mark attached to the casing of the moving camera from the captured image.

The marks are attached to the front and rear, left and right of the mobile camera housing so that the direction of the mobile camera can be seen. Further, the height at which the mark is attached to the casing of the mobile camera is determined in advance.
If the mark installation height with respect to the casing of the moving camera is known, the plane position of the planar projection image is obtained using the mark installation height instead of the installation height hc (see FIG. 5) of the cameras A to G which are fixed cameras. Can be requested.

FIG. 14 is an explanatory diagram illustrating an example in which the moving camera is moved to a position where the fixed camera cannot be captured.
For example, if there are two left and right wheels and one supporting wheel as wheels of the moving camera, it can move in the monitoring target area.
At this time, if an odometry measuring instrument for measuring the rotational speed of the wheel is installed for the two left and right wheels, the odometry measuring instrument can determine a change in the moving distance and direction from the rotational speed of the left and right wheels. it can.

The camera data acquisition unit 6 acquires the camera data indicating the position and direction of the moving camera as described above. The camera data output from the moving camera is, for example, the position and direction of the moving camera one second ago. Is the data indicating the displacement in the current position and direction with reference to.
The camera data acquisition unit 6 records the position and direction of the moving camera specified from the image taken by the built-in camera (not shown) or the camera for monitoring the moving camera (not shown) together with the time. If the built-in camera (not shown) or the like cannot detect the mark attached to the case of the moving camera, it moves using the position and direction of the moving camera as the initial value. The position and direction of the moving camera are updated by adding the position displacement and the direction displacement transmitted from the camera.

  As apparent from the above, according to the second embodiment, when a moving camera is included in a plurality of cameras, the camera detecting means for detecting the installation status of the moving camera is provided. It is possible to generate a plate model on which a planar projection image and a moving object region image are pasted from an image captured by a moving camera, and as a result, the presence of a shielding object makes it possible to capture with a fixed camera alone. Even for a moving object that cannot be performed, the moving camera can take an image and display the moving object.

In the first and second embodiments, the moving body region image extraction units 4-1 to 4-7 have shown the extraction of the moving body region from the images taken by the camera. If all the parts 4-7 indicate pixel values other than the floor area are extracted as moving object areas, for example, the bookshelves T1 and T2 in FIG. 2 can also be displayed as moving objects.
In addition, if a part showing pixel values other than the floor area is extracted as a moving object only for an image taken by a moving camera, a scene of a blind spot that cannot be photographed by a fixed camera is transferred to the moving camera. Can be reproduced and displayed.
FIG. 15 shows an example in which only a specific shelf is displayed transparently in conjunction with the zoom function of the camera.

  1-1 to 1-N camera, 2-1 to 2-N floor area image extracting unit (floor area image extracting means), 3-1 to 3-N floor area image memory, 4-1 to 4-N moving object Area image extraction unit (moving body area image extraction means), 5-1 to 5-N moving body area image memory, 6 camera data acquisition section (camera detection means), 7 camera data memory, 8 building structure data memory, 9 -1 to 9-N plane projection unit (plane projection unit), 10-1 to 10-N plane projection image memory, 11-1 to 11-N moving image pasting unit (moving body image pasting unit), 12-1 to 12-N board model memory, 13 image composition section (image composition means), 14 composite image memory, 15 image display section (image display means).

Claims (2)

  A plurality of floor area images that extract a floor area image of a floor surface from a plurality of cameras that shoot a monitoring target area from different directions and images of the monitoring target area captured by the camera. Extracting means, a plurality of moving object area image extracting means for extracting an image of a moving object area in which a moving object is imaged from an image of a monitoring target area imaged by the camera, and camera data indicating the installation status of the camera A plurality of planar projection means for projecting the floor area image extracted by the floor area image extraction means to a planar position in the orthogonal coordinate system of the monitoring target area to generate a planar projection image, and the monitoring A plate model is set in an area where the moving object exists on the orthogonal coordinate system of the target area, and the moving object region image extracted by the moving object region image extracting means is pasted on the surface of the plate model. A plurality of moving body image pasting means, and an image compositing means for synthesizing the planar projection image generated by the plurality of planar projecting means and the image of the plate model to which the moving body region image is pasted by the plurality of moving body image pasting means. And an image display means for displaying the image synthesized by the image synthesis means.   2. The composite display device according to claim 1, further comprising camera detection means for detecting an installation state of the moving camera when a plurality of cameras include a moving camera.

Images