JP2008015573A - Image processor, image processing method and image monitoring system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To favorably determine the presence of a predetermined region (a kinetic region) in a real space. <P>SOLUTION: A kinetic region (a region where a moving body exists) is detected from the respective images obtained by photographing with cameras 111, 112 by a kinetic region detecting part 122. The detected kinetic regions are converted to a kinetic region of a plane coordinate system in a coordinate conversion part 123 using conversion information for converting the respective camera images obtained by a conversion information obtaining part 121 to a specified plane coordinate system in the real space. An overlap of the respective converted kinetic regions is detected by an overlap detecting part 124, and based on the detection result, it is determined by a determination part 125 whether or not a kinetic region exists in a specified plane in the real space. For example, when an overlap region is enough large, the existence is decided, and when an overlap region is small, noise is decided and excluded. Thus, the existence of a moving body of such a size to extend over the respective planes can be determined by conversion to a plurality of plane coordinate systems. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image monitoring system. Specifically, the present invention detects a plurality of areas, which are predetermined areas, from a plurality of images obtained by photographing a real space, and converts each of the detected plurality of areas into a designated plane coordinate system in the real space. An image processing apparatus or the like that can determine the presence of a predetermined region in the real space by determining the presence of the predetermined region in the specified plane in the real space by overlapping the plurality of converted images. It is related to.

  In three-dimensional object measurement using images, a stereo image processing algorithm is generally used in many cases. However, the stereo image processing system has disadvantages such as a large amount of computation, which requires hardware, and it is difficult to lengthen the inter-camera interval (baseline), so accurate calibration is necessary. It has been.

  The visual volume intersection method does not need to acquire an accurate three-dimensional shape as compared with stereo image processing, and can measure a three-dimensional shape by a simple method using more cameras. In the view volume intersection method, silhouette images of a plurality of viewpoints are acquired, and their shapes are restored by back projecting them onto a three-dimensional space. Although the accuracy of three-dimensional measurement is lower than when using a stereo image processing algorithm, it is sufficient to determine the approximate size and structure of an object. This visual volume intersection method is described in Non-Patent Document 1, for example, and FIG. 11 shows a conceptual diagram thereof.

In this visual volume intersection method, the existence of an object in a certain space is determined by projecting the image obtained by each camera onto a certain plane in the real space, overlaying each image, and looking at the difference in texture. . If the textures of the converted images are the same, it is determined that there is an object there. An object detection method using this visual volume intersection method is described in, for example, Patent Document 1 and the like.
U Small Army, Toshikazu Wada, Shogo Tokai, Takashi Matsuyama, "Parallel volumetric intersection method using inter-plane perspective projection", Vol.42No.SIG6 (CVIM2) IPSJ Transactions on Computer Vision and Image Media, June 2001 . JP 2002-366932 A

  In the object detection method described in Patent Literature 1 and the like, the presence of an object is determined by looking at the difference in texture of each image projected on a certain plane in real space, and the load on the processing unit is large. In addition, when the texture is not sufficiently obtained at night or when there is a lot of noise, it is difficult to compare the textures, and thus the presence of the object cannot be determined satisfactorily.

  An object of the present invention is to make it possible to satisfactorily determine the presence of a predetermined area in real space.

The concept of this invention is
An area detection unit that detects first to Nth areas, which are predetermined areas, from first to Nth (N is an integer of 2 or more) images obtained by photographing a real space;
The first to N-th images detected by the region detection unit using the first to N-th coordinate conversion information for converting the first to N-th images respectively into a designated plane coordinate system in the real space. A coordinate conversion unit for converting each of the Nth region into a region of the planar coordinate system;
An overlap detection unit for detecting an overlap of a plurality of regions of the planar coordinate system obtained by the coordinate conversion unit;
An image processing apparatus comprising: a determination unit that determines presence of a predetermined region in a specified plane in the real space based on a detection result of the overlap detection unit.

  In the present invention, first to Nth areas, which are predetermined areas, are detected from first to Nth images obtained by photographing a real space. The first to Nth images are obtained, for example, by photographing the real space so that the fields of view overlap each other by the first to Nth image photographing units. The predetermined area is, for example, a moving object area (an area where a moving object exists in an image) or the like, and can be detected by some method. For example, the moving object region is detected by using a conventionally known inter-frame difference process or background difference process.

  The first to Nth images detected in this way are each converted into a designated plane coordinate system in the real space. In this case, first to Nth coordinate conversion information for converting each of the first to Nth images into a designated plane coordinate system in the real space is used. As the first to N-th coordinate conversion information, information set in advance or information obtained at an appropriate timing based on the first to N-th images provided with a conversion information acquisition unit is used. By acquiring the coordinate conversion information at an appropriate timing, for example, even when the image capturing unit (camera) moves in the wind or the like, the optimal coordinate conversion information can always be used.

  Overlap of a plurality of areas converted into the plane coordinate system is detected. Based on the detection result, the presence of the predetermined area in the real space is determined. For example, when the overlap area has a certain size or more, it is determined that the predetermined area exists on the plane. This determination result is presented, for example, on a monitor unit (PC).

  In this case, you may make it further provide the metadata conversion part which converts the position of the real space determined that the predetermined area | region exists into the metadata which shows this position. For example, by displaying the position where the predetermined area exists on the display unit with this metadata, the user can easily grasp the real space position where the predetermined area exists.

  As described above, the present invention detects a plurality of areas, which are predetermined areas, from a plurality of images obtained by photographing a real space, and each of the detected plurality of areas is designated in the real space. It is converted into a plane coordinate system, and the presence of a predetermined area in a specified plane in the real space is determined by the overlap of the plurality of converted images. As described in Patent Document 1 described above, each conversion is performed. The difference in the texture of the image is not observed, the load on the processing unit is reduced, and the presence of the predetermined area can be determined well even at night or when there is a lot of noise.

  The designated plane coordinate system in the real space is a plurality of plane coordinate systems respectively corresponding to a plurality of planes existing at a predetermined interval in a direction orthogonal to the plane, and the first coordinate system detected by the area detection unit. The 1st to Nth areas may be converted into areas of a plurality of plane coordinate systems, respectively. In this case, it is possible to determine the presence of a predetermined area having a certain size in the above-described orthogonal direction intersecting a plurality of plane coordinate systems.

  According to the present invention, a plurality of areas that are predetermined areas are detected from a plurality of images obtained by photographing the real space, and the detected plurality of areas are respectively converted into designated plane coordinate systems in the real space, The presence of the predetermined area in the designated plane in the real space is determined by the overlap of the plurality of converted images, and the presence of the predetermined area in the real space can be determined well.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 shows a schematic configuration of an image monitoring system 100 as an embodiment. The image monitoring system 100 includes a monitoring unit 110, a processing unit 120, and a monitor unit 130.

  The monitoring unit 110 has an image input function using a camera or the like. The monitoring unit 110 can be configured using two or more cameras, and the greater the number of cameras, the higher the determination accuracy of the presence of a predetermined area in the real space described later in the processing unit 120. In this embodiment, it is assumed that two cameras are used in order to simplify the description.

  The processing unit 120 has a function of detecting a predetermined area in the real space, in this embodiment, a moving object area (an area where a moving object exists), based on an image obtained by photographing with the camera of the monitoring unit 110. . The monitor unit 130 has a function of presenting the determination result of the processing unit 120 to the user.

  For example, the monitoring unit 110 is configured by a camera unit, and the processing unit 120 and the monitoring unit 130 are configured by a PC (personal computer). Further, for example, the monitoring unit 110 and the processing unit 120 are configured by a camera unit, and the monitor unit 130 is configured by a PC. In this case, the processing result is sent as metadata from the camera unit to the PC and presented.

  Further, for example, a part of the monitoring unit 110 and the processing unit 120 is configured by a camera unit, and a part of the processing unit 120 and the monitoring unit 130 are configured by a PC. In this case, the image and the processing result are sent from the camera unit to the PC as metadata, the remaining processing is performed by the PC, and the processing result is presented. By performing part or all of the processing unit in the camera unit, it is possible to reduce subsequent processing.

  In the monitor unit 130, an image based on an image signal from the camera of the monitoring unit 110 is displayed, and a moving object region detected by processing is displayed on the image so as to be superimposed. Further, the position of the real space in the moving object region is displayed. Displayed characters, symbols, marks, etc. are displayed. This display of characters, symbols, marks, etc. is made possible by converting the position (coordinates) of the real space in the moving object region into metadata indicating the position with reference to a table, for example.

  In addition, the monitor unit 130 can generate an alarm sound to notify the user when it is determined that a moving object area exists in the real space. In addition, warnings that can attract the user's attention, such as email transmission and vibration, can be used. It can also be used as a trigger for starting image recording at the time of determination.

  FIG. 2 shows a specific configuration of the image monitoring system 100.

  The monitoring unit 110 has two cameras 111 and 112. Each of these cameras 111 and 112 constitutes an image photographing unit. Each of the cameras 111 and 112 captures a real space so that a part of the field of view overlaps each other. For example, the cameras 111 and 112 may be arranged side by side like a so-called stereo camera, or may be arranged at a predetermined distance.

  The processing unit 120 includes a conversion information acquisition unit 121, a moving object region detection unit 122, a coordinate conversion unit 123, an overlap detection unit 124, and a determination unit 125. The conversion information acquisition unit 121 is coordinate conversion information (camera image coordinate system) for converting each of these images into a specified plane coordinate system in the real space based on the images acquired by the cameras 111 and 112. And conversion information between real space coordinate system).

  In this case, the conversion information acquisition unit 121 calculates the conversion information by performing calibration from the position of the marker installed in the real space on the image and the position in the real space, for example. Alternatively, the conversion information is calculated by taking correspondence between several points in the image and several points in the real space without clearly using a marker. For example, it is possible to take action by having a person whose height is known move around in the image. Further, when a characteristic object is reflected in the image, it is not necessary to attach a marker by using it as a clue.

  By calculating the conversion information in the conversion information acquisition unit 121 in advance before performing the determination process of the existence of the moving object region in the real space, the processing speed can be increased. When considering that the cameras 111, 112, etc. move due to a cold or vibration, it is possible to grasp and calibrate the positional relationship every hour.

  Here, the coordinate conversion information calculation process will be further described.

  The coordinate conversion information can be calculated, for example, by arranging a predetermined number, for example, four markers in the real space and photographing them with a camera. At this time, it is assumed that the position of the marker in the real space is known.

The position of a certain point Pi in the camera image coordinates is assumed to be (x _i , y _i , z _h ) ^T. z _h may be considered as a constant (eg, 1). At this time, if the position of the same point in the real space coordinates is expressed as (X _i , Y _i , Z _i ) ^T , the relationship between the camera seat image coordinates and the real space coordinates can be expressed by the following equation (1). Here, λ _i is a constant representing a magnification, and H is a projective transformation matrix.

  Here, the projective transformation matrix H is expressed by equation (2).

By obtaining this projective transformation matrix, the relationship between camera image coordinates and real space coordinates is obtained. By modifying equation (1), λ _i can be obtained as in equation (3).

  By substituting Equation (3) into Equation (1) again, the relational expressions of Equations (4) and (5) can be obtained.

  Then, when rewriting Formulas (4) and (5) in the format of Ah = 0, Formula (6) is obtained.

In order to obtain the projective transformation matrix h = (H ₁₁ H ₁₂ H ₁₃ H ₂₁ H ₂₂ H ₂₃ H ₃₁ H ₃₂ H ₃₃ ) ^T in Equation (6), a point on a plurality of camera images and a point on the real space It can be calculated by preparing a plurality of sets of positions. When a plurality of points are used, A is as shown in Equation (7).

For example, a sufficient number of points on the same plane in real space are prepared in both coordinate systems, and the matrix A is created, whereby h is obtained as an eigenvector for the minimum eigenvalue of A ^T A. As described above, the coordinate conversion information obtained in this way is used when a camera image is projected onto a designated plane in real space. The designation of this plane can be performed by operating the operation unit of the PC, for example.

  The moving body region detection unit 122 detects a moving body region (a region where an object moving in the image exists) from the images obtained by photographing with the cameras 111 and 112, respectively. The moving object region detection unit 122 detects the region of the moving object in the image by using conventionally known background difference processing and inter-frame difference processing.

  Background difference processing is a method of detecting a moving object region by creating a background image that does not include a moving object region and comparing the background image with an input image. The inter-frame difference process is a method of detecting a change area as a moving object area by comparing an input image (current image) with a past image.

  The coordinate conversion unit 123 is detected by the moving object region detection unit 122 from the images of the cameras 111 and 112 using the coordinate conversion information acquired by the conversion information acquisition unit 121 corresponding to the images obtained by the cameras 111 and 112, respectively. Each moving object region thus converted is converted into a designated plane coordinate system in the real space. By this processing, each moving object region is projected and converted to a designated plane in the real space. Here, the designated plane is not limited to a single plane, but may be a plurality of planes existing at a predetermined interval in a direction orthogonal to the plane.

  The overlap detection unit 124 detects an overlap of each moving object region converted into a designated plane coordinate system in the real space by the coordinate conversion unit 123. Based on the detection result of the overlap detection unit 124, the determination unit 125 determines the presence of a moving object region on a specified plane in the real space. For example, it is determined that the moving object region exists when the overlapping region has a certain size or more. Since the overlapping area is projected in the real space, the determination unit 125 can simultaneously specify the position in the real space where the moving object area exists.

  By using the overlapping results of moving object areas in a plurality of planes, for example, it is possible to obtain a determination result that a moving object area exists at the same point in a plurality of planes having different heights from the ground. Detection accuracy can be improved.

  The determination result of the determination unit 125 is sent to the monitor unit 130 as described above. Then, the monitor unit 130 presents the determination result, that is, the detection result of the moving object region in the real space to the user.

  The determination unit 125 can determine whether or not the moving object region exists, and at the same time obtain the position in the real space where the moving object region exists as described above. The position of the area where the moving object area overlaps coincides with the position of the projected real space. Therefore, the position of the moving object area can be sent to the monitor unit 130 at the subsequent stage, and the position of the moving object area can be presented to the user. For example, if the place being monitored is a train platform, detailed information on how many eyes the moving object area exists can be presented. At this time, the processing result obtained by the processing unit 120 is converted to metadata, for example, and sent to the monitor unit 130. In this case, the determination unit 125 also constitutes a metadata conversion unit.

  Next, the flow of processing in the image monitoring system 100 shown in FIGS. 1 and 2 will be described.

  FIG. 3 roughly shows the flow of processing. The processing is roughly divided into two, and includes processing for acquiring coordinate conversion information (step ST1) and processing for actually detecting a moving object region (animal body) in real space (step ST2). In FIG. 3, the moving object region in the real space is detected after the coordinate conversion information is acquired. However, the coordinate conversion information can be acquired while detecting the moving object region.

  FIG. 4 shows the flow of the moving object region detection process.

  After the processing is started, in step ST11, N indicating the number of the processing plane is set to 1, and M indicating the number of the processing camera is set to 1. In this embodiment, for example, the camera number of the camera 111 is 1, the camera number of the camera 112 is 2, and the maximum value Mmax of M is 2. In step ST <b> 12, an image obtained by shooting with the camera of number M is input to the moving object region detection unit 122.

  Next, in step ST13, the moving object region detection unit 122 detects the moving object region from the image of the camera with the number M input in step ST12. In step ST14, based on the coordinate conversion information for converting the image of the camera of number M acquired by the conversion information acquisition unit 121 by the coordinate conversion unit 123, in step ST13, the image of the camera of number M is used. The detected moving body region is converted into a coordinate system of a plane number N in real space.

  Next, in step ST15, it is determined whether or not the plane number N is smaller than the maximum value Nmax. When N <Nmax, the value of N is incremented by 1 in step ST16, and the process returns to step ST14 to shift to the process of converting the image of the camera with the number M into the coordinate system of the next plane. When the plane number N becomes the maximum value Nmax and N <Nmax is not satisfied, the process proceeds to step ST17.

  In step ST17, it is determined whether the camera number M is smaller than the maximum value Mmax. If M <Mmax, the value of M is increased by 1 in step ST18, and the process returns to step ST12 to shift to the coordinate conversion process for the next camera image. When the camera number M becomes the maximum value Mmax and M <Mmax is not satisfied, the process proceeds to step ST19.

  In this step ST19, the overlap detection unit 124 detects the overlap of the moving object regions related to the images of the respective cameras converted into the designated plane coordinate system in the real space as described above. For example, by performing AND processing on all moving object regions, it is possible to detect only regions where moving object regions exist on all planes related to all cameras.

  Next, in step ST20, the determination unit 125 determines whether or not a moving object region exists on a specified plane in the real space based on the detection result in step ST19. For example, if the area detected in step ST19 is sufficiently large, it is determined that the area exists. On the other hand, if the area detected in step ST19 is small, it is excluded as noise. It is also possible to remove the case where the area detected in step ST19 is too large.

  In the flow of the moving object detection process shown in FIG. 4, the coordinate conversion process from image input to each plane is performed sequentially on the image from each camera, but the images from each camera are arranged in parallel. May be processed automatically.

  Next, a specific example of moving object region detection processing in the image monitoring system 100 shown in FIGS. 1 and 2 will be described. In this specific example, the cameras 111 and 112 are installed along a train platform at a predetermined distance, for example, about 50 m apart.

  5A and 5B show examples of images obtained by being photographed by the cameras 111 and 112 and input to the conversion information acquisition unit 121 and the moving object region detection unit 122, respectively. In these image examples of FIGS. 5A and 5B, the person MN, which is a moving object, is on the platform PF and has not entered the track LW. On the other hand, FIGS. 6A and 6B show other examples of images obtained by being photographed by the cameras 111 and 112 and input to the conversion information acquisition unit 121 and the moving object region detection unit 122, respectively. In the image examples of FIGS. 6A and 6B, the person MN, which is a moving object, has entered the track LW. In the moving object region detection process, it is possible to detect that the person MN has entered the track LW in this way.

  7A and 7B show examples of images (images observed from above) obtained by converting the camera images shown in FIGS. 6A and 6B into a coordinate system in real space, respectively. However, in practice, the coordinate conversion unit 123 converts the moving object region detected by the moving object region detection unit 122 into a designated plane coordinate system in the real space, here, a coordinate system of a plane parallel to the ground. Therefore, the moving object areas detected from the camera images shown in FIGS. 6A and 6B are converted into moving object areas MAa and MAb of designated planes in the real space, as shown in FIGS. 8A and 8B, respectively. 8A and 8B show one converted into a coordinate system of one plane, but when a coordinate system of a plurality of height planes parallel to the ground is specified in the real space, Conversion is performed corresponding to the plane of

  8A and 8B, the overlap detector 124 detects the overlap of the moving object areas MAa and MAb converted into the plane coordinate system in the real space. FIG. 9 shows a state in which the moving body areas MAa and MAb overlap. FIG. 10 shows a state where the moving body areas MAa and MAb do not overlap.

  The determination unit 125 determines the presence of a moving object region on a specified plane in the real space based on the overlap detection result. For example, when the overlapping area of the moving object areas MAa and MAb is large as shown in FIG. 9, it is determined that the moving object exists on a specified plane in the real space, and as shown in FIG. When there is no or small MAb overlapping area, it is determined that there is no moving object on the specified plane in the real space.

  In this case, as described above, the coordinate system of a plurality of planes having a plurality of heights parallel to the ground is designated in advance in the real space, and the overlapping area of the moving object regions converted into the respective plane coordinate systems When is large, it can be determined that there is an animal body, such as a person, having a height straddling each plane on the track.

  According to the image monitoring system 100 shown in FIGS. 1 and 2, moving object regions are detected from images obtained by photographing the real space with the cameras 111 and 112, and the detection regions are designated in the real space. It is converted into a plane coordinate system, and the existence of the moving object region in the designated plane in the real space is determined by the overlapping of the converted moving object regions. The difference in texture of each converted image is not seen, the load on the processing unit 120 can be reduced, and the presence of a moving object region can be determined well even at night or when there is a lot of noise.

  In the above-described embodiment, it is possible to determine whether or not a moving object area exists on a plane designated in the real space based on images obtained by the cameras 111 and 112. However, the determination target is not limited to the moving object area, and may be another predetermined area such as a fixed obstacle on the road.

  The present invention can satisfactorily determine the presence of a predetermined region in real space, and can be applied to, for example, an image monitoring system for monitoring obstacles on a track.

It is a block diagram which shows schematic structure of the image monitoring system as embodiment. It is a block diagram which shows the specific structure of the image monitoring system as embodiment. It is a flowchart which shows the flow of a rough process in an image monitoring system. It is a flowchart which shows the flow of the moving body area | region detection process in an image monitoring system. It is a figure which shows an example of the camera image acquired by image | photographing with two cameras installed at predetermined distances along the platform of a train. It is a figure which shows an example (intrusion) of the camera image acquired by image | photographing with two cameras installed at predetermined distance along the platform of a train. An example of an image obtained by converting a camera image into a coordinate system in real space (an image observed from above) is shown. It is a figure which shows the example into which the moving body area | region detected from the camera image was converted into the moving body area | region of the designated plane coordinate system in real space. It is a figure which shows the example which the two moving body area | regions converted into the coordinate system of the designated plane in real space have overlapped. It is a figure which shows the example in which the two moving body area | regions converted into the coordinate system of the designated plane in real space have not overlapped. It is a figure which shows the concept of a visual volume intersection method.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 100 ... Image monitoring system, 110 ... Monitoring part, 111, 112 ... Camera, 120 ... Processing part, 121 ... Conversion information acquisition part, 122 ... Moving body area | region detection part, 123 * ..Coordinate conversion unit, 124 ... overlap detection unit, 125 ... determination unit, 130 ... monitor unit

Claims (7)

An area detection unit that detects first to Nth areas, which are predetermined areas, from first to Nth (N is an integer of 2 or more) images obtained by photographing a real space;
The first to N-th images detected by the region detection unit using the first to N-th coordinate conversion information for converting the first to N-th images respectively into a designated plane coordinate system in the real space. A coordinate conversion unit for converting each of the Nth region into a region of the planar coordinate system;
An overlap detection unit for detecting an overlap of a plurality of regions of the planar coordinate system obtained by the coordinate conversion unit;
An image processing apparatus comprising: a determination unit that determines presence of the predetermined region on a specified plane in the real space based on a detection result of the overlap detection unit. The image processing apparatus according to claim 1, wherein the predetermined area is a moving body area in which a moving object is present. Conversion information for acquiring the first to N-th coordinate conversion information for converting the first to N-th images into the designated plane coordinate system in the real space based on the first to N-th images, respectively. The image processing apparatus according to claim 1, further comprising an acquisition unit. The designated plane coordinate system in the real space is a plurality of plane coordinate systems respectively corresponding to a plurality of planes existing at a predetermined interval in a direction orthogonal to the plane,
The image processing apparatus according to claim 1, wherein the coordinate conversion unit converts the first to Nth areas detected by the area detection unit into areas of the plurality of planar coordinate systems. . The image processing apparatus according to claim 1, further comprising: a metadata conversion unit that converts the position of the real space determined by the determination unit to include the predetermined area into metadata indicating the position. . An area detection step of detecting first to Nth areas, which are predetermined areas, from first to Nth (N is an integer of 2 or more) images obtained by photographing a real space;
The first to N-th images detected in the region detection step using the first to N-th coordinate conversion information for converting the first to N-th images respectively into a designated plane coordinate system in the real space. A coordinate conversion step of converting each of the Nth region to the region of the planar coordinate system;
An overlap detection step of detecting an overlap of a plurality of regions of the planar coordinate system obtained in the coordinate conversion step;
An image processing method comprising: a determination step of determining the presence of the predetermined region on a specified plane in the real space based on a detection result of the overlap detection step. First to Nth (N is an integer greater than or equal to 2) image photographing units that photograph real space so that parts of the field of view overlap each other;
An area detecting unit for detecting first to Nth areas, which are predetermined areas, from the first to Nth images obtained by photographing with the first to Nth image photographing parts;
The first to N-th images detected by the region detection unit using the first to N-th coordinate conversion information for converting the first to N-th images respectively into a designated plane coordinate system in the real space. A coordinate conversion unit for converting each of the Nth region to the region of the planar coordinate system;
An overlap detection unit for detecting an overlap of a plurality of regions of the planar coordinate system obtained by the coordinate conversion unit;
A determination unit that determines the presence of the predetermined region in a specified plane in the real space based on a detection result of the overlap detection unit;
An image monitoring system comprising: a monitor unit that presents a determination result of the determination unit.

Images