JP2005352988A - Image extracting apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an apparatus and method for extracting an image of a moving object, corresponding to a trajectory. <P>SOLUTION: The image extraction apparatus, comprising an image capture unit 1 for capturing an image from a target area, a trajectory capture unit 2 for capturing the trajectory of the moving object, a processing unit 3 for having the functions: (1) integrating the trajectory captured by the image capture unit 1 with the trajectory captured by the trajectory capture unit 2 and (2) extracting the image of the moving object, located on the trajectory from the image captured by the image capture unit 1; while the image capture unit 1 the trajectory capture unit 2 and the processing unit 3 are connected by a network 4. The image capture unit 1 is equipped with a camera 11 and a terminal for camera 12, and the trajectory capture unit 2 is equipped with a laser scanner 21 and a terminal for a scanner 22. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image extraction apparatus for extracting an image of a moving object from an image in a certain area.

  As shown in the following Non-Patent Documents 1 to 3, the present inventors have proposed a technique for extracting a moving locus of a moving body (for example, a pedestrian). According to this technique, a trajectory can be extracted with high accuracy by using a plurality of laser scanners (laser sensors).

By the way, in order to determine the identity of the moving object corresponding to the trajectory acquired at different times, it is preferable to associate the image of the moving object with the trajectory.
Katsuyuki Nakamura, Tsuji, Ryosuke Shibasaki, Koji Sakamoto, Yayoi Osogori, Naosuke Suzukawa, Extraction of pedestrians using a multi-laser scanner, Proc. Of the 10th Image Sensing Symposium, 2004 H. Zhao, R. Shibasaki, "Pedestrian Tracking using Multiple Laser Range Scanners", Proc. Of Computers on Urban Planning and Urban Management, May. 2003 Sendai, Japan. H. Zhao, R. Shibasaki, N. Ishihara, "Pedestrian Tracking using Single-row Laser Range Scanners", Proc. Of IAPR Workshop on Machine Vision Application, Dec. 2002 Nara, Japan.

  The present invention has been made in view of such circumstances. An object of the present invention is to provide an apparatus and method for extracting an image of a moving object corresponding to a trajectory.

  An image extraction apparatus according to the present invention includes an image acquisition unit, a trajectory acquisition unit, and a processing unit. The image acquisition unit is configured to acquire an image of the target area. The locus acquisition unit is configured to acquire a locus of a moving object. The processing unit is (1) a function of integrating the coordinate system of the image acquired by the image acquisition unit and the coordinate system of the trajectory acquired by the trajectory acquisition unit, and (2) located on the trajectory. A function of extracting an image of the moving body from the image acquired by the image acquisition unit is provided.

  The image acquisition unit, the trajectory acquisition unit, and the processing unit may be connected by a network.

  The image acquisition unit may include a camera and a camera terminal.

  The trajectory acquisition unit may include a laser scanner and a scanner terminal.

  The trajectory acquisition unit may include a sensor that scans horizontally to obtain a range image and a sensor terminal. The sensor is, for example, a laser scanner, and the sensor terminal is, for example, a scanner terminal.

  An image processing apparatus according to the present invention includes an integration unit, an extraction unit, and a storage unit. The integration unit is configured to integrate the coordinate system in the image of the moving object acquired by the image acquisition device and the coordinate system in the locus of the moving object acquired by the locus acquisition device. The extraction unit is configured to extract an image of a moving body located on the trajectory from the image acquired by the image acquisition unit.

  The extraction unit is configured to attach an attribute corresponding to the orientation of the moving body estimated from the trajectory to the extracted moving body image and store the image of the moving body in the storage unit. Also good.

The image extraction method according to the present invention comprises the following steps:
(1) obtaining an image in the target region;
(2) obtaining a trajectory of the moving object;
(3) integrating the coordinate system of the acquired image and the coordinate system of the acquired trajectory;
(4) A step of extracting an image of the moving body located on the locus from the image in the target area.

  According to the present invention, it is possible to extract a moving body image corresponding to a trajectory relatively accurately.

(Configuration of the embodiment)
Hereinafter, an image extraction apparatus according to an embodiment of the present invention will be described with reference to the accompanying drawings. This apparatus is mainly composed of an image acquisition unit 1, a trajectory acquisition unit 2, and a processing unit 3 (see FIG. 1). These are connected to the network 4 and can transmit information to each other through the network 4.

  The image acquisition unit 1 includes a plurality of cameras 11 and camera terminals 12 connected to the cameras 11 (see FIG. 2). The camera 11 is usually a moving image camera, but may be a still image camera. The camera 11 is configured to acquire an image of the target area based on a control signal from the camera terminal 12. The target area is appropriately set by the user (described later). The camera terminal 12 is configured by a computer. The function of the camera terminal 12 will be described in detail in the description of the operation of the present embodiment.

  The locus acquisition unit 2 includes a plurality of laser scanners 21 and scanner terminals 22 connected to the scanners 21 (see FIG. 2). The laser scanner 21 can obtain a range image by measuring the distance to the object by scanning in the horizontal direction. Thereby, the locus | trajectory acquisition part 2 can acquire the locus | trajectory of a moving body. Since the configuration of the trajectory acquisition unit 2 is the same as that of the trajectory acquisition device described in the non-patent document, detailed description thereof is omitted. Instead of the laser scanner, other sensors that can obtain a range image (for example, radar using UWB (Ultra Wide Band), millimeter wave radar, etc.) can be used.

  The processing unit 3 is configured by a computer (so-called server). The processing unit 3 includes a coordinate integration unit 31, an extraction unit 32, and a storage unit 33 as functional elements using computer hardware and computer software. The coordinate integration unit 31 has a function of integrating the coordinate system of the image acquired by the image acquisition unit 1 and the coordinate system of the trajectory acquired by the trajectory acquisition unit 2. In this embodiment, the acquired images are integrated by the image acquisition unit 1. Of course, the processing unit 3 may perform image integration. The extraction unit 32 has a function of extracting an image of the moving body located on the trajectory from the image acquired by the image acquisition unit 1. Each function of the processing unit 3 will be described in detail in the description of the operation of the present embodiment.

(Operation of the embodiment)
Next, the operation of the image extraction apparatus according to this embodiment will be described. In this embodiment, the camera 11 of the image acquisition unit 1 and the laser scanner 21 of the trajectory acquisition unit 2 are arranged at predetermined positions (see FIG. 3). In this state, the camera 11 can acquire an image in an area set by the operator. The laser scanner 21 can acquire the position of the moving body in the region set by the operator. FIG. 4 shows a state in which the markers 51, 52, 53 and the calibration block 6 are arranged in the target area for a calibration operation (described later).

(Step Sa1-2)
As shown in FIG. 5, first, the processing unit 3 sends a synchronization signal to the camera terminal 12 and the scanner terminal 22 (generically referred to as a client in the figure).

(Steps Sb1-2 and Sc1-2)
As shown in FIG. 8, when the laser terminal 22 receives the synchronization signal, it performs time synchronization. Similarly, as shown in FIG. 9, when the camera terminal 12 receives the synchronization signal, it performs time synchronization. By performing time synchronization in this way, it is possible to match the acquisition times of data acquired at each terminal.

(Steps Sa3-6)
In the description of this embodiment, the entire calibration of the laser scanner 21 is referred to in step Sa3 in FIG. Step Sa3 has steps Sa4-6. In step Sa3, first, the processing unit 3 sends a request signal to each scanner terminal 22 (step Sa4). The operation of each scanner terminal 22 will be described later.

  When sample data is received from each scanner terminal 22 (step Sa5), the integration unit 31 of the processing unit 3 matches the sample data, and converts conversion parameters from the coordinate system of each laser scanner 21 to a certain integrated coordinate system. Obtain (step Sa6).

(Steps Sb3-4)
When each laser terminal 22 receives a request signal from the processing unit 3 (step Sb3), each laser scanner 21 scans a known moving body once to acquire sample data. (Step Sb4). Since the shape and position of the moving object acquired from the sample data are known, parameters for converting or integrating the coordinate system can be obtained by using the sample data.

(Steps Sa7-12)
Next, the entire calibration of the camera 11 will be described with reference to step Sa7.

  First, the processing unit 3 sends a data request signal to one of the camera terminals 12 (step Sa8). The operation of the camera terminal 12 will be described later.

  Next, the processing unit 3 receives sample data and internal orientation parameters from the camera terminal 12 (step Sa9).

  Next, the integration unit 31 of the processing unit 3 determines external orientation parameters from the coordinate system of the camera 11 connected to the camera terminal 12 to a certain integrated coordinate system (step Sa10).

  Next, the user sets an area where the camera 11 acquires an image (that is, an area that associates the coordinate system of the camera image and the locus) via the processing unit 3 (step Sa11). Hereinafter, this region may be referred to as a “tracking area”. The user also sets whether to perform real-time tracking and overlay display. After step Sa11 is completed, if there is a next camera 11, steps Sa8 to 12 are repeated (step Sa12).

(Steps Sc3-4)
The camera terminal 12 receives a data request signal (step Sa18) from the processing unit 3 via the network 4 (Sc3).

  Next, the camera terminal 12 acquires an image for one shot from the camera 11 (step Sc4). This image is, for example, an image including any of the plurality of markers 51 to 53 or the calibration block 6 whose positions are known. When the calibration block 6 is used for calibration, information such as the corners, sides, and surface positions of the block 6 is used. The camera terminal 12 sends this image to the control unit 3 as sample data. Furthermore, the control unit 3 sends the internal orientation parameters of the camera 11 to the control unit 3 (see step Sa9).

(Step Sa13)
The operator sets another tracking area and sets whether or not real-time tracking is necessary in that area.

(Step Sa14)
Next, the processing unit 3 sends a control start signal to each camera terminal 12 and scanner terminal 22.

(Steps Sb5 to 10: Operation of the trajectory acquisition unit)
When receiving the measurement start signal from the processing unit 3 (step Sb5), the scanner terminal 22 of the trajectory acquisition unit 2 operates the laser scanner 21 to acquire data for one scan (step Sb6).

  Further, the scanner terminal 22 generates background data from data for one scan. That is, in the first operation, the first acquired data becomes the background data. The scanner terminal 22 updates the background data every hour based on the subsequent scan result (step Sb7).

  Further, the scanner terminal 22 extracts moving object data based on the difference between the subsequent scan data and the background (step Sb8).

  Next, the scanner terminal 22 transmits moving body data (that is, position data of the moving body) to the processing unit 3 (step Sb9).

  Until the measurement end signal is received from the processing unit 3, the scanner terminal 22 repeats Steps Sb6 to 10 (Step Sb10).

(Steps Sc5-6: Operation of Image Acquisition Unit)
When receiving the measurement start signal from the processing unit 3 (Step Sc5), the camera terminal 12 of the image acquisition unit 1 operates the camera 11 to record a moving image at a constant frame rate (Step Sc6). Integration (so-called overlay) of the acquired image and the trajectory of the moving object on the image is performed in the image acquisition unit 1 after steps Sa15 to 24 of the processing unit 3 described later (steps Sc7 to 8 described later).

(Steps Sa15-25: Operation of the processing unit)
The processing unit 3 receives moving object data from the scanner terminal 22 (step Sa15) and stores it in the storage unit 33. Next, the coordinate integration unit 31 of the processing unit 3 performs coordinate conversion of the moving object data and integrates it into the same coordinate system as the image (step Sa16). Next, for each tracking area, a trajectory is extracted by the following subroutine (steps Sa18 to 24).

(Steps Sa18-24: Trajectory extraction subroutine)
First, when the corresponding tracking area performs real-time tracking and overlay display (step Sa18), the extraction unit 32 of the processing unit 3 extracts mobile body data in the corresponding tracking area (step Sa19). Save in the storage unit 33. Otherwise, the process proceeds to step Sa24.

  Furthermore, when performing real-time tracking (step Sa20), the extraction part 32 extracts the movement locus | trajectory of a moving body using applicable moving body data (step Sa21). The extracted trajectory information is also stored in the storage unit 33.

  Further, when overlay display is performed (step Sa22), the moving body data and the trajectory data are sent to the terminal 12 for the camera 11 that has captured the moving body (step Sa23). If there is a next tracking area (step Sa24), steps Sa18 to 24 are repeated. Otherwise, the process proceeds to step Sa25.

(Steps Sa25-26)
Next, when the trajectory measurement process is completed (step Sa25), the processing unit 3 sends a measurement end signal to each camera terminal 12 and each scanner terminal 22 (step Sa26). The end of the trajectory measurement process may be designated by the operator or may be automatically performed based on some condition.

(Steps Sc7-8)
The camera terminal 12 receives the trajectory data of the moving object from the processing unit 3 (see step Sa23). Further, the camera terminal 12 displays the moving image obtained from the camera 11 and the trajectory data in an overlay manner.

(Step Sb10)
Upon receiving the measurement end signal from the processing unit 3, the scanner terminal 22 stops the processing there.

(Step Sc9)
Similarly, when receiving the measurement end signal from the processing unit 3, the camera terminal 12 stops the processing there.

  In the processing described above, an image at a position corresponding to the trajectory can be cut out. For example, in the case where a moving image is captured while the moving body is moving, difference data located on the vertical line of the trajectory can be extracted as an image of the moving body. In general, when images of moving objects are overlapped, it is difficult to separate them only by image processing. According to this embodiment, since the image of the moving object is cut out using the trajectory information, the image of the moving object corresponding to the trajectory can be extracted relatively accurately. Note that if the moving body continues to be stationary during image shooting, or if the image is the result of still image shooting, when the image is cut out, for example, if the moving body is a pedestrian, the moving body And an image up to a predetermined height corresponding to the height may be cut out.

  In the conventional method, since a moving body is cut out using a large number of moving images, a large number of cameras are required. On the other hand, according to the method of the present embodiment, an image of a moving body can be cut out even if the number of cameras is small. Moreover, there are few restrictions on an installation location. Therefore, according to the apparatus of this embodiment, there is an advantage that the number of cameras and the degree of freedom of arrangement can be increased.

  Further, in the present embodiment, the camera with the best shooting conditions for the orientation of the moving body image (front image, side image, etc.) to be obtained based on the position and orientation that change in time series corresponding to the trajectory The photographing time can be selected, and the selected camera image at the position of the locus at the selected time can be cut out. Note that the criteria for determining good imaging conditions are, for example, (1) the closest to the desired moving body image orientation, and (2) that there is no other trajectory on the optical axis at which the camera captures the trajectory.

  An example using the apparatus of this embodiment will be described below.

(Example)
According to the apparatus of the present embodiment, it is possible to cut out an image of a moving body corresponding to a trajectory. At this time, if the trajectory information is used, the direction of the moving body can be determined. Therefore, the orientation of the moving object is attached to the cut out image and stored in the storage unit 33. Then, the moving body image can be classified and stored in a type such as a front image and a side image. When the moving body is stationary, it may be stored as unknown orientation.

An image with such attributes is used as follows, for example.
(Identification of moving objects when the trajectory stops)
The moving object may move outside the measurement area of the laser scanner 21 or in the shadow of another moving object, and the locus may be interrupted. Thereafter, when a trajectory appears again, the image corresponding to the new trajectory and the image stored as described above are collated in consideration of attribute (orientation) information. Thereby, the determination precision of the identity of a moving body can be improved.
(Identity determination with a mobile that has previously entered the area)
The moving body image corresponding to the trajectory that has entered the target area and the previously stored image are collated in consideration of attribute (orientation) information. Thereby, it is possible to accurately determine whether the mobile body that has entered has entered the area in the past. Then, it becomes easy to specify a person who enters the same area many times, and it is effective for suspicious person detection.

The description of the embodiment is merely an example, and does not indicate a configuration essential to the present invention. The configuration of each part is not limited to the above as long as the gist of the present invention can be achieved.
Furthermore, the above-described functional blocks may be combined into a single functional block. Further, the function of one functional block may be realized by cooperation of a plurality of functional blocks. Furthermore, the functional blocks described above only have to exist as functions, and any means such as hardware, computer software, or a combination thereof can be used as means for realizing the functions. One functional block may be realized by cooperation of hardware and computer software arranged at physically discrete positions.

It is a block diagram for demonstrating the whole structure of the image extraction apparatus which concerns on one Embodiment of this invention. It is a block diagram in the state where each terminal was arranged centering on a processing part. It is explanatory drawing for demonstrating the arrangement | positioning state of a camera and a laser scanner. It is explanatory drawing for demonstrating one method of calibration. It is a flowchart for demonstrating operation | movement of a process part. It is a flowchart for demonstrating operation | movement of a process part. It is a flowchart for demonstrating operation | movement of a process part. It is a flowchart for demonstrating operation | movement of the terminal for scanners. It is a flowchart for demonstrating operation | movement of the terminal for cameras.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image acquisition part 11 Camera 12 Camera terminal 2 Trajectory acquisition part 21 Laser scanner (sensor)
22 Scanner terminal (sensor terminal)
3 Processing Unit 31 Integration Unit 32 Extraction Unit 33 Storage Unit 4 Network

Claims (9)

An image acquisition unit, a trajectory acquisition unit, and a processing unit;
The image acquisition unit is configured to acquire an image of the target area,
The locus acquisition unit is configured to acquire a locus of a moving object,
The processor is
(1) a function of integrating the coordinate system of the image acquired by the image acquisition unit and the coordinate system of the trajectory acquired by the trajectory acquisition unit; and (2) an image of a moving body located on the trajectory. An image extraction apparatus having a function of extracting from an image acquired by the image acquisition unit.   The image extraction apparatus according to claim 1, wherein the image acquisition unit, the trajectory acquisition unit, and the processing unit are connected by a network.   The image extraction apparatus according to claim 1, wherein the image acquisition unit includes a camera and a camera terminal.   The image extraction apparatus according to claim 1, wherein the trajectory acquisition unit includes a laser scanner and a scanner terminal.   The image extraction apparatus according to claim 1, wherein the trajectory acquisition unit includes a sensor that scans horizontally to obtain a range image and a sensor terminal.   The image extracting apparatus according to claim 5, wherein the sensor is a laser scanner, and the sensor terminal is a scanner terminal. An integration unit, an extraction unit, and a storage unit;
The integration unit is configured to integrate a coordinate system in the image of the moving object acquired by the image acquisition device and a coordinate system in the locus of the moving object acquired by the locus acquisition device,
The image processing apparatus, wherein the extraction unit is configured to extract an image of a moving body located on the trajectory from an image acquired by the image acquisition unit.   The extraction unit attaches an attribute corresponding to the orientation of the moving object estimated from the trajectory to the extracted moving object image, and stores the image of the moving object in the storage unit. The image processing apparatus according to claim 7. An image extraction method characterized by comprising the following steps:
(1) obtaining an image in the target region;
(2) obtaining a trajectory of the moving object;
(3) integrating the coordinate system of the acquired image and the coordinate system of the acquired trajectory;
(4) A step of extracting an image of the moving body located on the locus from the image in the target area.

Images