JP2003259362A - Image processing apparatus and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus and an image processing method capable of detecting and recognizing an object with high accuracy. <P>SOLUTION: The image processing apparatus is provided with: an infrared ray camera 12 for performing infrared ray photographing to output an infrared ray image; a visual ray camera 11 for performing visible ray photographing with a visual field corresponding to the visual filed of the infrared ray camera and outputting a visible ray image; an extract section 15 for outputting position information representative of a position of a hot point denoting a heat generating body included in the infrared ray image from the infrared ray camera in the infrared ray image; and a processing section 17 for searching for a prescribed range from a position in the visible ray image from the visible ray camera corresponding to the position of the hot point from the extract section so as to detect and recognize the object. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an image processing method, and more particularly to a technique for detecting and recognizing a target object by processing an image obtained by photographing.

[0002]

2. Description of the Related Art Conventionally, an infrared image processing device is known as one of devices for detecting and recognizing a target object such as a vehicle in nature by image processing. This infrared image processing apparatus captures infrared rays generated by heat generation of a target with an infrared camera, and processes the infrared image obtained by this capturing to detect and recognize the target.

A visible light image processing apparatus is known as another apparatus for detecting and recognizing a target object by image processing. This visible light image processing apparatus captures a scene including a target with a visible light camera and processes the visible light image obtained by this capturing to detect and recognize the target.

[0004]

However, the above-mentioned conventional infrared image processing apparatus and visible light image processing apparatus have advantages and disadvantages, respectively, and cannot detect and recognize the target object with sufficient accuracy.

That is, in the infrared image processing apparatus, if the target is a heating element, it can be detected even in a dark place, but the information obtained from the infrared image depends on the wavelength used for detection, It is about the rough shape of an object. Therefore, there is insufficient information to recognize the type of target. Further, for example, an object warmed by sunlight is also detected as a heating element, so that there is a problem that false detection frequently occurs depending on the weather.

On the other hand, the visible light image processing apparatus can detect and recognize a target even if the target is not a heating element, but cannot detect or recognize the target in a dark place. Also, many features of the target object can be obtained from the visible light image compared to the infrared image, but when the target object has a similar color to the background, it is buried in the background and the target object is far away. In that case, the feature may not be obtained and the target may be missed.

The present invention has been made to solve the above problems, and an object thereof is to provide an image processing apparatus and an image processing method capable of accurately detecting and recognizing a target object.

[0008]

[Means for Solving the Problems] Means for solving the problems will be described below with reference to the numbers and symbols used in the embodiments of the present invention. These numbers and signs are added to clarify the correspondence between the description in [Claims] and the description in [Embodiment of the Invention], but in [Claims] It should not be used to interpret the technical scope of the described invention.

In order to achieve the above object, an image processing apparatus according to a first aspect of the present invention is compatible with an infrared camera (12) for taking an infrared image and outputting an infrared image, and a field of view of the infrared camera. Visible light camera (11) that shoots visible light in the field of view and outputs a visible light image
And an extraction unit (15) that outputs position information indicating the position in the infrared image of the hot point representing the heating element included in the infrared image from the infrared camera, and the hot point from the extraction unit. A processing unit (17) for searching a predetermined range from the position in the visible light image from the visible light camera corresponding to the position and detecting and recognizing the target.

According to the image processing apparatus of the first aspect, the hot point is detected by using the infrared camera (12), and whether or not the detected hot point may be a target object is detected. When there is a possibility of a target, the visible light image captured by the visible light camera (11) is checked to detect whether it is a target and to recognize the target. Therefore, as in the conventional art, erroneous detection of the target object by processing only the infrared image and overlooking of the target object by processing only the visible light image are eliminated. Can be improved.

In the image processing apparatus according to the first aspect, the infrared camera (12) and the visible light camera (11) are composed of a combined camera (10), and the combined camera (10 ) One lens system (10
0) and an infrared sensor (121) for detecting infrared rays incident through the lens system and outputting the infrared image.
And a visible light sensor (11) for detecting visible light incident through the lens system and outputting the visible light image.
1), and can be comprised.

According to this structure, the visible light image and the infrared image can be obtained by one compound camera, so that the structure of the image processing apparatus becomes simple and inexpensive. Further, it is not necessary to adjust the visible light camera (11) and the infrared camera (12) to have the same field of view, so that the handling becomes easy.

The image processing apparatus according to the first aspect further includes a map / terrain data memory (16) for storing data representing a map and topography, and the extraction unit (15).
Further determines whether or not the hot point may be the target object based on the map and the terrain data from the map / terrain data memory, and determines that the hot point may be the target object. In this case, the position information indicating the position of the hot point in the infrared image can be sent to the processing unit (17).

According to this structure, it is determined whether or not there is a possibility that the hot point in the infrared image is the target object in consideration of the map and the terrain data, so that the detection accuracy of the target object is improved.

For the same purpose as described above, the image processing method according to the second aspect of the present invention inputs the infrared image obtained by infrared photography, and the infrared of the hot point representing the heating element included in the infrared image. Outputting position information representing a position in the image, corresponding to the position of the hot point, searching a predetermined range from the position in the visible light image,
Thus, the target is detected and recognized.

Further, in the image processing method according to the second aspect, the step of further storing the data representing the map and the terrain and outputting the position information further comprises: When it is determined that the hot point may be the target object, and when it is determined that the hot point may be the target object, position information indicating the position of the hot point in the infrared image is output. Can be configured to.

The image processing method according to the second aspect configured as described above also provides the same operation and effect as the image processing apparatus according to the first aspect described above.

[0018]

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following, the target object is a vehicle existing in nature, but the target object of the present invention is not limited to the vehicle and can be applied to any object including a heating element (heat source).

(Embodiment 1) FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to Embodiment 1 of the present invention. This image processing apparatus includes a visible light camera 11, an infrared camera 12, a visible light image memory 13, an infrared image memory 14, a target extraction unit 15, and a map / terrain data memory 1.
6 and a processing unit 17.

The visible light camera 11 is a camera equipped with an image sensor which is sensitive to the wavelength range of visible light (about 400 to 700 nm). The visible light camera 11 includes a CCD sensor 1 that constitutes a lens system 110 and an image sensor.
11 and 11. The lens system 110 is composed of known optical elements such as a lens, a prism, and an optical filter, and forms an optical image on the photosensitive surface of the CCD sensor 111.

The CCD sensor 111 photoelectrically converts the optical image input from the lens system 110 to generate and output a digital signal. Although the CCD sensor is used as the image sensor in the first embodiment, an image sensor composed of a MOS type device and other solid-state imaging devices can be used. The digital signal output from the CCD sensor 111 is sent to the visible light image memory 13 as a visible light image signal.

The infrared camera 12 is a camera equipped with an image sensor which is sensitive to the wavelength range of far infrared rays (about 8 to 9 μm). This infrared camera 12 has a lens system 12
0 and an infrared sensor 121 forming an image sensor. Like the lens system 110 described above, the lens system 120 includes known optical elements such as lenses, prisms, and optical filters, and forms an optical image on the photosensitive surface of the infrared sensor 121. The digital signal output from the infrared sensor 121 is sent to the infrared image memory 14 as an infrared image signal.

The infrared sensor 121 photoelectrically converts the optical image input from the lens system 120 to generate and output a digital signal. As the infrared sensor 121, a thermal effect element utilizing a thermal effect or a quantum effect element utilizing a quantum effect can be used. The thermal effect element,
It includes bolometers, thermopiles, pyroelectric elements and the like.
In addition, the quantum effect device is a PC using the photoconductive effect.
A (Photo Conductive) type element and a PV (Photo Voltaic) type element utilizing the photovoltaic effect are included.

The visible light camera 11 and the infrared camera 12 are connected by a connecting member 18 and are adjustable so that the visual field of the visible light camera 11 and the visual field of the infrared camera 12 are substantially the same. . Therefore, from the visible light camera 11 and the infrared camera 12, an image obtained by photographing a scene in substantially the same range can be obtained. These visible light cameras 1
1 and the infrared camera 12 are controlled by a control unit (not shown).
It is controlled so that the shutter is released in synchronization.

The visible light image memory 13 stores the visible light image signal sent from the CCD sensor 111 in the visible light camera 11 as a visible light image. The visible light image stored in the visible light image memory 13 is sent to the processing unit 17.

The infrared image memory 14 is the infrared camera 1
The infrared image signal sent from the infrared sensor 121 in 2 is stored as an infrared image. If this infrared image contains a part (heating element) where the temperature has risen,
That part is displayed with high brightness. In this specification, the portion displayed with high brightness is referred to as a "hot point". Hot points appearing in the infrared image are caused by rocks heated by sunlight, places where fire is used, and the like, in addition to vehicles. The infrared image stored in the infrared image memory 14 is sent to the target extracting unit 15.

The map / terrain data memory 16 stores map data and terrain data. The map data and the terrain data will be described later in detail, but are used to determine the possibility that the hot point is a vehicle. This map
The map data and the terrain data stored in the terrain data memory 16 are sent to the target object extraction unit 15.

The target object extracting section 15 is provided in the infrared image memory 1
The infrared image from No. 4 is taken in, and the map data and the terrain data from the map / terrain data memory 16 are taken into consideration to determine whether the hot point may be a vehicle. This determination is performed based on, for example, the size and shape (for example, the aspect ratio) of the hot point, whether or not there is a road nearby, the filling rate, and the like. Then, the target object extracting unit 15
When determining that the vehicle may be a vehicle, sends the coordinates (X, Y) of the hot point to the processing unit 17.

The processing unit 17 reads the visible light image from the visible light image memory 13 and m × n pixels (m and n) around the coordinates (X, Y) sent from the target extracting unit 15.
Is a positive integer) and the vehicle is recognized as a processing area. In this processing, for example, the edge enhancement processing is performed by differentiating the image of the processing area, and the inclination (direction) of the line segment forming the line segment graphic obtained by this edge enhancement processing is statistically processed in the vehicle. Recognize whether there is.

In addition to the above-described method, the eigenspace method can be used for the vehicle recognition processing. The eigenspace method is
An image is represented using an eigenspace composed of the eigenvectors of the covariance matrix or autocorrelation matrix of the image set, and the distance between the image pattern of the target and the standard image pattern prepared in advance is identified in this eigenspace. This is a method of recognizing an object. When the processing unit 17 recognizes that the vehicle is a vehicle by the recognition processing using the method described above,
A message to that effect is sent to the outside.

The target extracting unit 15 and the processing unit 17 are
For example, microprocessor, personal computer,
It can be configured by an information processing device such as a workstation or a general-purpose computer.

Next, the operation of the image processing apparatus according to the first embodiment of the present invention having the above-mentioned configuration will be described with reference to the flowchart shown in FIG.

First, the visible light camera 11 and the infrared camera 12 perform photographing (step S10). This shooting is performed by synchronizing the shutters of both cameras. Thereby, from the visible light camera 11,
For example, a visible light image signal representing a scene in the field of view as shown by the solid line in FIG. 3 is obtained and stored in the visible light image memory 13. On the other hand, the infrared camera 12 obtains an infrared image signal of only hot points in the field of view as shown by the broken line in FIG.

The visible light camera 11 and the infrared camera 1
When the photographing by 2 and 3 is completed, then the target object extracting unit 1
5 reads the infrared image from the infrared image memory 14 to extract hot points, and sets the extracted hot points as vehicle candidates (step S11). FIG. 3 shows an example in which there are three candidates (hot points) such as candidate 1, candidate 2, and candidate 3.

Next, the target extracting unit 15 selects one candidate and checks whether or not the selected candidate is a detection target, that is, a vehicle (step S1).
2). At this time, the contents of the map / terrain data memory 16 are referred to and used as a part of the basis of the judgment.

For example, referring to the map data and the terrain data stored in the map / terrain data memory 16, the candidate 1 shown in FIG. 3 exists in a high mountain place and a vehicle can enter without a road nearby. It is recognized that it is not terrain. As a result, it is determined that the candidate 1 has no possibility of being a vehicle.

Further, referring to the map data and the terrain data stored in the map / terrain data memory 16, the candidate 2 shown in FIG. 3 is found to exist near the foot of the mountain and there is a road nearby. To be done. As a result, it is determined that the candidate 2 may be a vehicle.

Similarly, referring to the map data and the terrain data stored in the map / terrain data memory 16, the candidate 3 shown in FIG. 3 is near the place where the image processing apparatus is installed, It is recognized that the aspect ratio of candidate 3 is far from the vehicle. As a result, candidate 3 is
It is determined that there is no possibility of a vehicle.

Although detailed description is omitted as a basis for determining whether or not the hot point is a vehicle, it is not limited to the items shown in the above example, but the size and shape (for example, vertical and horizontal) of the hot point described above. Ratio), whether or not there are roads in the vicinity, and the filling rate, etc. are comprehensively taken into consideration.

When it is recognized in the process of step S12 that the candidate may be the detection target, that is, the vehicle, the target object extracting unit 15 determines the candidate (hot point).
The coordinates (X, Y) on the infrared image of are calculated (step S13). Then, the coordinate value that is the calculation result is processed by the processing unit 1.
Send to 7. In this case, the coordinate value of the center of the hot point can be sent.

Upon receiving the coordinate values from the target object extracting unit 15, the processing unit 17 extracts m × n pixels (m and n are positive integers) centered on the coordinates (X, Y) on the visible light image. , The extracted range is searched (step S14). These m and n can be set appropriately.

The target extracting unit 15 can be configured to send the coordinate value of the outer edge of the hot point to the processing unit 17 instead of the center coordinate of the hot point. In this case, the processing unit 17 can be configured to search for a range located at a predetermined distance from the range indicated by the received coordinate values, so that the search range can be optimized. As a result, there is an advantage that the search time can be shortened.

Then, the processing section 17 checks whether or not the object existing in the extracted range is a target object, that is, a vehicle (step S15). This is performed by the edge emphasis processing and the statistical processing of the image in the extracted range, as described above. Here, if it is determined that the object is the target object, the processing unit 17 outputs a message indicating that (step S16).

On the other hand, if it is determined in step S15 that the object is not the target object, the processing in step S16 is skipped. Then, it is checked whether or not the processing of all candidates has been completed (step S17). Here, if it is determined that the processing for all candidates has not been completed, the processing returns to step S12, and the same processing is repeatedly executed again. When it is determined in step S17 that the processing of all candidates has been completed in the process of this repeated execution, the series of processing ends.

As described above, according to the image processing apparatus according to the first embodiment, first, the hot point is detected by using the infrared camera 12, and the detected hot point may be the vehicle. To see if there is
When there is a possibility of a vehicle, the visible light image captured by the visible light camera 11 is examined to recognize whether it is a target or not. Therefore, it is possible to prevent erroneous detection of the target object by processing only the infrared image and miss of the target object by processing only the visible light image as in the related art.

Further, it is checked whether the hot point in the infrared image obtained from the infrared camera 12 may be a vehicle, and the recognition processing using the visible light image is performed only when the hot point is a vehicle. Therefore, the amount of calculation performed in the processing of the processing unit 17 can be reduced. As a result, the vehicle can be recognized in a short time, and this image processing apparatus is suitable for recognizing the target object in real time.

(Embodiment 2) Embodiment 2 of the present invention
Is a configuration in which the visible light camera 11 and the infrared camera 12 are integrally configured.

FIG. 4 is a block diagram showing the configuration of the image processing apparatus according to the second embodiment of the present invention. This image processing apparatus includes a compound camera 10 in which the visible light camera 11 and the infrared camera 12 according to the first embodiment described above are integrally configured.

The compound camera 10 comprises a lens system 110, a half mirror 101, a reflection mirror 102, a CCD sensor 111 as an image sensor, and an infrared sensor 121. The lens system 110 is composed of known optical elements such as lenses, prisms, and optical filters, and forms an optical image on the photosensitive surfaces of the CCD sensor 111 and the infrared sensor 121.

The half mirror 101 transmits a part of the light from the lens system 100 to guide it to the CCD sensor 111, and reflects the other part to guide it to the reflecting mirror 102.
The reflection mirror 102 totally reflects the light reflected from the half mirror 101 and guides it to the infrared sensor 121. C
The CD sensor 111 and the infrared sensor 121 are the same as those used in the first embodiment.

The operation of the image processing apparatus according to the second embodiment configured as described above is the same as that of the first embodiment already described.
The operation is the same as that of the image processing apparatus according to the above.

As described above, according to the image processing apparatus according to the second embodiment, it is possible to obtain a visible light image and an infrared image with a single compound camera. Therefore, compared with the configuration of the first embodiment. As a result, the structure of the image processing apparatus becomes simple and inexpensive. Further, unlike the first embodiment, since it is not necessary to adjust the visible light camera 11 and the infrared camera 12 to have the same field of view with the connecting member 18, the handling becomes easy.

[0053]

As described in detail above, according to the present invention,
An image processing device and an image processing method capable of accurately detecting and recognizing a target object can be provided.

More specifically, according to the image processing apparatus of the first aspect of the present invention, an infrared camera is used to detect a hot point, and the detected hot point may be a target object. If there is a possibility of a target object, the visible light image taken by the visible light camera is checked to detect whether it is a target object and recognize the target object. In addition, erroneous detection of the target object by processing only the infrared image and missing of the target object by processing only the visible light image are eliminated.
It is possible to improve the recognition accuracy of the target while improving the detection accuracy.

Further, in the image processing apparatus according to the first aspect of the present invention, the infrared camera and the visible light camera are constituted by a combined camera, so that the visible light image and the infrared light can be obtained by one combined camera. Since an image can be obtained, the configuration of the image processing device is simple and inexpensive. In addition, since it is not necessary to adjust the visible light camera and the infrared camera so that they have the same field of view, the handling becomes easy.

Further, in the image processing apparatus according to the first aspect of the present invention, the map and the terrain data are considered by further comprising the map / terrain data memory for storing the data showing the map and the terrain. Then, it is determined whether or not the hot point in the infrared image is likely to be the target object, so that the detection accuracy of the target object is improved.

Furthermore, according to the image processing method of the second aspect of the present invention, the same effect as that of the image processing apparatus of the first aspect can be obtained.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to a first embodiment of the present invention.

FIG. 2 is a flowchart showing an operation of the image processing apparatus according to the first embodiment of the present invention.

FIG. 3 is a diagram for explaining the operation of the image processing apparatus according to the first embodiment of the present invention.

FIG. 4 is a block diagram showing a configuration of an image processing device according to a second embodiment of the present invention.

[Explanation of symbols]

10 compound camera 11 visible light camera 12 infrared camera 13 Visible light image memory 14 infrared image memory 15 Target extraction unit 16 Map and terrain data memory 17 Processor 18 Connection member 100, 110, 120 lens system 101 half mirror 102 reflective mirror 111 CCD sensor 121 infrared sensor

   ─────────────────────────────────────────────────── ─── Continued front page    F-term (reference) 5B047 AA19 BA08                 5B057 AA13 AA16 BA02 CA02 CA08                       CA12 CA16 CB18 CC01 CH08                       DA07 DB02 DB05 DB09                 5C022 AA15 AC42 AC54 AC69                 5C054 AA01 AA05 AA07 CA04 CA05                       CC01 CC05 ED12 FC14 FC16                       HA18                 5L096 BA04 CA05 FA02 JA11

Claims (5)

[Claims] 1. An infrared camera for performing infrared imaging and outputting an infrared image; a visible light camera for performing visible light imaging and outputting a visible light image in a field of view corresponding to the field of view of the infrared camera; and the infrared camera. From the extraction unit that outputs position information indicating the position in the infrared image of the hot point representing the heating element included in the infrared image from, and the visible light corresponding to the position of the hot point from the extraction unit. An image processing apparatus, comprising: a processing unit that searches for a predetermined range from a position in the visible light image from a camera, and thereby detects and recognizes a target. 2. The infrared camera and the visible light camera are composed of a composite camera integrally configured, and the composite camera detects one infrared ray incident through the lens system and the infrared ray lens through the lens system. The image processing device according to claim 1, further comprising: an infrared sensor that outputs an infrared image; and a visible light sensor that detects visible light incident through the lens system and outputs the visible light image. . 3. A map / terrain data memory for storing data representing a map and terrain, wherein the extraction unit further includes the hot spot based on the data representing the map and terrain from the map / terrain data memory. When determining whether or not a point may be the target object, and determining that the point may be the target object, position information indicating the position of the hot point in the infrared image is provided to the processing unit. The image processing apparatus according to claim 2, which sends the image. 4. An infrared image obtained by infrared photography is input, position information indicating a position in the infrared image of a hot point representing a heating element included in the infrared image is output, and the position of the hot point is displayed. An image processing method, wherein a predetermined range is searched from a corresponding position in the visible light image, and thereby a target is detected and recognized. 5. The step of further storing data representing a map and terrain and outputting the position information further includes the possibility of the hot point being the target object based on the data representing the map and terrain. The image processing method according to claim 4, further comprising: outputting position information indicating a position of the hot point in the infrared image when it is determined that the target object may be the target object.