JP3659914B2 - Object recognition apparatus, object recognition method, program, and recording medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an apparatus and method for recognizing an object belonging to a specific category, and more particularly to an apparatus and method for recognizing an object using a plurality of images that represent the object using different attributes.
[0002]
[Prior art]
As a conventional technique for recognizing an object using a plurality of images representing the object using different attributes, a technique disclosed in Japanese Patent Application Laid-Open No. 8-287216, “Intrafacial region recognition method” is known. . In this prior art, a part (for example, mouth) in the face is obtained from a far-infrared light (light having a wavelength of 8 to 10 μm) image and a visible light image (a plurality of images expressing an object using different attributes). Be recognized. The far-infrared image represents the intensity of far-infrared light emitted from the object. Since the intensity of the far-infrared light emitted from the object can be correlated with the temperature of the object, a specific temperature from the far-infrared light image (for example, about 36 ° C., which is a normal temperature of human skin). Can be extracted.
[0003]
Using only a temperature image makes it difficult to accurately detect a person when an object with the same temperature as the person (such as an electrical appliance in a room) exists around the object. Reliable detection was realized with reference to the area.
[0004]
[Problems to be solved by the invention]
In the prior art described in the above publication, the skin temperature region extracted from the far-infrared image and the skin color region extracted from the visible light image are associated with each other in order to specify the position of the part to be recognized. In order to perform such association, (1) extraction of a skin temperature region (a region having a temperature of about 36 ° C.) from a far-infrared image and extraction of a skin color region from a visible light image are accurately performed. (2) It is necessary to associate the pixels in the far-infrared image with the pixels in the visible light image in advance.
[0005]
In order to associate the pixels in the far-infrared image with the pixels in the visible light image in advance, it is necessary to accurately align the optical axes of the visible light camera and the far-infrared light camera. There is a problem that the initial setting for recognition becomes complicated.
[0006]
In order to accurately extract the skin temperature region from the far-infrared image, it is necessary to frequently perform calibration to cancel the influence of the temperature of the optical system, circuit, element, etc. of the far-infrared camera that changes over time. . Alternatively, in order to eliminate the influence of the temperature of these optical systems, circuits, elements, etc., it is necessary to keep the entire far-infrared camera at a constant temperature (for example, to cool it). As a result, there is a problem that initial setting and maintenance of a recognition system including a far-infrared camera are complicated and costly.
[0007]
In addition, skin temperature varies greatly due to the effects of solar radiation and temperature. Especially in the outdoors, the skin temperature tends to deviate from the standard temperature around 36 ° C. and changes greatly in time during the day according to changes in conditions such as solar radiation and temperature. Thus, when the skin temperature changes, it becomes difficult to accurately extract the skin temperature region from the far-infrared image. In order to accurately extract the skin temperature region under various changing environmental conditions, an extraction algorithm corresponding to each condition must be prepared, and there is a problem that initial setting of the recognition system is not easy.
[0008]
Even in the visible light image, under the environment that is susceptible to the influence of artificial lighting such as sunlight and car headlights, such as outdoors, it is caused by the limitation of the dynamic range of the camera and the spectral distribution of the light source. It is difficult to always accurately detect the color of the object. In order to accurately extract the skin color region under various changing environmental conditions, an extraction algorithm corresponding to each condition must be prepared, and there is a problem that initial setting of the recognition system is not easy.
[0009]
Furthermore, the extraction of the skin temperature region from the far-infrared image and the extraction of the skin color region from the visible light image are both processing specialized for the attributes of the individual objects. Such processing does not work well when the recognition target changes. For example, in order to apply this prior art to animal recognition, the region extraction algorithm must be changed. Since an extraction algorithm must be prepared for each individual recognition target, initial setting of the recognition system is not easy.
[0010]
As described above, according to the related art, the initial setting for recognizing the object is not easy due to the necessity of the process of associating the region of the far-infrared image with the region of the visible light image, There is a problem of being easily affected by environmental conditions.
[0011]
The present invention has been made in view of such a problem, and has an object recognition device, a method for recognizing an object, a program, and an object recognition device that have high recognition reliability, are easily initialized, and are not easily affected by environmental conditions. An object is to provide a recording medium.
[0012]
[Means for Solving the Problems]
  The object recognition apparatus of the present invention is a first object.First image data which is image data of a visible light image ofSaid first objectSecond image data which is image data of a far-infrared light image ofAndIncludeFirst imagedataAn input unit for inputting a set;The input unit inputs the first image data set and is inputThe first imagedatasetIn the first image data and the second image dataIn at least one predetermined position, Orientation selectivity, position selectivity, at least one selectivity of spatial frequency characteristicsAt least one image filterRespectivelyBy applyingFrom the first image data and the second image data, respectivelyBased on the relationship between the feature quantity vector calculation unit for obtaining the first feature quantity vector in the feature quantity space, having at least one obtained filter output value as a component, and the first feature quantity vector and a predetermined identification parameter And a determination unit that determines whether or not the first object belongs to a specific category, thereby achieving the above object.
[0013]
  The first image data represents the first object by the intensity of visible light emitted or reflected from the first object, and the second image data is the first object. The first object may be expressed by the intensity of far-infrared light emitted or reflected from the object.
[0014]
  The input unit further inputs a second image data set and a third image data set each consisting of a plurality of image data to the feature vector calculation unit, and the second image data set and the third image set Each of the data sets is third image data that is image data of a visible light image of a second object belonging to the specific category, and image data of a far-infrared light image of the second object. And the feature quantity vector calculation unit includes the third image data and the fourth image data for each of the input second image data set and third image data set. Applying at least one image filter having the same selectivity as the image filter to at least one predetermined position in the third image data and the fourth image data A feature amount vector in the feature amount space having at least one filter output value respectively obtained from the image data as a component, and at least one feature amount vector in the feature amount space for the second image data set; And a learning unit for obtaining the identification parameter so as to identify at least one feature vector in the feature space for the third image data set..
[0015]
  The first object may be a human.
[0016]
  The learning unit includes a feature amount vector for the second image data set, a feature amount vector for the third image data set, and a feature amount vector in a temporary feature amount space having a larger number of dimensions than the feature amount space. The feature amount space may be defined by deleting at least one dimension from the temporary feature amount space based on a normal direction of a plane for identifying.
[0017]
  The identification parameter represents an identification plane in the feature amount space, and the determination unit determines the first feature vector based on which side the first feature amount vector is located with respect to the identification plane. It may be determined whether the object belongs to the specific category.
[0018]
  The determination unit may determine that the first object belongs to the specific category when a distance between the first feature vector and the identification plane is equal to or greater than a predetermined threshold..
[0019]
  The input unit includes the first object.It is image data of visible light image of5th imagedataAnd the first object objectIt is image data of far-infrared light image6th imagedataAnd furtherIn the feature vector calculatorinputIs supposed to,The fifth image data and the sixth image data are:The first imagedataAnd the second imagedataTogaFilmed1st timeFromTaken after a predetermined timeMay have been.
[0020]
  The first image data may be captured from a first location, and the second image data may be captured from a second location different from the first location..
[0021]
  The input unit includes the first object.It is image data of visible light image of5th imagedataAnd the first objectIt is image data of far-infrared light image6th imagedataAnd furtherIn the feature vector calculatorinputThe fifth image data and the sixth image data areThe first imagedataAnd the second imagedataTogaTo be photographedFirstWhat is a place?Taken from a different second locationMay be.
[0022]
  The present inventionThe method of recognizing an object is(A)FirstObjectsFirst image data which is image data of a visible light image ofSaid first objectSecond image data which is image data of a far-infrared light image ofAndIncludeFirst imagedataInputting a set; (b)Each of the input first image data and second image dataIn at least one predetermined position, Orientation selectivity, position selectivity, at least one selectivity of spatial frequency characteristicsAt least one image filterRespectivelyBy applyingFrom the first image data and the second image data, respectivelyObtaining a first feature quantity vector in the feature quantity space having at least one obtained filter output value as a component; and (c) based on a relationship between the first feature quantity vector and a predetermined identification parameter, Determining whether the first object belongs to the specific category, thereby achieving the above object.
[0023]
  The program of the present invention is a program for causing a computer to execute object recognition processing, and the object recognition processing includes:FirstObjectsFirst image data which is image data of a visible light image ofSaid first objectSecond image data which is image data of a far-infrared light image ofAndIncludeFirst imagedataInputting a set; (b)Each of the input first image data and second image dataIn at least one predetermined position, Orientation selectivity, position selectivity, at least one selectivity of spatial frequency characteristicsAt least one image filterRespectivelyBy applyingFrom the first image data and the second image data, respectivelyObtaining a first feature quantity vector in the feature quantity space having at least one obtained filter output value as a component; and (c) based on a relationship between the first feature quantity vector and a predetermined identification parameter, Determining whether the first object belongs to the specific category, thereby achieving the above object.
[0024]
  The recording medium of the present invention is a computer-readable recording medium that records a program for causing a computer to execute object recognition processing. The object recognition processing includes:FirstObjectsFirst image data which is image data of a visible light image ofSaid first objectSecond image data which is image data of a far-infrared light image ofAndIncludeFirst imagedataInputting a set; (b)Each of the input first image data and second image dataIn at least one predetermined position, Orientation selectivity, position selectivity, at least one selectivity of spatial frequency characteristicsAt least one image filterRespectivelyBy applyingFrom the first image data and the second image data, respectivelyObtaining a first feature quantity vector in the feature quantity space having at least one obtained filter output value as a component; and (c) based on a relationship between the first feature quantity vector and a predetermined identification parameter, Determining whether the first object belongs to the specific category, thereby achieving the above object.
[0025]
The operation will be described below.
[0026]
According to the present invention, an image set (first image set) input for recognition includes a first image representing an object (first object) using a first attribute, and And a second image representing the object using a second attribute different from the first attribute. Since it is determined based on the first attribute and the second attribute whether or not the object belongs to a specific category, the reliability of recognition of the object is increased. Further, a feature vector in the feature space having as a component a filter output value obtained by applying a predetermined image filter to a predetermined position of the predetermined number of images is obtained, and the image set is , Represented by this feature vector. Since this process does not require the process of associating the first image area and the second image area, the initial setting for recognizing the object is easy, and the recognition result depends on the influence of environmental conditions. It is hard to receive.
[0027]
DETAILED DESCRIPTION OF THE INVENTION
First, the principle of the present invention will be described with reference to FIGS.
[0028]
FIG. 1 shows the entire processing procedure of the object recognition method of the present invention. The object recognition method of the present invention includes a learning process 1001 (steps S1001a to S1001c) and a recognition process 1002 (steps S1002a to S1002c). Hereinafter, the procedure of the object recognition method of the present invention will be described by taking as an example the case where the object recognition method of the present invention is applied to recognize a human. The object recognition method shown in FIG. 1 is executed by an object recognition device 1 described later with reference to FIG.
[0029]
Step S1001a: A learning image set is input to the object recognition apparatus 1. In the following description, an image set refers to a set of two images of a visible light image and a far-infrared light image of the same object unless otherwise specified. The learning image set includes at least one image set (second image set) representing a human (second object belonging to the category “human”) and at least one other than the second image set. An image set (an image set representing a non-human object, a third image set). In step S1001a, a plurality of learning image sets are input.
[0030]
Step S1001b: A feature vector is obtained for each of the plurality of learning image sets input in step S1001a. The calculation of the feature vector from one image set will be described later with reference to FIGS. 2A and 2B. This feature vector can be regarded as one point in the feature space.
[0031]
Step S1001c: An identification plane in the feature amount space is determined so as to identify (separate) the feature amount vector for at least one second image set and the feature amount vector for at least one third image set. . The calculation of the identification plane will be described later with reference to FIG.
[0032]
Step S1002a: A recognition image set (first image set) is input to the object recognition apparatus 1.
[0033]
Step S1002b: A feature vector is obtained for each recognition image set input in step S1002a.
[0034]
Step S1002c: It is determined whether or not the object (first object) of the image set for recognition belongs to a specific category “human”. This determination is made based on the positional relationship between the identification plane obtained in step S1001c and the feature vector obtained in step S1002b.
[0035]
The learning process 1001 is a process for obtaining an identification surface (identification parameter) from a learning image set. In the recognition process 1002, this identification surface is used as a criterion for determining whether or not the object represented by the recognition image set belongs to a specific category.
[0036]
FIG. 2A shows image sets 610 to 613 (at least one second image set) input in step S1001a (FIG. 1). Each of the image sets 610 to 613 includes two images of a human visible light image and the same human far-infrared light image. In the example shown in FIG. 2A, the image set 610 includes a visible light image 601 (third image) and a far-infrared light image 602 (fourth image). The visible light image is an image representing the intensity of visible light (light in the wavelength band of 380 to 800 nm) emitted or reflected from the object of the image, and the far-infrared light image is the image. It is an image showing the intensity | strength of the far-infrared light ray (light ray of a wavelength band of wavelength 8-10 micrometers) radiated | emitted or reflected from the target object. The visible light image represents the object using an attribute of the object called the intensity (luminance) of visible light emitted or reflected from the object, and the far-infrared light image is the far light emitted or reflected from the object. It can be said that the object is expressed using the attribute of the object such as the intensity of infrared rays.
[0037]
The object 621 in the image 601 and the object 622 in the image 602 are the same object (the same person). The objects of the visible light image and the far-infrared light image included in the image set 611 are also the same object. However, the objects need not be the same among the image sets 610 to 611. The object only needs to belong to the same category (in this example, the category “human”) between the image sets 610 to 611.
[0038]
FIG. 2A shows four second image sets input in step S1001a (FIG. 1) (image sets 610 to 613), but the second image set input in step S1001a (FIG. 1). The number of sets is not limited to this.
[0039]
FIG. 2B shows image sets 660 to 663 (at least one third image set) input in step S1001a (FIG. 1). Each of the image sets 660 to 663 includes two images, a visible light image of a non-human object and a far-infrared light image of the same object. In the example shown in FIG. 2B, the image set 660 includes an image 651 (visible light image) and an image 652 (far infrared light image). The image 651 has the same size as the image 601 (FIG. 2A), and the image 652 has the same size as the image 602 (FIG. 2A).
[0040]
With reference to FIG. 2A again, the processing for obtaining a feature vector for an image set in step S1001b (FIG. 1) will be described.
[0041]
Assume that two types of image filters (image filter A and image filter B, not shown) are applied to each of the two positions 631 and 632 of the image 601. The image filter A and the image filter B are image filters having different characteristics, for example. A specific example of the image filter will be described later with reference to FIG. In FIG. 2A, locations 631 and 632 are shown as rectangles. This rectangle represents the size of the image filter A and the image filter B. Here, it is assumed that the image filter A and the image filter B have the same size.
[0042]
By applying one image filter to one position of the image 601, one scalar value (filter output value) is generated. In the example shown in FIG. 2A, four filter output values (10, 3, 11 and 5) are generated by applying image filters A and B to two positions 631 and 632 of image 601 respectively. . Specifically, filter output values “10” and “3” are generated by applying the image filter A to the positions 631 and 632, respectively, and the filter is applied by applying the image filter B to the positions 631 and 632. Output values “11” and “5” are respectively generated.
[0043]
Similarly, four filter output values (1, 7, 11, and 4) are generated by applying the above-described image filters A and B to the two positions 633 and 634 of the image 602, respectively.
[0044]
By combining the four filter output values for image 601 (10, 3, 11 and 5) and the four filter output values for image 602 (1, 7, 11 and 4), A feature vector (10, 3, 11, 5, 1, 7, 11, 4) is calculated. In this way, the information of the visible light image and the information of the far infrared light image are integrated using the filter output value.
[0045]
A feature quantity vector is similarly calculated for the image sets 611 to 613 shown in FIG. 2A. The feature vector has a filter output value as a component. This feature quantity vector can be regarded as one point in the 8-dimensional feature quantity space.
[0046]
For the image set 660 shown in FIG. 2B, the feature amount vector is calculated in the same manner. Specifically, four filter output values (8, 9, 0, and 2) are generated by applying the above-described image filter A and image filter B to the two positions 681 and 682 of the image 651, respectively. . By applying the image filter A and image filter B described above to the two positions 683 and 684 of the image 652, four filter output values (9, 12, 10 and 4) are generated. By combining the four filter output values for image 651 (8, 9, 0 and 2) and the four filter output values for image 652 (9, 12, 10 and 4), A feature vector (8, 9, 0, 2, 9, 12, 10, 4) is calculated. A feature quantity vector is similarly calculated for the image sets 661 to 663 shown in FIG. 2B.
[0047]
The image filter to be applied and the position to which the image filter is applied are determined in advance. In one embodiment of the present invention, an image filter to be applied and a position to which the image filter is applied are determined through a feature amount dimension deletion process described later with reference to FIG. In the example shown in FIGS. 2A and 2B, the same image filters as image filters A and B applied to image set 610 (FIG. 2A) are also applied to image set 660 (FIG. 2B). The positional relationship between the positions 631 and 632 with respect to the image 601 is equal to the positional relationship between the positions 681 and 682 with respect to the image 651, respectively. Note that the number of positions to which the image filter is applied in one image is not limited to two. Further, the number of image filters applied to one position in one image is not limited to two.
[0048]
Thus, in step 1001b (FIG. 1), each of a plurality of image sets (image sets 610 to 613 shown in FIG. 2A and image sets 660 to 663 shown in FIG. 2B) is preliminarily selected from the two images. A feature quantity vector in the feature quantity space is obtained that has at least one filter output value obtained by applying at least one predetermined image filter at at least one predetermined position as a component.
[0049]
FIG. 3 shows an example in which the image filter 354 is applied to the image 351. In the example shown in FIG. 3, the image filter 354 is applied to the position 353 of the image 351. FIG. 3 shows an enlarged view of the portion 352 of the image 351. In this enlarged view, the value inside the rectangle representing the outline of the portion 352 indicates the value of the pixel included in the image 351.
[0050]
In the example shown in FIG. 3, the image filter 354 has a size of 3 × 3. The value inside the rectangle representing the image filter 354 indicates nine filter coefficients. The filter output value obtained by applying the image filter 354 to the position 353 of the image 351 is the product of the filter coefficient of the image filter 354 and the pixel value corresponding to the filter coefficient. This is the sum of the coefficients. In this example, the filter output value is 765. The operation for obtaining the filter output value is called a filter operation. By the filter operation, local characteristic information of the image is extracted as a filter output value.
[0051]
FIG. 4 shows a state in which the feature amount vector obtained for each image set in step 1001b (FIG. 1) is plotted in the feature amount space 701. However, in the example shown in FIG. 4, the feature amount space 701 is represented as a two-dimensional space (that is, a plane) for the sake of explanation. The feature amount space 701 is defined by two dimensions, that is, a feature amount dimension 1 and a feature amount dimension 2. The feature amount vector is represented as one point in the feature amount space 701. In FIG. 4, each of the ◯ marks represents a feature amount vector for an image set (second image set) representing a human, and each of the x marks represents an image set (third) representing an object other than a human. ) Represents a feature vector. Hereinafter, a feature vector for an image set representing a human is simply referred to as a “feature vector representing a human”, and a feature vector for an image set representing a non-human object is simply referred to as “non-human”. It may be referred to as a “feature vector expressing the object”.
[0052]
The identification straight line 702 (identification plane) is determined in step 1001c (FIG. 1) so as to identify the feature quantity vector represented by the ◯ mark and the feature quantity vector represented by the X mark. In the example shown in FIG. 4, all feature quantity vectors represented by ◯ are on the upper side of the identification line 702 (arrow 703 side, first side), and all feature quantity vectors represented by x are identified. It is below the straight line 702 (arrow 704 side, second side). The identification line 702 can be determined using, for example, a support vector machine technique. Regarding the method of the support vector machine, for example, reference: V.C. See Vapnic, “The Nature of Statistical Learning Theory”, Springer Verlag, 1995. Alternatively, the identification line 702 may be determined using a technique such as learning of a linear perceptron or a discriminant analysis method. As the learning algorithm, a non-parametric learning algorithm such as a statistical parameter estimation method and a neural network may be employed.
[0053]
In the example shown in FIG. 4, in the two-dimensional feature amount space 701, the identification line 702 represents a feature amount vector for an image set (second image set) representing a person and an object other than a person. The feature quantity vector for the image set (third image set) is identified (separated). A feature amount vector for an image set representing a human (second image set) and a feature vector for an image set expressing a non-human target (third image set) are n-dimensional (n ≧ When represented in the feature amount space of 2), the feature amount vectors are identified by the identification plane. An n-dimensional feature space is dimension x₁, Dimension x₂,. . . , And dimension x_nIs defined by (Equation 1).
[0054]
[Expression 1]
a₁x₁+ A₂x₂+. . . + A_nx_n+ D = 0
Hereinafter, in this specification, “plane” means a point satisfying the relationship of (Equation 1) in an n-dimensional (n ≧ 2) feature amount space (x₁, X₂,. . . , X_n). In the case of n = 2, (Equation 1) represents a straight line, and this straight line is included in the definition of “plane” described above.
[0055]
FIG. 5 shows an example of a recognition image set (first image set) 510 input in step 1002a (FIG. 1). The image set 510 includes a visible light image 501 that represents the object 521 and a far-infrared light image 502 that represents the object 522. The target object 521 and the target object 522 are the same target object (first target object).
[0056]
In step S1002b (FIG. 1), a feature quantity vector (first feature quantity vector) in the feature quantity space for the image set 510 is calculated. In the example shown in FIG. 5, the feature vector for the image set 510 is calculated as (9, 4, 12, 6, 1, 6, 14, 3). The calculation of the feature amount vector is performed in the same manner as the calculation of the feature amount vector for the image set 610 described above with reference to FIG. 2A. That is, in step S1002b, for the image set 510, at least one image filter (image filters A and B) determined in advance in at least one predetermined position of two images (images 501 and 502). A feature quantity vector (first feature quantity vector) in the feature quantity space having at least one filter output value obtained by applying as a component is obtained.
[0057]
The mark ● shown in FIG. 4 indicates the first feature vector plotted in the feature space 701. However, in FIG. 4, for the sake of explanation, the first feature vector is represented as a two-dimensional feature vector (2, 10), not as an eight-dimensional feature vector.
[0058]
In step S1002c (FIG. 1), the object represented by the image set 510 (FIG. 5) is based on the positional relationship in the feature amount space 701 between the first feature amount vector indicated by the mark ● and the identification line 702. It is determined whether or not the user is human. In the example shown in FIG. 4, the feature amount vector indicated by the mark ● is on the upper side of the identification line 702 (the arrow 703 side). The arrow 703 side of the identification line 702 is the side on which the feature vector for an image set representing a person is located, and therefore it is determined that the object represented by the image set 510 (FIG. 5) is a person.
[0059]
Thus, according to the method of the present invention shown in FIG. 1, the first object (the visible light image 501 and the far-infrared light image 502 of the image set 510) represented by the image set 510 (FIG. 5). Are recognized as being human. The determination as to whether or not the first object belongs to the category “human” is based on the intensity of the visible light reflected or emitted by the object (first attribute) and the far infrared that the object reflects or emits. Since this is performed based on the intensity of the light beam (second attribute), the reliability of recognition of the object is increased. Furthermore, a feature quantity vector in the feature quantity space having a filter output value obtained by applying a predetermined image filter at a predetermined position of the image 501 and the image 502 as a component is obtained, and an image set 510 is obtained. Is represented by this feature vector. Since this process does not require the process of associating the area of the image 501 with the area of the second image 502, the initial setting for recognizing the first object is facilitated, and the recognition result is Less susceptible to environmental conditions.
[0060]
Embodiments of the present invention will be described below with reference to the drawings. The same components are denoted by the same reference numerals, and duplicate descriptions may be omitted.
[0061]
FIG. 6 shows a configuration of the object recognition apparatus 1 according to the embodiment of the present invention.
[0062]
The object recognition apparatus 1 includes a far-infrared light camera 100, a visible light camera 110, a storage device 120 that stores learning image data (a learning image set), and a filter processing unit 125 that applies an image filter to the image. And whether the object represented by the learning processing unit 130 and the image set acquired by the far-infrared light camera 100 and the visible light camera 110 belongs to a specific category (for example, whether the object is a human). A recognition processing unit 140, an identification parameter storage unit 150 that stores an identification parameter used as a determination criterion in the determination, a work memory 160, and a display unit 170 that displays a recognition result. Each component of the object recognition device 1 may be connected to each other via an internal bus or may be connected to each other via a network. Such a network may include any network such as a wireless network, a wired network, a telephone line network, and the like. Such a network may include the Internet.
[0063]
The far infrared light camera 100 captures a far infrared light image, and the visible light camera 110 captures a visible light image. In the embodiment of the present invention, a luminance image is used as the visible light image.
[0064]
The object recognition apparatus 1 can be applied to, for example, an outdoor intruder monitoring system, a pedestrian detection system mounted on a moving body such as an automobile, and a visual system mounted on a mobile robot.
[0065]
As described above, the object recognition apparatus 1 performs the learning process and the recognition process shown in FIG. 1 as a whole.
[0066]
The visible light camera 110 and the far-infrared light camera 100 are configured to input a learning image set in step S1001a (FIG. 1) and a recognition image set (first image) in step S1002a (FIG. 1). Group) input processing. The visible light camera 110 and the far-infrared light camera 100 function as an input unit 190 that inputs a learning image set and a recognition image set to the object recognition apparatus 1. Of course, a visible light camera and a far-infrared light camera that input a learning image set to the object recognition device, and a visible light camera and a far-infrared light camera that input a recognition image set to the object recognition device are separately provided. It may be provided.
[0067]
The learning image group is divided into an image group (second image group) representing a human object and an image group (third image group) representing a non-human object. The image data is temporarily stored in the storage device 120. The storage device 120 can be, for example, a hard disk. Alternatively, the storage device 120 can be any memory.
[0068]
The filter processing unit 125 calculates feature vectors in step S1001b (FIG. 1) and step S1002b (FIG. 1). The filter processing unit 125 (feature quantity vector calculation unit) can be, for example, a digital signal processor.
[0069]
The learning processing unit 130 (learning unit) performs processing for obtaining an identification plane in step S1001c (FIG. 1).
[0070]
The recognition processing unit 140 (determination unit) performs a process of determining whether or not the object of the recognition image set belongs to a specific category “human” in step S1002c (FIG. 1).
[0071]
The display unit 170 displays the recognition result in the recognition processing unit 140. An arbitrary display device can be used as the display unit 170. The display unit 170 may be omitted.
[0072]
FIG. 7A shows an example of the arrangement of the far-infrared light camera 100 and the visible light camera 110. In the example shown in FIG. 7A, the far-infrared light camera 100 and the visible light camera 110 are arranged in parallel.
[0073]
FIG. 7B shows another example of the arrangement of the far-infrared light camera 100 and the visible light camera 110. In the example shown in FIG. 7B, the far-infrared light camera 100 and the visible-light camera 110 are arranged so that the optical axis of the visible light camera 110 reflected by the cold mirror 802 is aligned with the optical axis of the far-infrared light camera 100. Has been. A cold mirror is a mirror that reflects visible light and transmits far-infrared light.
[0074]
The function of the far-infrared light camera 100 and the function of the visible light camera 110 may be realized by one camera.
[0075]
FIG. 7C shows an example in which a visible / far-infrared light camera 210 having both functions is used in place of the far-infrared light camera 100 and the visible light camera 110. The visible / far-infrared light camera 210 realizes the functions of a far-infrared light camera and the function of a visible light camera using an area sensor.
[0076]
FIG. 8A shows an example of a visible light image 803 that is captured by the visible light camera 110 and represents a human object.
[0077]
FIG. 8B shows an example of a far-infrared light image 804 that is captured by the far-infrared light camera 100 and represents a human object. The far-infrared light image 804 is obtained by photographing the same object as the visible light image 803 shown in FIG. 8A at substantially the same time.
[0078]
FIG. 9A shows an example of a visible light image 805 that is captured by the visible light camera 110 and represents a non-human object (tree).
[0079]
FIG. 9B shows an example of a far-infrared light image 806 representing an object (tree) other than a human being photographed by the far-infrared light camera 100. The far-infrared light image 806 is obtained by photographing the same object as the visible light image 805 shown in FIG. 9A at substantially the same time.
[0080]
The visible light image and the far-infrared light image shown in FIGS. 8A, 8B, 9A, and 9B constitute an image set (an image set for learning or an image set for recognition). It is necessary that the same object appears in the visible light image and the far-infrared light image, but the alignment between the visible light image and the far-infrared light image must be accurately performed in pixel units. Absent. For example, in the visible light image 803 (FIG. 8A), the object is shifted to the left from the center of the image, and in the far-infrared light image 804 (FIG. 8B), the object is shifted to the right from the center of the image. However, since the processing for associating the region of the visible light image 803 and the region of the far-infrared light image 804 is not necessary in the learning process and the recognition process of the present invention, such a shift is not a problem. Therefore, alignment of the far-infrared light camera 100 and the visible light camera 110 is easy, and the initial setting of the object recognition apparatus 1 is easy. However, the scale ratio (aspect ratio) of all visible light images included in the learning image set and the recognition image set must be the same, and the object must be shown at the same position. This may be realized by cutting out a predetermined area from a visible light image captured by the visible light camera 110. The same applies to the infrared light images included in the learning image set and the recognition image set.
[0081]
FIG. 10A schematically shows characteristics of an image filter used in the filter processing unit 125 (FIG. 6). The image filter shown in FIG. 10A, when applied to a specific position in the image, has an edge with a specific spatial frequency in a specific direction (vertical direction) at that specific position (the elliptical shape shown in FIG. 10A). Edges where the pixel values change sequentially within the range of the horizontal width (short axis) are selectively detected.
[0082]
FIG. 10B schematically shows the characteristics of an image filter that selectively detects horizontal edges.
[0083]
FIG. 10C schematically shows characteristics of an image filter that selectively detects an edge extending from the lower left to the upper right.
[0084]
FIG. 10D schematically illustrates the characteristics of an image filter that selectively detects an edge extending from the lower right to the upper left.
[0085]
The image filters shown in FIGS. 10A to 10D include orientation selectivity (detects only edges in a specific direction), position selectivity (detects edges at a specific position), and spatial frequency selectivity (specific spatial frequency). The edge where the pixel value changes). Here, the spatial frequency refers to a degree of change of a pixel value (for example, luminance) with respect to a position change in an image. An example of an image filter having such characteristics is a Gabor filter. By using multiple types of image filters having azimuth selectivity, position selectivity, and spatial frequency selectivity (a plurality of image filters with different selectivity), different image filters detect information related to the same edge repeatedly. Waste can be reduced and the number of required image filters can be reduced. Thereby, the amount of calculation required to execute the learning process 1001 and the recognition process 1002 can be reduced. As a result, an increase in the amount of calculation due to inputting a plurality of images (visible light image and far-infrared light image) having different attributes can be minimized.
[0086]
One filter output value represents information of an edge having a specific spatial frequency in a specific direction at a specific position of the image. Representing one set of images by a feature vector having a filter output value as a component is equivalent to simply expressing the shape of a common object of a visible light image and a far-infrared light image as a collection of edges. To do.
[0087]
11A to 11D show examples of filter coefficients of the Gabor filter. In the example shown in FIGS. 11A to 11D, each grid point corresponds to one filter coefficient of a Gabor filter having a size of 13 pixels × 13 pixels, and the value (real part) of the filter coefficient is represented by each grid point. Shown as the height of. 11A to 11D correspond to the image filters shown in FIGS. 10A to 10D, respectively.
[0088]
Hereinafter, detailed processing procedures of the learning process and the recognition process executed by the object recognition apparatus 1 will be described.
[0089]
<Learning process>
FIG. 12 shows the detailed procedure of the learning process executed by the object recognition device 1. Step S91 corresponds to step S1001a and step S1001b (FIG. 1), and steps S92 to S97 correspond to step S1001c (FIG. 1).
[0090]
Step S91: preparing a plurality of image sets (second image set) representing human beings and image sets (third image set) representing objects other than human beings, and obtaining feature vector for each image set. Ask. The set of feature quantity vectors obtained in step S91 is referred to as feature quantity data F. A more detailed processing procedure of step S91 will be described later with reference to FIGS. In the following description, it is assumed that a 1032-dimensional feature vector is calculated from each image set in step S91 (the present invention is not limited to this). This feature quantity vector has 1032 feature quantity dimensions and feature quantity dimensions x.₁, Feature dimension x₂, Feature dimension x_Three,. . . , Feature dimension x₁₀₃₂Expressed as a point in the space defined by.
[0091]
Step S92: Of the feature value dimensions, a feature value dimension used for learning is designated. First specify all dimensions. In this example, the first time is the feature dimension x₁, Feature dimension x₂, Feature dimension x_Three,. . . , Feature dimension x₁₀₃₂Are designated as feature quantity dimensions used for learning. In the second and subsequent times, the remaining dimensions removed in step S94, which will be described later, are designated as feature quantity dimensions used for learning. By repeating the processing from step S92 to step S96, the number of dimensions of the feature amount space decreases. This is referred to as “feature dimension dimension deletion processing”.
[0092]
Step S93: A lower-dimensional feature quantity space is defined using the designated feature quantity dimension (however, a 1032-dimensional feature quantity space is first defined), and each feature quantity vector included in the feature quantity data F is defined. Is represented as a feature quantity vector in this lower dimensional feature quantity space. The feature quantity vector in the lower dimensional feature quantity space is composed only of components corresponding to the feature quantity dimension designated in step S92 among the components of the 1032-dimensional feature quantity vector included in the feature quantity data F. It is expressed as a lower-dimensional feature vector. Since one component of the feature vector corresponds to one filter output value at one position in the image, this lower-dimensional feature vector is also determined in advance for at least one of the two images in the image set. As a component, at least one filter output value obtained by applying one predetermined image filter at a predetermined position is included.
[0093]
Next, in this lower-dimensional feature amount space, an identification plane for identifying (separating) a feature amount vector for an image set representing a person and a feature amount vector for an image set representing a non-human object. Is determined as a provisional identification plane.
[0094]
The position of the feature quantity vector with respect to the identification plane can be expressed by a value (weighted sum) obtained by weighting the components corresponding to the feature quantity dimensions of the feature quantity vector with the coefficients corresponding to the feature quantity dimensions of the identification plane. For example, consider a case where the identification plane in the three-dimensional space is represented as x + 2y + 3z = 0 and the feature vector is (x, y, z) = (− 1, 0, 4). When each component of the feature vector (−1, 0, 4) is weighted and added by the coefficient (1, 2, 3) of each feature value dimension of the identification plane, 1 × (−1) + 2 × 0 + 3 × 4 A value of = 11 is obtained. This value represents the distance between the feature vector (-1, 0, 4) and the identification plane. The positional relationship of the feature vector with respect to the identification plane can be expressed by the sign and magnitude of this value.
[0095]
As described above, a support vector machine method or the like is used as a method for determining an identification plane (temporary identification plane) that divides a category when points belonging to two categories are distributed in the feature amount space. obtain.
[0096]
Using such a technique, an identification plane for separating a feature vector for an image set representing a person in a feature space and a feature vector for an image set representing an object other than a human is temporarily identified. Determined as a plane. The separation does not necessarily have to be complete, and some feature vectors (for example, feature vectors for image sets representing humans) cross the identification plane (feature vector vectors for image sets that do not represent humans). It does not matter if it is in a distributed arrangement (misidentification). However, it is better that the number of misidentified feature vectors is small. The determination method of the identification plane may be selected from a plurality of methods so that the misidentification feature vector is reduced.
[0097]
Step S94: Among the coordinate axes corresponding to the feature quantity dimension designated in Step S92, d coordinate axes are designated in Step S92 in order from the coordinate axis having the smallest absolute value with respect to the temporary identification plane determined in Step S93. It is removed from the feature quantity dimension (feature quantity dimension used for learning). The value of d is a predetermined integer of 1 or more.
[0098]
For example, in a three-dimensional feature amount space (the coordinate axis is xyz), when the temporary identification plane is represented as x + 2y + 3z = 0, the absolute value of the angle formed by this temporary identification plane and the x, y, and z axes is x The axis increases in the order of the y-axis and the z-axis. In this case, if d = 1, the feature quantity dimension corresponding to the x-axis is excluded from the feature quantity dimension used for learning. Instead of paying attention to the angle formed between each coordinate axis and the identification plane, attention is paid to the angle δ formed between each coordinate axis and the normal of the identification plane, and d coordinate axes may be excluded from those having a large absolute value of δ. Similar results are obtained. The direction of the normal of the identification plane is represented by a normal vector having as a component the coefficient of the expression representing the temporary identification plane. For example, the direction of the normal line of the temporary identification plane x + 2y + 3z = 0 is represented by the normal vector (1, 2, 3).
[0099]
With reference to FIG. 4 again, the meaning of removing the feature value dimension from the feature value dimension used for learning will be described.
[0100]
In FIG. 4, the angle between the axis (horizontal axis) corresponding to the feature quantity dimension 1 and the identification line (identification plane) 702 is the same as the axis (vertical axis) corresponding to the feature quantity dimension 2 to the identification line (identification plane) 702. It is smaller than the angle to make. This is because the feature quantity dimension 1 is used to identify (separate) a feature quantity vector for an image set representing a human and a feature quantity vector for an image set representing a non-human object. Means less important than 2. That is, the value of the feature quantity dimension 2 (filter output value) greatly affects the determination of whether or not the object is a human. In the example shown in FIG. 4, in step S94 (FIG. 12), the feature quantity dimension 1 out of the feature quantity dimension 1 and the feature quantity dimension 2 is deleted.
[0101]
The value of one feature amount dimension corresponds to one filter output value at one position of one visible light image or far infrared light image included in the image set. In this way, by obtaining the identification line (or identification plane), it is possible to determine which feature quantity dimension (filter output) out of the plurality of filter outputs obtained from the visible light image and the plurality of filter outputs obtained from the far-infrared light image. ) Can be determined. Step S94 (FIG. 12) means that an unimportant feature amount dimension (a feature amount dimension that does not contribute to identification) is deleted from the feature amount dimension used for learning. Returning to FIG. 12, the detailed processing procedure of the learning process will be continued.
[0102]
Step S95: A feature amount space in which d coordinate axes are reduced is set, and the discrimination performance is evaluated in the newly set feature amount space. This evaluation is performed by using the feature vector representing the person included in the feature data F in the lower-dimensional feature quantity space newly defined by removing the d coordinate axes (feature quantity dimensions) in step S94 and the human. This is performed by examining how accurately the feature vector representing the object other than the above can be identified (separated). A more detailed processing procedure of step S95 will be described later with reference to FIG.
[0103]
Step S96: It is determined whether or not the identification performance satisfies a reference value. If the result of the determination in step S96 is “No”, the process proceeds to step S97. If the result of the determination in step S96 is “Yes”, the process returns to step S92 (further, the feature quantity dimension is deleted).
[0104]
Step S97: The feature quantity dimension used in step S93 is designated as the selection dimension. Further, the temporary identification plane obtained in step S93 is designated as the identification plane. This identification plane is used as a criterion in a recognition process performed later. The identification plane is a plane in the space (feature quantity space) defined by the feature quantity dimension designated as the selection dimension in step S97. In step S96, the first time may unconditionally move to step S92.
[0105]
As described above, in steps S92 to S96, the learning processing unit 130 (FIG. 6) sets an image group (FIG. 6) representing a person in a temporary feature amount space having a larger number of dimensions than the feature amount space used in the recognition process. The normal of the plane (temporary identification plane) for identifying the feature quantity vector for the second image set) and the feature quantity vector for the image set representing the non-human object (third image set) The feature amount space used in the recognition process is defined by deleting at least one dimension from the tentative feature amount space based on the orientation of.
[0106]
In step S97, 1032 feature quantity dimensions, feature quantity dimension x₁, Feature dimension x₂, Feature dimension x_Three,. . . , Feature dimension x₁₀₃₂Among them, m (m is an integer of 1032 or less) feature quantity dimension, feature quantity dimension x_a1, Feature dimension x_a2, Feature dimension x_a3,. . . , Feature dimension x_amAssuming that subscripts a1, a2, a3,... Am are integers greater than or equal to 1 and less than or equal to 1032 are specified as selection dimensions, a list of selection dimensions (feature quantity dimension x_a1, Feature dimension x_a2, Feature dimension x_a3,. . . , Feature dimension x_am) Indicates which filter output applied to the visible light image and the far-infrared light image included in the image set is used in a recognition process performed later. That is, it can be said that the list of selected dimensions defines how to combine the information of the visible light image and the information of the far infrared light image.
[0107]
The identification plane determined in step S97 is represented by a list of selected dimensions and coefficients for the selected dimensions. Parameters representing these identification planes are stored in the identification parameter storage unit 150 (FIG. 6).
[0108]
In the processing procedure described above, step S95 may be omitted, and the determination in step S96 may be replaced with a determination as to whether or not the number of deleted feature quantity dimensions has reached a predetermined value. That is, when the number of feature dimension deletions reaches a predetermined number, the process may proceed to step S97, and when it does not reach the predetermined number, the process may proceed to step S92. By performing such processing, the feature quantity dimension number can be set to a predetermined value.
[0109]
If the value d used in step S94 is increased, the number of repetitions of the procedure from step S92 to step S96 can be reduced, and the amount of calculation can be reduced. On the other hand, if the value of d is reduced, many feature quantity dimensions are not deleted at a time, so that it is possible to determine a sufficient number of feature quantity dimensions as selection dimensions necessary to realize a desired discrimination performance. Become.
[0110]
In the recognition process performed later, an identification plane (not limited to a plane) different from the identification plane determined in step S97 may be used as a determination criterion (identification parameter). Such an identification plane is set so as to identify a feature vector representing a person and a feature vector representing an object other than a human in a space defined by the selected dimension. An arbitrary identification method for identifying a feature vector representing a person and a feature vector representing a non-human object in a space defined by the selected dimension, and a recognition process performed later by an identification parameter used in the identification method , It can be employed to determine whether the object is human.
[0111]
In the recognition process performed later, a linear identification method may be employed, or a non-linear identification method may be employed. The linear identification method is, for example, whether the object is a human being based on which side of the identification plane represented by the identification parameter, which is determined in step S97, the feature vector representing the object is present. This is a method for determining whether or not. Examples of the non-linear identification method include a k-NN method, a perceptron using a non-linear element, LVQ, and a non-linear SVM. Hereinafter, an example of a non-linear identification method will be described with reference to FIGS. 13A to 13C.
[0112]
FIG. 13A is a diagram for explaining an identification method using a curved identification surface. A space 1380 is a feature amount space defined by the selected dimension. In FIG. 13A, the space 1380 is shown as a plane defined by two selection dimensions of the feature quantity dimension 1 and the feature quantity dimension 2, and the identification surface 1361 is shown as a curve. In FIG. 13A and FIGS. 13B and 13C described later, each of the ◯ marks represents a feature vector that represents a person, and each of the x marks represents a feature vector that represents an object other than a human.
[0113]
The identification surface 1361 identifies, in the feature quantity space 1380, a feature quantity vector that represents a person and a feature quantity vector that represents an object other than a human. In this example, the feature vector (○ mark) representing a human is located on the first side (the arrow 1362 side) of the identification surface 1361, and the feature vector (× mark) representing an object other than a human is , Located on the second side of the identification surface 1361 (the side of the arrow 1363). The identification surface 1361 can be represented by, for example, an expression having values of the feature value dimension 1 and the feature value dimension 2 as variables. The coefficient of such an expression is stored in the identification parameter storage unit 150 (FIG. 6) as an identification parameter.
[0114]
A point 1364 indicates a feature vector for an image set (recognition image set) input in a recognition process performed later. In the example shown in FIG. 13A, the feature quantity vector 1364 is located on the first side of the identification plane 1361, so it is determined that the object of the recognition image set is a human.
[0115]
FIG. 13B is a diagram for explaining an identification method using a distance in the feature amount space 1380. Examples of such identification methods include the k-NN method and LVQ. The representative point 1366 is a point representing a feature vector representing a person (a representative point indicating a category of “human”). The representative point 1366 is obtained, for example, as the centroid of all feature vectors representing humans. Similarly, the representative point 1367 is a point representing a feature vector representing a non-human object (a representative point indicating a category “non-human”). The representative point 1366 and the representative point 1367 are represented by coordinates representing the points in the feature amount space. Such coordinates are stored as identification parameters in the identification parameter storage unit 150 (FIG. 6).
[0116]
A point 1365 indicates a feature vector for an image set (recognition image set) input in a recognition process performed later. In this identification method, the category to which the representative point closest to the feature vector belongs is the recognition result for the feature vector. In the example shown in FIG. 13B, since the category indicated by the representative point (representative point 1366) closest to the feature quantity vector 1365 is the category “human”, it is determined that the object of the recognition image set is a human.
[0117]
FIG. 13C is a diagram for explaining an identification method using the distribution of feature quantity vectors in the feature quantity space 1382. As an example of such an identification method, a method such as a neural network such as a perceptron using a non-linear element may be used. In FIG. 13C, the feature quantity space 1382 is shown as a one-dimensional space (that is, a straight line) defined by one selected dimension (feature quantity dimension 1). A curve 1369 and a curve 1370 are respectively a distribution amount of a feature vector expressing a person in the feature space 1382 (a distribution intensity indicating a category of “human”) and a feature vector expressing an object other than a human. Distribution intensity (distribution intensity indicating a category of “non-human”). The curve 1369 and the curve 1370 can be represented by expressions using the value of the feature quantity dimension 1 as a variable. The coefficient of such an expression is stored in the identification parameter storage unit 150 (FIG. 6) as an identification parameter.
[0118]
A point 1368 indicates a feature vector for an image set (recognition image set) input in a recognition process performed later. In this identification method, the intensities of a plurality of distributions are compared at the position of the feature quantity vector, and the category indicated by the maximum distribution intensity is the recognition result for the feature quantity vector. In the example shown in FIG. 13C, the distribution intensity 1369 indicating the category “human” is greater than the distribution intensity 1370 indicating the category “non-human” at the position of the feature vector 1368. The object is determined to be a human.
[0119]
Thus, the identification parameter represents not only the identification plane but also a parameter used in an arbitrary identification method for identifying feature quantity vectors belonging to different categories in the feature quantity space.
[0120]
FIG. 14 schematically illustrates a change in identification performance associated with performing a feature quantity dimension deletion process. As shown in FIG. 14, initially, the identification performance is improved by deleting the feature quantity dimension. This is because extra information (noise) that adversely affects identification can be reduced by reducing feature quantity dimensions that do not contribute to identification.
[0121]
In general, simply combining information from multiple images with different attributes, such as a visible light image and a far-infrared light image, increases the amount of information, increases the identification process, and increases the number of samples required for learning (for learning Since the number of image sets) increases, sample collection becomes difficult. If the number of learning image sets is insufficient, the identification performance may deteriorate. However, in the embodiment of the present invention, after combining the information of the visible light image and the information of the far-infrared light image, the feature amount dimension is deleted in Steps S92 to S96. By deleting feature quantity dimensions and selecting information effective for identification, it is possible to improve (or maintain) identification performance while reducing the amount of calculation in recognition processing performed later.
[0122]
As a result of the simulations performed by the present inventors using 858 image sets representing humans and 11052 image sets representing non-human objects, the number of feature dimensions is 88 as a result of the processing in steps S92 to S96. %, And the probability of misidentification was reduced to 1/9.
[0123]
The determination in step S96 (FIG. 12) is performed as follows, for example. The change in the identification performance obtained in step S95 is monitored, the identification performance in the previous step S95 is compared with the performance in this step S95, and if the identification performance is improved or maintained, it is determined that the standard is satisfied, If the identification performance is degraded, it is determined that the standard is not satisfied. When such a determination is made, the maximum value of the identification performance indicated by the point 1302 in FIG. 14 can be realized.
[0124]
The determination in step S96 (FIG. 12) may be performed in other manners. For example, an absolute discrimination performance value (reference number 1301 shown in FIG. 14) may be designated in advance, and the feature amount dimension may be deleted as much as possible under the condition that the designated discrimination performance can be realized. In this case, as long as the identification performance value is satisfied, the number of feature amount dimensions is deleted to the maximum (point 1302).
[0125]
FIG. 15 shows a more detailed processing procedure of step S91 (FIG. 12). Step S101 corresponds to step S1001a (FIG. 1), and steps S102 to S104 correspond to step S1001b (FIG. 1).
[0126]
Step S101: A visible light image and a far-infrared light image are input. The visible light image and the far-infrared light image constitute an image set. In step S101, an image group representing a person (an image group in which a visible light image and a far-infrared light image represent the same person) and an image group (an visible light image) representing an object other than a person. And a far-infrared light image are input as an image set representing the same object other than a human being). Examples of such image sets have been described above with reference to FIGS. 2A and 2B.
[0127]
The visible light image and the far-infrared light image input in step S101 are captured using the far-infrared light camera 100 and the visible light camera 110, and are once stored in the storage device 120 as a learning image set. . Alternatively, the visible light image and the far-infrared light image may be input to the object recognition apparatus 1 by being read from a recording medium (not shown).
[0128]
Step S102: Normalization of pixel values is performed for each image (visible light image or far infrared light image). Normalization of pixel values is performed according to (Equation 2).
[0129]
[Expression 2]
I ′ (x, y) = (I (x, y) −m) / σ
here,
I (x, y): pixel value at coordinates (x, y) in the image before normalization
m: Average pixel value of the entire image
σ: Standard deviation from the average pixel value of the entire image
I '(x, y): normalized pixel value at coordinates (x, y) in the image
It is.
[0130]
Step S103: A plurality of Gabor filters having different characteristics are applied to a plurality of regions in the image with respect to the image on which the pixel value has been normalized.
[0131]
Step S104: A feature vector is obtained from the filter output value corresponding to each region (specific region) to which the Gabor filter is applied.
[0132]
FIG. 16 shows the relationship between the image 431 and the region 432 to which the image filter in the image 431 is applied. The image 431 is one of a visible light image or an infrared light image input in step S101. A region 432 indicates a region to which the image filter is applied. Hereinafter, the region 432 to which the image filter in the image 431 is applied is referred to as a “specific region”. Details of the processing in steps S103 and S104 (FIG. 15) will be described with reference to FIG.
[0133]
In the example illustrated in FIG. 16, the specific region 432 has a square shape with one side having a length L. The image 431 has a rectangular shape having a height H and a width W. A plurality of specific areas 432 are set in the image 431. By applying a Gabor filter that matches the size of the specific area 432 to each of the specific areas 432, a filter output value is generated in each of the specific areas 432. The method by which the filter output value is generated was described above with reference to FIG.
[0134]
For example, the filter processing unit 125 (FIG. 6) sets a plurality of specific areas 432 in the image 431 so that the specific area 432 covers the entire image 431 while allowing the image 431 to overlap. For example, in FIG. 16, when the specific area 432 has a size (size 1) of L = H / 8 and W = H / 2, the specific area 432 is divided by L / 2 in the vertical direction and the horizontal direction. When overlapping the entire surface of the image 431, the number of specific areas in the image 431 is (H / L × 2-1) × (W / L × 2-1) = 15 × 7 = 105. . Four image filters having different characteristics (different azimuth selectivity) (four image filters having four types of direction selectivity shown in FIGS. 11A to 11D) are applied to one specific region.
[0135]
Furthermore, specific areas having different sizes are arranged on the entire surface of the image 431. Image filters having different sizes are applied to specific regions having different sizes. If the specific area is a square having a size of L = H / 4 (size 2), the specific area is overlapped by L / 2 in the vertical direction and the horizontal direction so as to cover the entire surface of the image 431, and W = H Under the assumption of / 2, the number of specific regions is (H / L × 2-1) × (W / L × 2-1) = 7 × 3 = 21. Similarly, when the specific area is a square having a size of L = H / 2 (size 3), the number of specific areas is (H / L × 2-1) × (W / L × 2-1) = 3 × 1. = 3.
[0136]
The total number of specific areas of three different sizes (size 1, size 2 and size 3) in the image 431 is 105 + 21 + 3 = 129. When four types of image filters having different orientation selectivity are applied to each of the specific regions, 129 × 4 = 516 filter outputs can be obtained from the image 431. One image set (a learning image set or a recognition image set) includes a visible light image (luminance image) and a far-infrared light image. When the visible light image and the far-infrared light image have the same size (height H, width W), and the specific region is set at the same position in the visible light image and the far-infrared light image, 1 The number of filter outputs obtained from two images in one image set is 516 × 2 = 1032. Therefore, one image set is represented by a 1032-dimensional feature vector. Alternatively, a higher-dimensional feature quantity space may be set by setting a higher-dimensional feature quantity space and mapping this 1032-dimensional feature quantity vector to the higher-dimensional feature quantity space. By mapping the feature quantity vector to a higher-dimensional feature quantity space, the distance between the feature quantity vectors corresponding to each image set is increased, so that an identification plane is obtained in step S93 (FIG. 12) to be performed later. The advantage that it is easy is obtained.
[0137]
The number of orientations of the Gabor filter is not limited to four. The number of orientations of the Gabor filter may be changed depending on the size of the Gabor filter and / or the position of the specific region. By changing the number of orientations depending on the size of the Gabor filter and / or the location of a specific region, it is possible to change from a specific location in the image (eg, a location where the orientation is to be distinguished in detail) and / or a specific spatial frequency region. It becomes possible to acquire more information efficiently.
[0138]
Further, the size of the image filter is not limited to three types. The size of the image filter may be one or more types. The visible light image and the far-infrared light image have different spatial frequency characteristics of changes in pixel values (luminance). Therefore, it is possible to efficiently acquire a large amount of information from an image by changing the size of an applied image filter between a visible light image and a far-infrared light image.
[0139]
For the visible light image and the far-infrared light image, the size and position of the specific region are not necessarily equal. For visible light images and far-infrared light images, performance improvement can be expected by setting the size and position suitable for each. However, by making the size and arrangement of the specific areas for both images equal, there is the advantage that the processing of applying the image filter to both images can be executed in the same procedure, reducing the scale of hardware circuits and software Is possible.
[0140]
As the image filter, a filter having a shape similar to the Gabor filter or an image filter for obtaining another edge may be used. Furthermore, an image filter other than the filter for obtaining the edge may be used. However, by using a Gabor filter or an image filter having a similar shape, it is possible to efficiently acquire information on luminance changes localized in both the position space and the frequency space. Therefore, when a Gabor filter or an image filter having a similar shape is used, it is possible to acquire information on spatial changes at a specific spatial frequency more efficiently than when an edge filter such as a sobel filter is used. Become. As a result, information effective for recognition can be acquired efficiently from the amount of information increased by combining images with different properties such as a visible light image and a far-infrared light image. The information of the visible light image and the information of the far-infrared light image can be effectively combined and used for recognition without using the temperature information obtained from the far-infrared light image.
[0141]
In the example described with reference to FIG. 16, a plurality of specific areas (areas to which the Gabor filter is applied) are set in one size image 431. However, the same result can be obtained by setting specific regions (regions to which the Gabor filter is applied) of the same size for each of a plurality of images of different sizes obtained by photographing the same object at different resolutions.
[0142]
In the far-infrared light image and the visible light image, the object does not necessarily appear at the same position (they may be shifted in the vertical direction and / or the horizontal direction between the two images). As long as the same positional relationship is maintained even when other objects are photographed, the positions of the objects in the visible light image and the far-infrared light image do not need to match. This is because in the learning process and the recognition process of the present invention, it is not necessary to associate a region in the far-infrared light image with a region in the visible light image.
[0143]
FIG. 17 shows the detailed procedure of the process for evaluating the discrimination performance in step S95 (FIG. 12).
[0144]
Step S111: Of the components of the feature vector, only the component corresponding to the feature dimension specified in step S92 is validated. As the feature amount vector, all feature amount vectors included in the feature amount data F obtained in step S91 (FIG. 12) or some feature amount vectors included in the feature amount data F are used.
[0145]
Step S112: Learning is performed to identify a feature vector expressing a person and a feature vector expressing an object other than a human using a predetermined identification method. The predetermined identification method is a method used for determining whether or not the object is a human in a recognition process performed later.
[0146]
Step S113: The discrimination performance is calculated using the feature data for evaluation (a set of feature vectors). As the feature quantity data for evaluation, the feature quantity data F obtained in step S91 (FIG. 12) may be used, or feature quantity data not used in the learning process shown in FIG. 12 may be used. . Alternatively, a set of evaluation feature amount vectors may be separately created in advance by the same procedure as the feature amount data F. The identification performance includes, for example, a feature quantity vector (including a feature quantity vector representing a human and a feature quantity vector representing a non-human object) in which the dimension set in step S94 (FIG. 12) is validated. It is expressed as a ratio that can be correctly identified using the identification method after the learning in S112 is completed.
[0147]
<Recognition process>
FIG. 18 shows a detailed procedure of recognition processing executed by the object recognition apparatus 1. Step S121 corresponds to step S1002a (FIG. 1), steps S122 to S123 correspond to step S1002b (FIG. 1), and steps S124 to S125 correspond to S1002c (FIG. 1). ing.
[0148]
Step S121: A visible light image and a far-infrared light image are input. This image input is performed by the visible light camera 110 and the far-infrared light camera 100 (FIG. 6), similarly to step S101 (FIG. 15) in the learning process.
[0149]
Step S122: A recognition target region is cut out from the image (visible light image and far infrared light image). The recognition target area is cut out so that the shape of the recognition target area matches the shape of the image used in the learning process. The recognition target area may be fixed in the image, or a plurality of recognition target areas may be cut out from one image. In the example described with reference to FIG. 16, the shape of the recognition target region to be cut out is a rectangle having an aspect ratio of H to W. The shape of the visible light image in the learning process is the same as the shape of the recognition target region cut out from the visible light image in the recognition process, and the shape of the far infrared light image in the learning process and the far infrared light image in the recognition process As long as the shape of the recognition target region cut out from the same is the same, the shape of the recognition target region cut out from the visible light image may be different from the shape of the recognition target region cut out from the far-infrared light image.
[0150]
The clipping in step S122 is performed in consideration of the shooting positions of the visible light image and the far-infrared light image input in step S101 (FIG. 15) in the learning process. Specifically, the cutout in step S122 is performed so that the object appears at the same position in the visible light image in the learning process and the recognition target region cut out from the visible light image in the recognition process. Of course, you may install the visible light camera 110 which image | photographs a visible light image in a recognition process so that extraction may not be required. The same applies to the far-infrared light image. The same applies to the magnification of the visible light image and the far-infrared light image, and the magnification (number of pixels) of both the visible light image and the far-infrared light image may be different. And the far-infrared light image enlargement ratio (number of pixels) are adjusted to be the same in the learning process and the recognition process.
[0151]
Next, the size of the cut-out visible light image and far-infrared light image is normalized as necessary. In the case where the shape of both the visible light image and the far-infrared light image is a rectangle with an aspect ratio of 2: 1, the size is normalized to, for example, a rectangle of 64 pixels vertically and 32 pixels horizontally. By normalizing the size of the image, the size of the Gabor filter applied to the image in the next step S123 (the size of the specific area on which the filter operates) can be fixed. The visible light image and the far-infrared light image that are cut out and normalized in size constitute an image set for recognition.
[0152]
Step S123: A feature vector corresponding to the selected dimension determined in step S97 (FIG. 12) is obtained from the cut out image. The feature amount vector is calculated using the same Gabor filter as that used in the learning process. If the image size is not normalized in step S122 described above, a Gabor filter having a size corresponding to the image size is applied.
[0153]
When a part of the feature quantity dimension is deleted in the learning process, the calculation process of the Gabor filter corresponding to the deleted feature quantity dimension is unnecessary, and is excluded from the feature quantity vector calculation process in advance.
[0154]
Step S124: The similarity is obtained using the weighted sum of the feature vector and the coefficient of the identification plane. The similarity indicates the degree to which the object of the recognition image set resembles a human. As already described, the weighted sum of the feature vector and the coefficient of the identification plane represents the distance (positional relationship) between the feature vector and the identification plane. This distance is a positive value when it is on one side of the space separated by the identification plane (for example, the first side on which a feature vector representing humans is located), and the opposite side (for example, a non-human countermeasure) When the feature amount vector is on the second side), it can be expressed as a negative value. The absolute value of the distance increases as the feature vector moves away from the identification plane.
[0155]
Step S125: A person is recognized based on the similarity (that is, it is determined that the object is a person). For example, based on whether the similarity (distance between the feature vector and the identification plane) is positive or negative (that is, based on which side the feature vector is located) A determination is made as to whether the object is human. Alternatively, when the similarity is positive (that is, the feature vector is on the first side of the identification plane) and the similarity is greater than or equal to a predetermined threshold, it is determined that the object is a human being. You may do it. Such thresholds reduce the need for recognition accuracy (for example, whether it is desirable to reduce the possibility of misrecognizing a non-human object as a human being, or misrecognizing a human being not a human being). Is desired). A numerical value indicating the degree of similarity may be displayed on the display unit 170.
[0156]
As described above, the object recognition device 1 (FIG. 6) of the present invention obtains an identification parameter (for example, a parameter representing an identification plane) using the far-infrared light image and the visible light image of the learning image set, The identification parameter is used as a criterion for recognizing the object of the far-infrared light image and visible light image of the recognition image set (determining whether the object belongs to a specific category). Recognition of the object is based on the intensity of the visible light emitted or reflected from the object (first attribute) and the intensity of the far-infrared light emitted or reflected from the object (second attribute). Since this is done, the reliability of recognition of the object is increased.
[0157]
The present inventors set a set of visible light images and far-infrared light images taken outdoors day and night (an image set 858 representing a human and an image set 11052 representing an object other than a human) as a learning image set. The simulation of the learning process and the recognition process described above was performed. As a result of the simulation, the misrecognition rate was 0.2%. This error recognition rate is the error detection rate (2.7%) in the comparative example in which learning processing and recognition processing are performed using only visible light images, and learning processing and recognition processing are performed using only far-infrared light images. Compared to the erroneous recognition rate (3.5%) in the comparative example, the value is very low (1/10 or less). In this way, high reliability of object recognition is realized.
[0158]
The object recognition apparatus 1 of the present invention can learn a correlation between a visible light image and a far infrared light image by performing learning processing using a visible light image and a far infrared light image. The correlation is reflected in the recognition process. For example, consider a visible light image and a far-infrared light image obtained by photographing an object (human) outdoors in the daytime. Under an environmental condition where the object is exposed to direct sunlight, the brightness of the object in the visible light image increases, and the far-infrared light image indicates that the temperature of the object is high. On the other hand, under an environmental condition in which the object is not exposed to direct sunlight, the brightness of the object in the visible light image is low, and the far-infrared light image indicates that the temperature of the object is low.
[0159]
By using the visible light image and the far-infrared light image captured under such various environmental conditions, the object recognition apparatus 1 of the present invention can correlate between the visible light image and the far-infrared light image. You can learn the relationship. As a result, for example, when the brightness of the object in the visible light image of the recognition image set is high and the far-infrared light image has a low temperature of the object (an event that cannot occur when the object is a human being) If this occurs, the possibility of misrecognizing the object as a human is low.
[0160]
In recognition systems that use only visible light images for learning and recognition processing, it is possible to recognize objects as humans by performing learning processing using visible light images taken under various environmental conditions. The range becomes wider. As a result, there is a high possibility that a non-human object is erroneously recognized as a human. The same applies to a recognition system that performs learning processing and recognition processing using only a far-infrared light image.
[0161]
According to the object recognition device 1 of the present invention, by performing learning processing using a learning image set photographed under various environmental conditions (for example, illumination conditions, temperature conditions), the various environmental conditions are obtained. It becomes possible to correctly recognize the object of the image set for recognition taken in step S2. Such features include objects that are subject to changing environmental conditions such as outdoor intruder monitoring systems, pedestrian detection systems mounted on mobile objects such as automobiles, and visual systems mounted on mobile robots. It is particularly suitable for applications that require correct recognition.
[0162]
Furthermore, the learning process and the recognition process of the present invention described above are not processes specialized for the attributes of the object. This is because the object recognition apparatus 1 of the present invention is used for purposes other than human recognition (for example, an application for recognizing an animal or a vehicle for recognizing a vehicle) without changing the learning process and the recognition process of the present invention. Is also applicable. As described above, the object recognition apparatus 1 according to the present invention can easily perform the initial setting when the environmental condition for recognition is changed and when the recognition target is changed.
[0163]
In the object recognition apparatus 1 of the present invention, a process for extracting a specific temperature region from the far-infrared image is not necessary. Therefore, it is not necessary to perform a calibration process for canceling the influence of the temperature of the optical system, circuit, element, etc. of the far-infrared camera that changes over time, and the configuration and maintenance of the object recognition apparatus 1 can be simplified. Benefits are gained.
[0164]
In the embodiment described above, the selection dimension is set to one set in step S97 (FIG. 12) of the learning process. However, a plurality of sets of selected dimensions (a set of selected dimensions) may be prepared, and an identification parameter may be set for each set. In this case, in the recognition processing, human recognition may be performed based on the similarity obtained using any one set of identification parameters, or may be obtained using each of the plurality of sets of identification parameters. Human recognition may be performed based on the sum (average value) of similarities. Alternatively, human recognition may be performed based on the degree of similarity obtained using each of the plurality of sets of identification parameters, and the majority of the recognition results may be determined.
[0165]
In the above-described embodiment, the feature amount vector is obtained using the Gabor filter in step S123 for the recognition target region cut out in step S122. However, a Gabor filter may be applied to an image before being cut out. In this case, a specific area where the filter is applied is set in advance for the entire image, and the Gabor filter is applied to obtain a filter output for each position in the entire image in advance. Next, the feature quantity vector is calculated using only the filter output of the place that is the detection target area in the image. In this way, by obtaining the filter output in advance, a plurality of filter operations that apply the same Gabor filter to the same location in the image, for example, when the procedure of clipping and recognition is sequentially repeated while scanning a wide range in the image, etc. It is possible to avoid waste of time. Note that a person can be detected from an image in which the object is uncertain by performing a process of sequentially repeating the cutout and recognition procedures while scanning a wide area in the image. When such processing is performed, the object recognition device 1 (FIG. 6) can function as an object detection device.
[0166]
An image set (a set of visible light image and far-infrared light image) representing a non-human object used in learning processing is an object recognition device by photographing a real non-human object such as a tree or a dog. 1 may be input. Alternatively, a set of a visible light image and a far-infrared light image respectively generated by performing conversion processing on a visible light image and a far-infrared light image of an image set that represents a human being is a target other than a human being. An image set representing an object may be used in the learning process. Examples of such conversion processing include processing for performing affine transformation on an image and / or processing for adding noise to an image. The image after the conversion process is an image that is relatively similar to an image representing a person. By using such an image after the conversion process in the learning process, it is possible to learn a determination criterion that does not recognize an object having a shape different from a human shape as a human being.
[0167]
In the above-described embodiment, the learning process and the recognition process are performed by combining two images of a visible light image and a far-infrared light image captured at substantially the same time. The number of images to be combined is not limited to two. Further, a color image may be used as the visible light image instead of the luminance image. In this case, if a color image is represented by three images of RGB (three types of images representing the intensity of light in three different wavelength bands that are emitted or reflected from an object), an R image, a G image, A total of four images of the B image and the far-infrared light image are input to the object recognition apparatus 1 as one image set (an image set for learning and an image set for recognition). The learning process and the recognition process when four images are input are the same as the learning process and the recognition process when two images of a visible light image and a far-infrared light image are input.
[0168]
The image set input to the object recognition apparatus 1 may include images taken at different times. For example, the input unit 190 includes a visible light image (first image) and a far-infrared light image (second image) taken at time T (first time), and a time T + t (predetermined from the first time). The four images of the visible light image (fifth image) and the far-infrared light image (sixth image) taken at the time after the first time) are given to the object recognition apparatus 1 as one image set for recognition. It may be configured to input. Of course, in this case, each of the learning image sets also has a visible light image and a far-infrared light image captured at the same time, and a visible light image and a far-infrared image captured t hours after that time. It goes without saying that it must contain a light image. The learning process and the recognition process when four images are input are the same as the learning process and the recognition process when two images of a visible light image and a far-infrared light image are input.
[0169]
In this way, by combining a visible light image and a far-infrared light image with different shooting times, an object whose shape changes in a specific manner with time, such as a pedestrian, and an object whose shape does not change with time In addition, it is possible to distinguish an object whose shape changes with time, and the recognition accuracy is improved.
[0170]
If the predetermined time t is shortened, an object with fast movement can be efficiently recognized, and if the predetermined time t is lengthened, an object with slow movement can be efficiently recognized. Normally, when recognizing a person, vehicle, animal, or the like with movement outdoors, by setting the predetermined time t to 1 second or less, an object whose shape and / or position changes with time can be effectively used. Can be recognized. In this way, the identification performance is improved by effectively combining information of images at a plurality of times.
[0171]
According to the object recognition apparatus 1 of the present invention, even when the number of images included in an image set increases, only the feature quantity dimension (filter output) that contributes to identification is selected. An increase in the calculation amount of the recognition process is suppressed.
[0172]
The image set input to the object recognition apparatus 1 may include images with different viewpoints. For example, a visible light image and a far-infrared light image captured from the same position, and a visible light image and a far-infrared light image captured from a different position may constitute one image set.
[0173]
FIG. 19 shows still another example of the arrangement of the far-infrared light camera and the visible light camera. In the example shown in FIG. 19, the far-infrared light camera 100a and the visible light camera 110a are arranged at the point A, and the far-infrared light camera 100b and the visible light camera 110b are arranged at the point B. Four cameras are arrange | positioned so that the same target object may be image | photographed. As a whole, the far-infrared light cameras 100a and 100b and the visible light cameras 110a and 100b have an input unit 190 (FIG. 6) for inputting a learning image set and a recognition image set to the object recognition apparatus 1 (FIG. 6). Function as.
[0174]
In the recognition process, in addition to the visible light image (first image) and the far-infrared light image (second image) captured from the point A (first place), the input unit 190 further includes B Object recognition apparatus using four images of a visible light image (fifth image) and a far-infrared light image (sixth image) taken from a point (second place) as one image set for recognition 1 is input. The learning process and the recognition process when four images are input are the same as the learning process and the recognition process when two images of a visible light image and a far-infrared light image are input.
[0175]
In this way, by combining a visible light image and a far-infrared light image taken from different places, the accuracy of recognizing an object having a different shape depending on the viewing direction such as a person is improved.
[0176]
According to the object recognition apparatus 1 of the present invention, even when the number of images included in an image set increases, only the feature quantity dimension (filter output) that contributes to identification is selected. An increase in the calculation amount of the recognition process is suppressed.
[0177]
Further, a visible light image and a far-infrared light image taken from different positions may constitute one image set.
[0178]
FIG. 20 shows still another example of the arrangement of the far-infrared light camera 100 and the visible light camera 110. In the example shown in FIG. 20, the visible light camera 110 is disposed at the point C, and the far infrared light camera 100 is disposed at the point D. Two cameras are arrange | positioned so that the same target object may be image | photographed.
[0179]
In the example shown in FIG. 20, the far-infrared light camera 100 and the visible light camera 110 capture a visible light image (first image) from a point C (first place), and a point D (second place). The far-infrared light image (second image) is taken from the (location). According to such a configuration, the background of the object can be made different between the visible light image and the far-infrared light image. The possibility that the extra information of the background common to the visible light image and the far-infrared light image adversely affects the recognition result is reduced, and an advantage is obtained that the recognition result is less affected by the background. Similarly to the example shown in FIG. 19, the use of images taken from a plurality of different viewpoints improves the recognition accuracy of an object having a different shape depending on the viewing direction, such as a person.
[0180]
In the above-described embodiment, a visible light image and a far-infrared light image are given as examples of an image that expresses an object using different attributes, but the present invention is not limited to this. An image set may be constituted by a visible light image and a near-infrared light image, or an image set may be constituted by a visible light image and an ultraviolet light image. Or you may make it comprise an image group with a visible light image and a distance image. The pixel value in the distance image indicates the distance from the shooting point to the object. It can be said that the distance image is an image that expresses the object using the attribute of the distance from the shooting point to the object.
[0181]
Since the learning process and the recognition process of the present invention described above are not processes specialized for the attributes of the object, the above-described present invention can be used even when a type of image other than a visible light image and a far infrared light image is used. There is no need to change the learning process and the recognition process.
[0182]
The learning process and the recognition process of the present invention are typically realized by software on a computer. However, the learning process and the recognition process of the present invention may be realized by hardware, or may be realized by a combination of software and hardware. Furthermore, it is not essential for the object recognition apparatus 1 (FIG. 6) to execute the learning process. This is because the object recognition apparatus 1 can execute the recognition process (the processes of steps S1002a to S1002c shown in FIG. 1) as long as the learning process result (identification parameter) is stored in the identification parameter storage unit 150. Because it can. Such an identification parameter may be a predetermined parameter. Or such an identification parameter may be calculated | required by performing a learning process using an apparatus different from the object recognition apparatus 1. FIG. By storing the identification parameter obtained as a result of such learning processing in the identification parameter storage unit 150 of the object recognition device 1, the object recognition device 1 can perform recognition processing using the identification parameter as a determination criterion. become.
[0183]
When the object recognition device 1 does not perform the learning process, the learning processing unit 130 and the storage device 120 may be omitted.
[0184]
A program (learning program, recognition program) that expresses part or all of one or both of the learning process and the recognition process of the present invention is, for example, a memory (not shown) in the learning processing unit 130 or a recognition processing unit. 140 may be stored in a memory (not shown) within 140. Alternatively, such a program can be recorded on any type of computer-readable recording medium such as a flexible disk, a CD-ROM, and a DVD-ROM. A learning program or a recognition program recorded on such a recording medium is loaded into a computer memory via a disk drive (not shown). Alternatively, the learning program or the recognition program (or part thereof) may be downloaded to a computer memory through a communication network (network) or broadcast. When the CPU built in the computer executes the learning program or the recognition program, the computer functions as an object recognition device.
[0185]
【The invention's effect】
According to the present invention, an image set inputted for recognition includes a first image that represents an object using the first attribute, and a second attribute that is different from the first attribute. And a second image expressed using. Since it is determined based on the first attribute and the second attribute whether or not the object belongs to a specific category, the reliability of recognition of the object is increased. Further, a feature vector in the feature space having as a component a filter output value obtained by applying a predetermined image filter to a predetermined position of the predetermined number of images is obtained, and the image set is , Represented by this feature vector. Since this process does not require the process of associating the first image area and the second image area, the initial setting for recognizing the object is easy, and the recognition result depends on the influence of environmental conditions. It is hard to receive.
[Brief description of the drawings]
FIG. 1 is a flowchart showing an overall processing procedure of an object recognition method of the present invention.
FIG. 2A is a diagram showing image sets 610 to 613 input in step S1001a (FIG. 1).
FIG. 2B is a view showing image sets 660 to 663 inputted in step S1001a (FIG. 1).
3 is a diagram illustrating an example in which an image filter 354 is applied to an image 351. FIG.
FIG. 4 is a diagram showing a state in which feature quantity vectors obtained for each image set in step 1001b (FIG. 1) are plotted in a feature quantity space 701;
FIG. 5 is a diagram showing an example of a recognition image set 510 input in step 1002a (FIG. 1).
FIG. 6 is a block diagram showing the configuration of the object recognition apparatus 1 according to the embodiment of the present invention.
7A is a diagram showing an example of the arrangement of the far-infrared light camera 100 and the visible light camera 110. FIG.
7B is a diagram showing another example of the arrangement of the far-infrared light camera 100 and the visible light camera 110. FIG.
FIG. 7C is a diagram showing an example in which a visible / far-infrared light camera 210 having both functions is used instead of the far-infrared light camera 100 and the visible light camera 110;
FIG. 8A is a diagram showing an example of a visible light image 803 representing a human target image taken by the visible light camera 110;
FIG. 8B is a diagram showing an example of a far-infrared light image 804 representing a human target image taken by the far-infrared light camera 100.
FIG. 9A is a diagram showing an example of a visible light image 805 representing a non-human object (tree) taken by the visible light camera 110;
9B is a diagram showing an example of a far-infrared light image 806 representing an object other than a human being photographed by the far-infrared light camera 100.
FIG. 10A is a diagram schematically showing characteristics of an image filter used in the filter processing unit 125 (FIG. 6).
FIG. 10B is a diagram schematically showing the characteristics of an image filter that selectively detects edges in the horizontal direction.
FIG. 10C is a diagram schematically illustrating characteristics of an image filter that selectively detects an edge extending from the lower left to the upper right.
FIG. 10D is a diagram schematically illustrating the characteristics of an image filter that selectively detects an edge extending from the lower right to the upper left.
FIG. 11A is a diagram showing an example of filter coefficients of a Gabor filter.
FIG. 11B is a diagram showing an example of the filter coefficient of the Gabor filter.
FIG. 11C is a diagram showing an example of the filter coefficient of the Gabor filter.
FIG. 11D is a diagram showing an example of the filter coefficient of the Gabor filter.
FIG. 12 is a flowchart showing a detailed procedure of learning processing executed by the object recognition apparatus 1;
FIG. 13A is a diagram for explaining an identification method using a curved identification surface;
FIG. 13B is a diagram for explaining an identification method using a distance in the feature amount space 1380;
FIG. 13C is a view for explaining an identification method using a distribution of feature quantity vectors in a feature quantity space 1382;
FIG. 14 is a diagram schematically illustrating a change in identification performance due to a feature amount dimension deletion process;
FIG. 15 is a flowchart showing a more detailed processing procedure of step S91 (FIG. 12).
FIG. 16 is a diagram showing a relationship between an image 431 and a region 432 to which an image filter in the image 431 is applied.
FIG. 17 is a flowchart showing a detailed procedure of processing for evaluating identification performance in step S95 (FIG. 12).
FIG. 18 is a flowchart showing a detailed procedure of recognition processing executed by the object recognition apparatus 1;
FIG. 19 is a diagram showing still another example of the arrangement of the far-infrared light camera and the visible light camera.
20 is a diagram showing still another example of the arrangement of the far-infrared light camera 100 and the visible light camera 110. FIG.
[Explanation of symbols]
1 Object recognition device
100 far-infrared light camera
110 Visible light camera
120 storage device
125 Filter processing unit
130 Learning processing unit
140 Recognition processing unit
150 Identification parameter storage unit
160 Work memory
170 Display
190 Input section

Claims (13)

First image including a first image data is image data of the visible light image of the first object and a second image data is image data of the far-infrared light image of the first object An input unit for inputting a data set;
The input unit inputs the first image data set , and at the predetermined position in the first image data and the second image data of the input first image data set. , orientation selectivity, regioselectivity, at least 1 respectively obtained from the first image data and the second image data by applying each at least one image filtering with at least one selective spatial frequency characteristic A feature quantity vector calculation unit for obtaining a first feature quantity vector in the feature quantity space, having two filter output values as components;
An object recognition apparatus comprising: a determination unit that determines whether the first object belongs to a specific category based on a relationship between the first feature vector and a predetermined identification parameter. The first image data represents the first object by the intensity of visible light emitted or reflected from the first object, and the second image data is the first object. The object recognition apparatus according to claim 1, wherein the first object is represented by an intensity of far-infrared light emitted or reflected from the object. Wherein the input unit, each input to the second image data sets and the third addition the feature quantity vector calculation unit image data set comprising a plurality of image data, the second image data sets and the third image Each of the data sets is third image data that is image data of a visible light image of a second object belonging to the specific category, and image data of a far-infrared light image of the second object. 4th image data ,
The feature amount vector calculation unit may determine at least one position in the third image data and the fourth image data that is determined in advance for each of the input second image data set and third image data set. And applying at least one image filter having the same selectivity as the image filter to at least one filter output value respectively obtained from the third image data and the fourth image data as a component, Further obtain a feature vector in the feature space,
And at least one feature vector in the feature space for the second image data set, to identify at least one feature vector in the feature space for the third image data sets, wherein The object recognition apparatus according to claim 1, further comprising a learning unit for obtaining an identification parameter. The object recognition apparatus according to claim 1, wherein the first object is a human. The learning section, in feature space provisional with number of dimensions larger than the feature space, a feature amount vector for the second image data set, and the feature vector for said third image data set The object recognition device according to claim 3 , wherein the feature amount space is defined by deleting at least one dimension from the temporary feature amount space based on a direction of a normal line of a plane for identifying the feature amount.   The identification parameter represents an identification plane in the feature quantity space, and the determination unit determines whether the first feature quantity vector is located on the side of the identification plane. The object recognition apparatus according to claim 1, wherein it is determined whether an object belongs to the specific category. The determination unit determines a first feature vector, if the distance between the identification surface is equal to or greater than a predetermined threshold value, and said first object belongs to the specific category, claim 6 The object recognition apparatus described in 1. Wherein the input unit, the first and the fifth image data which is the image data of the visible light image of the object, a sixth image of an image data of the far-infrared light image of the first object object the data further being adapted to input to the feature quantity vector calculation unit, the image data of the image data and the sixth of the fifth, the first image data and the second image data is captured It has been from the first time it was taken after a predetermined time, the object recognition device according to claim 1. The first image data is captured from a first location, said second image data is the first location that was taken from a different second location, object recognition according to claim 1 apparatus. Wherein the input unit includes a fifth image data which is the image data of the visible light image of the first object, a sixth image data is image data of the far-infrared light image of the first object and further adapted to be input to the feature vector calculating section, the image data of the image data and the sixth of the fifth, the first image data and the second image data is captured the first location is one that is taken from a different second location, object recognition apparatus according to claim 1. A method for recognizing an object,
(A) including a first image data is image data of the visible light image of the first object thereof, and a second image data is image data of the far-infrared light image of the first object Inputting a first image data set;
(B) At least one selectivity of azimuth selectivity, position selectivity, and spatial frequency characteristics is provided to at least one predetermined position of each of the input first image data and second image data. the first feature vector in which the first image data and said having a second at least one filter output values respectively obtained from the image data as a component, feature quantity space by applying at least one image filters each having A step of seeking
(C) determining whether or not the first object belongs to the specific category based on a relationship between the first feature vector and a predetermined identification parameter. . A program for causing a computer to execute object recognition processing,
The object recognition process includes
(A) including a first image data is image data of the visible light image of the first object thereof, and a second image data is image data of the far-infrared light image of the first object Inputting a first image data set;
(B) At least one selectivity of azimuth selectivity, position selectivity, and spatial frequency characteristics is provided to at least one predetermined position of each of the input first image data and second image data. the first feature vector in which the first image data and said having a second at least one filter output values respectively obtained from the image data as a component, feature quantity space by applying at least one image filters each having A step of seeking
(C) determining whether the first object belongs to the specific category based on a relationship between the first feature vector and a predetermined identification parameter. A computer-readable recording medium storing a program for causing a computer to execute object recognition processing,
The object recognition process includes
(A) including a first image data is image data of the visible light image of the first object thereof, and a second image data is image data of the far-infrared light image of the first object Inputting a first image data set;
(B) At least one selectivity of azimuth selectivity, position selectivity, and spatial frequency characteristics is provided to at least one predetermined position of each of the input first image data and second image data. the first feature vector in which the first image data and said having a second at least one filter output values respectively obtained from the image data as a component, feature quantity space by applying at least one image filters each having A step of seeking
(C) a step of determining whether or not the first object belongs to the specific category based on a relationship between the first feature vector and a predetermined identification parameter.

Images