JPH08161499A - Object recognition device - Google Patents

Abstract

PURPOSE: To provide prescribed information in response to an operation content to an object by detecting the operation of the object. CONSTITUTION: A color extract circuit 31 extracts a skin color area from an image picked up by a wide angle color video camera 28 and a CPU (II) 33 detects a moving object from the movement of the skin color area. Then a notice point measurement unit U1 controls a notice direction of the detected moving object, a narrow angle color video camera 22 picks up an image, a radiation thermometer 21 measures the temperature and the CPU (II) 33 discriminates whether or not the object is a person based on the image information and the temperature measurement information. When the object is a person the operation content of the person is discriminated by a tracking processing section and a message in response to the operation content is outputted from a display device 41 and a voice output device 42 of an information output section 4 when the operation indicates an access motion or a crossing motion.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an object recognition device for remotely detecting information on an object such as color, temperature, distance, etc., and recognizing a target object from the detected information.
In particular, the present invention relates to an object recognition device that provides predetermined information to an object according to the movement of the object.

[0002]

2. Description of the Related Art Heretofore, an object recognition apparatus has been proposed which captures image information and temperature information of an object using a video camera and a radiation thermometer and recognizes the object from the image information and the temperature information. .

For example, in Japanese Unexamined Patent Publication No. 5-297152, a color video camera is attached to a finder portion of a radiation thermometer, and light and infrared rays from an object are provided in front of the finder portion with a color video camera and a radiation thermometer. A color video camera detects information such as color, size, and shape, and a radiation thermometer detects temperature. Based on the information about color, size, shape, and temperature, for example, There has been proposed an object recognition device that recognizes whether or not a person is a person.

Further, conventionally, there has been known one which generates a voice message such as "Welcome" in conjunction with the opening / closing operation of an automatic door.

[0005]

DISCLOSURE OF THE INVENTION Problems to be Solved by the Invention
The object recognition device of Japanese Patent Publication No. 152 recognizes a target object from image information and temperature information, and monitors the target object based on the recognition result. Does not have the function of providing the information of the above to the object.

[0006] Therefore, for example, when a human being is monitored as a target object, when the human being approaching approaches or crosses, a message such as "Welcome" or "Wait a moment" is given to the human. Is convenient, but the above-described conventional object recognition device cannot provide such information.

Further, a conventional voice message is generated in conjunction with an automatic door is a voice message generated in response to the opening / closing operation of the door, and detects a target object (human). Therefore, when the automatic door is opened / closed due to an act by a person other than the human being, the message is often provided unnecessarily due to a malfunction.

The present invention has been made in view of the above problems, and provides an object recognition apparatus excellent in versatility and applicability capable of providing predetermined information according to the movement of an object as well as object recognition. The purpose is to provide.

[0009]

The present invention is directed to an image capturing means for capturing an image, an image extracting means for extracting an image of a target object from an image captured by the image capturing means, and an extracting means. An object recognition unit that recognizes an object from an image, a motion detection unit that detects a motion of the recognized object, and whether or not the detected motion of the object matches a predetermined motion pattern set in advance. When the movement of the object and the discrimination means for discriminating matches the predetermined movement pattern,
An information output unit that outputs predetermined information set in advance corresponding to this operation pattern is provided.

[0010]

According to the present invention, when an image of a subject within a predetermined range in the gaze direction is captured, an image of the target object is extracted from this image, and the target object is extracted from the extracted image. It is determined (recognized) whether or not it is an object to be processed. Then, when the extracted image is recognized as the image of the target object, the motion of the target object is further detected, and if the detection result matches the preset predetermined motion pattern, Predetermined information corresponding to the operation pattern is output to the object.

[0011]

DESCRIPTION OF THE PREFERRED EMBODIMENTS An object recognition apparatus according to the present invention detects a target object by detecting a movement of a region of a predetermined color from an image picked up by a wide-angle color video camera, and a narrow-angle color video camera. The target object is tracked and imaged, and the temperature of the target object is measured by a radiation thermometer. Then, the object is recognized from the image information and the temperature information of the narrow-angle color video camera, and when it is confirmed that the object is the target object, the operation pattern of the object is further detected, and this operation pattern is detected. If it is a predetermined motion pattern, the target object is provided with predetermined information according to a predetermined motion.

As a concrete application example, the entrance / exit of a customer is monitored at the entrance of a store or a venue, and the customers who are entering or leaving are detected, and "Welcome." It is assumed that a message such as "Thank you" is provided, or that a customer who is crossing is detected and a message that calls attention such as "Please wait a moment" is provided to these customers.

The application of the object recognition apparatus according to the present invention is not limited to the above, but in various cases such as monitoring the operation of a worker or a production machine at a production site and detecting an abnormal operation to help improve security. However, in the present embodiment, a case of providing a message to the customer will be described as an example.

FIG. 1 is a block diagram of an object recognition apparatus according to the present invention. The object recognition device 1 includes a sensor unit 2 that captures information about attributes such as color, size, shape, and temperature of an object, and information captured to perform drive control of a sensor that captures the above information and recognition of the object. The image processing section 3 performs various arithmetic processing and image processing by using the, and an information output section 4 that outputs predetermined information with respect to the detected target object.

The sensor unit section 2 is for taking in image information and temperature information of the object, is equipped with a radiation thermometer 21 as a temperature information taking-in device, and has two video cameras, ie, image taking-in devices. , A narrow-angle color video camera 22 and a wide-angle color video camera 28.

The wide-angle color video camera 28 searches for an object in a wide field of view, has a single-focus photographing lens with a horizontal angle of 60 ° or more, and has a wide angle of view GA1 as shown in FIG. Capture the image of. As will be described later, the wide-angle color video camera 28 is fixed to a base that can rotate in a horizontal direction (hereinafter, referred to as a pan direction), and the monitoring direction (direction of the optical axis L0) is changed by rotating the base. It is possible.

On the other hand, the narrow-angle color video camera 22 (image capturing means) gazes at some objects within the range monitored by the wide-angle color video camera 28. It has a zoom lens with a narrow horizontal angle of view. The narrow-angle color video camera 22 is shown in FIG.
As shown in, the image of an arbitrary object included in the angle of view GA1 of the wide-angle color video camera 28 is captured with the angle of view GA2 narrower than the angle of view GA1.

The narrow-angle color video camera 22 is equipped with, for example, an automatic focusing (AF) mechanism as shown in FIG. 3 so that it is automatically focused on the target object.

That is, while moving the taking lens 221 by the lens feeding device 225, the image pickup device 2 such as CCD.
The object is imaged at a predetermined cycle at 22, and the imaged image signal is amplified by the amplifier 223 and then input to the image signal level detector 224. Video signal level detector 2
At 24, the high frequency component of the video signal is extracted in the distance measuring area provided at the center of the screen, and the signal level thereof is output as a detection signal. Then, this detection signal is input to the lens position control circuit 228.

On the other hand, the lens position of the taking lens is detected by the lens position detector 226, and this detection signal is sent to the amplifier 2
After being amplified by 27, it is input to the lens position control circuit 228. The lens position control circuit 228 moves the photographing lens 221 in the front-rear direction via the lens feeding device 225 based on the lens position information, and sets the photographing lens 221 to the position (focus position) where the high frequency component is maximum. .

The wide-angle color video camera 28 also has an automatic focusing (AF) mechanism similar to that described above. Further, the taking lens of the wide-angle color video camera 28 may be composed of a zoom lens so that the angle of view can be adjusted.

The angle-of-view adjusting device 23 is for adjusting the angle of view of the photographing screen of the narrow-angle color video camera 22. The view angle adjusting device 23 adjusts the zoom position of the narrow-angle color video camera 22 so that the view angle GA2 becomes the instructed view angle based on the control signal from the CPU (I) 29.

The radiation thermometer 21 receives infrared rays emitted from an object and measures the surface temperature of the object from the infrared energy. The radiation thermometer 21 is adapted to measure the spot temperature at the intersection of the optical axis L2 and the object, and the temperature measurement point is the tilt direction in the photographing screen GA2 of the narrow-angle color video camera 22 as described later. It can be changed to. The temperature information measured by the radiation thermometer 21 is stored in the control circuit (CPU (I)) 2 of the sensor unit 2
9 (hereinafter, referred to as CPU (I) 29).

That is, the object recognition apparatus 1 captures an image of a wide range by the wide-angle color video camera 28, and an object to be watched from the wide-angle image within the monitoring range GA1.
For example, if it is determined whether or not there is a person's face and it is determined that there is an object to be watched, the narrow-angle image of the object is captured by the narrow-angle color video camera 22 and the temperature is measured by the radiation thermometer 21. Is measured, and the object is recognized based on the image information and the temperature information.

The temperature measuring direction moving device 24 moves the temperature measuring direction of the radiation thermometer 21 in the tilt direction within a predetermined range. The distance measuring circuit 25 measures the distance to the object which the narrow-angle color video camera 22 is gazing at. The distance measuring circuit 25 uses the focus position information detected by the AF mechanism of the narrow-angle color video camera 22 (amplifier 2 in FIG. 3).
The actual distance D (mm) from the narrow-angle color video camera 22 to the object is detected based on the output (27). The distance D to the object detected by the distance measuring circuit 25 is calculated by the CPU (I)
29 is input.

The radiation thermometer 21, the narrow-angle color video camera 22, the temperature measuring direction moving device 24, and the distance measuring circuit 25 are
It captures image information and temperature information about a target object, and these devices constitute the gazing point measuring unit U1.

The gaze direction moving device 26 is a device that moves the gaze direction of the gaze point measuring unit U1 (direction of the optical axis L1 of the narrow-angle color video camera 22). The gazing point measurement unit U1 is rotatable in the pan direction and the tilt direction, and the gazing direction is moved in an arbitrary direction by combining the pan direction movement and the tilt direction movement. The gaze direction moving device 26 changes the gaze direction of the gaze point measuring unit U1 based on the turning control value (the amount of each rotation in the pan direction and the tilt direction) input from the CPU (I) 29.

The pan / tilt position detector 27 is a reference direction (front direction) of the optical axis L1 of the narrow-angle color video camera 22.
Rotation angle α in the pan direction and rotation angle θ in the tilt direction with respect to
Is to detect the tilt amount in the gaze direction with respect to the reference direction. Rotation angle α in the detected pan direction
The rotation angle θ in the tilt direction is input to the CPU (I) 29.

The CPU (I) 29 is a control circuit for centrally controlling the operation of the sensor unit section 2. CPU (I) 29
Has a memory 291 for performing various arithmetic processes. The CPU (I) 29 analyzes the temperature information input from the radiation thermometer 21 and the distance information to the target object input from the distance measuring circuit 25, and image-processes the temperature information and the distance information via a serial interface. Output to the part 3. Further, in response to a gaze direction control command from the image processing unit 3, the gaze direction moving device 26 is driven and controlled, and the gaze point measuring unit U1 is controlled in the direction of the target object. Further, the zoom position of the narrow-angle color video camera 22 is adjusted (angle of view adjustment) via the angle-of-view adjusting device 26 to adjust the size of the target object in the photographing screen GA2 to a suitable size.

FIG. 4 is a partially cutaway perspective view showing an example of the structure of the measurement block. In the figure, the base 5 is
It can be rotated in a horizontal plane by a drive mechanism (not shown),
The gazing directions of the color video cameras 22 and 28 are changed by rotating the base 5. The wide-angle color video camera 28 is fixed to the base 5 and rotates integrally with the base 5. The gazing point measuring unit U1 and the gazing direction moving device 26 are provided in the box body 10 fixed to the upper part of the wide-angle color video camera 28. The box body 10 also rotates integrally with the rotation of the base 5.

FIG. 5 is a schematic view of the internal structure of the gazing point measuring unit U1. Box 11 of gazing point measurement unit U1
A narrow-angle color video camera 22 is fixed to the lower part of the box body 11 by exposing the taking lens 221 to the front surface of the box body 11. At the upper part of the narrow-angle color video camera 22, a light receiving window 21A is provided on the front surface of the box body 11. The radiation thermometer 21 is attached so as to be exposed and rotatable in the tilt direction.

The radiation thermometer 21 has a rotating shaft 211 projecting at appropriate positions on both side surfaces, and the tip of the rotating shaft 211 is provided in the box body 11.
It is rotatably supported by. Alternatively, the rotary shaft 211 may be rotatably provided on both side surfaces of the radiation thermometer 21, and the tip of the rotary shaft 211 may be fixed to the box body 11.

The central position of the back surface of the radiation thermometer 21 (optical axis L
A lever 212 is provided so as to project at a position where the two intersect, and an eccentric cam 242 is provided at a position below the tip of the lever 212.

Further, the front end portion of the upper surface of the radiation thermometer 21 is urged upward by a spring 213 suspended in a proper place on the upper surface of the box body 11. Therefore, the lever 212 has the spring 213.
The urging force of the eccentric cam 242 causes the eccentric cam 242 to rotate in a clockwise direction about the rotating shaft 211 and is pressed against the peripheral surface of the eccentric cam 242.

The eccentric cam 242 is fixed to a rotary shaft 241 which is rotatably supported by the box body 11 and rotates integrally with the rotary shaft 241. The rotating shaft 241 is provided at a position where the lever 212 becomes parallel to the optical axis L1 when the pressure contact between the lever 212 and the eccentric cam 242 reaches the lowest point. Therefore, the temperature measuring direction of the radiation thermometer 21 (optical axis L2) can be tilted downward relative to the horizontal direction.

The rotary shaft 241 is integrated with the output shaft of the geared motor 243, and the rotary shaft 241 and the eccentric cam 242 are rotated by the drive of the geared motor 243.
As a result, the temperature measurement direction of the radiation thermometer 21 changes. The temperature measurement direction of the radiation thermometer 21 is an inclination angle γ with respect to the horizontal axis of the lever 212 (axis parallel to the optical axis L1 passing through the rotation axis 211).
Is adjusted and controlled.

When the rotational position of the eccentric cam 242 where the inclination angle γ is 0 ° (the peripheral surface position Pr1 where the minimum radius is the highest point) is the rotation reference position, the inclination angle γ is eccentric. Since it changes according to the rotation angle η of the cam 242 from the rotation reference position, specifically, the rotation angle η is controlled to adjust the temperature measurement direction of the radiation thermometer 21.

FIG. 6 is a diagram showing the relationship between the rotation angle of the eccentric cam 242 and the temperature measuring direction. If the temperature measuring direction of the radiation thermometer 21 is inclined, the optical axis L1 and the optical axis L2 intersect, and the distance between this intersection and the narrow-angle color video camera 22 increases (or decreases) of the inclination angle γ. ) Decrease (or increase) accordingly.

In order to perform object recognition with high accuracy, it is necessary to capture image information and temperature information at the same point on the object. Therefore, the inclination angle γ is controlled so that the intersection point occurs on the object. To be done. In this case, the distance is the distance (subject distance) D from the narrow-angle color video camera 22 to the measurement object, and the inclination angle γ is the correction angle (parallax correction angle) of the radiation thermometer 21 in the temperature measurement direction. .

FIG. 6 shows the relationship between the rotation angle η and the distance D by taking the distance D to the object to be measured as a variable indicating the temperature measuring direction of the radiation thermometer 21.

In the figure, the rotation angles η = 0 ° and 360 °
Indicates that the peripheral surface position Pr1 that is the minimum radius of the eccentric cam 242 is at the highest point, and the radiation thermometer 21 is in the temperature measurement direction (optical axis L2).
Is a position where the inclination angle γ is 0 ° (parallel to the optical axis L1 of the narrow-angle color video camera 22). Further, in the same figure, the rotation direction of the eccentric cam 242 is positive in the counterclockwise direction in the figure, and the rotation angle η slightly exceeds 180 ° (the peripheral surface position Pr2 which is the maximum radius of the eccentric cam 242 is the highest). The inclination angle γ of the lever 212 becomes maximum at a position slightly beyond the point).

When the distance D to the object to be measured is calculated by the distance measuring circuit 25, the eccentric cam 24 is calculated from this distance D and FIG.
Since the rotation angle ηto of 2 (the target rotation angle of control) is determined, the radiation thermometer 2 is driven by rotating the eccentric cam 242 from the current rotation angle ηt to the target rotation angle ηto.
Adjustment of the temperature measurement direction of 1 is performed.

Returning to FIG. 4, the gazing point measuring unit U1 is
It is attached so as to be rotatable inside the box 10. The turning mechanism of the gazing point measuring unit U1 is the gazing point measuring unit U1.
Hollow rectangular frame 12 for moving the gaze direction in the pan direction, a U-shaped guide member 13 curved in a semicircle for moving the gaze direction in the tilt direction, a drive mechanism 14 for the frame 12, and the guide. It is composed of a drive mechanism 15 for the member 13.

As shown in FIG. 7, the XYZ Cartesian coordinate system having the optical axis L1 of the narrow-angle color video camera 22 as the X-axis is set in the gazing point measuring unit U1. And a gazing point measuring unit U1 is fitted in the hollow portion of the frame body 12, and the tip end portion of the turning shaft 111 is a frame. It is fixed to the side surface of the body 12 and is rotatably supported by the frame body 12. It should be noted that the rotating shaft 111 may be fixed to the box body 11, and the tip end portion of the rotating shaft 111 may be rotatably supported by the frame body 12. The Y-axis and the Z-axis are parallel to the horizontal side and the vertical side of the rectangular image sensor 222 built in the narrow-angle color video camera 22, respectively.

The optical axis L1 of the narrow-angle color video camera 22 is orthogonal to the X-axis, Y-axis, and Z-axis when the optical axis L1 faces the front direction (parallel to the optical axis L0 of the wide-angle color video camera 28). If the coordinate system is the UVW coordinate system (see FIG. 7), the frame body 12 has a rotating shaft 141 in which pulleys 142 and 143 are fixed at central positions of the upper and lower side surfaces (positions where the W axis of the UVW coordinate intersects). The rotation shaft 141 is rotatably supported by the top plate and the bottom plate of the box body 10 and is rotatable in the pan direction. As a result, the frame body 12 is rotated around the W axis to fix the gazing point measuring unit U.
1 moves in the pan direction.

Further, the U-shaped guide member 13 is provided outside the frame body 12. Guide member 13
Is provided so that a curved guide portion 131 is formed on the rear side of the frame body 12. The guide member 13 is provided with a rotating shaft 151, to which pulleys 152 and 153 are fixedly attached, on the outside of both ends of the guide member 13. The rotating shaft 151 serves as the left and right side plates of the box body 10 in the UVW coordinate system. Are mounted in the box body 10 so as to be rotatably supported at positions where the V axes intersect.

The curved guide portion 131 of the guide member 13.
A guide groove 132 is formed along the longitudinal direction (see FIG. 7).
The rod-shaped engaging member 16 is slidably engaged with the guide groove 132 at the position where the optical axis L1 (X axis) intersects with the rear side surface of the box body 11. There is. That is, as shown in FIG. 9, a cylindrical roller 162 rotatably provided at the tip of the engaging member 16 penetrates the guide groove 132 of the guide member 13.

As a result, when the guide member 13 is rotated around the V axis, the engagement member 16 is rotated, and the gazing point measuring unit U1 is rotated around the Y axis to rotate the radiation thermometer 21 and the narrow angle. The gazing direction of the color video camera 22 moves in the tilt direction.

A shaft 144 is rotatably attached to the rear of the frame body 12 in the box body 10 in parallel with the rotary shaft 141 of the pulleys 142 and 143. This axis 14
4, the pulleys 145 and 146 are fixed to the positions parallel to the pulleys 142 and 143 at both ends, and the pulley 145
The gear 147 is fixed at a position near the position. Pulley 14
2 and the pulley 145, and between the pulley 143 and the pulley 146, timing belts TB1 and TB2 are installed, respectively, and the gear 147 includes the drive gear 1 of the geared motor 149.
48 is meshed. Also, the timing belt TB1
A pan rotation angle detector 17 for detecting the rotation angle of the frame body 12 in the pan direction is meshed with.

A shaft 154 is rotatably mounted in parallel with the rotary shaft 151 of the pulleys 152, 153, and pulleys 155, 156 are provided at both ends of the shaft 154 in parallel with the pulleys 152, 153. The gear 157 is fixed and further fixed to a position near the pulley 156. Timing belts TB3 and TB4 are provided between the pulleys 152 and 155 and between the pulleys 153 and 156, respectively.
And a drive gear 158 of a geared motor 159 is meshed with the gear 157. A tilt rotation angle detector 18 that detects the rotation angle of the frame body 12 in the tilt direction is meshed with the timing belt TB3.

The above pulleys 142, 143, 145, 14
6, the shaft 144, the gear 147, the geared motor 149, and the pan rotation angle detector 17 constitute a drive mechanism of the frame body 12, that is, a pan-direction drive mechanism of the gazing point measurement unit U1, and the pulleys 152, 153, and The reference numerals 155, 156, the shaft 154, the gear 157, the geared motor 159, and the tilt rotation angle detector 18 constitute a drive mechanism of the guide member 13, that is, a drive mechanism in the tilt direction of the gazing point measurement unit U1.

The pan rotation angle detector 17 and the tilt rotation angle detector 18 detect the movement amounts of the timing belt TB1 and the timing belt TB3, respectively, and correspond to the pan / tilt position detector 27. The detection data of the pan rotation angle detector 17 and the tilt rotation angle detector 18 are
It is output to the CPU (I) 29 by a control cable (not shown). The CPU (I) 29 calculates the rotation amount of the pulley 142 or the pulley 153 using the above detection data, and further, based on the calculation result, the front direction of the optical axis L1 (optical axis L2 of the wide-angle camera).
The current rotation angle α in the pan direction and the rotation angle θ in the tilt direction with respect to the (direction parallel to) are detected.

Then, the control of the gaze direction of the gaze point measuring unit U1 (the movement control of the optical axis L1 in the pan direction or the tilt direction) is performed based on the rotation angle α (or the rotation angle θ). This is performed by controlling the driving of the motor 159).

Next, in the above configuration, the locus of the optical axis L1 when the gazing point measuring unit U1 is tilted in the tilt direction (angle θ ≠ 0) and moved in the pan direction is shown in FIGS. This will be described using 8.

As is apparent from FIG. 7, when the frame body 12 is rotated around the W axis, the gazing point measuring unit U1 also rotates around the W axis and is projected from the rear side surface of the gazing point measuring unit U1. The engaging member 16 also rotates around the W axis. At this time, the engagement member 16 has the guide groove 13 of the guide member 13.
The optical axis L1 coaxial with the engaging member 16 always moves in a plane including the guide groove 132 and the V axis because the optical axis L1 is engaged with the guide groove 132 and slides along the guide groove 132.

When the angle θ = 0 °, the guide groove 132
And the plane including the V axis is horizontal, the optical axis L1 is UV.
Although it moves in the plane, when the angle θ ≠ 0 °, the plane including the guide groove 132 and the V axis is inclined by the angle θ with respect to the horizontal plane, so that the optical axis L1 is in the plane inclined by the angle θ from the horizontal plane. To move.

In FIG. 8, S1 is a UV plane consisting of a U axis and a V axis in UVW coordinates. Also, S2 is
It is a vertical surface that is virtually provided in front of the narrow-angle color video camera 22 in the gazing direction and that is composed of a V axis and a W axis that are orthogonal to the horizontal plane S1. The x-axis and the y-axis are orthogonal coordinates provided on the vertical plane S2, the x-axis is a straight line where the horizontal plane S1 and the vertical plane 2 intersect, and the y-axis is the intersection O'of the U-axis and the vertical plane S2. It is a straight line perpendicular to the passing x-axis. Therefore, the vertical surface S2 is parallel to the VW plane composed of the V axis and the W axis, and the y axis is parallel to the W axis.

The point S is a narrow-angle color video camera 2
2 is an intersection of the optical axis L1 and the vertical surface S2 when the optical axis L1 is tilted upward from the position in the front direction (direction in which the optical axis L1 coincides with the U axis) by an elevation angle θ. That is,
An intersection point between the optical axis L1 and the vertical surface S2 when the guide member 13 is rotated clockwise by an angle θ in the drawing from the initial position (θ = α = 0 °) of the turning control around the V axis. Is.

At point R, the optical axis L1 of the narrow-angle color video camera 22 is tilted upward from the position in the front direction with respect to the U axis by an elevation angle θ, and is further counterclockwise with respect to the U axis by an angle α. The optical axis L1 and the vertical plane S2 when the pan is moved
Is the intersection with That is, the guide member 1 from the initial position
This is the intersection of the optical axis L1 and the vertical plane S2 when the frame 12 is rotated counterclockwise about the W axis by an angle α after rotating 3 by the angle θ.

When the frame 12 is rotated around the W axis while the guide member 13 is kept at the rotation angle θ, the locus of the intersection R between the optical axis L1 and the vertical plane S2 is the optical axis L1. Guide groove 1
Since it lies in a plane including 32 and the V axis, it is represented by an intersection line H between this plane, that is, the plane S3 including the origin O of the UVW coordinates and the points S and R and the vertical plane S2. On the other hand, the plane S
Since 3 is inclined upward by an angle θ with respect to the horizontal plane S1, it can be seen that the intersection line x axis between the horizontal plane S1 and the vertical plane S2 is parallel to the above intersection line H, and the point R moves horizontally. .

Although the semicircular guide member 13 is used in the present embodiment, the shape of the guide member 13 is not limited to the semicircular shape, and may be a semielliptical curved shape or a U-shape. It may be bent. Further, in the present embodiment, the cylindrical roller 162 rotatably provided at the tip of the engaging member 16 is passed through the guide groove 132, but the guide member 13 and the engaging member 16 are engaged with each other. The structure is not limited to this structure, and may be the structure shown in FIGS. 10 and 11, for example.

In the second embodiment of the engaging structure shown in FIG. 10, the guide member 13 is constituted by a bar member which is curved in a semicircle centered on the origin O of the UVW coordinates in FIG.
A roller holder 164 rotatable around the central axis of the shaft 161 is provided at the tip of the engaging member 16, and the roller holder 16 is further provided.
4 is provided with a roller 163 rotatably held around an axis orthogonal to the central axis of the shaft 161, and the roller 163 is brought into contact with the curved guide member 13 from the inside. When the frame body 12 is rotated in the pan direction, the roller 163 of the engaging member 16 is rotationally moved on the guide member 13, and the optical axis L1 of the narrow-angle color video camera 22 is aligned with the shaft 161 and the guide member 13.
Move in the pan direction in the plane including and.

The third embodiment of the engagement structure shown in FIG. 11 is a modification of FIG. 10, and the guide member 13 has the same shape as that of FIG. In the structure shown in FIG. 11, a roller holder 168 rotatable around the central axis of the shaft 161 is provided at the tip of the engaging member 16, and the roller 168 is provided with a shaft 165a parallel to the central axis of the shaft 161. A roller 165 rotatably held around and a roller 166 rotatably held by a rotary shaft 166a parallel to these two shafts 161 and 165a and movable in a plane formed by these two shafts 161 and 165a. Is attached so as to hold the guide member 13.

Rotating shaft 165a of roller 165 and roller 166
A spring 167 is provided between the rotating shafts 166 a of the rollers 165, and the rollers 165 and 166 are pressed against the guide member 13 by the urging force of the spring 167. When the frame body 12 is rotated in the pan direction, the rollers 165 and 166 of the engagement member 16 rotate and move on the guide member 13, and the optical axis L1 of the narrow-angle color video camera 22 becomes the axis 161 and the guide member 13. Move in the pan direction within the plane containing.

Further, in this embodiment, in order to widen the field of view of the gazing point measuring unit U1, the geared motor 149, which is a driving source in the pan direction and the tilt direction, is designed to reduce the width dimension of the box body 10 as much as possible. 159 is the gazing point measurement unit U1
The geared motor 149
And the geared motor 159, the shaft 141 and the shaft 15 respectively.
Alternatively, the frame 12 and the guide member 13 may be directly connected to each other and directly driven. Further, a stepping motor may be used as a drive source instead of the geared motor. When the stepping motor is used, the pan rotation angle detector 17 and the tilt rotation angle detector 18 are unnecessary, so that the configuration can be simplified.

Returning to FIG. 1, the image processing section 3 determines a predetermined color, for example, a preset color from the image pickup signals of the narrow-angle color video camera 22 and the wide-angle color video camera 28 which are input from the sensor unit section 2. A color extraction circuit 31 (image extraction means) for extracting a skin color, an image memory 32 for storing the image pickup signal, and a control circuit (CPU (II)) 33 (object recognition means for centrally controlling the operations of the image processing unit 3).
It is composed of motion detecting means and discriminating means).

The color extraction circuit 31 includes a color image processing circuit 311 and a region extraction circuit 313 which perform color extraction processing on an image pickup signal (hereinafter referred to as a wide angle image signal) picked up by the wide angle color video camera 28, and a narrow angle. Color video camera 22
A color image processing circuit 312 and a region extraction circuit 314 that perform color extraction processing on an image pickup signal imaged in (hereinafter referred to as a narrow-angle image signal).

The color image processing circuits 311 and 312 convert and form an image signal of hue (H), lightness (L) and saturation (S) from an RGB image signal and a smoothing circuit (not shown) for removing noise. A color signal conversion circuit (not shown), and the color image processing circuit 311 converts the wide-angle image signal (RGB image signal) to LS.
The color image processing circuit 313 converts the narrow-angle image signal (RGB image signal) into an LSH image signal.
Each of the converted LSH image signals has a region extraction circuit 31.
3 and the area extracting circuit 314.

The area extracting circuits 313 and 314 are for extracting an area having the same color as a preset color (for example, skin color) existing in the photographing screen using the HLS image signal. The area extracting circuit 313 is The skin color area in the wide-angle screen GA1 is extracted, and the area extraction circuit 314 extracts the skin color area in the narrow-angle screen GA2. Each screen GA extracted
The data of the skin color area in 1 and GA2 is output to the CPU (II) 33.

The image memory 32 stores images taken by the narrow-angle color video camera 22 and the wide-angle color video camera 28 under the control of the CPU (II) 33.

The CPU (II) 33 has a memory 331 and a timer 332 for performing various arithmetic processing. CP
The U (II) 33 calculates the barycentric position for each corner skin color region extracted by the wide-angle image signal, and further, based on the calculation result and the distance to the object input from the sensor unit 2, the gazing point measuring unit U1. Of the slewing control value is calculated, and the object corresponding to each skin color area is detected by the narrow-angle color video camera 22.
In order to make the operator pay attention to
To the CPU (I) 29.

This is because when a plurality of skin color regions are extracted in the wide-angle image shooting screen, the face determination processing is sequentially performed on each skin color region until it is determined that one of the skin color regions is a human face. , To execute. Therefore, CPU (I
If it is not determined by I) 33 that the current skin color area is the face of a person, the gaze directions of the narrow-angle color video camera 22 and the radiation thermometer 21 are changed to another skin color area. Then, the extraction of the skin color area, the temperature measurement operation, and the face determination process are executed until the face is determined to be a person,
This operation is repeated.

Further, the CPU (II) 33 uses the timer 332 to calculate the movement amount of the barycentric position of each skin color area extracted by the wide-angle image signal at a predetermined cycle (for example, a cycle in which an image for one frame is captured). The moving skin color area, that is, the moving person in the wide-angle screen GA1 is detected based on the calculated moving amount. The detection result is output to the sensor unit section 2 so that the person corresponding to the skin color area can be tracked by the narrow-angle color video camera 22.

Further, the CPU (II) 33 calculates the barycentric position, size, shape and the like of the skin color area extracted by the narrow-angle image signal, and outputs these calculation results and the sensor unit 2
It is determined whether or not the extraction area is a person's face based on the temperature information and the distance information to the target object input from.
When it is determined that the face is a person's face, the sensor unit is arranged so that the extraction region is located at the center of the shooting screen GA2 of the narrow-angle color video camera 22 (so that the face of the person is located at the screen center). The control value of the gaze direction of the unit 2 is calculated, and the calculation result is output to the sensor unit unit 2.

Further, the CPU (II) 33, in the tracking process of the object determined to be a human, turns information (pan rotation angle α and tilt rotation angle θ of the gazing point measuring unit U1 input from the sensor unit section 2). Information) and information on the distance D to the measurement object at a predetermined cycle, and whether or not the person who is gazing is performing a predetermined predetermined motion from the change pattern of the turning information and the distance information. judge.

If the person who is gazing is performing a predetermined action, the information output unit 4 outputs a start command together with the action content of the person, and outputs a predetermined message according to the action content to the person. Let For example, if a person is approaching the object recognition apparatus 1 side, a message "Welcome." Is output, and if a person is crossing the object recognition apparatus 1, a message "Please wait." Is output. .

The information output unit 4 (information output means) outputs a predetermined message corresponding to the motion of a target object performing a predetermined motion, and includes a display device 41, a voice output device 42, and a control unit. It is composed of a circuit (CPU (III)) 43.

The display device 41 is a CRT (Cathode Ray Tu).
be), an LCD (Liquid Crystal Display), and the like, and a display driver to visually display the above message. The voice output device 42 has a speaker and an amplifier, and outputs the above message audibly.

The CPU (III) 43 centrally controls the operation of the information output section 4 and has a memory 431 for performing the above-mentioned display or voice output processing. This memory 43
In 1, a message such as “Welcome to you” or “Wait a moment” is stored in advance corresponding to a predetermined operation.

Next, the operation of the object recognition device 1 according to the present invention will be described. FIG. 12 is a main flowchart showing the operation of the object recognition device 1. In the flow shown in the figure, a target moving object (here, a human face) is detected from the monitoring target, and when a predetermined pattern operation is performed while tracking the target object, a predetermined operation corresponding to the operation is performed. The message is provided.

Regarding the main processing, a diagram showing the relationship between the image after each processing and the temperature measurement position is attached. That is, V0 is the narrow-angle color video camera 22 after adjusting the angle of view.
An example of an RGB image taken in from is shown, in which the face of the person G1 is placed in the center of the shooting screen, and the temperature measurement position Pc (indicated by a dotted circle in the figure) is placed above the head of the person G1. ing. V1 indicates an RGB image in which the temperature measurement direction is adjusted while the gaze direction of the narrow-angle color video camera 22 is maintained, and the temperature measurement position Pc is adjusted in the center of the person's face.

When power is supplied to the object recognizing device 1, the temperature measuring operation of the radiation thermometer 21 and the photographing operation of the narrow-angle color video camera 22 and the wide-angle color video camera 28 are started. First, the flowchart shown in FIG. The skin color area is extracted from the image captured by the wide-angle color video camera 28 according to (# 2).

That is, the RGB image V2 is captured by the wide-angle color video camera 28 (# 40). Subsequently, after the image signal is smoothed by the color image processing circuit 311 so as to remove noise in the image signal (# 42), the color signal conversion process (R
GB-LSH conversion) is performed (# 44), and an image signal indicating three attributes of color of lightness (L), saturation (S), and hue (H) is generated.

Subsequently, the area extraction circuit 313 extracts the area having the same color as the skin color from the obtained image signal indicating the three attributes of the color (# 46). The extraction of the skin color region is performed based on the skin color condition set in advance. That is,
As the skin color condition in this case, for example, h0 <H <h1,
A threshold value of s1 <S <s2 is set, and with such threshold value,
Binarization processing is performed on the hue signal (H) and the saturation signal (S).

In this embodiment, the flesh color area is judged by the saturation (S) and the hue (H).
The determination may be made based on the R output / G output and the B output / G output.

Next, the product of the two images after the binarization processing is taken, and the area of the same color as the skin color is extracted (# 4
6) Return. For example, it is assumed that the environment image shown by the RGB image V2 has been obtained. In this environmental image,
In addition to the figures G1 and G2, a desk and a pear G placed on this desk
4. Various objects such as the poster G3 stuck on the wall are displayed. By performing the area extraction processing, the human faces H1 and H2 such as the skin color area extraction image V3 are displayed.
Only the face H3 of the person and the pear H4 close to the skin color in the poster are extracted.

Returning to FIG. 12, next, an object which seems to be a person is detected from the movement of the skin color area extracted in the wide-angle image.

FIG. 14 is a flowchart showing the process of detecting a moving person. First, a label for area identification is attached to each of the extracted skin color areas H1 to H4 (# 5
0). Then, the barycentric position representing the area position is calculated for each label area (# 52). The calculation result is stored in the memory 331 in the CPU (II) 33.

Here, a method of detecting the area position will be described. FIG. 15 is a flowchart showing the procedure of the area position detection processing, and FIG. 15 shows the processing of two types of area position detection methods.

FIG. 10A shows a method (hereinafter referred to as the first method) of filling the skin color area with one color and calculating the center of gravity of this area as position information of the area, and FIG.
The figure shows a method (hereinafter, referred to as a second method) of calculating a portion having the highest flesh color density from the entire flesh color image as position information of the area. Further, FIG. 7C shows a processed image corresponding to each procedure of the both area position detection method, using the portion of the person's face H1 as an example.

First, FIG. 15A shows the first method.
Will be explained. Since the flesh color region extraction image V4 is usually an image containing noise, isolated points of the flesh color are also present around the extracted flesh color region (image V4). First, an isolated point removal process is performed on the flesh color area extraction image V4 (# 60) to remove the isolated points around the flesh color area (image V5).

Subsequently, the flesh color areas H1 to H4 are subjected to a painting process (# 62), and the flesh color areas H1 to H4 are extracted as an integrated area (image V6). Next, labeling is performed to identify each extracted skin color area (#
64). Then, the skin color area is selected according to the order of the label, and the center of gravity P of the area is calculated for each skin color area using a known center of gravity calculation method (# 66, V7). The coordinates of the calculated center of gravity P are used as the detection area position of the skin color area.

Next, the second method will be described with reference to FIG. First, the skin color region extraction image V4 is equally divided into blocks of a required size (# 70, V8).
Next, the density of the skin color area in each block area is obtained (# 72, V9). Further, the block having the maximum density (the block having the density 53 in the image V9) is extracted from the obtained densities (# 74, V10). Then, the moment calculation is executed using the density of this block and the density of blocks around the extracted block, and the position coordinates P of the skin color region are obtained (# 76, V11). According to this method, the isolated point removal processing and the filling processing (# 6
Since it is not necessary to perform 0, # 62), the processing can be simplified.

Returning to FIG. 14, when the barycentric position of each label area is calculated, the movement amount δ of each label area is subsequently calculated (# 54). The movement amount δ of each label area is such that, when the coordinates of the center of gravity position P are P (m, n), the previously calculated center of gravity position P (m1, n1) and the currently calculated center of gravity position P.
From (m2, n2) and δ = √ ((m2-m1) ² + (n2
-N1) ² ) is calculated.

Subsequently, the calculated movement amount δ is compared with a preset reference value δo to determine whether or not there is a skin color region (person) estimated to be moving (# 5).
6). That is, it is determined whether or not there is a skin color area that satisfies the movement amount δ ≧ the reference value δo. If the movement amount δ <the reference value δo (NO in # 56), the moving person If not, go back to # 2. On the other hand, if any of the skin color areas has a movement amount δ ≧ reference value δo (# 5
6 is YES), the object corresponding to the skin color area is determined to be a moving person, and the CPU performs the gazing and temperature measurement processing on the object by the gazing point temperature measurement unit U1. (I) Output a gazing / temperature measurement command to 29 (# 5
8) Return.

The color extraction processing and the moving person detection processing in the wide-angle image are performed on the picked-up image each time an image for one frame is captured (every 1/30 seconds). When a moving person is detected in the captured image in the frame, as will be described later,
The narrow-angle color video camera 22 performs tracking processing of the person and temperature measurement processing by the radiation thermometer 21.

Returning to FIG. 12, when a moving object is detected by the color extraction processing of the wide-angle image, the gaze direction of the narrow-angle color video camera 22 is subsequently adjusted to the direction of the detected person (# 6). The CPU (II) 33 calculates a turning control value (target pan rotation angle αo and tilt rotation angle θo) of the gazing point measurement unit U1 from the center of gravity position P of the moving skin color area, and the calculation result is described above. It is output to the CPU (I) 29 together with the gaze / temperature measurement command. The CPU (I) 29 turns the gazing point measuring unit U1 based on the input turning control value and sets the gaze direction of the narrow-angle color video camera 22 to the direction of the detected person.

Subsequently, color extraction processing is performed on the narrow-angle image captured by the narrow-angle color video camera (# 8).
FIG. 16 is a flowchart showing the color extraction processing of a narrow-angle image. The color extraction processing of the narrow-angle image is performed in the same procedure as the color extraction processing of the wide-angle image, and in each step of # 80 to # 86, the image captured by the narrow-angle color video camera 22 is represented by # in FIG. The same processing as 40 to # 46 is performed.

For example, in the RGB image V2, the person G1
Is detected as a moving person, first, the image of the person G1 is read by aligning the gaze direction of the narrow-angle color video camera 22 with the direction of the center position P of the skin color area H1 (# 80, RGB image V12. ), Color image processing circuit 3
After the smoothing process and the color signal conversion process are performed in step 12 (#
82, # 84), the area extracting circuit 31 from the image V12.
A skin color area H5 is extracted in step 4 (# 86, extracted image V1).
3). Then, regarding the extracted skin color region H5,
The center of gravity position P is detected by the area position extraction and detection method shown in FIG. 15 (# 88).

Subsequently, returning to FIG. 12, it is judged whether or not the detected barycentric position P of the skin color region H5 is located at the center of the photographing screen GA2 (# 10), and the barycentric position of the skin color region H5 is determined. When is not located at the center of the shooting screen GA2, the narrow-angle color video camera 22 is panned according to the flowchart shown in FIG. 17 so that the gaze direction of the narrow-angle color video camera 22 is the barycentric position P of the skin color area H5. The gaze direction is adjusted by rotating in the tilt direction or the tilt direction (# 12).

That is, first, the rotation angle αo 'in the pan direction (hereinafter referred to as the target pan rotation angle) and the rotation angle θo in the tilt direction in which the gaze direction of the narrow-angle color video camera 22 matches the barycenter position P of the skin color area H5. ′ (Hereinafter referred to as the target tilt rotation angle) is calculated (# 90). This operation is
The narrow-angle image is divided into several regions, and a pan / tilt relative angle required to match the injection direction in each region is used for setting a table. A table different from this table is prepared for wide-angle images, and the process of # 2 is performed using this table.

Then, the pan / tilt position detector 27 is used to rotate the current rotation angle α of the narrow-angle color video camera 22 in the pan direction (hereinafter referred to as the current pan rotation angle α) and the rotation angle θ of the tilt direction (current tilt rotation). Angle θ) is detected (# 92), and the current pan rotation angle α and the current tilt rotation angle θ are respectively compared with the target pan rotation angle αo ′ and the target tilt rotation angle θo ′ (# 94, # 98). , # 100).

If the current pan rotation angle α does not match the target pan rotation angle αo '(NO in # 94), the narrow-angle color video camera 22 moves the target pan rotation angle α in the pan direction.
It is rotated to o '(# 96), and if the tilt rotation angle θ does not currently match the target tilt rotation angle θo'(# 98,
(NO in # 100), the narrow-angle color video camera 22 is rotated in the tilt direction to the target tilt rotation angle θo ′ (# 10).
2) If the current pan rotation angle α and the current tilt rotation angle θ match the target pan rotation angle αo ′ and the target tilt rotation angle θo ′, respectively (YES in # 94, # 98, and # 100), the gaze direction adjustment process Ends.

Then, returning to # 8 and # 10, it is determined again whether the barycentric position P of the skin color area H5 is located at the center of the photographing screen GA2.

The barycentric position P of the skin color area H5 is the extracted image V1.
3 or when the center of gravity of the skin color area H5 is located in the center of the extracted image V13 after the gaze direction of the narrow-angle color video camera 22 is corrected (YES in # 10), the measurement is performed. The object distance D from the narrow-angle color video camera 22 to the person G1 is measured by the distance circuit 25 (#
14). Although the subject distance is calculated from the AF information in the present embodiment, the subject distance is estimated from the size of the skin color region in the extracted image V13 when such a mechanism is not provided or as a simple method. Good.

Subsequently, the size and shape of the extracted skin color area H5 on the screen are calculated (# 16), and the actual size of the skin color area H5 is calculated from the calculation result and the object distance D related to this size. It is estimated and finally it is determined whether or not it is a human face (# 18). If it is determined that the extracted skin color region H5 is not a human face (NO in # 18), the process returns to # 2, and if it is determined that it is a human face (YE in # 18).
S), the angle of view of the narrow-angle color video camera 22 is adjusted (# 20). This angle of view adjustment is performed so that the size of the narrow-angle extracted image V13 of the object having the flesh-colored area determined to be a human face becomes appropriate.

Then, the radiation thermometer 21 is adjusted in the temperature measuring direction and the temperature is measured according to the flow chart shown in FIG. 18 (# 22, # 24). That is, the eccentric cam 242 described above is obtained from the information on the subject distance D detected by the distance measuring circuit 25.
The target rotation angle ηto (hereinafter, target cam position η
is calculated (# 110). The target cam position ηto is the distance D to the formulation target and the eccentric cam 242.
6 has a relationship as shown in FIG. 6, and is calculated using a conversion formula or a table in which the target cam position is set for the subject distance D in advance.

Next, the pan / tilt position detector 27 detects the current cam position ηt of the eccentric cam 242 (hereinafter referred to as the current cam position ηt) (# 112), and further the current cam position ηt is set as the target cam position ηt. It is determined whether or not it matches ηto (# 114). If the current cam position ηt does not match the target cam position ηto (N in # 114,
O), the eccentric cam 242 is rotated to the target cam position ηto (# 116).

Then, if the current cam position ηt matches the target cam position ηto, or if it matches by the above adjustment (YES in # 114), the radiation thermometer 21 is operated and the narrow-angle color video camera 22 gazes. The temperature of the flesh-colored area of the target object is measured (# 118).

Returning to FIG. 12, when the adjustment of the temperature measurement direction and the temperature measurement process are completed, it is subsequently determined from the measurement result whether or not the skin color region is a human (# 26).

In this judgment, the temperature of human skin is about 30 ° C.
Since it is (t ₀ ) to 34 ° C. (t ₁ ),
Measured temperature T is performed by whether or not entered t ₀ ~t _1, if t ₀ <T <t _1, it is determined that the human-like. That is, the measurement target is the person image G drawn on the poster.
The skin color areas H3 and H4 such as 3 and pear G4 are not judged to be human since they are outside the above temperature range, and only the skin color areas H1 and H2 are measured to be human.

If the measured temperature T is outside the human skin temperature range (NO in # 26), the process returns to # 2, and if it is within the human temperature range (YES in # 26), the process proceeds to # 28 and the human The motion content of the target object is detected while tracking the moving target object that is determined to be.

Here, a method of determining the motion content of the object will be described. In the present embodiment, the motion of the target object is determined by examining the change pattern of the distance D to the target target object (human face), the pan rotation angle α and the tilt rotation angle θ of the gazing point measuring unit U. I am trying.

FIG. 19 shows the position of the object Q using the UVW coordinates shown in FIG. From the figure, assuming that the coordinates of the object Q are (u, v, w), the size | OQ | of the vector OQ corresponds to the distance D to the object, and the orthogonal projection OQ ′ of the vector OQ onto the UV plane. Is the pan rotation angle, the orthogonal projection OQ ″ of the vector OQ on the UW plane.
Is the tilt rotation angle, so D = √
(U ² + v ² + w ² ), tan θ = w / u, and tan α = v / u.

For example, when the gazing point measuring unit U1 is arranged below and gazing obliquely upward to gaze at a human face, the gazing person Q approaches the gazing point measuring unit U1 side straight at a constant speed. , The tilt rotation angle θ and the pan rotation angle α when the person Q is tracked.
20 is changed as shown in FIG.

That is, since the human Q moves forward and backward with respect to the gazing point measuring unit U1 at a constant speed, the pan rotation angle α hardly changes and the distance D decreases at a predetermined rate according to the measurement time. The tilt rotation angle θ is almost constant when the person Q is out of the predetermined range, and is rapidly increased when the person Q is in the predetermined range.

When the gazing person Q moves straight away from the gazing point measuring unit U1 at a constant speed, the pan rotation angle α hardly changes, and the distance D increases at a predetermined rate according to the measurement time. Thus, the tilt rotation angle θ rapidly decreases when the human Q is within the predetermined range, and is almost constant when the human Q is outside the predetermined range.

On the other hand, when the gazing person Q crosses the front of the gazing point measuring unit U1 at a constant speed, right beside, the distance D, the tilt rotation angle θ and the pan rotation angle α when the person is tracked are The change is as shown in FIG.

That is, since the human Q moves left and right at a constant speed, the tilt rotation angle θ hardly changes and the distance D is U.
The pan rotation angle α changes from the − side to the + side (or +).
From the side to the − side, it changes so as to monotonically increase (or monotonically decrease) so as to cross the 0 point.

Therefore, when the reference change patterns are set for the distance D, the tilt rotation angle θ, and the pan rotation angle α in advance corresponding to the operation to be discriminated, and the object being watched is tracked for a certain period of time. Distance D, tilt rotation angle θ
It is possible to determine whether or not the object is performing a specific motion by detecting the change patterns of the pan rotation angle α and comparing the change patterns with the reference change patterns.

In this embodiment, if the change pattern of the distance D, the tilt rotation angle θ and the pan rotation angle α is as shown in FIG. 20, it is judged that the object is approaching, and if it is as shown in FIG. The object is determined to cross.

FIG. 22 is a flow chart for detecting the operation content of the object. The operation content detection process is performed in parallel with the above-described moving object recognition process by the CPU (II) 33.
Is being done in.

When the operation content detection process is started, first, various flags and registers are initialized, and then H
The alt state is set (# 120, # 122). This Hal
The t state is a state of waiting for the input of an interrupt signal input from the timer 332 at a predetermined cycle (for example, every second).

When the interrupt signal is input from the timer 332, the distance D from the CPU (I) 29 to the object, the current pan rotation angle α and the current tilt rotation angle θ are fetched, and these information are stored in the memory 331. Are arranged and stored in (# 1
24). Then, the amount of change in the distance D, the pan rotation angle α, and the tilt rotation angle θ is calculated from the information that has already been captured and the information that has been captured this time. It is determined whether or not it is performed (# 126).

The approaching motion is determined by, for example, the discriminant of the following formula 1, and the crossing motion is determined by the discriminant of the following formula 2, for example.

[0126]

[Equation 1]

[0127]

[Equation 2]

When it is determined that the operation content is the approaching operation, the start command (I) is output to the information output unit 4 (#
128), if it is determined that the operation content is a transverse operation, the start command (II) is output to the information output unit 4 (# 13
0), returning to # 122, and if the operation content is neither the approach operation nor the crossing operation, the operation returns to # 122 without outputting a start command to the information output unit 4, and the detection of the operation content is continued.

FIG. 23 is a flow chart of information output processing which is processed in parallel by the information output unit 4 in correspondence with the above-described operation content determination processing.

When the information output process is started, various flags, registers, etc. are first initialized (# 140), and the start command (I) or the start command from the CPU (II) 33 of the image processing unit 3 is started. The state of waiting for input of (II) is entered (# 142).

When the activation command is input from the CPU (II) 33 (YES in # 142), it is determined whether or not it is the activation command (I) (# 144), and if it is the activation command (I), For example, a message "Welcome to you." Corresponding to the approaching operation is called from the memory 431, displayed on the display device 41, and sounded by the voice output device 42.

On the other hand, if it is the start instruction (II), the memory 43
A message corresponding to the crossing action, for example, "Please wait a moment." Is called from 1, and is displayed on the display device 41 and sounded by the voice output device 42.

In the above embodiment, only the approaching movement and the crossing movement are detected. However, the moving away movement may be detected and a message such as "Thank you." May be provided.

[0134]

As described above, according to the present invention,
The target object is recognized from the image information, and the motion of the target object is detected. If the detection operation is a preset predetermined operation, the predetermined information corresponding to the detection operation is output to the target object. By doing so, it is possible to call attention to a recognized object such as a person or provide useful information, and it can be effectively used as an object recognition device and an information providing device.

[Brief description of drawings]

FIG. 1 is a block configuration diagram of an object recognition device according to the present invention.

FIG. 2 is a diagram showing a relationship between an angle of view of a wide-angle color video camera and an angle of view of a narrow-angle color video camera.

FIG. 3 is a diagram showing an example of a circuit configuration diagram of an AF mechanism of a horn angle color video camera.

FIG. 4 is a partially cutaway perspective view showing an example of a structure of a sensor unit portion.

FIG. 5 is a schematic diagram showing an internal structure of a gazing point measuring unit.

FIG. 6 is a diagram showing a relationship between a rotation angle of an eccentric cam and a distance to an object to be measured.

FIG. 7 is a diagram showing XYZ coordinates and UVW coordinates set in a gazing point measuring unit.

FIG. 8 is a diagram for explaining a trajectory in a gaze direction when the gaze point measurement unit is moved in the pan direction.

FIG. 9 is a cross-sectional view showing a first embodiment of an engaging structure of a guide member and an engaging member.

FIG. 10 is a sectional view showing a second embodiment of the engagement structure of the guide member and the engagement member.

FIG. 11 is a cross-sectional view showing a third embodiment of the engagement structure of the guide member and the engagement member.

FIG. 12 is a main flowchart showing a recognition operation of the object recognition device according to the present invention.

FIG. 13 is a flowchart showing color extraction processing of a wide-angle image.

FIG. 14 is a flowchart for detecting an object from a moving color area.

FIG. 15 is a flowchart showing a procedure of region position detection processing, in which (a) is a flow of painting a skin color region and calculating the center of gravity thereof as position information, and (b) is a position of the highest skin color density from the entire skin color image. A flow calculated as information, (c) is a diagram showing a processed image for each step of the both area position detecting method.

FIG. 16 is a flowchart showing color extraction processing for a horn image.

FIG. 17 is a flowchart showing a gaze direction adjustment process.

FIG. 18 is a flowchart showing temperature measurement direction adjustment and temperature measurement processing.

FIG. 19 is a diagram showing a position of an object in UVW coordinates.

FIG. 20 is a waveform diagram showing changes in the distance D to the object, the pan rotation angle θ, and the tilt rotation angle α when the object is approaching.

FIG. 21 is a waveform diagram showing changes in the distance D to the object, the pan rotation angle θ, and the tilt rotation angle α when the object is crossing.

FIG. 22 is a flowchart for detecting operation content.

FIG. 23 is a flowchart of information output processing.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Object recognition device 2 Sensor unit part 21 Radiation thermometer 211 Rotating shaft 212 Lever 213 Spring 22 Narrow-angle color video camera (image capturing means) 23 Angle of view adjusting device 24 Temperature measuring direction moving device 241 Rotating shaft 242 Eccentric cam 243 Geared motor 25 Distance measuring circuit 26 Gaze direction moving device 27 Pan / tilt position detector 28 Wide-angle color video camera 29 Control circuit (CPU (I)) 291 Memory 3 Image processing unit 31 Color extraction circuit (image extraction means) 311 312 color image processing circuit 313, 314 area extraction circuit 32 image memory 33 control circuit (CPU (II)) (motion detection means, determination means) 331 memory 332 timer 4 information output unit (information output means) 41 display device 42 audio output Device 43 CPU (III) 431 Memory 5 Base 10, 11 Box 12 Frame 1 Guide member 14 Pan direction drive mechanism 142,143,145,146 Pulley 149 Geared motor 15 Tilt direction drive mechanism 152,153,155,156 Pulley 159 Geared motor 16 Engagement member 17 Pan rotation angle detector 18 Tilt rotation angle detector TB1 to TB4 Timing belt U1 Gaze point measuring unit

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁶ Identification number Office reference number FI technical display location // G01J 5/48 C

Claims (1)

[Claims] 1. An image capturing means for capturing an image, an image extracting means for extracting an image of a target object from the images captured by the image capturing means, and an object for recognizing the target object from the extracted image. Recognition means, motion detection means for detecting the motion of the recognized object, judgment means for judging whether the detected motion of the object matches a predetermined motion pattern set in advance, and the object And an information output unit that outputs predetermined information set in advance corresponding to the operation pattern when the operation of the above item matches the above-mentioned predetermined operation pattern.