JP6342874B2 - Image recognition device - Google Patents

Description

  The present invention relates to an image recognition apparatus for recognizing human behavior based on a signal input from a distance image sensor capable of three-dimensional recognition of an imaging target.

  For example, when a driver operates various in-vehicle devices mounted on a vehicle, it is usually necessary to operate dedicated buttons provided at various locations in order to operate each in-vehicle device. . However, every time an operation is performed, the position where the target button exists must be searched, and the finger must be aligned to that position. For this operation, the driver temporarily shifts his / her line of sight from the front, Special attention must be paid to the movement of hands and fingers, which hinders safe driving.

  Therefore, techniques for eliminating the need for driver's button operation have been developed. For example, an operation input device is known in which when a driver's behavior is image-recognized by a camera provided in a vehicle interior and a predetermined behavior is detected, a device mounted on the vehicle operates according to the behavior ( Patent Document 1).

JP 2013-218391 A

  When a human behavior is detected by recognizing an image captured by a camera as in Patent Document 1, a general camera can only capture a two-dimensional image, so that the position and behavior in the depth direction cannot be detected. The degree of freedom of the operation pattern (gesture) cannot be increased, and the behavior may be detected when the detection is unnecessary.

  In recent years, a TOF (Time Of Flight) distance image sensor (hereinafter referred to as a TOF camera) capable of three-dimensional recognition of an imaging target is commercially available. In addition to the TOF camera, there is a camera capable of three-dimensional recognition of an imaging target. Since the TOF camera can detect the time until the light from the light source hits the measurement target and returns for each pixel, it can capture a stereoscopic image including position information corresponding to the distance in the depth direction.

  On the other hand, in the operation input device as in Patent Document 1, the behavior of the driver is detected only in a specific two-dimensional or three-dimensional space (hereinafter referred to as behavior detection space) determined in advance, and the operation of the in-vehicle device is performed. It is assumed to be reflected. For example, if the driver's hand is in the vicinity of the steering wheel (handle) as in a normal driving state, and control is performed so that the in-vehicle device is operated according to the behavior of the driver's hand or finger, In a situation where the driver moves his / her hand unconsciously, the in-vehicle device will not operate.

  However, since the operation input device as in Patent Document 1 monitors only the behavior of the driver's hand and fingers, it is difficult to increase the types of operation patterns that can be recognized, and it is also difficult to increase the recognition accuracy. For example, when trying to distinguish the differences in the shape of the driver's delicate hands and fingers, the device may incorrectly recognize the shape of the hand or finger, and the gesture operation pattern intended by the driver and the device There is a possibility that a malfunction may occur due to a difference from the operation pattern that is actually recognized.

  Therefore, for example, it is conceivable to simultaneously detect the behavior of other parts in addition to the driver's hand. However, in order to make it possible to detect the behavior of various human parts, it is necessary to increase the angle of view of the camera to be photographed or to increase the number of cameras installed in the vehicle. However, when the angle of view of the camera is increased, the detection accuracy of the shape and position is lowered and the recognition accuracy is lowered. In addition, when the number of cameras installed in the vehicle is increased, it is inevitable that the cost of the apparatus increases significantly.

  In addition, an error may occur in the distance measured by the TOF camera. In particular, in a high-temperature environment, there is a tendency that the angle of irradiation from the light source varies with temperature and the error increases. Therefore, when the TOF camera is installed in a vehicle room where the temperature changes rapidly, there is a concern about the influence of the distance measured by the TOF camera and the error of the three-dimensional coordinates.

  When the error in the distance measured by the TOF camera increases, the behavior detection space is enlarged, reduced, or displaced. As a result, even when the driver's hand is present outside the behavior detection space, there is a possibility that the in-vehicle device may operate in response to the behavior of the hand performed unconsciously. That is, an operation unintended by the driver is performed without permission. Since such a malfunction is an operation that is not expected by the driver, the driver's thinking may be confused and the driving may be hindered.

  The present invention has been made in view of the above-described circumstances, and an object thereof is to increase the types of recognizable gesture operation patterns and improve operation detection accuracy while suppressing an increase in apparatus cost. An object of the present invention is to provide an image recognition apparatus capable of performing the above.

In order to achieve the above-described object, an image recognition apparatus according to the present invention is characterized by the following (1) to (5).
(1) An image recognition device for recognizing human behavior based on a signal input from a distance image sensor capable of three-dimensional recognition of an imaging target,
As a space to recognize human behavior,
A first behavior detection space;
A second behavior detection space that is at least partially located on a line connecting the distance image sensor and the first behavior detection space and that does not overlap the first behavior detection space;
Is determined in advance,
The first behavior detection space and the second behavior detection space are assigned to positions separated from each other on a line connecting the distance image sensor and the first behavior detection space,
A result of recognizing human behavior in at least one of the first behavior detection space and the second behavior detection space is reflected in the output;
Image recognition device.

  According to the image recognition device having the configuration (1), it is possible to increase the types of gesture operation patterns that can be recognized and to improve operation detection accuracy while suppressing an increase in device cost. That is, the types of operation patterns can be increased by recognizing human behavior in each of the first behavior detection space and the second behavior detection space. In addition, since the first behavior detection space and the second behavior detection space are arranged on the same line as viewed from the distance image sensor, both behaviors can be obtained without increasing the angle of view when the distance image sensor captures an image. The detection space can be monitored simultaneously. Therefore, the number of pixels of the image data corresponding to each recognized recognition object increases, and the recognition accuracy is improved.

(2) The first behavior detection space is assigned to an area where human facial expressions can be recognized,
The second behavior detection space is assigned to an area where human hand or finger movement can be recognized.
The image recognition apparatus according to (1) above.

  According to the image recognition device having the configuration (2) above, it is possible to detect the behavior related to the facial expression of the human face by processing the image in the first behavior detection space, and simultaneously process the image in the second behavior detection space. By doing so, it is possible to detect a behavior related to the movement of a human hand or finger.

(3) Recognizing a human behavior in each of the first behavior detection space and the second behavior detection space, and identifying an operation performed by the human in accordance with a recognized combination of human behaviors.
The image recognition apparatus according to (1) above.

  According to the image recognition device having the configuration (3), the combination of the behavior in the first behavior detection space and the behavior in the second behavior detection space is used, and therefore the types of operations corresponding to these overall behaviors. Can be recognized with higher accuracy. In other words, the probability that the driver will perform a plurality of special behaviors other than a specific gesture at the same time is very low, so it is easy to distinguish whether a gesture is a specific gesture without recognizing subtle changes in each behavior detection space. it can.

(4) A first measurement distance to a predetermined specific photographing target recognized based on a signal input from the distance image sensor and a distance from the distance image sensor to the specific photographing target are determined in advance. A ratio with a reference distance obtained by actual measurement is calculated and recognized based on a parameter specifying the first behavior detection space and the second behavior detection space, or a signal input from the distance image sensor. A measurement value correction unit that corrects the second measurement distance to an arbitrary point of the behavior monitoring target by a correction amount based on the ratio;
The image recognition apparatus according to (1) above.

  According to the image recognition device having the configuration (4), even when a large error is included in the result of the distance measured by the distance image sensor, for example, when a TOF camera is used, the error Can be automatically corrected, so that it is possible to prevent malfunctions caused by behaviors detected in regions other than the first behavior detection space and the second behavior detection space.

(5) At least one of the first behavior detection space and the second behavior detection space is a region adjacent to the fixed portion, or a region around the fixed portion, with a specific fixed portion in the vehicle interior as a reference Assigned to the
The image recognition apparatus according to (1) or (2) above.

  According to the image recognition device having the configuration of (5) above, the first behavior detection space and the second behavior detection space are determined with reference to a specific fixed part in the passenger compartment. The driver's gesture can be recognized only under a specific condition such as the case of the posture. Therefore, it is possible to prevent the device from misrecognizing a behavior that the driver has unconsciously performed as a specific gesture.

  According to the image recognition device of the present invention, it is possible to increase the types of recognizable gesture operation patterns and improve operation detection accuracy while suppressing an increase in device cost. That is, it is possible to increase the types of operation patterns by recognizing human behavior in each of the first behavior detection space and the second behavior detection space. In addition, since the first behavior detection space and the second behavior detection space are arranged on the same line as viewed from the distance image sensor, without increasing the angle of view when the distance image sensor images, Both behavior detection spaces can be monitored simultaneously. Therefore, the number of pixels of the image data corresponding to each recognized recognition object increases, and the recognition accuracy is improved.

  The present invention has been briefly described above. Further, the details of the present invention will be further clarified by reading through a mode for carrying out the invention described below (hereinafter referred to as “embodiment”) with reference to the accompanying drawings. .

FIG. 1 is a side view showing an example of the positional relationship of each part when the interior of a vehicle equipped with an image recognition device according to an embodiment of the present invention is viewed from the right side of the vehicle. FIG. 2 is a block diagram illustrating a configuration example of an in-vehicle system including the image recognition device according to the embodiment of the present invention. FIG. 3 is a front view showing a configuration example of the interior of a vehicle in which the image recognition device according to the embodiment of the present invention is mounted. FIG. 4 is a flowchart showing an operation example of the image recognition apparatus according to the embodiment of the present invention. 5A and 5B are front views showing specific examples of gestures based on the behavior of the driver's hand. FIG. 6 is a front view illustrating a configuration example of the interior of a vehicle in which the image recognition device according to the modification is mounted.

  Specific embodiments relating to the present invention will be described below with reference to the drawings.

<Specific example of environment using image recognition device>
<Example of vehicle interior configuration>
FIG. 3 shows an example of the arrangement of main components in the vicinity of the driver's seat in the vehicle interior. The image recognition apparatus of the present invention is used in a state where it is mounted on a vehicle as shown in FIG. 3, for example.

  As shown in FIG. 3, a driver seat 34 and a passenger seat 35 are provided in the vehicle interior, and a steering wheel 31 is disposed in front of the driver seat 34. A display unit 32 having a rectangular display screen is installed near the center of the front dashboard 33. The display unit 32 can be used as a display unit of a car navigation device or a car audio device.

  As a display device that does not use the display unit 32, the main body of a head-up display (HUD) is stored below the dashboard 33. When this head-up display is used, a transparent combiner (reflecting plate) that reflects display light rises from the inside of the dashboard 33 and appears on the dashboard 33, and is positioned at a position where the driver can visually recognize it. .

  An operation lever 36 having a movable structure is arranged at a position where the driver on the left front side of the driver seat 34 can operate. The operation lever 36 is used to switch the transmission mode of the vehicle transmission.

  Further, the TOF camera 21 is installed on a dashboard 33 slightly ahead of the steering wheel 31 in a fixed state. The TOF camera 21 is a TOF (Time Of Flight) distance image sensor capable of three-dimensional recognition of an imaging target. That is, the TOF camera 21 can detect the time until the light from the light source hits the measurement target and returns for each pixel, and can capture a stereoscopic image including position information corresponding to the distance in the depth direction.

<Specific examples of positional relationship>
FIG. 1 shows an example of the positional relationship of each part when the interior of a vehicle equipped with the image recognition device of the embodiment of the present invention is viewed from the right side of the vehicle.

  The TOF camera 21 shown in FIG. 3 is installed at a position facing the steering wheel 31, and can capture a range in which both the entire steering wheel 31 and the driver's hand operating the steering wheel 31 are reflected simultaneously. As described above, the shooting direction and the angle of view are adjusted in advance.

  More specifically, as shown in FIG. 1, the fixed range on the vehicle such as the face 30 f of the driver 30, the hand 30 h, and the steering wheel 31 is included in the shooting range 25 </ b> A of the TOF camera 21. It is positioned so that

  In this embodiment, two behavior detection spaces ArF and ArH are determined in advance as a three-dimensional behavior detection space for detecting a driver's gesture. The behavior detection space ArF is a space for recognizing the behavior of the face 30f of the driver 30, and is assigned to an area including the position of the face 30f when the driver 30 is in a normal driving posture as shown in FIG. It has been. The behavior detection space ArH is a space for recognizing the behavior of the hand 30h of the driver 30 and is assigned to an area including the position of the hand 30h when the driver 30 is in a normal driving posture as shown in FIG. It has been. Each of the behavior detection spaces ArF and ArH is a rectangular parallelepiped three-dimensional space having a fixed size.

  In an actual vehicle, since the driving posture and position of the driver 30 can be adjusted, the behavior detection spaces ArF and ArH are automatically adjusted according to the adjusted actual driving posture and position. For example, the behavior detection space ArF for detecting the face 30f is assigned to a region separated by a predetermined distance with reference to this position after calculating the actual position of the headrest 34h of the driver seat 34. Further, the behavior detection space ArH for detecting the hand 30h is allocated to a region separated by a predetermined distance with reference to this position after calculating the actual posture (tilt angle, etc.) and position of the steering wheel 31. .

  Further, as shown in FIG. 1, two behavior detection spaces ArH and ArF are assigned to positions shifted from each other in the depth direction (Z direction) along the axis Az in the same shooting direction of the TOF camera 21. When viewed from the TOF camera 21 side, only two portions of the two behavior detection spaces ArH and ArF overlap each other, and a hand and a face in each space can be photographed simultaneously.

  In this case, since the two behavior detection spaces ArH and ArF are arranged on the same axis Az, only one TOF camera 21 is required without greatly expanding the angle of view of the TOF camera 21 corresponding to the imaging range 25A. Thus, it is possible to simultaneously photograph both photographing objects of the two behavior detection spaces ArH and ArF. Image recognition accuracy is improved by reducing the angle of view. Moreover, since it is not necessary to install a plurality of TOF cameras 21, an increase in cost can be suppressed.

  In addition, it is possible to increase the types of gestures that can be used by recognizing the behavior for each of the plurality of behavior detection spaces ArH and ArF. In addition, it is possible to increase gesture recognition accuracy, and it is also possible to employ gestures that are easy for the user to use.

<In-vehicle system configuration example>
A configuration example of an in-vehicle system including the image recognition device 10 according to the embodiment of the present invention is shown in FIG. The in-vehicle system shown in FIG. 2 is mounted on the vehicle shown in FIGS.

  The image recognition apparatus 10 shown in FIG. 2 can automatically recognize an operation instruction for the in-vehicle device from a predetermined gesture of the driver, that is, a gesture or a gesture, and can control the corresponding in-vehicle device.

  Actually, the TOF camera 21 captures the driver's hand, finger, face, steering wheel 31 and the like, and recognizes the behavior of the hand, finger, face, etc. as a gesture from the three-dimensional image obtained by the capturing. Therefore, the driver can operate the in-vehicle device by a gesture without operating a special button or the like. Therefore, when the driver operates the in-vehicle device, it is not necessary to search for a target button or to release the hand from the steering wheel 31 for the operation, and a function useful for improving safe driving can be provided.

  However, in the present embodiment, the image recognition apparatus 10 detects the behavior of the hand, finger, face, etc. only when they exist inside the predetermined behavior detection spaces ArH, ArF, etc. is there. As a result, for example, it is possible to prevent the driver from unintentionally recognizing the movement of the hand moved unconsciously as a specific gesture, and to prevent the in-vehicle device from performing an unexpected operation.

  As shown in FIG. 2, the TOF camera 21 includes a light source unit 21a and a light receiving unit 21b. The light source unit 21a can irradiate the object to be imaged with pulsed light. The light receiving unit 21b includes a two-dimensional image sensor formed of a CMOS or the like. In addition, for each pixel constituting the two-dimensional image detected by the light receiving unit 21b, distance information corresponding to the time (Time Of Flight) until the light from the light source unit 21a hits a photographing object such as a hand and returns to the light receiving unit 21b is obtained. A circuit for detection is built in the TOF camera 21. Therefore, the TOF camera 21 can capture a three-dimensional image.

  Information on the three-dimensional image captured by the TOF camera 21 is applied to the input of the image recognition apparatus 10. As shown in FIG. 2, the image recognition device 10 includes an image recognition processing unit 11 and a gesture monitoring control unit 12.

  The image recognition processing unit 11 executes information processing on the image information output from the TOF camera 21 at high speed, recognizes a pattern of a specific shape registered in advance, and positions and dimensions of the recognized pattern on the three-dimensional coordinates. It has the function to measure color, movement, shape change and so on.

  The gesture monitoring control unit 12 identifies whether or not the driver's hand exists inside the predetermined behavior detection space. Further, when the driver's hand is present inside the predetermined behavior detection space, the driver's hand, finger, face, etc. are monitored based on the recognition result of the image recognition processing unit 11, and the driver's hand is detected in advance. It is identified whether or not it matches with the gesture pattern registered in. When a behavior matching a specific gesture is detected, predetermined control is performed.

  The host ECU (electronic control unit) 22 sends information indicating the position and posture of the steering wheel 31, information indicating the position and posture of the driver's seat 34, information indicating ignition on / off of the vehicle, and the like to the gesture monitoring control unit 12. Can be given.

  In the in-vehicle system shown in FIG. 2, the HUD unit 23, the car navigation device 24, and the car audio device 26 are connected to the output of the image recognition device 10. The image recognition device 10 can control each of the HUD unit 23, the car navigation device 24, and the car audio device 26 based on a driver's gesture.

  For example, when the operation of the HUD unit 23 is activated by a specific gesture, a transparent combiner (not shown) included in the HUD unit 23 rises from below the dashboard 33 shown in FIG. 3 and appears on the dashboard 33. The driver can see the vehicle. In this state, the display light projected from the HUD unit 23 is reflected by the combiner, and a virtual image visible at the position of the driver's eyes is formed. When the operation of the HUD unit 23 is ended by a specific gesture, the combiner is lowered and stored below the dashboard 33.

  In addition, various buttons for operating the car navigation device 24 and functions similar to various buttons for operating the car audio device 26 can be assigned to various gestures that can be recognized by the image recognition device 10. It is.

<Specific examples of gestures>
Specific examples of gestures based on the behavior of the driver's hand are shown in FIGS. 5 (A) and 5 (B), respectively.

  For example, when performing a gesture operation for activating the operation of the HUD unit 23, the driver 30 keeps the left hand LH and the right hand RH touching the steering wheel 31 as shown in FIG. Move the left hand LH to trace from the bottom to the top. Further, the positions of the left hand LH and the right hand RH at this time are adjusted so as to be located within the region of the behavior detection space ArH. This operation is recognized as a specific gesture by the image recognition device 10 based on the image captured by the TOF camera 21, and the image recognition device 10 sends a control signal for activation to the HUD unit 23.

  Further, when performing a gesture operation to end the operation of the HUD unit 23, the driver 30 keeps the left hand LH and the right hand RH touching the steering wheel 31 as shown in FIG. Move to trace the left hand LH from top to bottom. Further, the positions of the left hand LH and the right hand RH at this time are adjusted so as to be located within the region of the behavior detection space ArH. This operation is recognized as a specific gesture by the image recognition device 10 based on the image captured by the TOF camera 21, and the image recognition device 10 sends a control signal for ending the operation to the HUD unit 23.

  In order to easily distinguish between a normal driving operation and a gesture, a more complicated operation than that shown in FIGS. 5A and 5B may be performed. For example, a gesture pattern that expresses a special hand shape by bending or stretching a specific finger or repeating a tracing operation a plurality of times may be employed.

  In the present embodiment, since the image recognition apparatus 10 can also recognize the facial expression of the face 30f in the behavior detection space ArF, it is conceivable to employ a gesture that combines the behavior of the hand and the behavior of the face. For example, the HUD unit 23 is activated only when the behavior of the hand as shown in FIG. 5A is detected and the direction of the line of sight of the eyes is facing forward in the face 30f in the behavior detection space ArF. Recognize as a gesture to do. In addition, the operation of the HUD unit 23 is performed only when the behavior of the hand as shown in FIG. 5B is detected and the direction of the line of sight of the eyes faces the front in the face 30f in the behavior detection space ArF. Is recognized as a gesture for ending.

  The behavior in the behavior detection space ArF and the behavior in the behavior detection space ArH may be assigned to independent device control. For example, the hand behavior in the behavior detection space ArH as shown in FIGS. 5A and 5B is assigned as a gesture for operating the HUD unit 23, and the facial behavior in the behavior detection space ArF is the car. It is conceivable to assign it as a gesture for operating a device such as the navigation device 24, the car audio device 26, or an air conditioner.

<Explanation of measurement error>
Since the image recognition apparatus 10 shown in FIG. 2 recognizes a three-dimensional image obtained by photographing with the TOF camera 21, the position of the recognition target hand or face is within the range of the behavior detection spaces ArF and ArH determined in advance. It can be identified whether or not it exists.

  However, a relatively large error may occur in the measurement distance from the TOF camera 21 to the recognition target. Actually, since the irradiation angle and the angle of view of the light from the light source unit 21a greatly fluctuate in a high temperature environment, the measurement distance in the photographing direction (depth direction Z) and the other axial directions (X, Y) An error also occurs in the coordinate position. In a vehicle, the environmental temperature in the passenger compartment may fluctuate greatly, so the measurement error of the TOF camera 21 becomes so large that it cannot be ignored.

  When a large error occurs in the distance measured by the TOF camera 21, the position of the hand to be recognized that is recognized based on the three-dimensional image is greatly deviated from the actual position. Even when the actual hand position is outside the range of the behavior detection spaces ArH and ArF, the image recognition device 10 may detect a gesture in response to the behavior of the hand or face. That is, since the image recognition apparatus 10 erroneously detects a gesture in a situation where the driver does not intend to perform a gesture, an in-vehicle device such as the HUD unit 23 performs an operation that is not expected by the driver. In order to prevent such a malfunction, the image recognition apparatus 10 has a function of automatically correcting the position of the recognition target recognized based on the three-dimensional image, as will be described later.

<Operation of Image Recognition Device 10>
<Outline of processing procedure>
FIG. 4 shows a main operation example of the image recognition apparatus 10 in the embodiment of the present invention. That is, the computer (not shown) or the image recognition processing unit 11 built in the gesture monitoring control unit 12 of the image recognition apparatus 10 shown in FIG. 2 responds to the driver's gesture according to the procedure shown in FIG. Implement the control.

  The procedure shown in FIG. 1 includes a process for correcting the position of the hand to be recognized that is recognized based on the three-dimensional image output from the TOF camera 21. Specifically, a correction ratio R1 is obtained with reference to the position of the steering wheel 31, and the recognized hand or face position is corrected.

  Since the steering wheel 31 is basically fixed to the vehicle body, the distance from the TOF camera 21 to a specific position on the steering wheel 31 can be treated as known. Therefore, this distance is measured in advance and registered in the constant table TB1 on the image recognition apparatus 10 as a distance reference value Lref.

  However, in an actual vehicle, the steering wheel 31 often has a posture adjustment function for changing the tilt angle and the length of the steering shaft, and the distance from the TOF camera 21 to a specific position on the steering wheel 31 is also large. It is variable. Therefore, a plurality of distance reference values Lref actually measured in each of a plurality of postures of the steering wheel 31 are registered in the constant table TB1, and an optimum distance reference value Lref is selected according to the actual posture of the steering wheel 31. Use it.

A distance measurement value L1 from the TOF camera 21 to the specific position can be calculated based on the three-dimensional coordinates of the specific position on the steering wheel 31 recognized based on the three-dimensional image captured by the TOF camera 21. Here, if a measurement error occurs, a difference appears between the distance reference value Lref and the distance measurement value L1. Therefore, the ratio R1 is calculated and used as a correction coefficient for correcting a distance error.
R1 = L1 / Lref (1)

  That is, the distance measurement value L2 from the position of the TOF camera 21 to the position of the monitored hand recognized based on the three-dimensional image captured by the TOF camera 21 includes a distance measurement error caused by the characteristics of the TOF camera 21. In order to reduce this measurement error, the distance measurement value L2 and the like are corrected using the ratio R1.

  Using the corrected distance measurement values and the like, the position of the hand or face is compared with the threshold value of each range of the behavior detection spaces ArH and ArF, so that the position of the hand or face is the range of each behavior detection space. It is possible to correctly identify whether it is within or not.

  Further, when the behavior detection spaces ArF and ArH are fixed in advance, the behavior detection spaces ArF and ArH may deviate from the desired spaces in accordance with the adjustment of the actual driving posture of the driver. . For example, if the position of the driver's seat 34 is adjusted in the front-rear direction, the position of the actual face 30f during driving may move back and forth, and the face may not be detected in the behavior detection space ArF.

  Therefore, when determining the behavior detection space ArF, the position of the behavior detection space ArF is determined as a space in the vicinity of the headrest 34h of the driver's seat 34 fixed on the vehicle. However, since the position of the headrest 34h changes as the driver's seat 34 moves and adjusts, the information on the position and orientation of the driver's seat 34 is obtained, the position of the headrest 34h is calculated, and the behavior is detected as a result. This is reflected in the position of the space ArF.

  Further, when determining the behavior detection space ArH, the position of the behavior detection space ArH is determined as a space near the periphery of the steering wheel 31 fixed on the vehicle. However, since the position and height of the steering wheel 31 change with the posture adjustment such as the tilt angle of the steering wheel 31, the information on the posture of the steering wheel 31 is acquired and the actual position of the steering wheel 31 is specified. The result is reflected in the position of the behavior detection space ArH.

<Details of processing procedure>
When the ignition of the vehicle is turned on, the process executed by the gesture monitoring control unit 12 proceeds from step S11 to S12 in FIG. In step S <b> 12, the gesture monitoring control unit 12 acquires information such as a tilt angle (corresponding to a difference in height) of the steering wheel 31 and the position and posture of the driver seat 34 from the host ECU 22.

  In the next step S13, the gesture monitoring control unit 12 specifies the actual position of each part of the steering wheel 31 based on the attitude information of the steering wheel 31 acquired in S12, and the behavior detection space ArH based on this position. Determine the position. Further, the actual position of the headrest 34h is calculated based on the position and posture information of the driver seat 34 acquired in S12, and the position of the behavior detection space ArF is determined based on this position. It should be noted that the constants representing the relationship between the positions necessary for specifying the positions of the respective parts of the steering wheel 31 and the headrest 34h and the positions of the respective parts and the relative distances of the respective parts are obtained from the constant table TB1.

  For example, the position in the front-rear direction (Z direction) of the behavior detection space ArF is determined so as to be adjacent to the front of a position separated from the position of the headrest 34h by a distance about half the size of a standard human head. . Further, the positions in the horizontal direction (X) and the vertical direction (Y) of the behavior detection space ArF are determined so as to coincide with the substantially central position of the headrest 34h.

  In step S14, the gesture monitoring control unit 12 uses the posture information acquired in S12 as a parameter, and acquires one distance reference value Lref associated therewith from the constant table TB1. That is, a value representing an actual distance from the position of the TOF camera 21 to a specific position on the fixed steering wheel 31 is acquired as the distance reference value Lref.

  In the next step S15, the gesture monitoring control unit 12 controls the TOF camera 21 to start photographing. Thereafter, data of a three-dimensional image obtained by photographing with the TOF camera 21 is sequentially input to the image recognition processing unit 11 and the gesture monitoring control unit 12 for each frame of the image.

  In step S16, the image recognition processing unit 11 processes the input three-dimensional image data and executes predetermined image recognition. That is, by comparing various feature amounts extracted from the inputted three-dimensional image with reference data such as the shape of the steering wheel 31 registered in advance, the shape of the hand, and the shape of the finger, the steering wheel 31. Recognize each recognition object such as a hand, a finger, and a face.

  In step S <b> 17, the gesture monitoring control unit 12 specifies the position coordinates of a specific position on the steering wheel 31 that is set in advance as a reference position based on the recognition result of the image recognition processing unit 11 in S <b> 16, and from the TOF camera 21 A distance measurement value L1 to the specific position is calculated. The “specific position” can be easily specified by using, for example, a special shape such as a protrusion, or a characteristic mark such as coloring or a mark.

  In step S18, the gesture monitoring control unit 12 calculates the ratio R1 of the expression (1) based on the distance reference value Lref acquired in S14 and the distance measurement value L1 calculated in S17.

  For example, when the actual distance from the TOF camera 21 to the steering wheel 31 is 30 cm, the distance reference value Lref is 30 cm. Further, when the distance measurement value L1 obtained by the image recognition is 50 cm, an error is included since it is different from the distance reference value Lref. Therefore, if the ratio R1 (50/30) is used as a correction value, it is possible to correct the error between the distance measurement value L1 and the distance reference value Lref.

  In step S19, the gesture monitoring control unit 12 acquires the hand position coordinates based on the recognition result of the image recognition processing unit 11, and calculates the distance in the depth direction (Z direction) from the position of the TOF camera 21 to the hand position. A distance measurement value L2 to be expressed is calculated. Further, based on the recognition result of the image recognition processing unit 11, face position coordinates are acquired, and a distance measurement value L3 representing a distance in the depth direction (Z direction) from the position of the TOF camera 21 to the face position is calculated.

In step S20, the gesture monitoring control unit 12 corrects the distance measurement value L2 to the hand position acquired in S19 using the ratio R1 obtained in S18, and acquires the corrected distance measurement value L21. Further, the distance measurement value L3 to the face position acquired in S19 is corrected using the ratio R1 obtained in S19, and the corrected distance measurement value L31 is acquired. Further, the position of each coordinate other than the distance is also corrected using the ratio R1.
L21 = L2 / R1 (2)
L31 = L3 / R1 (3)

  In step S21, the gesture monitoring control unit 12 compares the three-dimensional coordinates of the position of the driver's hand including the corrected distance measurement value L21 with a threshold value that specifies the range of the behavior detection space ArH, so that the hand behaves. It is identified whether or not it is within the detection space ArH. Here, if the position of the hand is within the range of the behavior detection space ArH, the process proceeds to the next S22, and if it is out of the range, the process proceeds to S23.

  In step S22, the gesture monitoring control unit 12 is based on the image recognition result of the image recognition processing unit 11, for example, the driver's hand in the behavior detection space ArH as shown in FIG. 5 (A) and FIG. 5 (B). And monitor finger behavior.

  In step S23, the gesture monitoring control unit 12 compares the three-dimensional coordinates of the position of the driver's face including the corrected distance measurement value L31 with a threshold value that specifies the range of the behavior detection space ArF, and the behavior of the face is determined. Whether it is within the detection space ArF is identified. If the face position is within the range of the behavior detection space ArF, the process proceeds to the next S24, and if it is out of the range, the process proceeds to S25.

  In step S24, the gesture monitoring control unit 12 monitors the behavior such as the facial expression of the driver in the behavior detection space ArF based on the image recognition result of the image recognition processing unit 11. For example, the direction of the line of sight of the left and right eyes, the shape of the mouth, the orientation of the face, the tilt of the head, etc. are monitored.

  In step S25, the gesture monitoring controller 12 compares the behavior pattern of the hand or finger detected in S22 and the behavior pattern such as the facial expression detected in S24 with the gesture reference pattern registered in advance, Identify whether they match.

  As a result of the comparison in S25, if the gesture monitoring control unit 12 detects a registered gesture, the process proceeds from S26 to S27. And the gesture monitoring control part 12 outputs a control signal with respect to vehicle equipment by S27 so that the control matched with the applicable gesture may be performed.

  For example, when the gesture of the hand shown in FIG. 5A is detected and the direction of the line of sight of the face is detected to be forward, the gesture monitoring control unit 12 activates the operation on the HUD unit 23. Outputs a signal to Further, when the gesture shown in FIG. 5B is detected and it is detected that the direction of the line of sight of the face is forward, the gesture monitoring control unit 12 ends the operation with respect to the HUD unit 23. The signal is output.

<Description of modification>
FIG. 6 shows a configuration example of the interior of a vehicle in which the image recognition device 10B according to the modification is mounted. In the modification shown in FIG. 6, it is assumed that the behavior detection spaces ArL and ArD are used instead of the behavior detection spaces ArF and ArH described above.

  The behavior detection space ArL shown in FIG. 6 is assigned as a rectangular parallelepiped region surrounding the position of the knob of the operation lever 36 with the position of the operation lever 36 as a reference. In addition, the behavior detection space ArD is assigned as an adjacent rectangular parallelepiped area so as to face the screen of the display unit 32 with the position of the display unit 32 as a reference.

  In the example shown in FIG. 6, the TOF camera 21 </ b> B is installed on the ceiling of the vehicle interior, and the photographing direction is directed downward. That is, the shooting range 25B and the direction of the TOF camera 21B are adjusted so that the vicinity of the behavior detection spaces ArD and ArL located below the ceiling in the passenger compartment can be simultaneously shot.

  Here, since the position of the operation lever 36 and the position of the display unit 32 do not change, the distance from the TOF camera 21 to the operation lever 36 and the display unit 32 is registered in the constant table TB1 as a constant similar to the above-described distance reference value Lref. It can be used as a reference value for calculating the ratio R1.

Therefore, in the image recognition device 10B shown in FIG. 6, the following gesture by the driver 30 can be recognized.
(1) Hand and finger shape and movement pattern when the driver 30 brings his hand close to the operation lever 36 (2) Hand and finger shape when the driver 30 brings his hand close to the screen of the display unit 32 And movement pattern

<Advantages of Image Recognition Device 10>
In the image recognition apparatus 10 shown in FIG. 1, a plurality of behavior detection spaces ArH and ArF are provided on an extension line of the same axis Az indicating the photographing direction of the TOF camera 21. Therefore, it is possible to photograph both recognition objects in the plurality of behavior detection spaces ArH and ArF at the same time without increasing the angle of view of the TOF camera 21. Therefore, it is possible to prevent the recognition accuracy from being lowered. Moreover, since it is not necessary to mount a plurality of cameras, an increase in cost can be suppressed.

  In addition, since human behavior can be detected for each of the plurality of behavior detection spaces ArH and ArF, it is possible to increase the types of gestures that can be recognized. In addition, since a combination of a plurality of behaviors can be assigned as one gesture, a highly flexible gesture can be realized, and for example, an easy-to-use user interface that can be operated intuitively can be provided.

  In the image recognition apparatus 10 described above, the TOF camera 21 is employed as a distance image sensor capable of three-dimensional recognition of a subject to be photographed. However, other distance image sensors may be used. Further, the number of behavior detection spaces may be increased to 3 or more if possible.

Here, the features of the above-described embodiment of the image recognition apparatus according to the present invention are briefly summarized and listed in the following [1] to [5], respectively.
The characteristic items related to the image recognition apparatus 10 are listed below.
[1] An image recognition device (10) for recognizing a human behavior based on a signal input from a distance image sensor (TOF camera 21) capable of three-dimensional recognition of an imaging target,
As a space to recognize human behavior,
A first behavior detection space (ArF or ArH);
A second behavior detection space (ArH or ArF) that is at least partially located on a line connecting the distance image sensor and the first behavior detection space and does not overlap with the first behavior detection space;
Is determined in advance,
The result of recognizing human behavior in at least one of the first behavior detection space and the second behavior detection space is reflected in the output (S26, S27),
Image recognition device.

[2] The first behavior detection space is allocated to an area (ArF) where human facial expressions can be recognized,
The second behavior detection space is assigned to an area (ArH) where human hand or finger movement can be recognized,
The image recognition apparatus according to [1] above.

[3] Recognize human behavior in each of the first behavior detection space and the second behavior detection space (S22, S24), and perform an operation performed by the human according to the recognized combination of human behaviors. Identify (S25, S26),
The image recognition apparatus according to [1] above.

[4] A first measurement distance (distance measurement value L1) to a predetermined specific imaging target recognized based on a signal input from the distance image sensor, and the specific imaging target from the distance image sensor A parameter (R1) for calculating a ratio (R1) to a reference distance (distance reference value Lref) obtained by actually measuring the distance up to and determining the first behavior detection space and the second behavior detection space; Alternatively, a measurement value correction unit that corrects the second measurement distance (distance measurement value L2) to an arbitrary point of the behavior monitoring target recognized based on the signal input from the distance image sensor by the correction amount based on the ratio. (S20)
The image recognition apparatus according to [1] above.

[5] At least one of the first behavior detection space and the second behavior detection space is based on a specific fixed portion (steering wheel 31, headrest 34h, operation lever 36, display unit 32) in the vehicle interior. Assigned to the area adjacent to the fixed part, or the area around the fixed part,
The image recognition apparatus according to the above [1] or [2].

DESCRIPTION OF SYMBOLS 10 Image recognition apparatus 11 Image recognition process part 12 Gesture monitoring control part 21 TOF camera 21a Light source part 21b Light-receiving part 22 Host ECU
23 HUD unit 24 Car navigation device 25A, 25B Shooting range 26 Car audio device 30 Driver 30f Face 30h Hand 31 Steering wheel 32 Display unit 33 Dashboard 34 Driver's seat 34h Headrest 35 Passenger seat 36 Operation lever LH Left hand RH Right hand TB1 Constant table Az Shooting direction axis ArH, ArF, ArL, ArD Behavior detection space Lref Distance reference value L1, L2, L3 Distance measurement value R1 ratio

Claims (5)

An image recognition apparatus for recognizing a human behavior based on a signal input from a distance image sensor capable of three-dimensional recognition of an imaging target,
As a space to recognize human behavior,
A first behavior detection space;
A second behavior detection space that is at least partially located on a line connecting the distance image sensor and the first behavior detection space and that does not overlap the first behavior detection space;
Is determined in advance,
The first behavior detection space and the second behavior detection space are assigned to positions separated from each other on a line connecting the distance image sensor and the first behavior detection space,
A result of recognizing human behavior in at least one of the first behavior detection space and the second behavior detection space is reflected in the output;
Image recognition device. The first behavior detection space is assigned to an area where a human facial expression can be recognized,
The second behavior detection space is assigned to an area where human hand or finger movement can be recognized.
The image recognition apparatus according to claim 1. Recognizing a human behavior in each of the first behavior detection space and the second behavior detection space, and identifying an operation performed by the human according to a recognized combination of human behaviors;
The image recognition apparatus according to claim 1. A first measurement distance to a predetermined specific photographing target recognized based on a signal input from the distance image sensor and a distance from the distance image sensor to the specific photographing target are measured in advance. A behavior monitoring target that is recognized based on a parameter that calculates a ratio with the obtained reference distance and that specifies the first behavior detection space and the second behavior detection space, or a signal input from the distance image sensor A measurement value correction unit that corrects the second measurement distance to an arbitrary point by a correction amount based on the ratio,
The image recognition apparatus according to claim 1. At least one of the first behavior detection space and the second behavior detection space is assigned to a region adjacent to the fixed portion or a region around the fixed portion with a specific fixed portion in the vehicle interior as a reference. ing,
The image recognition apparatus according to claim 1 or 2.

Images