WO2024014182A1

WO2024014182A1 - Vehicular gesture detection device and vehicular gesture detection method

Info

Publication number: WO2024014182A1
Application number: PCT/JP2023/020909
Authority: WO
Inventors: 拓也田村; 慶一梁井
Original assignee: 株式会社アイシン
Priority date: 2022-07-13
Filing date: 2023-06-06
Publication date: 2024-01-18

Abstract

A gesture detection device (90) comprises a determination unit (92) which determines whether or not a user has performed a gesture for operating a vehicle door (40), on the basis of positional change of a body part of the user in each of a plurality of images photographed by a camera (60) at different times. The determination unit (92) determines that the user has performed the gesture if a state, in which the position of the body part is moving in a first direction, has continued, and then a state, in which the position of the body part is moving in a second direction opposite to the first direction, has continued.

Description

Vehicle gesture detection device and vehicle gesture detection method

The present disclosure relates to a vehicle gesture detection device and a vehicle gesture detection method.

Patent Document 1 describes a vehicle body provided with an opening, a vehicle gate that opens and closes the opening, a gate actuator that drives the vehicle gate, a camera that photographs the surroundings of the vehicle, and a vehicle that controls the gate actuator. A vehicle is described that includes a gesture detection device. When the vehicle gesture detection device determines that the user has performed a predetermined gesture based on the image taken by the camera, the vehicle gate is opened by the gate actuator.

Japanese Patent Application Publication No. 2016-196798

The above-mentioned vehicle gesture detection device still has room for improvement in terms of accurately detecting user gestures.

In one aspect of the present disclosure, a gesture detection device for a vehicle is provided. The vehicle gesture detection device is applied to a vehicle that includes a door drive unit that opens and closes a door opening of a vehicle body, and a camera that photographs the area around the door opening, and operates the door. Detect user gestures. The gesture is an action in which the user moves a body part in a first direction and then moves it in a second direction opposite to the first direction. The vehicular gesture detection device performs a gesture determination process of determining whether the user has performed the gesture based on a change in the position of the body part in a plurality of images taken by the camera at different times. A determination unit is provided to perform the determination. In the gesture determination process, the determination unit is configured to perform, when the position of the body part continues to move in the second direction after the position of the body part continues to move in the first direction, It is determined that the user has performed the gesture.

In another aspect of the present disclosure, a gesture detection method for a vehicle is provided. The gesture detection method for a vehicle is applied to a vehicle that includes a door drive unit that opens and closes a door opening of a vehicle body, and a camera that photographs the vicinity of the door opening, and operates the door. Detect user gestures. The gesture is an action in which the user moves a body part in a first direction and then moves it in a second direction opposite to the first direction. The gesture detection method for a vehicle includes a determination step of determining whether the user has performed the gesture based on a change in the position of the body part in a plurality of images taken by the camera. The determination step includes determining whether the user makes the gesture when the position of the body part continues to move in the second direction after the position of the body part continues to move in the first direction. This includes determining that the same has been implemented.

FIG. 1 is a schematic diagram of a vehicle. FIG. 2 is an example of an image taken by the camera. FIG. 3 is an example of the quantized movement direction of the user's foot. FIG. 4 is a graph showing how the foot moves when different users perform kick gestures. FIG. 5 is an example of a gesture pattern. FIG. 6 is a flowchart illustrating the flow of processing performed by the gesture detection device to detect a user's kick gesture. FIG. 7 is a table showing changes in the displacement vector of the foot position when the user performs a large kick gesture. FIG. 8 is a table showing changes in the displacement vector of the foot position when the user performs a small kick gesture.

Hereinafter, one embodiment of a vehicle gesture detection device and a vehicle gesture detection method (hereinafter referred to as "gesture detection device and gesture detection method") will be described with reference to the drawings.

<Configuration of this embodiment>
As shown in FIG. 1, the vehicle 10 includes a vehicle body 20, a front door 30, a sliding door 40, a door driving section 50, a camera 60, a wireless communication device 70, a door control device 80, and a gesture detection device. 90. Further, a portable device 100 is tied to the vehicle 10.

<Vehicle body 20>
The vehicle body 20 has a front opening 21 that is opened and closed by a front door 30, and a rear opening 22 that is opened and closed by a sliding door 40. The front opening 21 and the rear opening 22 are parts that the user passes through when going back and forth between the inside and outside of the vehicle 10. In this embodiment, the front opening 21 and the rear opening 22 are partitioned by the B pillar of the vehicle body 20. In other embodiments in which the vehicle body 20 does not include a B-pillar, the front opening 21 and the rear opening 22 may be integrated as one opening. The rear opening 22 corresponds to a "door opening".

<Front door 30>
The front door 30 includes a door main body 31, a door knob 32 provided near the rear end of the door main body 31, and a side mirror 33 provided near the front end of the door main body 31. The front door 30 is displaced between a fully closed position where the front opening 21 is fully closed and a fully open position where the front opening 21 is fully opened by swinging around an axis extending in the vertical direction with respect to the vehicle body 20. .

<Sliding door 40 and door drive unit 50>
The sliding door 40 includes a door body 41 and a door knob 42 provided near the front end of the door body 41. The sliding door 40 is displaced between a fully closed position where the rear opening 22 is fully closed and a fully open position where the rear opening 22 is fully opened by sliding in the longitudinal direction with respect to the vehicle body 20. The opening direction of the sliding door 40 is backward, and the closing direction of the sliding door 40 is forward. The sliding door 40 is opened and closed by a door drive unit 50 between a fully closed position and a fully open position. The sliding door 40 can also be called a rear door in that it is located at the rear of the front door 30.

<Camera 60>
The camera 60 is installed on the side mirror 33 so as to face downward and rearward. As shown in FIG. 1, the photographing area AP of the camera 60 includes the area around the rear opening 22. The camera 60 outputs the captured image to the gesture detection device 90 frame by frame. The camera 60 may be, for example, a surroundings monitoring device for automatic driving or a camera that captures an original image for synthesizing a bird's-eye view of the surroundings of the vehicle. The frame rate of the camera 60 may be about 30 fps, for example.

<Wireless communication device 70 and portable device 100>
The portable device 100 includes a switch that is operated to open, close, or stop the sliding door 40. The portable device 100 may be a so-called electronic key, a smartphone, or another communication terminal. The wireless communication device 70 determines whether or not the portable device 100 is associated with the vehicle 10 by performing wireless communication with the portable device 100 located around the vehicle 10 . In this respect, the wireless communication device 70 can determine whether or not a user with the portable device 100 exists within the communication area AC set around the vehicle 10. Communication area AC is a larger area than photographing area AP.

When the switch for operating the sliding door 40 is operated in the portable device 100, the wireless communication device 70 outputs any one of an open operation command signal, a close operation command signal, and a stop command signal, depending on the operated switch. is output to the door control device 80. The opening operation command signal is a command signal for opening the sliding door 40. The closing operation command signal is a command signal for closing the sliding door 40. The stop command signal is a command signal for stopping the sliding door 40 during opening/closing operation. Further, when the portable device 100 is present within the communication area AC, the wireless communication device 70 outputs a signal indicating this to the gesture detection device 90.

<Door control device 80>
The door control device 80 includes, for example, a processing circuit including a CPU and a memory. Door control device 80 controls door drive unit 50 according to a program stored in memory. The door control device 80 controls the door drive unit 50 based on the contents of the input command signal. Specifically, the door control device 80 operates the sliding door 40 to open when the opening operation command signal is input. The door control device 80 closes the sliding door 40 when the closing operation command signal is input. The door control device 80 stops the sliding door 40 in operation when a stop command signal is input.

<Gesture detection device 90>
The gesture detection device 90 includes, for example, a processing circuit including a CPU and a memory. The gesture detection device 90 detects a gesture that causes a user's body part to reciprocate, and outputs a command signal to the door control device 80, according to a program stored in a memory.

As shown by the bold line arrow in FIG. 2, the gesture in this embodiment is a kick gesture in which the user's foot Ft approaches the vehicle body 20 and then moves the user's foot Ft away from the vehicle body 20. In this embodiment, the direction in which the user's foot Ft approaches the vehicle body 20 corresponds to a "first direction," and the direction in which the user's foot Ft moves away from the vehicle body 20 corresponds to a "second direction." When the user's foot Ft approaches the vehicle body 20 and when the user's foot Ft leaves the vehicle body 20, the user's foot Ft moves on substantially the same trajectory. As shown in FIG. 1, the gesture detection device 90 includes a storage section 91 and a determination section 92.

<Storage section 91>
The storage unit 91 stores a learned model that has been subjected to machine learning using teacher data in which images photographed in advance are associated with the user's foot position Pf. In other words, the learned model is a model that inputs an image captured by the camera 60 and outputs the user's foot position Pf within the image. The learned model is created, for example, when designing the vehicle 10 and written into the storage unit 91 when the gesture detection device 90 is manufactured. In this embodiment, the foot position Pf means the position of the user's foot Ft, more specifically, the position of the user's toe.

The method for generating a trained model will be described below.
The method for generating a trained model includes a preparation step of preparing teacher data, and a learning step of performing machine learning based on the teacher data. The preparation process includes an acquisition process of acquiring images taken with the user standing in the photography area AP under various conditions, and a specification process of specifying the user's foot position Pf in the plurality of images acquired in the acquisition process. and, including.

The acquisition step is performed using, for example, the actual vehicle 10. In the acquisition step, it is preferable to acquire many images taken by changing the conditions related to the user and the conditions related to the environment around the vehicle 10. This makes it possible to obtain a trained model that is adaptable to various situations, in other words, a highly versatile trained model. In the designation step, the user's foot position Pf is designated with respect to the acquired image. To specify the position, for example, coordinates using pixels in an image may be used.

In the learning process, a model is generated by machine learning using multiple teaching data as learning data. Various machine learning methods can be selected, and an example is a convolutional neural network (CNN). The learned model outputs the foot position Pf of a person in the image when a photographed image is input. On the other hand, the trained model does not output the foot position Pf when an image that does not include the person's foot Ft is input. Note that even if the trained model can output the foot position Pf, the outputted foot position Pf may not be the position of the user's toe depending on the accuracy of the trained model. However, in the following description, it is assumed that the output foot position Pf is the tip of the user's foot.

<Determination unit 92>
The determination unit 92 acquires the foot position Pf within the image by inputting the image captured by the camera 60 into the learned model. Then, the determination unit 92 performs a start determination process of determining whether the user is present in the vicinity of the vehicle 10, and determines whether the user has performed a kick gesture, based on the acquired foot position Pf. Gesture determination processing is performed.

In the start determination process, the determining unit 92 determines whether the user's foot Ft exists within the determination area AG set in the image captured by the camera 60. The determination unit 92 performs gesture determination processing when the user's foot position Pf exists within the determination area AG. On the other hand, the determination unit 92 does not perform the gesture determination process when the user's foot position Pf cannot be acquired or when the user's foot position Pf does not exist within the determination area AG. Note that the determination area AG is set within the photographing area AP of the camera 60. As shown in FIG. 1, the determination area AG includes a first determination area AG1 that is close to the camera 60 and a second determination area AG2 that is far from the camera 60. The first determination area AG1 can also be said to be an area close to the vehicle 10, and the second determination area AG2 can also be said to be an area distant from the vehicle 10.

In the gesture determination process, the determination unit 92 calculates a displacement vector Vd of the user's foot Ft between the plurality of images based on the plurality of images taken at different times. Specifically, the determination unit 92 calculates a displacement vector Vd from the foot position Pf in the N-th image to the foot position Pf in the N+1-th image. Here, the Nth image is the Nth image taken by the camera 60, and the N+1st image is the N+1st image taken by the camera 60, and the frame after the Nth image. This is an image of

For example, the determination unit 92 matches the feature amount of the area including the foot position Pf in the Nth image with respect to the N+1th image, thereby determining the N+1 corresponding to the foot position Pf in the Nth image. The foot position Pf in the th image is obtained. The determination unit 92 may obtain the foot position Pf in the N+1-th image by inputting the N+1-th image to the trained model. Then, the determination unit 92 calculates the displacement vector Vd of the foot position Pf based on the foot position Pf in both images. Subsequently, the determination unit 92 calculates a displacement vector Vd from the foot position Pf in the N+1-th image to the foot position Pf in the N+2-th image. Furthermore, the determination unit 92 calculates a displacement vector Vd from the foot position Pf in the N+2-th image to the foot position Pf in the N+3-th image.

In this way, the determination unit 92 calculates the displacement vector Vd of the foot position Pf every time a new image is taken. The determination unit 92 may calculate the displacement vector Vd of the foot position Pf for each frame, or may calculate the displacement vector Vd of the foot position Pf for each plurality of frames. The direction of the displacement vector Vd of the foot position Pf indicates the direction of movement of the user's foot Ft. Further, the magnitude of the displacement vector Vd of the foot position Pf indicates the amount of displacement of the foot Ft between one frame, that is, the speed of the user's foot Ft.

The determination unit 92 determines that a kick gesture has been performed when the direction of the displacement vector Vd of the foot position Pf changes according to the gesture pattern corresponding to the kick gesture. As shown in FIG. 3, in this embodiment, the direction of the displacement vector Vd of the foot position Pf is quantized into eight directions. In other embodiments, the direction of the displacement vector Vd of the foot position Pf may be quantized in 4 directions, 16 directions, or any other number of directions. Here, as shown in FIGS. 2 and 3, the X direction is the horizontal direction of the image taken by the camera 60, and the Y direction is the vertical direction of the image taken by the camera 60. The gesture pattern defines the order in which the direction of the displacement vector Vd of the foot position Pf changes.

In the present embodiment, the determination unit 92 alternately calculates the displacement vector Vd of the foot position Pf and matches the displacement vector Vd with the gesture pattern. In other embodiments, the determination unit 92 may perform the calculation of the displacement vector Vd of the foot position Pf and the verification of the gesture pattern at the same time. Then, when the direction of the displacement vector Vd of the user's foot position Pf changes according to the gesture pattern, that is, when it is determined that the user has performed a kick gesture, the determination unit 92 issues an opening operation command to the door control device 80. Output a signal.

As described above, the determination unit 92 determines whether the user has performed a kick gesture if the direction of the displacement vector Vd of the foot position Pf when the user performs the kick gesture changes according to a preset gesture pattern. It is determined that the Here, if different users perform the kick gesture, there may be differences in the manner in which the kick gesture is performed. Furthermore, even if the same user performs the kick gesture, there may be differences in the manner in which the kick gesture is performed. Therefore, the determination unit 92 needs to accurately determine whether the user has performed a kick gesture, regardless of the manner in which the user performs the kick gesture.

Next, with reference to FIG. 4, an example of a change in foot position Pf when three different users perform a kick gesture will be described. In the graph shown in FIG. 4, the horizontal axis indicates the coordinates of the foot position Pf in the X direction within the image, and the vertical axis indicates the coordinates of the foot position Pf in the Y direction within the image. Three types of markers indicate three users. Furthermore, two markers connected by a line segment indicate the foot position Pf in the N-th image and the foot position Pf in the N+1-th image. Therefore, when two markers are connected in the order in which the images are taken, the above-mentioned displacement vector Vd is obtained.

As shown in FIG. 4, there is a difference in the swing width of the foot Ft when performing the kick gesture among the three users. Specifically, there is a difference between the three users in the total amount of movement of the foot Ft in the X direction and the total amount of movement in the Y direction when performing the kick gesture. Furthermore, there is a difference in the movement speed of the foot Ft when performing the kick gesture among the three users. In other words, there is a difference in the time required from the start to the end of the kick gesture between the three users. Therefore, if a gesture pattern is set for a user whose foot Ft moves a large amount when performing a kick gesture, the gesture pattern will be suitable for a user whose foot Ft moves a small amount when performing a kick gesture. It will stop happening.

On the other hand, no matter which user performs the kick gesture, the user's foot position Pf continues to move toward the upper right, and then continues to move toward the lower left. When the direction of the displacement vector Vd of the user's foot position Pf is quantized as shown in FIG. The state of movement in the "3" direction continues. In particular, the direction of movement of the foot position Pf of the three users tends to change in the same way before and after the position where the direction of movement of the user's foot position Pf is reversed (hereinafter also referred to as the "return position"). It has become. In other words, before reaching the turn-back position, the user's feet Ft tend to move in the same direction, and after reaching the turn-back position, the user's feet Ft tend to move in the same direction.

Therefore, the determination unit 92 determines that when the foot position Pf as the detection target position continues to move in the "7" direction and then continues to move in the "3" direction, the user It is determined that a kick gesture has been performed. Therefore, in this embodiment, the gesture pattern set in advance is as shown in FIG. 5. As a result, if the direction of the displacement vector Vd of the foot position Pf changes to the "7" direction four times and then the direction of the displacement vector Vd of the foot position Pf changes to the "3" direction three times, the user performs a kick gesture. It is determined that this has been carried out.

As described above, the gesture pattern preferably corresponds to the period before and after the turning position when the user performs the kick gesture. Therefore, even if the user performs a kick gesture with a small movement, it is preferable that the gesture pattern be matched with a part of the kick gesture. Therefore, it is preferable that the gesture pattern is determined based on the results of recording kick gestures of a plurality of users in advance.

Furthermore, the duration of movement of the foot position Pf in the "7" direction (hereinafter referred to as "first determination time") required by the gesture pattern is the duration of movement of the foot position Pf in the "3" direction (hereinafter referred to as "first determination time"). 2 judgment time.) Specifically, it is necessary to check the displacement vector Vd in the direction "7" four times, whereas the displacement vector Vd in the direction "3" only needs to be checked three times. It is shorter than the time. In this respect, the determination unit 92 determines that after the state in which the foot position Pf moves in the "7" direction continues for the first determination time, the state in which the foot position Pf moves in the "3" direction continues for less than the first determination time. If it continues for the second determination period, it is determined that the user has performed the kick gesture.

Furthermore, when the user changes his or her standing position on the side of the vehicle 10, or when the user approaches the vehicle 10 and changes the direction of movement, the direction of the user's foot position Pf changes according to the gesture pattern described above. There are cases. In such a case, it is preferable that the determination unit 92 not determine that the user has performed a kick gesture. Here, when the user changes the standing position or the moving direction, the time the user's foot position Pf stays at the same position increases when the moving direction of the user's foot position Pf changes. Cheap. On the other hand, when the user performs a kick gesture, when the moving direction of the user's foot position Pf changes, the time during which the user's foot position Pf remains at the same position is less likely to increase.

Therefore, in the gesture determination process, if the user's foot position Pf remains at the same position, the determination unit 92 cancels the gesture determination process. Specifically, the determination unit 92 stops the gesture determination process when the state in which the magnitude of the displacement vector Vd is less than the lower limit determination value Vlth continues for a predetermined stop determination time. In other words, the determination unit 92 determines that the user is not performing a kick gesture. In other words, in the present embodiment, the determination unit 92 continues the gesture determination process if the time during which the magnitude of the displacement vector Vd continues to be less than the lower limit determination value Vlth is less than the predetermined stop determination time. . In this respect, it can be said that the determination unit 92 allows the foot Ft to stop for a short time when the user performs the kick gesture. Here, the lower limit determination value Vlth is a speed determination value for determining whether the user's foot position Pf is stopped.

Further, if the magnitude of the displacement vector Vd of the foot position Pf is extremely large, there is a high possibility that the user has not performed a kick gesture. Therefore, the determination unit 92 also stops the gesture determination process when the magnitude of the displacement vector Vd of the foot position Pf is greater than or equal to the upper limit determination value Vuth. In other words, the determination unit 92 determines that the user is not performing a kick gesture. Here, the upper limit determination value Vuth is a speed determination value for determining whether the user's foot position Pf is moving at a very high speed.

In this embodiment, the displacement vector Vd is calculated based on the user's foot position Pf in the image taken by the camera 60. Therefore, even if the user performs the kick gesture in the same way, there will be a difference in the magnitude of the displacement vector Vd when comparing the cases where the user performs the kick gesture at a position near and far from the camera 60. . Therefore, the determination unit 92 corrects the upper limit determination value Vuth and the lower limit determination value Vlth according to the distance from the camera 60 to the user's foot position Pf. Specifically, when the user's foot position Pf is within the first determination area AG1 close to the camera 60, the determination unit 92 does not correct the upper limit determination value Vuth and the lower limit determination value Vlth. That is, the default values of the upper limit determination value Vuth and the lower limit determination value Vlth are adopted. On the other hand, if the user's foot position Pf is within the second determination area AG2 far from the camera 60, the determination unit 92 corrects the upper limit determination value Vuth and the lower limit determination value Vlth to be smaller than the default values. do. The amount of correction is preferably set according to the number of pixels and focal length of the camera 60.

<Gesture detection method>
The flow of processing performed by the gesture detection device 90, that is, the gesture detection method will be described below. This process is a process that is executed in a predetermined control cycle when the user carrying the portable device 100 enters the communication area AC and the sliding door 40 is located in the fully closed position.

As shown in FIG. 6, the gesture detection device 90 acquires an image taken by the camera 60 (S11). Subsequently, the gesture detection device 90 acquires the foot position Pf by inputting the image captured by the camera 60 into the learned model stored in the storage unit 91 (S12). After that, the gesture detection device 90 determines whether the foot position Pf in the image exists within the determination area AG (S13). If the foot position Pf does not exist within the determination area AG (S13: NO), the gesture detection device 90 ends this process. Even when the foot position Pf cannot be acquired in step S12, the gesture detection device 90 ends this process.

On the other hand, if the foot position Pf exists within the determination area AG (S13: YES), the gesture detection device 90 acquires a new image captured by the camera 60 (S14). The image acquired in step S14 is an image taken after the previously acquired image. Subsequently, the gesture detection device 90 acquires the foot position Pf in the new image (S15).

Then, the gesture detection device 90 calculates the displacement vector Vd of the foot position Pf at the image capturing interval of the camera 60 (S16). The displacement vector Vd of the foot position Pf is a vector directed from the previously specified foot position Pf to the currently specified foot position Pf.

Subsequently, the gesture detection device 90 determines whether the magnitude of the displacement vector Vd of the foot position Pf is greater than or equal to the upper limit determination value Vuth (S17). Here, the upper limit judgment value Vuth is a value set depending on which area of the first judgment area AG1 or the second judgment area AG2 the foot position Pf acquired in the most recent step S15 is located. There is. When the magnitude of the displacement vector Vd is greater than or equal to the upper limit determination value Vuth (S17: YES), in other words, when the magnitude of the displacement vector Vd deviates from the movement speed of the user's foot when performing the kick gesture. , the gesture detection device 90 ends this process.

On the other hand, if the magnitude of the displacement vector Vd is less than the upper limit determination value Vuth (S17: NO), the gesture detection device 90 determines whether the magnitude of the displacement vector Vd of the foot position Pf is less than the lower limit determination value Vlth. (S18). Similarly to step S17, the lower limit judgment value Vlth is a value set depending on which area of the first judgment area AG1 or the second judgment area AG2 the foot position Pf acquired in the most recent step S15 is located. It becomes. When the magnitude of the displacement vector Vd is less than the lower limit determination value Vlth (S18: YES), in other words, when the position of the foot position Pf has not substantially changed, the gesture detection device 90 sets the stop counter Cnt by "1". Add (S19). The stop counter Cnt is a variable for counting the number of times the magnitude of the displacement vector Vd becomes less than the lower limit determination value Vlth. The stop counter Cnt is initialized to "0" at the timing of starting this process and at step S22, which will be described later.

The gesture detection device 90 determines whether the stop counter Cnt is equal to or greater than the stop determination number Cntth (S20). When the stop counter Cnt is equal to or greater than the stop determination number Cntth (S20: YES), for example, when the user stops or the user stops performing the kick gesture, the gesture detection device 90 ends this process. On the other hand, if the stop counter Cnt is less than the stop determination number Cntth (S20: NO), the gesture detection device 90 moves the process to step S14. Here, if the number of stoppage determinations Cntth is set to a small number, if the state in which the foot position Pf does not move continues for a short period of time, an affirmative determination is made in the process of step S20. On the other hand, if the number of stop determinations Cntth is set to a large number, an affirmative determination will not be made in the process of step S20 unless the foot position Pf remains stationary for a long period of time. The stop determination count Cntth is a number corresponding to the stop determination time described above.

In step S18, if the magnitude of the displacement vector Vd is greater than or equal to the lower limit determination value Vlth (S18: NO), the gesture detection device 90 determines whether the direction of the displacement vector Vd matches the gesture pattern (S21). For example, when matching up to the "N+2"th gesture pattern has been completed, it is determined whether the direction of the displacement vector Vd matches the "7" direction corresponding to the "N+3"th gesture pattern. Ru.

Then, the gesture detection device 90 initializes the stop counter Cnt to "0" (S22). Subsequently, the gesture detection device 90 determines whether all gesture pattern matching is completed (S23). If the gesture pattern matching is not completed, the gesture detection device 90 moves the process to step S14. On the other hand, if the gesture pattern matching is completed (S23: YES), the gesture detection device 90 outputs an opening operation command signal to the door control device 80 (S24). That is, the sliding door 40 is operated to open.

Note that in the flowchart shown in FIG. 6, steps S14 to S23 correspond to a "judgment step".
<Action of this embodiment>
Hereinafter, the manner in which kick gestures are detected when different aspects of kick gestures are performed will be described.

FIG. 7 shows a change in the direction of the displacement vector Vd of the foot position Pf when the user performs a kick gesture with a large movement. As shown in FIG. 7, when the user performs a kick gesture with a large swing width of the foot Ft, the direction of the displacement vector Vd of the user's foot position Pf becomes the "7" direction from the first timing. The condition continues for a relatively long period of time. Thereafter, when the user's foot Ft reaches the turning position at a timing between the 10th and 11th, the direction of the displacement vector Vd of the user's foot position Pf becomes the "3" direction. In the case shown in FIG. 7, the direction of the displacement vector Vd of the foot position Pf changes according to the gesture pattern shown in FIG. 5 during the period from the seventh timing to the thirteenth timing. Therefore, it is determined that the kick gesture has been performed at the 13th timing. Therefore, the directions of the displacement vectors Vd of the fourteenth and subsequent foot positions Pf are not used for determining whether to perform a kick gesture. That is, the opening operation of the sliding door 40 is started while the user is performing the kick gesture.

FIG. 8 shows a change in the direction of the displacement vector Vd of the foot position Pf when the user performs a kick gesture with a small movement. As shown in FIG. 8, when the user performs a kick gesture with a small swing width of the foot Ft, the direction of the displacement vector Vd of the user's foot position Pf becomes the "7" direction from the first timing. The condition continues for a relatively short period of time. Thereafter, when the user's foot Ft reaches the turning position between the seventh and eighth timings, the direction of the displacement vector Vd of the user's foot position Pf becomes the "3" direction. Here, in the case shown in FIG. 8, the direction of the displacement vector Vd of the foot position Pf changes according to the gesture pattern shown in FIG. 5 during the period from the fourth timing to the tenth timing. Therefore, it is determined that the kick gesture has been performed at the 10th timing. Therefore, the directions of the displacement vectors Vd of the 11th and subsequent foot positions Pf are not used for determining whether to perform a kick gesture. That is, the opening operation of the sliding door 40 is started while the user is performing the kick gesture.

<Effects of this embodiment>
(1) Regardless of whether the user performs the kick gesture with a small movement or a large movement, when the user performs the kick gesture, the user's foot Ft continues to move in the direction approaching the vehicle 10. After that, a state of moving away from the vehicle 10 continues. In this respect, the gesture detection device 90 can determine whether or not the user has performed a kick gesture, regardless of the magnitude of the movement of the foot Ft when the user performs the kick gesture. In other words, the gesture detection device 90 can accurately detect the user's kick gesture.

(2) In the gesture pattern shown in FIG. 5, the number of matches in the "7" direction of the displacement vector Vd is 4 times, while the number of matches in the "3" direction of the displacement vector Vd is 3 times. It has become. Therefore, the gesture detection device 90 can determine whether or not the user performed the kick gesture at an earlier timing than in the comparative example in which the number of verifications is opposite to that of the present embodiment. Specifically, when the user performs the kick gesture, the gesture detection device 90 can determine that the user has performed the kick gesture immediately after the user's foot Ft begins to move away from the vehicle 10.

(3) The gesture detection device 90 cancels the gesture determination process if the user's foot Ft continues to be stopped during the gesture determination process. Therefore, the gesture detection device 90 can cancel the gesture determination process when the user cancels the kick gesture or when the user stops on the side of the vehicle 10. Therefore, the gesture detection device 90 can improve the accuracy of determining the user's kick gesture.

(4) When the distance from camera 60 to foot Ft is long, the magnitude of displacement vector Vd tends to be smaller than when the distance from camera 60 to foot Ft is short. In this regard, the gesture detection device 90 sets the upper limit determination value Vuth and the lower limit determination value Vlth to different values depending on the distance from the camera 60 to the foot Ft. Therefore, the gesture detection device 90 can improve the accuracy of determining the user's kick gesture regardless of the position of the foot Ft in the image.

<Example of change>
- The stop determination process using the stop counter Cnt may be performed only when the moving direction of the user's foot position Pf is reversed. Specifically, the stop determination process using the stop counter Cnt is performed from the time when the direction of the N+4th displacement vector Vd is completed until the time when the direction of the N+5th displacement vector Vd is completed in the gesture pattern shown in FIG. It may be implemented only in certain cases.

-Also, in the stop determination process using the stop counter Cnt, the stop determination number Cntth may be set to a different value depending on whether the moving direction of the user's foot position Pf is reversed or not.

-Also, the gesture detection device 90 does not need to perform the stoppage determination process using the stop counter Cnt. Specifically, in the flowchart shown in FIG. 6, the processes of steps S18 to S20 may be omitted.

- The kicking gesture may be a kicking gesture of opening and closing the tip of the foot centering on the heel of the foot Ft.
- The gesture performed by the user does not have to be a kick gesture as long as it is a gesture in which the user moves a body part back and forth. For example, the gesture may be a gesture of raising and lowering a hand, or a gesture of changing the direction of the face.

・The gesture pattern can be changed as appropriate. For example, in the gesture pattern, the number of movements in the "7" direction may be the same as the number of movements in the "3" direction, or may be less than the number of movements in the "3" direction.

- The camera 60 does not have to be installed on the side mirror 33. For example, the camera 60 may be installed at the upper end of the rear opening 22 or may be installed at the sliding door 40.
- The gesture detection device 90 may output a closing operation command signal for closing the sliding door 40 to the door control device 80 based on the user's kick gesture. Furthermore, the gesture detection device 90 may output one of the opening operation command signal and the closing operation command signal to the door control device 80 depending on the position of the sliding door 40 when the user performs the kick gesture.

- The determination unit 92 may calculate the distance from the camera 60 to the user's foot position Pf. In this case, the determination unit 92 may correct the upper limit determination value Vuth and the lower limit determination value Vlth according to the distance from the camera 60 to the user's foot position Pf. In this case, the amount of correction is preferably proportional to the distance from the camera 60 to the user's foot position Pf.

- The determination unit 92 does not need to correct the upper limit determination value Vuth and the lower limit determination value Vlth according to the distance from the camera 60 to the user's foot position Pf. In this case, the determination unit 92 may correct the magnitude of the displacement vector Vd according to the distance from the camera 60 to the user's foot position Pf.

- The vehicle 10 may include a front door drive unit that drives the front door 30. In this case, the gesture detection device 90 may detect a kick gesture for opening the front door 30, similar to the case for opening the sliding door 40.

- The vehicle 10 may include a back door that opens and closes an opening opening to the rear of the vehicle body 20, a back door drive unit that drives the back door, and a back camera that photographs the rear of the vehicle 10. In this case, the gesture detection device 90 may detect a kick gesture for opening the back door, based on an image taken by the back camera, as in the case of opening the sliding door 40.

- The door control device 80 and the gesture detection device 90 are not limited to processing circuits that include a CPU and a ROM and execute software processing. For example, the door control device 80 and the gesture detection device 90 may include a dedicated hardware circuit that executes at least a portion of the various processes executed in each of the above embodiments. An example of the dedicated hardware circuit is an ASIC. ASIC is an abbreviation for "Application Specific Integrated Circuit." That is, the door control device 80 and the gesture detection device 90 may have any of the following configurations (a) to (c).

(a) A processing circuit that includes a processing device that executes all of the above processing according to a program, and a program storage device such as a ROM that stores the program.
(b) A processing circuit comprising a processing device and a program storage device that execute part of the above processing according to a program, and a dedicated hardware circuit that executes the remaining processing.

(c) A processing circuit that includes a dedicated hardware circuit that performs all of the above processing.
Here, there may be a plurality of software execution devices including a processing device and a program storage device, and a plurality of dedicated hardware circuits.

[Summary of this embodiment]
This embodiment includes at least the following configurations.
The vehicle gesture detection device (90) of this embodiment includes a door drive unit (50) that opens and closes a door (40) that opens and closes a door opening (22) of a vehicle body (20), and a door drive unit (50) that opens and closes a door (40) that opens and closes a door opening (22) of a vehicle body (20). ), and detects a user's gesture for operating the door (40). The gesture is an action in which the user moves the body part (Ft) in a first direction and then moves it in a second direction opposite to the first direction. The vehicle gesture detection device (90) determines whether the user performed the gesture based on a change in the position of the body part (Ft) in a plurality of images taken by the camera (60) at different times. A determination unit (92) is provided that performs gesture determination processing to determine whether the gesture is a gesture or not. In the gesture determination process, the determination unit (92) determines whether the position of the body part (Ft) moves in the second direction after the position of the body part (Ft) continues to move in the first direction. If the moving state continues, it is determined that the user has performed the gesture.

Regardless of whether the user performs the gesture with small movements or the user performs the gesture with large movements, when the user performs the gesture, the user's body part continues to move in the first direction. After that, a state of movement in the second direction continues. Therefore, the vehicular gesture detection device can determine whether the user has performed the gesture, regardless of the magnitude of the movement of the body part when the user performs the gesture. In other words, the vehicular gesture detection device can detect the user's gestures with high accuracy.

In the present embodiment, in the gesture determination process, the determination unit (92) determines whether the body part (Ft) moves in the first direction after the body part (Ft) continues to move in the first direction for a first determination time. It is preferable to determine that the user has performed the gesture when a state in which the position of Ft) moves in the second direction continues for a second determination time. It is preferable that the second determination time is shorter than the first determination time.

For example, consider a comparative example in which the second determination time is longer than the first determination time. In this comparative example, it cannot be determined whether the user has performed a gesture until the relatively long second determination time has elapsed after the moving direction of the user's body part is switched to the second direction. On the other hand, the vehicle gesture detection device configured as described above makes the second determination time shorter than the first determination time. Therefore, the vehicular gesture detection device determines whether the user has performed the gesture or not after the relatively short second determination time has elapsed after the moving direction of the user's body part switches to the second direction. can.

In the present embodiment, it is preferable that the determination unit (92) calculates a displacement vector indicating the amount and direction of movement of the body part (Ft) within the two images taken at different timings. In the gesture determination process, the determination unit (92) determines whether the direction of the displacement vector continues to be in the second direction after the direction of the displacement vector continues to be in the first direction. , it is preferable to determine that the user has performed the gesture. It is preferable that the determination unit (92) cancels the gesture determination process if the magnitude of the displacement vector continues to be less than a lower limit determination value (Vlth) during the gesture determination process.

During gesture determination processing, when the user continues not to move a body part, such as when the user cancels a gesture or when the user performs an action that is not a gesture, the vehicle gesture detection device detects the following: Gesture determination processing can be canceled. Therefore, the vehicular gesture detection device can improve the accuracy of determining the user's gesture.

In this embodiment, the determination unit (92) determines that when the distance from the camera (60) to the body part (Ft) is long, the distance from the camera (60) to the body part (Ft) is long. It is preferable to make the lower limit judgment value (Vlth) smaller than when it is short. For example, if the distance from the camera (60) to the body part (Ft) is the first distance, then the second distance from the camera (60) to the body part (Ft) is shorter than the first distance. The lower limit determination value (Vlth) can be made smaller than when it is a distance.

When the distance from the camera to the body part is long, the magnitude of the displacement vector tends to be smaller than when the distance from the camera to the body part is short. In this regard, the vehicle gesture detection device changes the magnitude of the lower limit determination value depending on the distance from the camera to the body part. Therefore, the vehicle gesture detection device can improve the accuracy of determining the user's gesture regardless of the position of the body part in the image.

The vehicle gesture detection method of the present embodiment includes a door drive unit (50) that opens and closes a door (40) that opens and closes a door opening (22) of a vehicle body (20), and a door drive unit (50) that opens and closes a door opening (22) of a vehicle body (20), and a periphery of the door opening (22). The present invention is applied to a vehicle (10) equipped with a camera (60) for photographing the door (40), and detects a user's gesture for operating the door (40). The gesture is an action in which the user moves the body part (Ft) in a first direction and then moves it in a second direction opposite to the first direction. The vehicle gesture detection method includes a determination step of determining whether the user has performed the gesture based on a change in the position of the body part (Ft) in a plurality of images taken by the camera (60). Equipped with In the determination step, when the position of the body part (Ft) continues to move in the second direction after the position of the body part (Ft) continues to move in the first direction, The method includes determining that the user has performed the gesture.

The vehicular gesture detection method can obtain the same effects as the above-mentioned vehicular gesture detection device.

DESCRIPTION OF SYMBOLS 10...Vehicle, 20...Vehicle body, 40...Sliding door (an example of a door), 50...Door drive unit, 60...Camera, 70...Wireless communication device, 80...Door control device, 90...Gesture detection device, 91...Storage unit , 92... Judgment unit, AG (AG1, AG2)... Judgment area, Cnt... Stop counter, Cntth... Number of stop judgments, Ft... Foot (an example of body part), Pf... Foot position, Vd... Displacement vector, Vlth... Lower limit judgment value, Vuth...upper limit judgment value.

Claims

The present invention is applied to a vehicle that includes a door drive unit that opens and closes a door opening of a vehicle body, and a camera that photographs the area around the door opening, and detects a user's gesture for operating the door. A gesture detection device for a vehicle, the device comprising:
The gesture is an action in which the user moves a body part in a first direction and then moves it in a second direction opposite to the first direction,
comprising a determination unit that performs gesture determination processing to determine whether the user has performed the gesture based on changes in the position of the body part in a plurality of images taken by the camera at different times;
In the gesture determination process, the determining unit is configured to perform, when the position of the body part continues to move in the second direction after the position of the body part continues to move in the first direction, A gesture detection device for a vehicle that determines that the user has performed the gesture.
In the gesture determination process, the determination unit determines that after a state in which the position of the body part moves in the first direction continues for a first determination time, a state in which the position of the body part moves in the second direction continues. It is determined that the user has performed the gesture if the gesture continues for a second determination period,
The vehicle gesture detection device according to claim 1, wherein the second determination time is shorter than the first determination time.
The determination unit includes:
calculating a displacement vector indicating the amount and direction of movement of the body part in the two images taken at different timings;
In the gesture determination process, the user performs the gesture when the direction of the displacement vector continues to be in the second direction after the direction of the displacement vector continues to be in the second direction. It is determined that
The gesture detection device for a vehicle according to claim 1 or 2, wherein the gesture determination process is stopped if the magnitude of the displacement vector continues to be less than a lower limit determination value while the gesture determination process is being performed. .
4 . The vehicle according to claim 3 , wherein the determination unit makes the lower limit determination value smaller when the distance from the camera to the body part is long than when the distance from the camera to the body part is short. Gesture detection device.
The present invention is applied to a vehicle that includes a door drive unit that opens and closes a door opening of a vehicle body, and a camera that photographs the area around the door opening, and detects a user's gesture for operating the door. A gesture detection method for a vehicle, the method comprising:
The gesture is an action in which the user moves a body part in a first direction and then moves it in a second direction opposite to the first direction,
a determination step of determining whether the user has performed the gesture based on a change in the position of the body part in a plurality of images taken by the camera;
The determination step includes determining whether the user makes the gesture when the position of the body part continues to move in the second direction after the position of the body part continues to move in the first direction. A vehicle gesture detection method including determining that a gesture has been performed.