JP2020118024A - Device operating system and device control method - Google Patents

Abstract

To provide a device operating system and a device control method that can perform opening/closing operation for windows and the like without touching a switch even if the position, shape, and the like of the switch are unknown.SOLUTION: A gesture of a user is photographed. The user's instruction through the gesture is recognized by analyzing the captured image data. Based on gesture recognition results, a power window device is operated according to the user's instruction. Features are selected from the captured image data and a feature vector that follows each feature is calculated. Feature vectors from the previous and next image data are compared, and features for which feature vectors are similar are detected and temporal movement of the features is grasped. The movement of features is calculated as movement vectors and chronological data that indicates temporal changes in movement vectors is analyzed using a predetermined determination threshold. Based on results of the analysis, the user's instruction through gesture is determined, and according to the instruction, the power window device is operated.SELECTED DRAWING: Figure 5

Description

The present invention relates to a device operation system capable of operating a switch of the device by a gesture operation of the device operator (movement of a body such as a gesture or a hand gesture), and is particularly applied to opening and closing a window glass in a vehicle such as an automobile. Regarding effective technology.

Normally, a window (door window) on the side of an automobile can be opened and closed, and a recent automobile is generally equipped with a power window device that automatically opens and closes the door window. This power window device uses an electric motor (hereinafter abbreviated as a motor) housed in a door panel as a drive source, and the motor is operated by a switch operation by a driver or a passenger to open and close a window. In this case, the opening/closing operation of the window is performed by operating an opening/closing switch attached to a door panel, a center console, or the like, but a switch arranged on the door panel is mainly used.

Japanese Utility Model Publication No. 6-51462

However, the opening and closing switch of the power window often has its mounting position, shape, and operation method different depending on the manufacturer and vehicle model. For this reason, when you are unfamiliar with the car or when it is difficult to see the switch at night, you may not be able to open and close the window as you do not know where the switch is. In particular, with the widespread use of rental cars and car sharing, there are more and more opportunities to drive other than a specific vehicle, and there is a problem that the user finds a different switch depending on the vehicle type and takes time to operate it. Further, when a plurality of switches such as a lock function are provided, it is often difficult to find the open/close switch itself.

An object of the present invention is to provide an apparatus operation system and an apparatus control method capable of opening and closing a window without touching the switch without knowing the position or shape of the switch.

An apparatus operation system of the present invention is an operation system for an operated apparatus that operates based on a switch operation by an apparatus user, and captures a drive unit for operating the operated apparatus and a gesture of the apparatus user. And a gesture recognition unit that analyzes image data captured by the photographing unit and recognizes an instruction of the device user by the gesture, and the device user based on a recognition result in the gesture recognition unit. And a control unit that controls the operation of the drive unit in accordance with the instruction of 1.

In the present invention, an image capturing unit such as a camera captures a gesture of a device user (for example, a body motion such as a hand movement), and a gesture recognition unit analyzes the captured image data, Recognize the device user's instruction by gesture. Then, based on the recognition result of the gesture recognition unit, the operation of the drive unit of the actuated device is controlled according to the instruction of the device user. Accordingly, the device user can perform a switch operation of the operated device such as a door window without touching the switch by a gesture of the device user. Therefore, even if the position and shape of the switch are not known, it is possible to operate the actuated device according to the intention of the user of the device.

In the device operation system, the gesture is a motion of a body (for example, a hand) of the device user, the device user instructs the operation of the actuated device by the motion of the body, and the imaging unit, The movement of the body of the device user is photographed, the gesture recognition unit recognizes an instruction of the device user from the movement of the body, and the control unit gives an instruction by the movement of the body of the device user. The operation of the drive unit may be controlled based on the above. Further, the control unit may control an operation form of the actuated device based on a movement direction of the gesture.

In addition, the gesture recognition unit, a feature point extraction unit that selects a plurality of feature points in the gesture from the image data, a feature vector calculation unit that calculates a feature vector associated with each feature point, shooting A characteristic vector tracking unit that compares the characteristic vectors of the plurality of image data at different time points, detects the characteristic points that the characteristic vectors approximate, and grasps the temporal movement of the characteristic points; A motion vector calculation unit that calculates the motion of the feature point grasped by the vector tracking unit as a motion vector of the feature point, and time-series data indicating a temporal change of the motion vector from the plurality of motion vectors Based on the analysis result of the motion vector processing unit, the motion vector analysis unit that analyzes the time-series data obtained by the motion vector processing unit using a predetermined determination threshold, and the motion vector analysis unit. It may be configured to include a gesture determination unit that determines an instruction by a gesture of the device user.

In this case, the motion vector analysis unit performs FFT analysis on each of the time-series data, acquires an amplitude value for each frequency of the motion vector, and the gesture determination unit determines a frequency range from the amplitude value. The gesture of the device user may be detected based on a predetermined threshold value of the amplitude, and the device user's instruction by the gesture may be determined. Further, the motion vector analysis unit performs FFT analysis on each of the time-series data, and the gesture determination unit detects the gesture of the device user based on the phase difference of the frequency having the maximum amplitude, and the gesture is detected. The instruction of the user of the device may be determined.

Further, the actuated device may be a power window device of an automobile, in which case the photographing unit photographs a gesture by an occupant of the automobile, and the gesture recognition unit detects the gesture of the occupant from the occupant. The controller controls the operation of the drive unit based on the window opening/closing instruction by the occupant's gesture, and opens/closes the window. Further, the gesture may be a movement of the occupant's hand, the occupant directs an opening/closing operation of a window of the automobile by the movement of the hand, and the photographing unit causes the movement of the hand of the device user to move. The gesture recognition unit recognizes an instruction for opening and closing the window by the occupant from the movement of the hand, and the control unit opens and closes the window according to the movement direction of the hand of the occupant. Control. Further, the control unit operates the drive unit so as to open the window when the occupant's hand makes a rotational movement in the first direction, and when the occupant's hand makes a rotational movement in the second direction. The drive may be actuated to close the window.

On the other hand, the device control method according to the present invention is a device control method for an actuated device that operates based on a switch operation by the device user, and captures a gesture of the device user and analyzes the captured image data. Thus, the instruction of the device user by the gesture is recognized, and the operated device is operated according to the instruction of the device user based on the recognition result of the gesture.

In the present invention, the gesture of the device user is photographed, the photographed image data is analyzed, and the instruction of the device user by the gesture is recognized. Then, based on the recognition result of the gesture, the operated device is operated according to the instruction of the device user. With this, a gesture of the device user (for example, movement of a hand) allows the switch operation of the operated device such as the door window to be performed without touching the switch. Therefore, even if the position and shape of the switch are not known, it is possible to operate the actuated device according to the intention of the user of the device.

In the device control method, the gesture may be a movement of a hand of the device user, and in this case, the device user instructs the operation of the actuated device by the movement of the hand, and the movement of the hand. By analyzing image data of the captured image, the instruction of the device user is recognized from the movement of the hand, and the operated device is operated based on the instruction of the device user recognized from the movement of the hand. Let

In addition, a plurality of feature points are selected from the image data, feature vectors associated with the respective feature points are calculated, the feature vectors of the plurality of image data having different imaging times are compared, and the feature vector Is detected, the temporal movement of the characteristic point is grasped, the movement of the characteristic point is calculated as the movement vector of the characteristic point, and the time of the movement vector is calculated from the plurality of movement vectors. Time series data indicating a dynamic change is created, the time series data is analyzed using a predetermined determination threshold value, and an instruction by a gesture of the device user is determined based on the analysis result of the motion vector analysis unit. You may do it.

In this case, FFT analysis is performed on each of the time-series data, the amplitude value for each frequency of the motion vector is acquired, and the amplitude value is obtained from the amplitude value based on a predetermined threshold value for the frequency range and the amplitude magnitude. The gesture of the device user may be detected, and the instruction of the device user due to the gesture may be determined. In addition, FFT analysis is performed on each of the time-series data, the gesture of the device user is detected based on the phase difference of the frequency having the maximum amplitude, and the instruction of the device user by the gesture is determined. Is also good.

Further, the operated device may be a power window device of an automobile, and the gesture may be a movement of a hand of an occupant of the automobile. In this case, the occupant may open and close a window of the automobile by the movement of the hand. Instructing, by analyzing the image data obtained by shooting the movement of the occupant's hand, recognize the instruction of the occupant from the movement of the hand, based on the instruction of the occupant recognized from the movement of the hand, the The power window device is operated to open and close the window.

According to the device operation system of the present invention, in the operation system of the operated device that operates based on the switch operation by the device user, the image capturing unit that captures the gesture of the device user and the image data captured by the image capturing unit. And a gesture recognition unit that recognizes an instruction of the device user by the gesture, and a control unit that controls the operation of the drive unit of the actuated device according to the instruction of the device user, based on the recognition result in the gesture recognition unit, Since the device is provided, the switch operation of the actuated device can be performed without touching the switch by the gesture of the device user. Therefore, even if the position and shape of the switch are not known, it is possible to operate the actuated device according to the intention of the user of the device.

According to the device control method of the present invention, in the device control method in the operated device that operates based on the switch operation by the device user, by capturing the gesture of the device user, by analyzing the captured image data, Since the device user's instruction by the gesture is recognized and the actuated device is operated according to the device user's instruction based on the recognition result of the gesture, the device user's gesture causes a switch operation of the actuated device. Can be done without touching the switch. Therefore, even if the position and shape of the switch are not known, it is possible to operate the actuated device according to the intention of the user of the device.

It is explanatory drawing which shows the structure of the power window apparatus using the apparatus operating system which is Embodiment 1 of this invention. It is a block diagram which shows the structure of the apparatus operating system which is Embodiment 1 of this invention. It is explanatory drawing which shows an example of the installation position of a camera. It is a block diagram which shows the structure of a gesture recognition apparatus. It is explanatory drawing which shows the flow of the analysis process in a gesture recognition apparatus. 6 is a data flow diagram showing the flow of analysis data in the processing of FIG. 5. It is explanatory drawing which shows the extraction process of a feature point. It is explanatory drawing regarding a feature point and its tracking process. It is explanatory drawing which shows an example of the result (the same (a)) which carried out the FFT analysis of the time series data (FIG.9(b)) of a motion vector. It is a block diagram which shows the structure of the apparatus operating system which is Embodiment 2 of this invention. It is a block diagram which shows the structure of a gesture stop recognition apparatus. It is explanatory drawing regarding the tracking process of the feature point in a gesture stop recognition apparatus, and subsequent gesture stop determination. It is explanatory drawing which shows the flow of the analysis process in the gesture determination apparatus in the apparatus operation system of FIG. It is explanatory drawing which shows another example of the installation position of a camera.

(Embodiment 1)
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is an explanatory diagram showing the configuration of a power window device (actuated device) using the device operation system according to the first embodiment of the present invention, and the method of the present invention is also implemented by the power window device of FIG. To be done. The power window device 1 of FIG. 1 is a device for opening and closing a window (door window) 3 of a door 2 provided on a side surface of an automobile, and is driven by a motor (drive unit) 4 for opening and closing the window. In the following description, the door window on the driver's seat side will be described as an example, but the system can also be applied to windows on the passenger seat side and rear seats.

In the power window device 1, an X-arm type window regulator 5 is arranged between the door window 3 and the motor 4, and when the motor 4 is operated, the window regulator 5 moves the door window 3 up and down to open the window opening 6 Is opened and closed. In this case, the operation of the door window 3 can be operated by the power window opening/closing switch 8 arranged on the door panel 7 as in the case of a normal automobile, but in the power window device 1, the passenger (user: device user) can operate. The door window 3 can be opened and closed by a gesture motion (a motion of the body such as a motion of a hand). Here, in the case of the door located on the left side when the occupant in the vehicle is looking in the front direction of the vehicle, when the user draws a circle in the left (counterclockwise) direction with the finger, the door window 3 opens and the right ( The door window 3 is set to close when a circle is drawn in the clockwise direction.

FIG. 2 is a block diagram showing a configuration of a device operation system 10 (hereinafter, appropriately abbreviated as system 10) used in the power window device 1. As described above, in the system 10, the movement of the door window 3 can be controlled by the movement (gesture) of the occupant (here, the driver). Therefore, as shown in FIG. 2, the system 10 includes a camera (imaging unit) 11 that captures the movement of the driver's hand and a gesture recognition that recognizes and analyzes the gesture of the driver using the image of the camera 11. And a device (gesture recognition unit) 12. The analysis result of the gesture recognition device 12 is sent to the window opening/closing control device (control unit) 13 as user operation data indicating the user's intention regarding the opening/closing operation of the window.

On the other hand, the power window device 1 is provided with a power window opening/closing switch 8 (hereinafter abbreviated as switch 8) as in the conventional case, and operates based on a switch operation by a user. The state of the switch 8 (open/closed window) is also sent to the window opening/closing control device 13 as user operation data. The window opening/closing control device 13 operates the motor 4 and raises/lowers the door window 3 based on the user operation data input from the gesture recognition device 12 or the switch 8.

The camera 11 of the system 10 constantly monitors the operation of the occupant, and when the occupant gives a sign that the window is to be opened/closed, the camera 11 captures the gesture and sends it to the gesture recognition device 12. In the system 10, the camera 11 is installed at a position where an object to be photographed (here, the occupant's hand) can always be photographed so that the gesture of the occupant can be reliably grasped, and a place about 300 to 500 mm away from the hand. At, it is possible to take an image of a range of approximately 600 to 800 mm in diameter.

FIG. 3 is an explanatory diagram showing an example of the installation position of the camera. As shown in FIG. 3, in the present embodiment, the camera 11 is installed at the front end of the door 2 (on the front side of the window opening 6 where the tweeter of the car audio or the like is arranged), and the occupant's hand is present. A photograph is taken between the handlebar 9 and the door 2, which is likely to occur. Further, the camera 11 shoots at a frame rate (about 30 fps to 60 fps) necessary for recognizing a gesture, and the image data is sent to the gesture recognition device 12.

The gesture recognition device 12 recognizes a gesture of an occupant based on the image data from the camera 11. Image data of an occupant's hand is sequentially input from the camera 11 to the gesture recognition device 12, and points called “feature points” that are easy to track are extracted from the input image. Then, the movement of the feature point is analyzed to determine the movement of the occupant's hand, and the intention of the user indicated by the movement is sent to the window opening/closing control device 13 as user operation data. FIG. 4 is a block diagram showing the configuration of the gesture recognition device 12 that performs such recognition/analysis processing.

As shown in FIG. 4, the gesture recognition device 12 is provided with a feature point extraction unit 21 that selects “feature points” from image data, and a feature vector calculation unit 22 that calculates a feature vector associated with each feature point. There is. Further, in the subsequent stage of the feature vector calculation unit 22, a feature vector tracking unit 23 that compares the feature vectors and detects the movement of the feature point, and a movement of the feature point (change in position) detected by the feature vector tracking unit 23. A motion vector calculation unit 24 is provided for calculating as the motion vector of each feature point.

Further, the gesture recognition device 12 uses a motion vector processing unit 25 that calculates time-series data regarding a temporal change of a motion vector, and time-series data obtained by the motion vector processing unit 25 using a predetermined determination threshold. A motion vector analysis unit 26 for performing analysis according to the above, and a gesture determination unit 27 for determining the meaning (window open/window closed) of the occupant's gesture based on the analysis result of the motion vector analysis unit 26. It should be noted that the gesture recognition device 12 uses a RAM 28 that stores a feature vector and a motion vector of each feature point, time-series data of the motion vector, etc. There is also provided a ROM 29 that stores the determination threshold value and the like.

In such a device operation system 10, the user's gesture is grasped and the door window 3 is operated as follows. FIG. 5 is an explanatory diagram showing a flow of analysis processing in the gesture recognition device 12, and FIG. 6 is a data flow diagram showing a flow of analysis data in the gesture recognition device 12. As shown in FIG. 5, in the gesture recognition device 12, first, the “feature point” P is extracted by the feature point extraction unit 21 from the image data sent from the camera 11 (step S1). The feature points P are selected based on the contrast (brightness/darkness) difference in the image such that the edge portion and the corner portion of the image are selected according to the index of the characteristic. At that time, a predetermined selection threshold for brightness difference or the like is used as an index of the feature-likeness, and about 300 to 500 feature points P are extracted at the contour portion of the hand or the fingertip as shown in FIG. 7 (in FIG. 7, , Only representative points are shown for simplicity).

The characteristic point P extracted by the characteristic point extraction unit 21 is associated with a unique “feature vector”, and the characteristic vector calculation unit 22 calculates the characteristic vector from the data of each characteristic point P (step S2). In this case, the feature vector represents the feature quantity of the feature point in the form of a vector. For example, the document [JRM Vol.28 No.6 pp. 887-898 “Autonomous Mobile Robot Navigation Using Scene Matching with Local Features”] is a vector in which the original information of each feature point is not lost as much as possible. Here, with the direction that changes from light to dark as the reference direction, one feature point is divided into 4×4 areas, and the total sum of brightness changes in the X and Y directions (Σdx, Σdy) and the amount of change The sum of absolute values of (Σ|dx|, Σ|dy|) is vectorized, and there are 4×4×4=64 feature vectors at each feature point. The feature vector calculation unit 22 calculates 64 feature vectors of each feature point, and these are stored in the RAM 28 (step S3).

After calculating the feature vector and storing it in the RAM 28, this time, for the image data sent from the camera 11, the feature point P is extracted and the feature vector is calculated and stored in the RAM 28 ( Steps S1→S3). After the feature vectors are obtained in this way for a plurality of image data at different time points, the feature vector tracking unit 23 performs image data for the previous frame (FIG. 8A) and image data for the current frame (FIG. 8). Compare (b)). The feature vector tracking unit 23 mutually compares the feature points P extracted from the image data of the previous frame and the feature data of the current frame, searches for a set having the closest feature vector, and combines them (tracking process: step S4). .. At this time, for each of the feature points in the previous frame image, each feature vector included in them is compared with the feature vectors of all the feature points in the current frame image. That is, as shown in FIG. 8C, the point P1 in FIG. 8A is compared with all the feature points in FIG. 8B, and at this time, the feature point among the feature points in FIG. The point P2 having the closest vector is detected, and both are combined. As a result, the temporal movement of each feature point P (previous frame→current frame) is grasped. In this way, the feature vector tracking unit 23 analyzes the image data moment by moment and performs the feature vector tracking process. Thereby, the gesture of the user is sequentially analyzed, and the recognition accuracy of the gesture is improved.

When the tracking succeeds in the images before and after, the position change of the combined feature point P is calculated for each set by the motion vector calculation unit 24 to create a “motion vector” (motion vector sequence) (FIG. 8(d): step S5). The motion vector thus obtained is statistically processed by the motion vector processing unit 25, and those having a small motion and those having an extremely large motion are removed by setting a threshold value. Then, the average of the remaining motion vectors is calculated and used as the representative value of the motion vector at that time (step S6), and this is successively stored in the RAM 28 as the average vector (step S7).

When the process of step S6 is executed every time an image is sequentially input, the temporal change of the motion vector can be known. Therefore, the motion vector processing unit 25 creates temporal change data (time series data) of the motion vector (average vector) as shown in FIG. 8E from the average vector accumulated in the RAM 28. At that time, when the camera 11 is shooting at a frame rate of 30 fps, when the average vectors for 1 second are collected, time-series data of 30 average vectors is obtained.

After creating the time-series data of the motion vector in this way, the motion vector analysis unit 26 performs frequency analysis (eg, FFT (Fast Fourier Transform) analysis is performed (step S8). By the FFT analysis, the amplitude value for each frequency according to time and the number of samples as shown in FIG. 9A is obtained. It should be noted that when the hand is moved in a circle, the X and Y components of the motion vector become sinusoidal with the phase shifted by 90° as shown in FIG. 9B, and the motion can be easily identified. Therefore, the system 10 employs a circular motion gesture (hand-turning motion) for the user's instruction.

After the analysis by the motion vector analysis unit 26, the gesture determination unit 27 determines, from the analysis result, a predetermined threshold value for each item such as the magnitude of the amplitude of the motion vector, the frequency range, and the phase difference between the X component and the Y component. And the meaning of the gesture motion (user's intention) is determined (step S9). In this case, the following determination thresholds are set for the items (1) amplitude, (2) frequency range, and (3) phase difference, and the gesture determination is a point system. The gesture determination unit 27 adds the points set to the respective threshold values, and when it exceeds a predetermined reference value, determines that the user has issued an open/closed sign. These thresholds and points are stored in the ROM 29.

<<Example of discrimination threshold>>
(1) Amplitude-Magnitude of amplitude First lower limit: 150 [pix/dt]→2pt
Second magnitude lower limit of amplitude: 100 [pix/dt]→1 pt
(2) Frequency range ・Lower frequency limit: 0.5 [Hz]
・Upper limit of frequency: 3.1 [Hz]
→ 2pt if the upper and lower limits are entered and the frequencies of the X and Y components are the same
→ 1 pt if both X and Y component frequencies fall within the upper and lower limits
(3) Phase difference (X component and Y component)
First phase difference range: 60-120 [deg]→2pt
Second range of phase difference: 30-150 [deg]→1pt

(1) Amplitude In the discrimination threshold, the magnitude of the amplitude corresponds to the range of movement of the user's hand, and in the case of the hand-turning operation, there are peak values for both the X component and the Y component (when the hand is slid, If it is, the peak value of the component different from the sliding direction is hard to appear). The value of the amplitude corresponds to the diameter of the circle. In this case, the state in which the amplitude is less than 100 [pix/dt] means that the movement of the user is small, and since the noise also increases, it is determined that there is no sign of the user. When the magnitude of the amplitude is 100 to 150, the movement of the user's hand is large to some extent, and there is a possibility of circular movement, but there is also a possibility that it is not a sign. On the other hand, when it exceeds 150, it can be determined that the circular motion is a sign. Therefore, as an evaluation point, 1 pt is given when the amplitude is 100 to 150, and 2 pt is given when the amplitude exceeds 150.

(2) Frequency range Next, the frequency range corresponds to the speed of movement of the user's hand. When the frequency at which the amplitude shows the peak value is higher than the upper limit, the sign is too fast, and when the frequency is lower than the lower limit, the hand is moving. Too late. Therefore, if the upper and lower limits are exceeded, it is unlikely that it is the user's signature. On the other hand, if it falls within the range from the upper limit to the lower limit, it is highly likely that it is the user's signature. Therefore, 1 pt is given when the frequency falls between the upper limit and the lower limit. Further, in the case of a hand-turning operation, if it is an ideal circular movement, it is highly possible that the X component and the Y component have the same frequency. Therefore, when the frequency is in the upper limit to the lower limit and the X component and the Y component have the same value, 2pt is given. In FIG. 9A, there is a peak value of amplitude near 2 Hz, which indicates a state in which the user is issuing a sign of a circular motion of about two revolutions per second.

(3) Phase Difference The phase difference between the X component and the Y component shown in FIG. 9B is 90 degrees if the user's sine is a perfect circular motion. Therefore, if it is in the first range, it is determined to be circular motion and it is 2 pt, and if it is in the second range, there is room for doubt, so it is set to 1 pt. Then, the gesture determination unit 27 adds the points of each item and determines whether the three elements are satisfied. That is, when 5 pt is acquired, it is determined to be “with gesture: circular motion”. On the other hand, when it is less than 5 pt, it is determined that there is a gesture, but it is not the user's signature for opening and closing the window, and “no operation” is determined. When determining the phase difference, it is preferable to use the phase difference of the motion vector of the frequency having the maximum amplitude (2 Hz in FIG. 9A) from the viewpoint of ease of calculating the phase difference.

When the gesture determination unit 27 determines that the movement of the user's hand is a circular movement from the analysis result of the motion vector, the gesture determination unit 27 further determines the movement direction based on the phase difference. Then, it determines whether the sign is open or closed, and sends it to the window opening/closing control device 13 as user operation data. That is, when the gesture determination unit 27 determines that the movement of the user's hand is left rotation (first direction), the gesture determination unit 27 determines that the user operation data “open window” is right rotation (second direction). Outputs the user operation data of "window closing" to the window opening/closing control device 13. In the case of "no operation", no signal is output, as in the case where the switch 8 is not operated.

In this way, the gesture recognition device 12 analyzes the image data captured by the camera 11, recognizes the user's instruction, and outputs the user operation data. Then, the window opening/closing control device 13 that has received this drives the motor 4 based on the recognition result of the gesture recognition device 12. As a result, the door window 3 is moved up and down according to the user's gesture instruction, and the window opening/closing operation is performed.

As described above, in the device operating system 10 of the present invention, the door window 3 can be freely opened and closed by the gesture of the user (hand movement in this case). Specifically, the window can be opened and closed by making one turn of the finger so as to draw a circle having a diameter of about 15 cm. Therefore, even if the position, shape, etc. of the power window opening/closing switch are not known, the window opening/closing operation can be performed according to the user's intention without touching the switch. Therefore, it is possible to open and close the window with a certain gesture even when driving cars of different manufacturers and car types such as car rental and car sharing, and it is easy to open and close the window even in an unfamiliar car or in the dark. it can. Further, for example, even a person whose hands are dirty by eating at the passenger seat can open and close the window.

Further, for example, even a person who has a difficulty in operating a conventional operation switch, such as a hand or finger obstacle, can open and close the window. In addition, since the switch operation is simple and convenient, it is possible to open and close the window without taking the driver's attention, which improves safety during driving. It should be noted that the gesture of the user is not limited to the movement of the hand, and the device can be operated according to the movement of the body of other parts of the user such as the head, legs, and arms. In this case, since even a person with a hand or foot disability can operate various devices, the device operation system of the present invention can be applied as an operation interface of a welfare device.

(Embodiment 2)
Next, as a second embodiment of the present invention, a system will be described in which the window is opened and closed only while the user is issuing a sign to open and close the window, and the door window can be stopped at an arbitrary position. FIG. 10 is a block diagram showing the configuration of the device operation system according to the second embodiment of the present invention. In the second embodiment, the same parts, members, processes and the like as those in the first embodiment are designated by the same reference numerals as those in the first embodiment and the description thereof is omitted.

As shown in FIG. 10, in the device operation system 30 (hereinafter, appropriately abbreviated as system 30) of the second embodiment, the gesture recognition device 12 and the gesture stop recognition device are provided between the camera 11 and the window opening/closing control device 13. A gesture determination device 32 including 31 is provided. FIG. 11 is a block diagram showing the configuration of the gesture stop recognition device 31, and a gesture stop operation determination unit 33 is provided instead of the gesture determination unit 27 (see FIG. 4) in the gesture recognition device 12. It should be noted that the gesture stop recognition device 31 and the gesture recognition device 12 share a part except the gesture stop operation determination unit 33 and the gesture determination unit 27, and the gesture determination device 32 processes the gesture stop and the gesture recognition at the same time. ..

In the gesture stop recognition device 31, the motion vector statistically processed by the motion vector processing unit 25 is analyzed by the motion vector analysis unit 26 and the gesture stop operation determination unit 33, and the opening/closing sign of the user is stopped. When it is determined that the movement of the hand has stopped), the operation stop data is output to the window opening/closing control device 13. The window opening/closing control device 13 determines the operation of the window based on the operation data input from the gesture recognition device 12 and the operation stop data input from the gesture stop recognition device 31, and controls the power supplied to the motor 4. , Activate or deactivate the door window 3.

In such a system 30, the stop of the gesture is determined as follows. In this case, also in the system 30, the characteristic point P of the user's hand is extracted from the image data of the camera 11 and tracking processing is performed by the same processing as the system 10 of the first embodiment to create a motion vector. Then, after statistically processing this motion vector, the absolute value sum of the motion vectors is calculated and compared with a predetermined threshold value. If the sum total is less than the threshold value, it is determined that the opening/closing sign has stopped. FIG. 12 is an explanatory diagram relating to the feature point and its tracking process in the gesture stop recognition device 31 at that time, and a gesture stop determination after that, and FIG. 13 is an explanatory diagram showing a flow of the analysis process in the gesture determination device 32.

In the system 30, the gesture stop recognition device 31 also extracts the “feature point” P from the image data sent from the camera 11 by the feature point extraction unit 21 similarly to the gesture recognition device 12 (step S1). After the characteristic points P are extracted by the characteristic point extracting unit 21, the characteristic vector calculating unit 22 calculates a "characteristic vector" unique to each characteristic point P from the data of each characteristic point P (step S2). The calculated feature vector is stored in the RAM 28 (step S3). After storing the current feature vector in the RAM 28, the feature point P is extracted and the feature vector is calculated for the next image data, and the extracted feature vector is stored in the RAM 28 (steps S1→S3).

After obtaining the feature vector for a plurality of image data at different time points, the feature vector tracking unit 23 performs the image data of the previous frame (FIG. 12A) and the image data of the current frame (FIG. 12B). 12C, as shown in FIG. 12C, a set having the closest feature vector is searched (for example, point P1 and point P2) and combined (tracking process: step S4). At this time, similar to the first embodiment, the feature vectors of all feature points are compared. As a result, the temporal movement of each feature point P (previous frame→current frame) is grasped. In this way, the feature vector tracking unit 23 analyzes the image data moment by moment and performs the feature vector tracking process.

When the tracking succeeds in the images before and after, the position change of the combined feature point P is calculated for each set by the motion vector calculation unit 24 to create a “motion vector” (motion vector sequence) (FIG. 12(d): Step S5). The motion vectors are statistically processed by the motion vector processing unit 25, and those having a small motion or those having an extremely large motion are removed by setting a threshold value (step S6).

In the gesture stop recognition device 31, the motion vector analysis unit 26 obtains the X component and the Y component of each of the remaining motion vectors statistically processed in this manner, and the sum of the two (sum of absolute values ΣV=Σ |dX|+Σ|dY|) is obtained (FIG. 12(e): step S11). Thereby, the magnitude of the movement of the user's hand is grasped as a numerical value. Then, the gesture stop operation determination unit 33 compares the total sum ΣV with a predetermined threshold Th (FIG. 12(f): step S12). When the total sum ΣV is less than the threshold Th (ΣV<Th), the user's gesture is It is judged that the user has stopped, that is, the user's opening/closing sign is stopped, and the operation stop data is sent to the window opening/closing control device 13. When the operation stop data is input, the window opening/closing control device 13 stops the motor 4, thereby stopping the opening/closing operation of the door window 3.

On the other hand, if the total sum ΣV is equal to or greater than the threshold Th (ΣV≧Th) in step S12, it is determined that the user's gesture has not stopped, and the process proceeds to step S7. In that case, the processing from step S7 onward is performed by the gesture recognition device 12 similarly to the system 10 of the first embodiment, and the operation data is sent to the window opening/closing control device 13.

As described above, in the system 30 according to the second embodiment, the gesture stop recognition device 31 can recognize that the gesture of the user has stopped. Therefore, the window opening/closing operation can be performed only while the user is gesturing. In this respect, the system 10 according to the first embodiment operates until the window is completely opened and closed when the power window device is operated based on the recognition result of the gesture, but in the system 30, the window is stopped during the opening and closing operation. It is also possible to make fine adjustments to the opening and closing positions of the windows. Further, in addition to the full opening/closing position of the window, it is possible to open/close the window at an intermediate position.

It is needless to say that the present invention is not limited to the above-mentioned embodiment and can be variously modified without departing from the gist thereof.
In the above-described embodiment, the data after the FFT analysis is determined by the threshold value stored in the ROM 29 in advance, but the neural network determination by AI (Artificial Intelligence) of supervised learning may be performed. In this case, for example, the image data of the opening/closing gesture operation for a plurality of people (such as 50 people), the FFT analysis result, the image data other than the opening/closing gesture and the FFT analysis result are stored in the ROM 29, and the gesture discrimination unit 27 is made to learn the pattern. To make gesture determination.

Further, the installation position of the camera 11 is not limited to the front end portion of the door 2 as shown in FIG. 3, and the lower end portion of the window opening 6 (FIG. 14) and the door panel 7 may be provided as long as the gesture of the occupant can be detected. It can also be installed on the upper edge, armrests, instrument panel, etc. Furthermore, the configuration of the power window device 1 is not limited to the X arm type as shown in FIG.

On the other hand, the gesture motion of the user may be a reciprocating motion such as up and down or left and right instead of a hand-turning motion, and in this case, analysis/discrimination using FFT is also possible. It is also possible to change the open/closed state depending on the hand-turning speed. For example, when the user is turning the hand quickly, the full-open/full-close (auto) mode may be set. Further, it is possible to add a function of stopping the window at the position pointed by the user. In addition to the gesture operation, it is possible to use an instruction by voice recognition together.

The device operation system according to the present invention can be applied to not only the power window device of an automobile, but also an air conditioner or a car audio as an actuated device that operates based on a switch operation. In this case, the switches of the air conditioner and car audio can be driven by a motor, and the gesture is used to instruct up and down of temperature and volume (for example, manual rotation of right rotation is up, left rotation is down). Is also possible. Further, it is also possible to apply an electric sunroof device, an electric slide door, an electric rear gate, an electric reclining device, a wiper device, and the like to the actuated device, and apply the present invention to the operation thereof.

1 power window device 2 door 3 door window 4 motor 5 window regulator 6 window opening 7 door panel 8 power window opening/closing switch 9 handle 10 device operation system 11 camera 12 gesture recognition device 13 window opening/closing control device 21 feature point extraction unit 22 feature vector Calculation unit 23 Feature vector tracking unit 24 Motion vector calculation unit 25 Motion vector processing unit 26 Motion vector analysis unit 27 Gesture determination unit 28 RAM
29 ROM
30 device operation system 31 gesture stop recognition device 32 gesture determination device 33 gesture stop operation determination unit P feature point

Claims (15)

An operating system for an operated device, which operates based on a switch operation by a device user,
A drive unit for operating the actuated device,
A photographing unit for photographing the gesture of the device user,
A gesture recognition unit that analyzes image data captured by the image capturing unit and recognizes an instruction of the device user due to the gesture;
A device operation system comprising: a control unit that controls the operation of the drive unit according to an instruction from the device user based on a recognition result of the gesture recognition unit. The device operating system according to claim 1,
The gesture is a movement of the device user's body, and the device user instructs movement of the actuated device by the movement of the body,
The photographing unit photographs the movement of the body of the device user,
The gesture recognition unit recognizes an instruction of the device user from the movement of the body,
The device operation system, wherein the control unit controls the operation of the drive unit based on an instruction by the body movement of the device user. The device operating system according to claim 1 or 2,
The device operation system, wherein the control unit controls an operation form of the actuated device based on a movement direction of the gesture. The device operation system according to any one of claims 1 to 3,
The gesture recognition unit,
A feature point extraction unit that selects a plurality of feature points from the image data,
A feature vector calculation unit that calculates a feature vector associated with each feature point,
A feature vector tracking unit that compares the feature vectors of the plurality of image data at different image capturing times, detects the feature points that the feature vectors approximate, and grasps the temporal movement of the feature points.
A motion vector calculation unit that calculates a motion of the feature point grasped by the feature vector tracking unit as a motion vector of the feature point;
A motion vector processing unit that creates, from a plurality of the motion vectors, time-series data indicating temporal changes in the motion vectors;
A motion vector analysis unit that analyzes the time-series data obtained by the motion vector processing unit using a predetermined discrimination threshold;
A gesture determination unit that determines an instruction by a gesture of the device user based on an analysis result of the motion vector analysis unit, the device operation system. The device operation system according to claim 4,
The motion vector analysis unit performs FFT analysis on each of the time-series data to obtain an amplitude value for each frequency of the motion vector,
The gesture determination unit detects a gesture of the device user from the amplitude value based on a predetermined threshold value for a frequency range and a magnitude of the amplitude, and determines an instruction of the device user by the gesture. Device operation system characterized by. The device operation system according to claim 4,
The motion vector analysis unit performs FFT analysis on each of the time series data,
The device operation system, wherein the gesture determination unit detects a gesture of the device user based on a phase difference between frequencies having a maximum amplitude, and determines an instruction of the device user based on the gesture. The device operation system according to any one of claims 1 to 6,
The operated device is a power window device of an automobile,
The photographing unit photographs a gesture by an occupant of the automobile,
The gesture recognition unit recognizes an instruction from the occupant regarding opening and closing of a window from a gesture of the occupant,
The apparatus operation system, wherein the control unit controls the operation of the driving unit based on an instruction to open/close the window by the occupant's gesture to open/close the window. The device operation system according to claim 7,
The gesture is the movement of the occupant's hand, and the occupant instructs the opening/closing operation of the window of the automobile by the movement of the hand,
The photographing unit photographs the movement of the hand of the device user,
The gesture recognition unit recognizes an instruction for opening and closing the window by the occupant from the movement of the hand,
The apparatus operation system, wherein the control unit controls the opening/closing operation of the window according to the movement direction of the hand of the occupant. The device operation system according to claim 8,
The control unit operates the drive unit to open the window when the occupant's hand makes a rotational movement in the first direction, and the window when the occupant's hand makes a rotational movement in the second direction. A system for operating a device, characterized in that the drive is actuated to close the. A method for controlling an actuated device that operates based on a switch operation by a device user, comprising:
Taking a picture of the gesture of the device user,
By analyzing the captured image data, the instruction of the device user by the gesture is recognized,
A device control method comprising operating the actuated device in accordance with an instruction from the device user based on the recognition result of the gesture. The device control method according to claim 10,
The gesture is a movement of the device user's body, and the device user instructs the operation of the actuated device by the movement of the body,
By analyzing the image data obtained by photographing the movement of the body, the instruction of the device user is recognized from the movement of the body,
A device control method comprising operating the actuated device based on an instruction from the device user recognized from the movement of the body. The device control method according to claim 10 or 11,
Select a plurality of feature points from the image data,
Calculating a feature vector associated with each of the feature points,
Comparing the feature vectors of the plurality of image data at different shooting times, detecting the feature points to which the feature vectors approximate, grasping the temporal movement of the feature points,
Calculating the motion of the feature point as a motion vector of the feature point,
From a plurality of the motion vectors, to create time series data showing the temporal change of the motion vector,
Analyzing the time series data using a predetermined discrimination threshold,
An apparatus control method, comprising: determining an instruction by a gesture of the apparatus user based on an analysis result of the motion vector analysis unit. The device control method according to claim 12,
FFT analysis is performed on each of the time series data to obtain an amplitude value for each frequency of the motion vector,
From the amplitude value, a device control characterized by detecting a gesture of the device user based on a predetermined threshold value for a frequency range and a magnitude of the amplitude, and determining an instruction of the device user by the gesture. Method. The device control method according to claim 12,
FFT analysis is performed on each of the time series data,
A device control method comprising detecting a gesture of the device user based on a phase difference between frequencies having the maximum amplitude, and determining an instruction of the device user due to the gesture. The device control method according to any one of claims 10 to 14,
The operated device is a power window device of an automobile,
The gesture is a movement of a hand of an occupant of the automobile, and the occupant instructs the opening/closing operation of a window of the automobile by the movement of the hand,
By analyzing the image data of the movement of the occupant's hand, the instruction of the occupant is recognized from the movement of the hand,
A device control method comprising operating the power window device to open and close the window based on an instruction from the occupant recognized from the movement of the hand.

Images