KR20120089992A - Gesture recognition device, gesture recognition method, computer readable recording medium recording control program - Google Patents

Abstract

PURPOSE: A gesture recognition apparatus, a method thereof, and computer readable recording medium are provided to implement a non-contact type input function of high process or high accuracy regardless of brightness of a used place by recognizing a gesture of hands based on a moving path of a variation area. CONSTITUTION: A temperature sensor is composed of infrared sensors. A variation specific unit(32) specifies a variation area whether temperature varies as a hand area based on temperature which the infrared sensors sense. A gesture recognizing unit(34) recognizes a gesture of hands. A center of gravity determining unit(33) determines the center of gravity of the variation area. The recognizing unit traces the center of gravity.

Description

Gesture recognition device, gesture recognition method, computer-readable recording medium recording control program {GESTURE RECOGNITION DEVICE, GESTURE RECOGNITION METHOD, COMPUTER READABLE RECORDING MEDIUM RECORDING CONTROL PROGRAM}

The present invention relates to a gesture recognition apparatus for recognizing a gesture, a gesture recognition method, a control program and a recording medium.

Background Art Conventionally, input functions for electronic devices such as mobile phones, PDAs (Personal Digital Assistants), televisions, recording / playback devices, personal computers, digital cameras, and the like have been realized in various forms. As a method of realizing an input function, first, there is a method of performing input by bringing an object (a finger, a touch pen, etc.) into contact with an input device (button, touch panel) (for example, Patent Documents 1, 2, etc.). Secondly, there is a method in which an input device (a camera or the like) is recognized to input an object in a non-contact manner to an input device or an electronic device (for example, Patent Document 3).

In the input contact realization method of the first contact type, various inputs can be made to the electronic device reliably and immediately, but there are the following problems.

If the screen (touch panel) is too large, it is difficult to bring the object (finger, etc.) into contact from the end of the screen to the end, and a decrease in operability becomes a problem. It is inconvenient because when the screen is viewed or used away from the electronic device and the operation is required for the electronic device, it must be brought close to a place where the electronic device can be operated. Since fingerprints or contaminants on the screen are attached to the screen, it is an unsuitable input method for users who are concerned about screen contamination. This is inconvenient because the hands are wet, contaminated or cannot be operated without touching the screen at one time.

In addition, in an electronic device having a projector function, a use scene is assumed that allows the electronic device to stand still and projects display contents (slideshows, etc.) on the screen, but turns a slide (page), etc. If you touch the screen when you want to perform the operation, it is inconvenient because the image of the screen is shaken because the position of the electronic device placed in the proper position is shifted. In addition, it is cumbersome for a user presenting by the screen to move to operate an electronic device.

In a use case in which a contact type input method is inadequate where the problem described above occurs, a second non-contact type input realization method such as Patent Document 3 has been adopted as the input function of an electronic device. In the technique of Patent Document 3, a gesture recognition system is disclosed in which image data of a hand captured by a camera is image-processed to recognize a hand and detect the movement (hand gesture) of the recognized hand. By recognizing the various hand gestures on the electronic device, it is possible to perform operation input on the electronic device without touching the electronic device.

Japanese Patent Application Publication No. 2009-071708 (published April 2, 2009) Japanese Patent Application Publication No. 2009-168647 (published July 30, 2009) Japanese Patent Application Laid-Open No. 08-315154 (published November 29, 1996)

However, when the contactless input method is realized by the technique of Patent Document 3 described above, the following problems arise.

For example, in a dark place (indoors where lighting is dimmed, for example, outdoors at night, during projector startup, etc.), photographing of a subject (hand) by a camera is difficult, and image data that can be recognized as a hand cannot be obtained. . Therefore, the system of patent document 3 has a problem that it cannot apply to the input device of the electronic device used in a dark place. Moreover, in the system of patent document 3, in order to handle the image picked up by the camera, it is necessary to perform image processing with high processing load. Therefore, there exists a problem that a highly functional information processing apparatus is needed, or processing time becomes long. Further, in order to recognize the shape of the hand, a process of distinguishing the background image from the image of the hand is performed, but when the texture similar to the hand is included in the background image, the shape (area) of the hand cannot be extracted accurately, Error recognition of gestures is likely to occur.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems, and an object thereof is to control a gesture recognition device, a gesture recognition method, and a gesture recognition device that realize a high-speed processing and a high accuracy non-contact input function regardless of the brightness of the place of use. There is provided a recording medium on which a program and the control program are recorded.

In order to solve the above problem, the gesture recognition apparatus of the present invention includes a temperature sensor in which a plurality of infrared sensors are disposed, and a change region in which a temperature change occurs based on a temperature detected by each of the infrared sensors of the temperature sensor, A change area specifying means for specifying as an area representing a hand and the change area specifying means are provided with gesture recognition means for recognizing a movement trajectory of a specific change area and recognizing a gesture of a hand.

According to the above configuration, in the gesture recognition apparatus, information for detecting the movement of an object (hand) is obtained as temperature information from a plurality of infrared sensors arranged. Then, the change area specifying means specifies the change area where the temperature change has occurred as the area indicating the hand, and the gesture recognizing means specifies the movement trajectory of the specified change area to recognize the gesture of the hand.

In this way, the gesture recognizing apparatus can analyze the movement of the object only by the temperature information, even if the data amount does not handle the massive image data, and it becomes possible to recognize the gesture. As a result, in the gesture recognition apparatus, it is possible to realize an improvement in processing efficiency and a reduction in processing load. That is, it is possible to realize an input device with a fast reaction speed.

Moreover, the gesture recognition apparatus of this invention detects the movement of an object by the difference of the temperature of an object and the surrounding environmental temperature by acquiring the surface temperature of an object using an infrared sensor. Therefore, in the configuration of recognizing a gesture by imaging an object with an optical system camera, use in a dark place is limited, but the gesture recognition device of the present invention can function as an input device in a dark place without limitation.

In addition, in gesture recognition using an optical camera, when there is no large difference between the texture of the object and the background, there is a problem that the position, shape, etc. of the object cannot be accurately recognized, and error recognition of the gesture increases. On the other hand, since the gesture recognition apparatus of the present invention detects an object (hand, etc.) having a surface temperature different from the ambient environment temperature based on the temperature information, the problem of error recognition due to similarity of texture cannot occur.

As a result, the gesture recognition apparatus of the present invention has an effect that it is possible to realize a high-speed processing and a highly accurate non-contact input function regardless of the brightness of the use place.

The gesture recognition apparatus further includes a center of gravity determining means for determining the center of gravity of the change area specified by the change area specifying means, wherein the gesture recognizing means tracks the position of the center of gravity for the movement trajectory. It is desirable to specify.

This makes it possible to specify the movement trajectory of the change area in a simplified processing procedure.

The gesture recognition apparatus further includes change amount calculating means for calculating a temperature change amount for each infrared sensor by comparing respective temperatures detected by each infrared sensor with a predetermined reference temperature, and the center of gravity determining means includes infrared rays. The center of gravity of the change area may be determined based on the temperature change amount obtained for each sensor.

According to the above configuration, the center of gravity of the change area is obtained by considering the amount of temperature change, not the center of gravity determined only based on the geometry of the change area.

By tracking the trajectory of the center of gravity thus obtained, it is possible to more accurately detect the actual hand movement and improve the accuracy of gesture recognition.

The gesture recognition apparatus further includes a mask storage unit that stores a mask indicating a valid invalid placement pattern in the plurality of infrared sensors in association with the type of application of an electronic device to which the device itself connects, and the electronic device. Is provided with mask selection means for selecting a mask associated with an application during startup from the mask storage portion, and the change area specifying means has a temperature detected by an infrared sensor valid for the mask selected by the mask selection means. The change area may be specified based on the bay.

When the infrared sensor is selectively used, the amount of information to be processed can be reduced to increase the processing speed, and the recognition accuracy can be improved when a specific gesture is recognized.

The gesture recognition apparatus further includes deviation calculation means for calculating a deviation of the change area specified by the change area specifying means, wherein the gesture recognition means is based on the increase and decrease of the deviation with the passage of time. You may recognize a gesture.

The gesture recognizing means recognizes that when the deviation of the change area increases, both hands or fingers move in the direction of spreading, and when the deviation of the change area decreases, the gesture recognition means moves in the direction of gathering both hands or fingers. I can recognize it.

In order to solve the said subject, the gesture recognition method of this invention is the temperature acquisition step which acquires the temperature for every infrared sensor from the temperature sensor in which the several infrared sensor was arrange | positioned, and each said said infrared sensor acquired by the said temperature acquisition step. On the basis of the detected temperature, the gesture of the hand is recognized by specifying a change area specifying step where the temperature change has occurred as an area indicating a hand and a movement trajectory of the change area specified in the change area specifying step. And a gesture recognition step.

The gesture recognition apparatus may be realized by a computer. In this case, the control program of the gesture recognition apparatus which realizes the gesture recognition apparatus in a computer by operating the computer as the above means and a computer-readable recording medium recording the same. Also, the scope of the present invention.

The gesture recognition apparatus of the present invention identifies a change region in which a temperature change has occurred, as a region indicating a hand, based on a temperature sensor in which a plurality of infrared sensors are arranged and a temperature detected by each of the infrared sensors of the temperature sensor. The change area specifying means and the change area specifying means are provided with gesture recognition means for identifying the movement trajectory of the specific change area and recognizing the gesture of the hand.

The gesture recognition method of this invention is based on the temperature acquisition step which acquires the temperature for every infrared sensor from the temperature sensor in which the several infrared sensor was arrange | positioned, and the temperature detected by each said infrared sensor acquired by the said temperature acquisition step. And a change region specifying step of specifying a change region in which a temperature change has occurred as an area representing a hand, and a gesture recognition step of specifying a movement trajectory of a specific change region in the change region specifying step to recognize a gesture of a hand. It is characterized by.

Therefore, regardless of the brightness of the place of use, the high-speed processing and the high accuracy non-contact input function can be realized.

BRIEF DESCRIPTION OF THE DRAWINGS The block diagram which shows the detail of the principal part structure of the data processing part with which the gesture recognition apparatus in embodiment of this invention is equipped.
It is a figure which shows the external appearance of the electronic device (smartphone) in which the gesture recognition apparatus in embodiment of this invention is mounted.
3 is an exploded perspective view of a sensor unit embedded in the electronic device.
4 is a diagram showing an example of the arrangement of infrared detection elements in the temperature sensor chip of the sensor unit.
Fig. 5 is a block diagram showing a main part configuration of a gesture recognition apparatus according to the embodiment of the present invention.
Fig. 6 is a diagram schematically showing a data structure of temperature information after the coordinate arrangement unit of the gesture recognition apparatus plots the temperature from each sensor on the XY coordinate system.
FIG. 7 is a diagram showing a specific example of temperature information plotted in an XY coordinate system. FIG.
Fig. 8 is a diagram illustrating the relationship between (1) the positional relationship between a smartphone and a hand in each detection timing and (2) the temperature information detected and acquired at the corresponding timing.
FIG. 9 is a diagram showing a specific example of information generated by the change amount calculating unit of the gesture recognizing apparatus and information generated by the center of gravity determination unit for the periods of timings T ₁ to T ₃ shown in FIG. 8; FIG.
10 is a view showing a specific example of the result of specifying the movement trajectory of the center of gravity by the gesture recognition unit of the gesture recognition apparatus.
Fig. 11 is a diagram showing the relationship between (1) the positional relationship between a smartphone and a hand in each detection timing and (2) the temperature information detected and acquired at the corresponding timing.
12 is a diagram showing an example of a gesture pattern stored in a gesture pattern storage unit of the gesture recognition apparatus.
13 is a flowchart showing a flow of gesture recognition processing performed by each data processing unit included in the gesture recognition apparatus.
Fig. 14 is a flowchart showing in detail the flow of gesture recognition processing executed by the gesture recognition unit of the gesture recognition apparatus.
Fig. 15 is a diagram showing a specific example of a mask stored in a mask storage unit of the gesture recognition apparatus.
16 is a diagram showing an example of a data structure of a mask stored in a mask storage unit of a gesture recognition apparatus.
It is a figure which shows typically the data structure of the temperature information after the coordinate arrangement part of the gesture recognition apparatus plotted the temperature from each sensor in the XY coordinate system in another embodiment of this invention.
Fig. 18 is a diagram showing the relationship between (1) the positional relationship of both hands and (2) the temperature information and the change area acquired at the timing in each detection timing.
FIG. 19 shows information generated by the change amount calculating unit of the gesture recognition apparatus, information generated by the center of gravity determining unit, and information generated by the deviation calculating unit at the timings of timings T ₁ and T _{2 shown} in FIG. 18. Drawing which shows the specific example of the.
20 is a diagram illustrating an example of a gesture pattern stored in a gesture pattern storage unit of a gesture recognition device according to another embodiment.
21 is a flowchart showing in detail the flow of gesture recognition processing executed by the gesture recognition unit of the gesture recognition apparatus according to the other embodiment.

`` First embodiment ''

EMBODIMENT OF THE INVENTION Embodiment of this invention is described based on drawing. In embodiment described below, as an example, the gesture recognition apparatus of this invention is mounted in a smart phone, and a gesture recognition device implements the input function of a smart phone. In addition to the smart phone, the gesture recognition device of the present invention may be a mobile phone, a PDA, a portable game machine, an electronic dictionary, an electronic notebook, a digital camera, a digital video camera, a personal computer, a laptop computer, a television, a recording generator, a home game machine, or the like. Can be mounted on various electronic devices, and can take charge of the input function of these electronic devices.

[Configuration of Gesture Recognition Device]

2 is a diagram illustrating an appearance of a gesture recognition apparatus according to the embodiment of the present invention. In this embodiment, the gesture recognition apparatus of this invention is equipped with the sensor part 2, as shown in FIG. 2, and is integrated in the smart phone 100. As shown in FIG.

The sensor part 2 detects the temperature of the target object in a detectable range (for example, in the broken line range of FIG. 2).

3 is an exploded perspective view of the sensor unit 2 incorporated in the smartphone 100. As shown in FIG. 3, the sensor part 2 is comprised as follows. Specifically, the signal processing unit 7 and the temperature sensor chip 6 are mounted on the substrate 21, and the intermediate case 22 for protecting these precision components and fixing the lens 23 thereon is mounted thereon. ) Is installed. On the intermediate case 22, a lens 23 is mounted, and an outer case 24 is provided to cover all of them.

The temperature sensor chip 6 is configured by arranging a plurality of infrared ray detection elements (infrared sensors) on the surface. Each infrared sensor detects the temperature information of a detection object based on the temperature rise which arises by receiving infrared light.

4 is a diagram illustrating an example of the arrangement of the infrared detection element in the temperature sensor chip 6. For example, as shown in FIG. 4, the temperature sensor chip 6 is made up of 4 x 4 infrared sensors arranged horizontally and horizontally. Addresses from 0 to 15 are assigned to each sensor, for example, and the positional relationship on the surface of each sensor is grasped by this address.

Thus, by arrange | positioning the some sensor managed by address on the surface, the surface temperature of the detection object in the said detectable range can be acquired in two dimensions. For example, as shown in FIG. 2, when a user's hand moves in the detectable range, the movement can be detected by the difference of the position of the sensor which detects the surface temperature of the hand.

Next, the functional structure of the gesture recognition apparatus which performs a gesture recognition process using the temperature information obtained from the temperature sensor chip 6 as mentioned above is demonstrated.

5 is a block diagram showing the configuration of main parts of the gesture recognition apparatus 1 according to the embodiment of the present invention. As shown in FIG. 5, the gesture recognition apparatus 1 collectively controls the sensor unit 2 including the temperature sensor chip 6 and the signal processing unit 7 and the unit including the sensor unit 2. The input control part 3, the memory | storage part 4, the temporary memory | storage part 8, and the connector part 5 are comprised.

The signal processing unit 7 acquires an infrared signal obtained from the infrared rays received by each sensor of the temperature sensor chip 6, and performs signal amplification processing and the like. The signal processing unit 7 is realized by, for example, an application specific integrated circuit (ASIC).

In the present embodiment, the signal processing unit 7 includes a signal selection unit 13 and a signal amplifier unit 14. The signal selector 13 selectively acquires an infrared signal from the temperature sensor chip 6. Specifically, the signal selector 13 can selectively acquire the infrared signal from the infrared sensor at a specific address under the control of the input control unit 3.

The signal amplifier 14 amplifies the infrared signal acquired by the signal selector 13 so as to be digitally processed by the input controller 3 and supplies it to the input controller 3.

The infrared signal acquired and processed by the signal processing unit 7 in this manner is digitized in the input control unit 3 and used as temperature information.

The input control unit 3 collectively controls the operation of each unit of the gesture recognition apparatus 1 and is realized by a microcomputer or the like. Or the function of the input control part 3 may be the structure with which the main control part of the smartphone 100 main body is equipped. In this case, the smartphone 100 itself functions as the gesture recognition apparatus 1 of the present invention.

The input control unit 3 uses the AD conversion unit 12 for converting the infrared signal supplied from the signal processing unit 7 into an analog signal to a digital signal, and the main control unit of the main body of the smartphone 100 through the connector unit 5. The communication control part 11 which communicates with a part, and the data processing part 10 which performs a gesture recognition process using the digitized temperature information are provided. The structure and operation of the data processing unit 10 as a functional block will be described in detail later with reference to the separate section.

The connector part 5 connects the gesture recognition apparatus 1 and the main control part of the smart phone 100 electrically and communicatively. The result of the gesture recognition by the data processing unit 10 is supplied to the main control unit of the main body of the smartphone 100 via the communication control unit 11 and the connector unit 5. On the contrary, the information regarding the state of the smartphone 100 such as which application the smartphone 100 is currently running is supplied from the main controller of the smartphone 100.

The temporary storage unit 8 is a so-called working memory that temporarily stores data to be used for calculations, calculation results, and the like in the course of various processes executed by the gesture recognition apparatus 1.

The storage unit 4 is an application program for executing the various functions of the gesture control device 1, the control program, the OS program, and the input control unit 3 that the input control unit 3 executes. It stores various data to be read when the application program is executed. In particular, the storage unit 4 stores various programs and data to be read when the gesture recognition apparatus 1 performs a gesture recognition process.

In this manner, in the gesture recognition apparatus 1, the data processing unit 10 of the input control unit 3 performs movement of an object (such as a user's hand) based on temperature information acquired from the temperature sensor chip 6. It recognizes as a gesture, and the communication control unit 11 supplies this recognition result to the smartphone 100 main body as an input to the smartphone 100. Accordingly, the smartphone 100 may execute an operation assigned to the recognized gesture.

That is, the gesture recognition apparatus 1 can function as a non-contact input device of the smart phone 100.

[Configuration of Data Processing Unit]

FIG. 1 is a block diagram showing the details of the configuration of main parts of the data processing unit 10 of the gesture recognition apparatus 1.

The data processing unit 10 is configured to include at least a temperature information acquisition unit 30, a coordinate arrangement unit 31, a change amount calculation unit 32, and a gesture recognition unit 34 as functional blocks. It is preferable that the data processing unit 10 further includes a center of gravity determination unit 33. In addition, the data processing unit 10 may further include a mask selection unit 35 and a deviation calculation unit 36.

The storage unit 4 further includes at least a gesture pattern storage unit 40 for storing a pattern of gestures to be identified by the data processing unit 10. The storage unit 4 may further include a mask storage unit 41 and a reference value storage unit 42.

Each functional block of the data processing unit 10 includes a central processing unit such as a micro processing unit (MPU) or a central processing unit (CPU) such as read only memory (ROM) and non-volatile random access memory (NVRAM). The program stored in the memory device (memory section 4) realized by, for example, can be realized by reading and executing a random access memory (RAM) (temporary memory section 8) or the like.

The temperature information acquisition unit 30 acquires the digitally converted temperature information d1 from the AD conversion unit 12. In the present embodiment, the temperature information d1 has a data structure in which addresses of the sensors from 0 to 15 correspond to values of temperatures detected by the sensors. In the present embodiment, the temperature information acquisition unit 30 uses the temperature information d1 detected in real time by each sensor of the temperature sensor chip 6 at regular time intervals (for example, every 0.5 seconds). It acquires and stores it in the temporary storage part 8 from time to time. At this time, the temperature information acquisition unit 30 associates timing information T _i (i = 0, 1, 2, ..., n) for identifying the detected timing with the temperature information d1 to be acquired.

The coordinate arrangement unit 31 plots the temperature information acquired from each sensor on two-dimensional coordinates so as to correspond to the arrangement of each sensor. In this embodiment, temperature information acquired from each sensor is plotted on the XY coordinate system so as to correspond to the arrangement of 4x4 = 16 sensors from address 0 to address 15.

FIG. 6: is a figure which shows typically the data structure of the temperature information after the coordinate arrangement part 31 plotted the temperature from each sensor on the XY coordinate system.

For example, as shown in FIG. 4, in the temperature sensor chip 6, when 16 infrared sensors of 0-15 addresses are arrange | positioned so that it may become 4 * 4 vertically and horizontally, the coordinate arrangement part 31 is The temperature of each sensor is plotted using the left lower end of the temperature sensor chip 6 as the origin and the left lower end of the XY coordinate system as the origin.

Specifically, the temperature detected by the sensor at address 12 is plotted at a position of (X, Y) = (1, 1), and the temperature of the sensor at address 13 on the right side is (X, Y) = ( Plot at the position of 2, 1). Similarly, the coordinate arranging unit 31 plots the temperature in the XY coordinate system so as to correspond to the actual arrangement of the sensor. Finally, the temperature of the sensor at the rightmost top three is plotted at the position of (X, Y) = (4, 4).

The temperature information plotted in the XY coordinate system in this way is stored in the temporary storage unit 8 in association with the timing information T _i . 7 is a diagram illustrating a specific example of temperature information plotted on an XY coordinate system. 7 shows an example in which an XY coordinate system is created for each of the temperature information acquired between timings T ₀ to T _n . The two digits stored in 16 cells of the XY coordinate system indicate the temperature (in this case, the unit is ° C) detected by each sensor.

The change amount calculation unit 32 compares the predetermined threshold value with the actual temperature acquired from each sensor as described above, and determines the presence or absence of a temperature change or a detection target (hand, finger, etc.). In addition, the amount of change in the detection temperature of each sensor is calculated by comparing the predetermined reference value with the actual temperature.

The change amount calculation unit 32 determines whether or not the temperature of each sensor detects the temperature of the target object in comparison with a predetermined threshold value. For example, when it is desired to detect a human hand as a detection object, a lower limit (which may be an upper limit in some cases) that can be assumed as human body temperature is set as a threshold. Here, it is set to 33 degreeC.

The change amount calculating part 32 compares the temperature of each sensor with the threshold value of 33 degreeC, and recognizes the sensor which detected the temperature of 33 degreeC or more as a sensor which detected a hand. That is, if the position of the sensor which detected a hand is confirmed, it can be treated as a position where a hand exists as it is.

The threshold value may be appropriately set to an effective temperature for specifying the object in consideration of the properties of the desired detection object. The set threshold value is previously stored in the reference value storage section 42, and is read by the change amount calculation section 32 as necessary.

FIG. 8: is a figure which shows the positional relationship of the smartphone 100 and a hand in each detection timing, and the relationship of the temperature information detected and acquired in the said timing. As shown in Figure 8, at time T _1, and that the hand is located in the left end of the detection range of the sensor unit (2). In this case, the coordinate arranging unit 31 generates the XY coordinate system associated with the timing T ₁ shown in FIG. 8. The change amount calculating unit 32 compares the temperature stored at each coordinate position of the XY coordinate system with the threshold value 33, specifies the coordinate position of the temperature of the threshold value or more, and specifies this as the area of the hand. In the example shown in FIG. 8, the change amount calculation part 32 specifies the area | region of a left end (area of a halftone point) as an area of a hand also in an XY coordinate system similarly to the actual position of the said hand. Next, at time T _2, and that the hand is moved to the front of the smart phone (100) the sensor section (2). In this case, the coordinate arranging unit 31 generates the XY coordinate system associated with the timing T ₂ (FIG. 8). And the change amount calculation part 32 specifies the whole surface (area of a halftone point) of an XY coordinate system as an area | region of a hand. At timing T ₃ , the hand was further moved to move to the right end of the detection range. In this case, the coordinate arranging unit 31 generates the XY coordinate system associated with the timing T ₃ shown in FIG. 8. The change amount calculation part 32 specifies the area | region of the right end part (region of a half-dot) which is threshold value 33 degreeC or more as an area | region of a hand.

The change amount calculation unit 32 calculates the difference between the temperature actually detected and the reference value as the change amount ΔT. The change amount calculation unit 32 calculates the change amounts ΔT (X, Y) respectively with respect to the temperature of each address (coordinate position).

Here, the reference value may be the same as the threshold value for detecting the above-described object, or an appropriate value may be determined in consideration of the environmental temperature when there is nothing. In the present embodiment, since the ambient temperature around the smartphone 100 is 25 ° C, the reference value is determined to be 25 ° C in advance. The reference value of this regulation is stored in the reference value storage section 42, and is read by the change amount calculation section 32 as necessary.

The reference value may be a structure that is dynamically updated. For example, it is said that the temperature is measured from time to time by the temperature sensor chip 6, and it has elapsed for a certain period of time without a great fluctuation in the amount of change. At this time, the average value calculating part which is not shown in the data processing part 10 may calculate the average value of the temperature measured in this period, and may store this in the reference value storage part 42 as a reference value.

In addition, the change amount calculation unit 32 adds all the change amounts ΔT (X, Y) obtained for each address of each cell, and sums up the change amount ΔT (X, Y) of the XY coordinate system at the timing T _i (hereinafter, SUM). ) Specifically, in the present embodiment, the change amount calculation unit 32 has the following formula: SUM = ΔT (1, 1) + ΔT (1, 2) + ΔT (1, 3) + ΔT (1, 4) + ΔT (2, 1) + ΔT (2, 2) + ΔT (2, 3) + ΔT (2, 4) + ΔT (3, 1) + ΔT (3, 2) + ΔT (3, 3) + ΔT (3, 4) + ΔT (4, 1) Calculate + ΔT (4, 2) + ΔT (4, 3) + ΔT (4, 4).

Thereby, the total amount of change SUM generated in the detection range (in the broken line range in FIG. 2) in the timing T _i becomes apparent.

The change amount calculation unit 32 associates the change amount ΔT (X, Y) and the sum SUM obtained for each address of each cell as described above with the timing T _i , and plots the result in the XY coordinate system to store the temporary storage unit 8. Store in

By referring to the change amount ΔT obtained by the change amount calculation unit 32, it is possible to determine at what timing and at which position the change in temperature is seen. In addition, the change amounts ΔT and SUM are read and used by the functional blocks described below to determine the presence or absence of a gesture recognition object and the coordinate positions (Gx, Gy) of the center of gravity of the area of the object (area of large change amount). do.

In addition, the change amount calculation unit 32, based on the temperature information acquired from each sensor, the timing from when to when the user's operation period (the user tries to input to the smartphone 100, It is preferable to determine whether or not it is acceptable to recognize the period of time when the hand is intentionally moved.

Some of the methods for determining the period of time that the user is operating will be described in detail below.

The first method is a method of using a threshold value stored in the reference value storage section 42. When the change amount calculation unit 32 compares each temperature plotted on the XY coordinate system at a certain timing with a threshold value (for example, 33 ° C), and there are two or more sensors that detect 33 ° C or more, It is judged that there is a change (operation start). And the period in which two or more sensors which detected 33 degreeC or more can be determined as an operation period.

The second method is a method of using the change amount SUM of the entire XY coordinate system obtained for each detection timing and the reference value stored in the reference value storage section 42. When the change amount calculation unit 32 sets the reference value to Ta ° C, the change amount calculation unit 32 sets (35 ° C-Ta ° C) x 2 as the threshold of the change amount, and when the SUM is (35 ° C-Ta ° C) x 2 or more, the change It is judged that there is (operation start). And the period in which SUM is (35 degreeC-Ta degreeC) x2 or more can be judged as an operation period.

According to the above configuration, only the temperature information of the period determined as the operation period by the change amount calculating section 32 can proceed to the subsequent gesture recognition processing, and the gesture recognition apparatus 1 can be reduced by reducing the amount of information involved in the processing. ), And the processing load can be reduced.

The center of gravity determination unit 33 is a region in which a change is recognized in the XY coordinate system based on the change amounts ΔT and SUM of each sensor, specifically, a change region in which a temperature rise occurs due to the detection of a hand (see FIG. 8). Area of halftone). The center of gravity determination unit 33 determines the coordinates Gx and Gy of the center of gravity G of the change area in accordance with the following procedure while referring to the changes ΔT and SUM.

The center of gravity determination unit 33 obtains the X coordinate Gx of the center of gravity G. Gx is computed by adding up the product of the change amount (DELTA) T of each coordinate position multiplied by the coefficient according to the X coordinate of each coordinate position, and dividing this by the said SUM. Specifically, the center of gravity determination unit 33 is represented by the following equation, Gx = {(ΔT (1, 1) + ΔT (1, 2) + ΔT (1, 3) + ΔT (1, 4)) × 1 + (ΔT ( 2, 1) + ΔT (2, 2) + ΔT (2, 3) + ΔT (2, 4) × 2 + (ΔT (3, 1) + ΔT (3, 2) + ΔT (3, 3) + ΔT (3, 4) ) X 3 + (ΔT (4, 1) + ΔT (4, 2) + ΔT (4, 3) + ΔT (4, 4)) × 4} ÷ SUM.

In addition, the center of gravity determination unit 33 obtains the Y coordinate Gy of the center of gravity G. Gy is computed by multiplying the change quantity (DELTA) T of each coordinate position multiplied by the coefficient according to the Y coordinate of each coordinate position, and dividing this by the said SUM. Specifically, the center of gravity determination unit 33 is represented by the following equation, Gy = {(ΔT (1, 1) + ΔT (2, 1) + ΔT (3, 1) + ΔT (4, 1)) × 1 + (ΔT ( 1, 2) + ΔT (2, 2) + ΔT (3, 2) + ΔT (4, 2) × 2 + (ΔT (1, 3) + ΔT (2, 3) + ΔT (3, 3) + ΔT (4, 3) ) X 3 + (ΔT (1, 4) + ΔT (2, 4) + ΔT (3, 4) + ΔT (4, 4)) × 4} ÷ SUM.

The center of gravity determination unit 33 stores the coordinates Gx and Gy of the center of gravity G of the change area at the timing T _i obtained as described above in the temporary storage unit 8 in association with the timing T _i . do. The center of gravity determination unit 33 may plot the center of gravity G (Gx, Gy) on the XY coordinate system generated by the coordinate arrangement unit 31.

FIG. 9 shows a specific example of the information generated by the change amount calculation unit 32 and the information generated by the center of gravity determination unit 33 for the period of the timings T ₁ to T ₃ shown in FIG. 8. Drawing.

The change amount calculation unit 32 obtains the temperature information of the timing T ₁ (XY coordinate system T _{1 in} FIG. 8), calculates a difference between the temperature of each sensor and the reference value of 25 ° C., and plots the value of the change amount ΔT of each sensor. do. The results of this plot shows the XY coordinate system T ₁ in Fig. Similarly, Figure 9 of the XY coordinate system T _2, T _3, respectively, indicates that the timing information from the temperature T _2, T ₃ in Fig. 8, the determined value of the change amount ΔT.

And the change amount calculation unit 32 also calculates a total amount of change of the SUM at the timing T ₁ on the basis of the XY coordinate system T ₁ in Fig. 9 by "105". Similarly, the total amount of change SUM at timing T ₂ is calculated as "168", and the total amount of change SUM at timing T ₃ is calculated as "84".

Subsequently, the center of gravity determination unit 33 determines the center of gravity G of the change area (the halftone dot area in FIG. 8) by using the change amounts ΔT and SUM of each sensor obtained as described above. In the example shown in Figure 9, the center-of-gravity determining section 33, the timing T weight coordinate of the center G of the ₁ (Gx, Gy) to (1.8, 2.5) as determined, the weight in the timing T ₂ The coordinates Gx and Gy of the center G are determined as (2.5, 2.5), and the coordinates Gx and Gy of the center of gravity G at the timing T ₃ are determined as (3.5, 2.5).

By following this position of the center of gravity G, it is possible to recognize in a subsequent process how and when the object (hand) moved at the timing.

As described above, by specifying the coordinates of the center of gravity G of the change area for each detection timing, the trajectory of the movement of the change area (hand) can be obtained with a small amount of calculation, so that the configuration and the center of gravity of the center of gravity determiner 33 are determined. It is preferable to adopt a method for determining the ratio from the viewpoint of improving the processing efficiency and reducing the processing load.

In addition, in order to recognize the gesture of the object (hand), it is necessary to track the trajectory of the movement of the hand. As a method of tracking the movement trajectory, the above-described method of determining the center of gravity and tracking the center of gravity are simply It is an example and the structure of this invention is not limited to this.

The gesture recognition unit 34 tracks the position of the object and recognizes the movement of the object as a gesture according to the trajectory of the movement of the object. In this embodiment, specifically, the position of the center of gravity of the object determined by the center of gravity determination unit 33 is tracked, the trajectory of the center of gravity is specified, and the gesture of the object is identified according to the trajectory of the specific center of gravity. .

10 is a diagram showing a specific example of the result of the gesture recognition unit 34 specifying the trajectory of the center of gravity. In addition, the trajectory of the center of gravity shown in FIG. 10 has described in detail the example of the trajectory which the gesture recognition part 34 specifies, and in fact, such a trajectory is presented so that a user can visually confirm. no.

As shown in Fig. 10, if the plot of each of the center of gravity of a timing T ₁ to T ₃ shown in FIG. 9, a single coordinate system, it is clear that the locus of movement of the center of gravity.

The gesture recognition unit 34 tracks the center of gravity from the start timing T _s of the operation period to T1, T2, T3, ... and the operation period end timing T _e , and recognizes the movement. do. In the example shown in FIG. 10, the center of gravity moves from the left to the right from the position of T _{1 to} the position of T ₃ . Therefore, the gesture recognition unit 34 can recognize that the gesture "move from left to right" was performed. The gesture recognition unit 34 moves "from right to left", "moves from top to bottom", "moves from bottom to top", "moves clockwise", and "counterclockwise" according to the trajectory of the center of gravity. And various gestures such as “move to "

Further, according to the gesture recognition apparatus 1 of the present invention, the gesture recognition unit 34 detects not only the movement in the X-axis direction or the Y-axis direction, but also the movement in the Z-axis direction, and identifies it as a gesture. It is possible.

11 is a diagram showing the positional relationship between the smartphone 100 and the hand at each detection timing, and the relationship between the temperature information detected and acquired at the timing. As shown in Figure 11, at time T _1, and that the hand is positioned furthest from within the detection range of the sensor unit (2). In this case, coordinate configuration unit 31, and generates the XY coordinate been timing T ₁ are related as shown in Fig. Since the hand is located at a position distant on the Z axis from the sensor portion 2 of the smartphone 100, the hand is small in terms of the temperature sensor chip 6 of the sensor portion 2. Therefore, the change area (dotted dot area) is specified locally, not on the entire surface of the temperature sensor chip 6.

On the other hand, at time T _2, when all of the hand, the timing T _1, said to be in a position close to the smartphone 100. The In this case, as shown in FIG. 11, the coordinate arranging unit 31 generates an XY coordinate system associated with the timing T ₂ . Since the hand is in a position close to the Z axis from the sensor unit 2 of the smartphone 100, the hand is largely grasped in the surface of the temperature sensor chip 6. In the example shown in FIG. 11, the whole surface of the temperature sensor chip 6 is grasped large. Therefore, the change area is specified on the whole surface of the temperature sensor chip 6. That is, the larger the area of the change area (the number of sensors having a threshold value of 33 ° C. or higher), the closer the object is to the vicinity of the smartphone 100, and the smaller the area of the change area (the fewer the number of sensors having a threshold value of 33 ° C. or higher). It may be determined that the object is located far from the smart phone 100.

Therefore, the gesture recognition unit 34 tracks the movement in the Z-axis direction of the object in accordance with the increase and decrease of the area of the change area, which changes every detection timing, to identify the gesture. Specifically, the gesture recognizing unit 34 can identify various gestures such as "close hands" or "away hands" according to the increase and decrease of the area of the change area.

In this embodiment, the memory | storage part 4 is equipped with the gesture pattern memory | storage part 40, and memorize | stores some patterns of a predetermined gesture here. The gesture recognition unit 34 determines which gesture pattern stored in the gesture pattern storage unit 40 corresponds to the trajectory of the center of gravity or the increase or decrease of the change area to identify the gesture.

FIG. 12 is a diagram illustrating an example of a gesture pattern stored in the gesture pattern storage unit 40.

As shown in FIG. 12, in the gesture pattern storage unit 40, some gesture patterns that the gesture recognition unit 34 can recognize are stored. In this embodiment, the gesture ID for identifying these gestures is stored in correspondence with each gesture pattern.

For example, when the gesture recognizing unit 34 recognizes the gesture of the object as "move from left to right" according to the locus of the center of gravity shown in FIG. 10, the gesture recognition unit 34 performs a gesture of "move from left to right". The gesture ID " 00 " indicating is output to the communication control unit 11 as the recognition result d2 (see Fig. 1).

The recognition result d2 is input into the main body of the smartphone 100 via the connector unit 5. The main control unit of the smartphone 100 executes the operation assigned to the gesture ID "00" in response to the input of the gesture ID "00".

In addition, the pattern of the gesture shown in FIG. 12 is an example, and does not limit this invention. The gesture recognition unit 34 of the present invention may recognize patterns of movement of various objects (for example, movement in an oblique direction, etc.) as gestures other than the gestures shown in FIG. 12.

[Gesture recognition processing flow]

13 is a flowchart showing the flow of gesture recognition processing performed by each unit of the data processing unit 10 included in the gesture recognition apparatus 1 of the present invention. 14 is a flowchart showing in detail the flow of the gesture recognition processing executed by the gesture recognition unit 34.

In the present embodiment, the sensor unit 2 detects the temperature information of the target object at predetermined detection timings T ₀ , T ₁ , T ₂ ,..., T _n for a predetermined period, and at any time, the data processing unit 10. I assume the constitution supplying to).

First, when the predetermined detection timing has been reached, the temperature information acquisition unit 30 acquires the detection timing T _i (S101). Then, the temperature information d1 associated with the acquired timing T _i is obtained from the temperature sensor chip 6 (S102). The coordinate arrangement unit 31 plots the temperature for each sensor included in the acquired temperature information d1 in the XY coordinate system corresponding to the arrangement of each sensor (S103). The coordinate arranging unit 31 stores the data generated in the same manner as in S103 in the temporary storage unit 8 in association with the acquired timing Ti (S104).

The change amount calculating part 32 compares the threshold value (for example, 33 degreeC) with the temperature plotted in each coordinate position of an XY coordinate system, and determines whether there are two or more sensors which detected 33 degreeC or more. (S105). If there is more than two (in S105 YES), the change amount calculation unit 32, in the art the timing T _i, and the object (hand) is present in the detection range, the point of timing T _i is the period that the user operates I think it is included in. And the change amount calculation part 32 calculates the change amount (DELTA) T of the temperature of each coordinate position, respectively, and calculates SUM by adding up the change amount (DELTA) T of all coordinate positions (S106). Subsequently, the center of gravity determining unit 33 determines the coordinates Gx and Gy of the center of gravity G in the area of the sensor that detects the temperature of the hand, that is, the change area, based on the change amounts ΔT and SUM ( S107).

The center of gravity determination unit 33 plots the determined center of gravity in the XY coordinate system stored in S104, and the change amount calculation unit 32 plots the change amount ΔT and the change area in the XY coordinate system and associates the SUM with each other. It saves in the temporary storage part 8 (S108).

On the other hand, when there are no two or more sensors that detect 33 ° C. or more (NO in S105), the change amount calculation unit 32 does not exist in the detection range at the timing T _i . At the timing T _i , the user determines that the input operation is not performed. The processing of S106 to S108 is omitted.

If the predetermined time period for performing the gesture recognition is not finished, that is, if the unprocessed temperature information d1 is supplied from the sensor unit 2 (NO in S109), the data processing unit 10 determines the timing T _i. Is increased by one (S110), and then the processes of S102 to S108 are repeated with respect to the temperature information d1 at the detection timing.

When the timing T _i reaches T _n and the processing is completed for all of the supplied temperature information d1 (YES in S109), the gesture recognition unit 34 continues to the temporary storage unit 8. Based on the stored data, a gesture corresponding to the movement of the detected hand is recognized (S111). Specifically, among the gesture patterns stored in the gesture pattern storage unit 40 by tracking the center of gravity position in the predetermined periods T ₀ to T _n or increasing or decreasing the area of the change area, Specifies the gesture to match.

The gesture recognition unit 34 outputs the gesture ID indicating the specific gesture to the communication control unit 11 as the recognition result d2 (S112).

The flow of gesture recognition processing executed by the gesture recognition unit 34 in S111 will be described in detail with reference to FIG. 14 as follows.

First, the gesture recognizing unit 34 acquires the movement trajectory of the center of gravity of the hand based on the center of gravity coordinates Gx and Gy determined for each detection timing T _i (S201).

Here, if the length of the movement distance of the center of gravity is sufficient (YES in S202), the gesture recognition unit 34 determines that the hand has moved in the X-axis direction or the Y-axis direction, and the movement trajectory and movement of the center of gravity are determined. The direction is specified (S203).

If the movement direction of the center of gravity is the left-right direction (X-axis direction) (1 in S204), the gesture recognition unit 34 determines whether the hand has moved from left to right based on the movement trajectory ( S205). When it is determined that the hand has moved from left to right (YES in S205), the gesture recognition unit 34 recognizes the gesture input in this period as "00: move from left to right" (S206). On the other hand, when it is determined that the hand has moved from the right side to the left side (NO in S205), the gesture recognition unit 34 recognizes the gesture input in this period as "01: move from right side to left side" (S207).

Or if the movement direction of a center of gravity is an up-down direction (Y-axis direction) (2 in S204), the gesture recognition part 34 will determine whether a hand moved from the upper side to the lower side (S208). When it is determined that the hand has moved from the upper side to the lower side (YES in S208), the gesture recognition unit 34 recognizes that the gesture input in this period is "10: Move from upper side to lower side" (S209). On the other hand, when it is determined that the hand has moved from the lower side to the upper side (NO in S208), the gesture recognition unit 34 recognizes the gesture input in this period as "11: Move from the lower side to the upper side" (S210).

Alternatively, when the movement trajectory of the center of gravity is rotating as if drawing a circle (3 in S204), the gesture recognition unit 34 determines whether the hand has moved clockwise (S211). When it is determined that the hand has moved clockwise (YES in S211), the gesture recognition unit 34 recognizes the gesture input in this period as "30: move clockwise" (S212). On the other hand, when it is determined that the hand has moved in the counterclockwise direction (NO in S211), the gesture recognition unit 34 recognizes the gesture input in this period as "31: moves in the counterclockwise direction" (S213).

On the other hand, in S202, if the length of the movement distance of the center of gravity is not sufficient (NO in S202), the gesture recognition unit 34 determines that the hand did not move in the X axis direction or in the Y axis direction. . Subsequently, the process proceeds to processing for judging whether or not it has moved in the Z-axis direction. That is, the gesture recognition part 34 acquires the area (number of sensors) of the change area | region specified by the change amount calculation part 32 for every detection timing T _i (S214).

The gesture recognition unit 34 determines whether the area of the change area tends to increase over time (S215). If the area tends to increase (YES in S215), the gesture recognition unit 34 recognizes the gesture input in this period as "20: hand close" (S216). On the other hand, when the area tends to decrease (NO in S215), the gesture recognition unit 34 recognizes the gesture input in this period as "21: Keep hands away" (S217).

In addition, although the case shown in FIG. 14 does not explain the case where neither the movement of the center of gravity nor the increase or decrease of the area is described, such a case is determined by the gesture recognition apparatus 1 whether the object is recognized by error or the object. This may occur when the gesture recognition apparatus 1 recognizes that the object is stationary because the motion of the object is not sufficient. Therefore, in such a case, the gesture recognition apparatus 1 displays a message urging the user to change the reference value or the threshold value via the smart phone 100, or displays an error indication indicating that the gesture recognition has failed. Or display a message that facilitates re-entry of the gesture.

As mentioned above, the gesture recognition apparatus 1 of this invention is equipped with the sensor part 2 which detects the temperature of an object, and the data processing part 10 which performs a series of gesture recognition processes mentioned above. Thereby, the contactless input device of the smart phone 100 can be realized.

According to the above configuration and method, in the gesture recognition apparatus 1, information for detecting the movement of the object is acquired as temperature information from a plurality of infrared sensors arranged. And the data processing part 10 can analyze the movement of an object only by temperature information, and can recognize a gesture, even if a data amount does not handle huge image data. As a result, in the gesture recognition apparatus 1 as the input device, it is possible to realize an improvement in processing efficiency and a reduction in processing load. That is, it is possible to realize an input device with a fast reaction speed.

Moreover, the gesture recognition apparatus 1 of this invention detects the movement of an object by the difference of the object's temperature and surrounding environmental temperature by acquiring the surface temperature of an object using an infrared sensor. Therefore, in the configuration of recognizing a gesture by capturing an object with an optical system camera, use in a dark place is limited, but the gesture recognition device 1 of the present invention can function as an input device in a dark place without limitation.

In addition, in gesture recognition using an optical camera, when there is no large difference between the texture of the object and the background, there is a problem that the position, shape, etc. of the object cannot be accurately recognized, and error recognition of the gesture increases. On the other hand, since the gesture recognition apparatus 1 of the present invention detects an object (such as a hand) having a surface temperature different from the ambient temperature based on the temperature information, the problem of error recognition due to the similarity of texture cannot occur. .

When the gesture recognition apparatus 1 of the present invention is used as an input device of an electronic device having a projector function, the advantage of the gesture recognition apparatus 1 of the present invention can be maximally enjoyed.

For example, a use case is assumed that uses a projector function of an electronic device to project a slide such as a photo stored in the electronic device onto a screen located at a spaced distance to give a presentation. Since the projector function is generally used by darkening the indoors, it has been difficult to make an input device for gesture recognition using a conventional optical system camera function during a presentation. However, according to the gesture recognition apparatus 1 of the present invention, it is possible to function as a non-contact input device even in a dark place, so that a user who makes a presentation at a position away from the electronic device is in contact with the electronic device. Without this, the slide can be easily operated.

[Variation example]

In the above-mentioned embodiment, the gesture recognition apparatus 1 processed the temperature information acquired from the whole surface (all infrared sensors) of the temperature sensor chip 6, and recognized the gesture. The gesture recognition apparatus 1 of this invention is not limited to this structure, You may perform gesture recognition using the temperature information from the infrared sensor arrange | positioned at the temperature sensor chip 6 partially.

When the infrared sensor is selectively used, the processing speed can be reduced by reducing the amount of information to be processed, and the recognition accuracy can be improved when a specific gesture is recognized.

The smartphone 100 is multifunctional and is configured to execute various applications, and the instruction input format for the application varies depending on the application. For example, in a book reading application or a slide show display application, an instruction input involving a left-right movement will be performed to turn a page, and in a web browsing application, an up-down movement is performed to scroll a screen. Instruction input will be made. In addition, in the music reproduction application, it is assumed that the movement of turning the dial in multiple stages is input for adjusting the volume.

Therefore, the gesture recognition apparatus 1 of the present invention limits the gesture to be recognized based on which application the smartphone 100 is running, and efficiently executes the gesture recognition process suitable for the recognition of the limited gesture. Can be. As a structure for that, the gesture recognition apparatus 1 is provided with the following.

That is, as shown in FIG. 1, the data processing part 10 of the gesture recognition apparatus 1 in this modification is provided with the mask selection part 35 in addition to the structure mentioned above.

The mask selector 35 limits the types of gestures to be recognized by the main body of the smartphone 100 according to the type of application being activated, and selects a mask suitable for recognition of the limited gestures.

The mask in this invention is pattern information which shows which infrared sensor among the infrared sensors arrange | positioned at the temperature sensor chip 6 is used for the gesture recognition process, or not.

Masks are designed in advance and are prepared in plural kinds so that the gestures can be recognized efficiently and with high accuracy depending on which gestures are desired to be recognized. In this modification, as shown in FIG. 1, the memory | storage part 4 of the gesture recognition apparatus 1 is equipped with the mask memory | storage part 41. As shown in FIG. The mask memory | storage part 41 memorize | stores several types of mask, and the mask which should be referred for every application is associated.

15 is a diagram illustrating a specific example of a mask.

As shown in the scene 50 shown in FIG. 15, when the hand is specialized in a gesture of moving the left and right directions, the hand is properly detected by a sensor at the left end and the right end of the infrared sensor arrangement surface of the temperature sensor chip 6. If you can, you can recognize gestures efficiently and accurately. Therefore, the "left and right detection mask" for recognizing the movement of right and left is designed to make the column at the left end and the column at the right end of the placement surface valid, and to invalidate the center column.

In the mask shown in FIG. 15, the diagonal line area is valid and the blank area is invalid. Specifically, the arrangement of 4x4 shown in FIG. 4 will be described as an example. The "left and right detection mask" includes only temperature information from infrared sensors at addresses 0, 4, 8, 12 and 3, 7, 11 and 15. It is designed to be used.

When the hand is specialized in a gesture of moving the hand in the up and down direction as in the scene 51, the gesture can be recognized efficiently and efficiently if the hand can be detected with good timing by the sensors of the upper and lower parts of the placement surface. Therefore, the "up and down detection mask" for recognizing the movement of the up and down is designed to make the row of the upper end and the lower part of the placement surface valid, and invalidate the center row.

As shown in FIG. 15, the "up and down detection mask" is designed so that only the temperature information from the infrared sensor of 0-3 and 12-15 may be used, for example.

Specializing in the gesture of the hand turning like the scene 52, when a hand is detected by the sensor outside of the said arrangement surface, a gesture of rotation can be recognized correctly efficiently. Therefore, the "rotation detection mask" for recognizing rotation is designed to use only temperature information from infrared sensors at addresses 0-4, 7, 8, 11-15, for example.

In this way, a plurality of types of masks according to the scene (movement of a hand to be detected) are stored in the mask storage unit 41. FIG. 16 is a diagram illustrating an example of a data structure of a mask stored in the mask storage unit 41.

As shown in FIG. 16, the mask memory | storage part 41 memorize | stores several types of masks, and also stores the mapping of the mask which should be referred for every application.

Thereby, the mask selection part 35 can always select the optimal mask according to the application in which the smartphone 100 is starting. In more detail, the mask selection part 35, through the connector part 5 and the communication control part 11, which application the main body of the smart phone 100 is currently starting from the main control part of the smart phone 100 is. The state information d3 indicating is obtained (see FIG. 1). The mask selection unit 35 specifies an application during startup based on the state information d3. The mask selection unit 35 then references the mask storage unit 41 to select a mask corresponding to the specific application.

Once an application is specified, the application is limited in what movements are input as gestures. For example, in the web browsing application, the movement in the up and down direction is input, in the book browsing application, the movement in the left and right directions is input, and in the music reproduction application, the rotation movement is input.

For example, in the example shown in FIG. 16, the mask for right and left detection is associated with the TV application (TV_Channel). That is, the mask selection part 35 selects the mask for right and left detection shown in FIG. 15, when it determines with the smartphone 100 starting up a TV application.

When the mask selection unit 35 selects the left and right detection masks, each unit of the data processing unit 10 can recognize the movement in the left and right directions efficiently and accurately. Specifically, the temperature information acquisition unit 30 acquires the temperature only from the infrared sensor of the effective area, and executes each subsequent subsequent shading process only for the temperature stored in the effective area.

During the use of the TV application, the user inputs only the left and right gestures for changing the channel, so that the operability of the smart phone 100 can be improved by enabling the user to recognize the left and right movements efficiently and accurately.

In addition, the example shown in FIG. 16 regarding an association of an application and a mask is an example for demonstrating invention, and there is no intention to limit the structure of invention. For the application, the gesture necessary for the application is taken into consideration, and an association with the optimum mask is performed in advance.

As described above, the mask selected by the mask selector 35 is a mask designed for gesture recognition specialized in each of the above-described movements and efficiently and with high precision. Therefore, by selecting a mask suitable for each application, it becomes possible to efficiently and accurately recognize the operation gesture input to the application.

In addition, the mask selection unit 35 specifies the address of the effective sensor based on the selected mask, and instructs the signal selection unit 13 of the signal processing unit 7 to acquire an infrared signal only from the sensor of the effective address. The structure may be sufficient. Alternatively, the mask selection unit 35 may be configured to instruct each unit of the data processing unit 10 to process only the temperature from the sensor at the effective address.

`` Second embodiment ''

In the above-described embodiment, when the temperature sensor chip 6 has a configuration in which a 4x4 infrared sensor is arranged, the gesture recognition device 1 which processes 4x4 temperature information obtained therein and recognizes a gesture is provided. The composition of the.

However, the configuration of the gesture recognition apparatus 1 of the present invention is not limited to the above. Arrangement of the infrared sensor of the temperature sensor chip 6 can be configured in any way. For example, it is also possible to arrange | position an infrared sensor not only square but a rectangle or circular shape. Moreover, you may make the temperature sensor chip 6 high resolution by increasing the number of the infrared sensors arrange | positioned. According to this configuration, it is possible to detect a plurality of objects (for example, the operation of both hands) only by using the temperature information, or to precisely detect the shape as well as the position of the object. As a result, the calculation amount does not become large as in the image processing technology, and it becomes possible to identify more various gestures, and to perform complicated operation input on the electronic device.

In the following description, the temperature sensor chip 6 is configured such that 8 x 8 = 64 infrared sensors are arranged, and the gesture recognition device 1 processes the temperatures of the 64 sensors to recognize gestures by both hands. The structure and gesture recognition method of the gesture recognition apparatus 1 in case of doing so are demonstrated. In addition, for the convenience of description, the same code | symbol is attached | subjected about the member which has a function similar to the drawing demonstrated in 1st Embodiment mentioned above, and the description is abbreviate | omitted.

[Configuration of Gesture Recognition Device]

In the gesture recognition apparatus 1 in this embodiment, the data processing part 10 is a structure provided with the deviation calculation part 36 as shown in FIG.

The deviation calculation unit 36 calculates the deviation of the change area based on various information generated by the coordinate arranging unit 31, the change amount calculation unit 32, and the center of gravity determination unit 33. Deviation calculating section 36 calculates a deviation (σy ²⁾ of, and variance (σx ²⁾ in the X-axis direction of the change area, Y-axis direction.

First, the coordinate arrangement | positioning part 31 plots temperature information on the 8 * 8 XY coordinate system by the method similar to what was demonstrated in 1st Embodiment. FIG. 17: is a figure which shows typically the data structure of the temperature information after the coordinate arrangement part 31 plotted the temperature from each sensor on the XY coordinate system.

Next, the change amount calculation part 32 specifies the change area for every detection timing based on the temperature information of the XY coordinate system obtained for every detection timing. FIG. 18 is a diagram showing the positional relationship between the two hands at each detection timing, and the relationship between the temperature information acquired at the timing and the change region.

In the timing T ₁ where both hands are close to each other, as shown in the XY coordinate system T ₁ , the change area (the area of the halftone point) is gathered at the center. On the other hand, in timing T ₂ with both hands spaced apart in the horizontal direction, as shown in the XY coordinate system T ₂ , the change area is scattered in the left and right directions.

Subsequently, the change amount calculation unit 32 obtains the change amount ΔT for each coordinate position, and also calculates the SUM. The center of gravity determination unit 33 determines the coordinate positions Gx and Gy of the center of gravity G for each detection timing based on the change amounts ΔT and SUM. FIG. 19 shows the information generated by the change amount calculating unit 32, the information generated by the center of gravity determining unit 33, and the deviation calculating unit at the timings of the timings T ₁ and T ₂ shown in FIG. 18. It is a figure which shows the specific example of the information produced by 36).

As shown in FIG. 19, when the change amount (DELTA) T (X, Y) of each coordinate position, the sum SUM of the change amount, and the coordinate position (Gx, Gy) of the center of gravity G of a change area | region are calculated | required, based on these values The deviation calculating unit 36 calculates the deviation of the change area. Specifically, the deviation calculator 36 calculates the deviation σx ^{2 in} the X-axis direction and the deviation σy ^{2 in the} Y-axis direction of the change area based on the following equations, respectively. The following equation means

And,

to be.

In the example shown in FIG. 19, according to the above-described equation, the deviation calculator 36 sets the deviation σx ^{2 in} the X-axis direction 2.394 and the deviation σy ^{2 in the} Y-axis direction 1.222 in the timing T ₁ . Calculate Further, the variation (σx ²⁾ at the timing T ₂ is calculated as 6.125, 1.018 deviations (σy ^2).

Finally, the gesture recognition unit 34 recognizes the gestures of both hands based on the change of the deviation for each detection timing calculated by the deviation calculator 36 and the change of the area of the change area. Similarly to the first embodiment, the gesture pattern of both hands is also stored in the gesture pattern storage unit 40.

20 is a diagram illustrating an example of a gesture pattern stored in the gesture pattern storage unit 40 according to the present embodiment.

As shown in FIG. 20, in addition to the gesture pattern shown in FIG. 12, the gesture pattern memory | storage part 40 memorizes several gesture patterns of both hands which the gesture recognition part 34 can recognize. Using both hands, the user can input various types of gestures to the device by changing the distance of both hands (collecting or spreading both hands).

In the present embodiment, the gesture recognition unit 34 can identify in which direction the two hands are gathered or spread with reference to the deviation calculated by the deviation calculator 36.

Specifically, as shown in the example of FIG. 19, the gesture recognition unit 34 determines that both hands are spread in the horizontal direction when the value of the deviation in the X-axis direction increases with the passage of time. On the contrary, when the value of the deviation decreases, it is judged that the two hands are collected in the horizontal direction from the open state. Similarly, when the value of the deviation in the Y-axis direction increases with the passage of time, it is determined that both hands are spread in the vertical direction. On the contrary, when the value of the deviation decreases, it is judged that both hands are collected in the vertical direction from the state in which the hands are vertically spread.

In addition, when the deviation in the X-axis direction and the deviation in the Y-axis direction increase together, the gesture recognition unit 34 checks whether the hands are spread in an oblique direction or the two hands are held in the Z-axis direction without changing the distance between the two hands. We believe there is a possibility of getting close. In this case, the gesture recognizing unit 34 further determines whether the area is wider or closer to the oblique direction depending on whether the area of the change area is increased.

In addition, when the deviation in the X-axis direction and the Y-axis direction decrease together, the gesture recognition unit 34 determines whether both hands are gathered in an oblique direction, or both hands are separated from the device in the Z-axis direction while the distance between the two hands is the same. I think there is a possibility. In this case, the gesture recognizing unit 34 also determines whether the area of the change area is gathered in an oblique direction or away from the oblique direction.

In addition, when the deviation in the X-axis direction and the deviation in the Y-axis direction do not change so much together, the gesture recognizing unit 34 determines that the distance between the two hands is not changed and is also not moved in the Z-axis direction. Therefore, in this case, since the gesture recognition procedure of one hand of the first embodiment can be applied, it is determined by the procedure shown in Fig. 14 how both hands move in the X-axis direction or the Y-axis direction.

[Gesture recognition processing flow]

21 is a flowchart showing in detail the flow of gesture recognition processing executed by the gesture recognition unit 34 of the present embodiment.

First, referring to FIG. 13, the flow of gesture recognition processing performed by each part of the data processing unit 10 according to the present embodiment will be described. The difference from the first embodiment is different from the first embodiment in S107. 36) in the deviation (σX ^2, and that, to obtain S108 σY ²⁾ for each detection timing, and the timing is a point to be stored is constructed deviation (σX ^2, σY ²⁾ relative to T _i.

The gesture recognition process of S111 in this embodiment is demonstrated, referring FIG.

A gesture recognition unit 34, first, at timings of a predetermined period T _i (at time T ₀ to T _n), and acquires the deviation (σX ^2, σY ²⁾ (S301). It is then determined how the value of the deviation changes (increases, decreases, or does not change) with time.

A gesture recognition unit 34, if there is a change in the deviation (σX ²⁾ (NO in S302), it is determined whether the deviation (σX ²⁾ is increased to reduce a tendency that the trend (S303).

A gesture recognition unit 34 is in the tendency to decrease when the variation (σX ²⁾ (in S303 NO), and then determines whether or not, the deviation (σY ²⁾ has a decreasing trend (S304).

When the deviation σY ² does not tend to decrease, that is, when the deviation σ Y ² tends to increase, or when there is no change (NO in S304), the movement in the X-axis direction is referred to. The movement detected in the predetermined period of time is recognized as a gesture of "03: Put both hands in left and right directions" (S305). On the other hand, the deviation if (σY ²⁾ is in a decrease trend, that is, the deviation (σX ^2, σY ²⁾ a case where the tendency to decrease together (in S304 YES), the gesture recognition unit 34, the timing T _i Each time, the area of the change area is obtained (S306). Then, it is determined whether or not the area of the change area is in a decreasing trend with the passage of time (S307). When the area does not tend to decrease, that is, when there is no change in the area (NO in S307), the movement detected in the predetermined period is recognized as a gesture of "43: Put both hands in an oblique direction" (S308). On the other hand, when the area tends to decrease (YES in S307), the movement detected in the predetermined period is recognized as a "21: Keep hands away" gesture (S309).

On the other hand, the gesture recognition unit 34 if the trend increases the variation (σX ²⁾ (in S303 YES), it is determined whether the following, the deviation (σY ²⁾ is in the increasing tendency (S310).

When the deviation σ Y ² does not tend to increase, that is, when the deviation σ Y ² tends to decrease or there is no change (NO in S310), the predetermined value is based on the movement in the X-axis direction. The movement detected in the period is recognized as a gesture of " 02: hands are spread in left and right directions " (S311). On the other hand, when the deviation σ Y ² tends to increase, that is, when the deviations σ X ² and σ Y ² tend to increase together (YES in S310), the gesture recognition unit 34 changes every timing T _i . The area of the area is acquired (S312). Then, it is determined whether the area of the change area tends to increase with the passage of time (S313). When the area does not tend to increase, that is, when there is no change in area (NO in S313), the movement detected in the predetermined period is recognized as a gesture of " 42: spread both hands in an oblique direction " (S314). On the other hand, when the area tends to increase (YES in S313), the movement detected in the predetermined period of time is recognized as a "20: hand close" gesture (S315).

In the other hand, the gesture recognition unit 34 is S302, the deviation, if (σX ²⁾ that is increased or decreased is not recognized to have the following, (for example, in S302), and determines the presence or absence of the increase or decrease in the deviation (σY ²⁾ (S316 ).

A gesture recognition unit 34 has deviation when there is no change in the (σY ²⁾ That is, the deviation (σX ^2, σY ²⁾ is in the trajectory of the absence of sensitizer, the weight shown in (for example, in S316), FIG. 14 central with Based gesture recognition processing is performed to recognize a gesture. On the other hand, the gesture recognition unit 34, if there is a change in the deviation (σY ²⁾ (NO in S316), it is determined whether the deviation (σY ²⁾ is increased to reduce a tendency that the trend (S317).

When the deviation sigma Y ² tends to decrease (NO in S317), the gesture recognition unit 34 recognizes the movement detected in the predetermined period as the "13: put both hands up and down" gesture (S318). . On the other hand, when the deviation σY ² tends to increase (YES in S317), the gesture recognition unit 34 recognizes the movement detected in the predetermined period as the gesture "12: spread both hands upward and downward" ( S319).

According to the said structure and method, compared with 1st Embodiment, by acquiring more high-resolution temperature information d1, it becomes possible to recognize even a some object or the shape of an object. Therefore, it is possible to realize an input device capable of performing various types of operation inputs without dealing with a huge amount of information as in image processing technology.

In addition, in the above-mentioned embodiment, although the structure which the gesture recognition apparatus 1 recognizes the open / close gesture of both hands was demonstrated, the structure of the gesture recognition apparatus 1 of this invention is not limited to this. The gesture recognition apparatus 1 of the present invention can recognize the opening and closing gesture of a finger of one hand based on the above procedure.

[Modification 2]

In each of the above-described embodiments, the case where the sensor unit 2 and the gesture recognition device 1 (smartphone phone 100) are integrally described has been described. However, the configuration of the gesture recognition device 1 according to the present invention is described herein. It is not limited.

For example, unlike the gesture recognition apparatus 1, the sensor part 2 is installed remotely and the structure which acquires the detection signal of the sensor part 2 to the gesture recognition apparatus 1 using a suitable communication means is also shown. It is included in the invention. That is, the gesture recognition apparatus 1 does not need to be provided with the sensor part 2, In this case, what is necessary is just to provide the appropriate communication means which acquires a detection signal from the sensor part 2.

The present invention is not limited to the above-described embodiments, but various modifications are possible within the scope shown in the claims, and the embodiments obtained by appropriately combining the technical means disclosed in the other embodiments are also included in the technical scope of the present invention. do.

Finally, each block of the gesture recognition apparatus 1, in particular, each part of the data processing unit 10 may be configured by hardware logic, or may be implemented by software using a CPU as follows.

That is, the gesture recognition apparatus 1 includes a central processing unit (CPU) for executing instructions of a control program for realizing each function, a read only memory (ROM) storing the program, and a random access memory (RAM) for expanding the program. ), And a storage device (recording medium) such as a memory for storing the program and various data. An object of the present invention is to provide a computer-readable recording medium for recording program code (executable program, intermediate code program, source program) of a control program of the gesture recognition apparatus 1, which is software for realizing the above functions. It can also be achieved by supplying to the gesture recognizing apparatus 1 and reading out and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy disks / hard disks, and optical disks such as CD-ROM / MO / MD / DVD / CD-R. A disk system, a card system such as an IC card (including a memory card) / optical card, or a semiconductor memory system such as a mask ROM / EPROM / EEPROM / flash ROM can be used.

In addition, the gesture recognition apparatus 1 may be configured to be connectable with the communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited, and for example, the Internet, an intranet, an extranet, a LAN, an ISDN, a VAN, a CATV communication network, a virtual private network, a telephone line network, a mobile communication network, and a satellite communication network can be used. Do. In addition, the transmission medium constituting the communication network is not particularly limited, and, for example, infrared rays such as IrDA or a remote controller, Bluetooth, even a wire such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc. (Registered trademark), 802.11 wireless, HDR, mobile telephone network, satellite line, terrestrial digital network and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is implemented by electronic transmission.

The gesture recognition apparatus of the present invention includes various electronic devices such as a smartphone, a mobile phone, a PDA, a portable game machine, an electronic dictionary, an electronic notebook, a digital camera, a digital video camera, a personal computer, a notebook computer, a television, a recording generating device, and a home game machine. It can be mounted in a device, and can play the input function of these electronic devices. In particular, as an input device of an electronic device with a projector function, it is possible to use very effectively.

1: Gesture recognition device
2: sensor
3: input control unit
4: Memory
5 connector
6: temperature sensor chip (temperature sensor)
7: signal processing unit
8: temporary storage
10: data processing unit
11: communication control unit
12: AD conversion unit
13: signal selector
14: signal amplifier
21: substrate
22: middle case
23: lens
24: outer case
30: temperature information acquisition unit
31: coordinate arrangement
32: change amount calculation unit (change area specifying means / change amount calculating means)
33: center of gravity determining unit (weight center determining unit)
34: gesture recognition unit (gesture recognition means)
35 mask selection unit (mask selection means)
36: deviation calculation part (deviation calculation means)
40: gesture pattern memory
41: mask memory
42: reference value storage unit
100: smart phone

Claims (8)

A temperature sensor in which a plurality of infrared sensors are arranged,
Change region specifying means for specifying, as a region indicating a hand, a change region in which a temperature change has occurred based on a temperature detected by each of the infrared sensors of the temperature sensor;
And the gesture recognizing device is provided with gesture recognition means for recognizing a gesture of a hand by specifying the movement trajectory of the specific change area. The apparatus according to claim 1, further comprising: center of gravity determining means for determining a center of gravity of the change region specified by said change region specifying means,
And the gesture recognizing means specifies the movement trajectory by tracking the position of the center of gravity. The apparatus according to claim 2, further comprising change amount calculating means for calculating a temperature change amount for each infrared sensor by comparing respective temperatures detected by each infrared sensor with a predetermined reference temperature,
And the center of gravity determining means determines the center of gravity of the change area based on the temperature change amount obtained for each infrared sensor. The mask memory according to any one of claims 1 to 3, wherein the mask memory, which stores a mask indicating an invalid invalid placement pattern in the plurality of infrared sensors, is stored in association with the type of application of an electronic device to which the device itself is connected. Wealth,
Mask selection means for selecting from the mask storage section a mask associated with an application during which the electronic device is starting up,
The change area specifying means,
And the change region is specified based only on the temperature detected by the infrared sensor which the mask selected by the mask selecting means is valid. The apparatus according to any one of claims 1 to 3, further comprising deviation calculation means for calculating a deviation of the change region specified by the change region specifying means,
And the gesture recognizing means recognizes the gesture on the basis of the increase and decrease of the deviation with the passage of time. The method of claim 5, wherein the gesture recognition means,
When the deviation of the change area is increased, it is recognized that both hands or fingers moved in the direction of spreading, and when the deviation of the change area is reduced, it is recognized that the movement in the direction in which both hands or fingers are collected. , Gesture recognition device. A temperature acquisition step of acquiring a temperature for each infrared sensor from a temperature sensor in which a plurality of infrared sensors are arranged;
A change region specifying step of specifying a change region in which a temperature change has occurred, as a region indicating a hand, based on the temperature detected by each of the infrared sensors acquired in the temperature obtaining step;
And a gesture recognition step of recognizing a gesture of a hand by specifying a movement trajectory of a specific change area in the change area specifying step. The computer-readable recording medium which recorded the control program for making a computer function as each means of the gesture recognition apparatus in any one of Claims 1-3.

Images