JP5791557B2 - Contact operation support system, contact operation support device, and contact operation method - Google Patents

Description

  The present invention relates to a contact operation support system, a contact operation support device, and a contact operation method.

Conventionally, when a person is contacted (accessed) using a mobile phone, an operation for determining a contact partner (access partner) and a contact means (access means, for example, access to a telephone, mail, blog) through a series of procedures. Therefore, there is a problem in user convenience. For example, as shown in FIG. 18, a contact partner (for example, Emiko Mobile) is first selected (first step), and then a contact means (for example, mail) with the contact partner is selected (second step). It took a lot of time and effort to decide who to contact and how to contact through a series of hierarchical steps.
In addition, when there is a problem in convenience as described above, there is a possibility that the opportunity of contact at the corner may be missed. For example, while looking at a memorable album and trying to get in touch with an old friend, the task of finding an old friend's contact information from the address book is cumbersome. It seems that it is not uncommon for the situation not to be taken.

  On the other hand, a technique has been disclosed for the purpose of allowing a user to easily use address book information stored in a mobile communication terminal (see, for example, Patent Document 1). The mobile phone terminal described in Patent Document 1 generates facial feature point data from facial image data input from a camera, compares it with previously stored facial feature point data, and a registrant corresponding to the matched facial feature point data Information such as name and phone book data is read and displayed on the display or notified by voice.

JP 2009-105652 A

However, the technique of Patent Document 1 has the following problems.
(First problem)
The mobile phone terminal described in Patent Document 1 performs up to the first step (selection of a contact partner) among a series of hierarchical steps by a photographing operation, but does not perform the second step (selection of contact means). After all, there is a problem that the user has to perform the second step separately.

(Second problem)
Further, it is considered that the mobile phone terminal described in Patent Document 1 assumes a case where there is one subject. Therefore, especially in the case of browsing a photo, when there are multiple subjects (when multiple people are shown in the photo), even a person who should be a “desired contact” cannot be specified. There is a problem.

  This invention is made | formed in view of such a situation, and it aims at providing the technique which can determine a desired contact partner and a desired contact means simply.

  In order to solve the above problems, a contact operation support system according to an aspect of the present invention is a contact operation support system that determines a contact partner and a contact means using an image, and recognizes at least a person. A storage unit that stores facial feature data and associates and stores a finger movement pattern indicating a finger movement pattern tracing the image and contact means identification information for identifying the contact means; and a finger movement on the image by the user A recognition unit for recognizing the face area designated by the finger movement on the image, and the face feature data of the face in the face area on the image recognized by the recognition unit is selected from the storage unit and selected. The finger movement obtained from the finger movement recognized by the recognition unit as a contact means for determining the person identified by the face feature data as a contact partner and as a contact means with the contact partner to be determined Said contact means identification information associated with the pattern selected from said storage unit, characterized in that it comprises a determining unit for determining a communication means that is identified by the selected said contact means identification information.

  According to the contact operation support system, it is possible to easily determine a desired contact partner and a desired contact means. That is, according to the contact operation support system, the access is based on a simple procedure of tracing a photograph with a finger instead of a conventional series of procedures (for example, the first step and the second step described above). In addition to the partner, the access means can also be determined. Further, according to the contact operation support system, even when there are a plurality of subjects, it is possible to specify the subject by tracing with a finger, so that it is possible to determine the access partner and the access means without any problem. .

In the contact operation support system, the recognition unit includes a photographing unit that photographs an image and a user, a finger motion recognition unit that recognizes the finger motion on the image by the user from a photographed image photographed by the photographing unit, A face area recognition unit that recognizes a face area in a captured image captured by the imaging unit and that is designated by the finger motion recognized by the finger motion recognition unit, and the determination unit includes: A face feature data generation unit that generates face feature data by extracting a face feature amount in the face region recognized by the face region recognition unit, the face feature data generated by the face feature data generation unit, and the storage unit A face feature data matching unit for matching the face feature data stored in
As a result of the collation by the face feature data collation unit, the face feature data with matching features is selected from the storage unit, and a contact partner determination unit that determines a person identified by the selected face feature data as a contact partner; A finger action pattern generation unit that generates finger action patterns by measuring time-series data of finger positions from finger actions on an image recognized by the finger action recognition unit, and the finger action pattern generation part that generates the finger action pattern A finger motion pattern matching unit that matches a finger motion pattern and the finger motion pattern stored in the storage unit, and the finger motion pattern that matches the pattern as a result of matching by the finger motion pattern matching unit from the storage unit. A contact means determination unit for selecting and determining a contact means identified by the contact means identification information associated with the selected finger movement pattern Unishi may be.

  The contact operation support system further includes a communication unit that communicates with an external communication device, and the communication unit includes an external name of the contact partner determined by the determination unit and the contact means identification information selected by the determination unit. You may make it transmit to a communication apparatus or the command which performs the contact by the contact means determined by the determination part with the contact party determined by the determination part to an external communication apparatus.

  According to the contact operation support system, in addition to a desired contact partner and a desired contact means on a communication device (for example, a notebook PC) to which the contact operation support system is applied, an external communication device (for example, a mobile phone, a notebook) It is possible to easily determine a desired contact partner and desired contact means on a PC or tablet terminal.

  In order to solve the above problem, a contact operation support device according to another aspect of the present invention includes a transparent touch panel formed in a frame of a housing so that the other surface side can be seen from one surface side of the housing. A contact operation support device for determining a contact partner and a contact means using an image seen through the touch panel, storing at least facial feature data for recognizing a person, and on the touch panel A storage unit for storing a finger movement pattern indicating a finger movement pattern and a contact means identification information for identifying a contact means in association with each other, and when the housing is placed on an image, A recognition unit for recognizing a finger motion on the touch panel and a face region on the image seen through the indication region by the finger motion on the touch panel; and a face in the face region on the image recognized by the recognition unit. Previous Face feature data is selected from the storage unit, a person identified by the selected face feature data is determined as a contact partner, and as a contact means with the determined contact partner, from the finger movement recognized by the recognition unit A determining unit that selects the contact means identification information associated with the obtained finger movement pattern from the storage unit and determines a contact means that is identified by the selected contact means identification information; .

  According to the contact operation support device, the desired contact partner and the desired contact means can be easily determined in the same manner as the contact operation support system.

  In the communication operation support device, the recognition unit detects a contact position on the touch panel, and captures a finger motion recognition unit that recognizes the finger motion on the touch panel and an image that can be seen through the touch panel. And a face area recognition unit for recognizing a face area in a photographed image photographed by the photographing unit and corresponding to an instruction area on the touch panel by the finger motion recognized by the finger motion recognition unit; The determination unit includes: a face feature data generation unit that generates face feature data by extracting a face feature amount in the face region recognized by the face region recognition unit; and the face feature data generation unit includes: The facial feature data matching unit that matches the generated facial feature data with the facial feature data stored in the storage unit, and the facial features whose features match as a result of matching by the facial feature data matching unit From the storage unit, a contact partner determining unit that determines a person identified by the selected face feature data as a contact partner, and a finger from the finger motion on the touch panel recognized by the finger motion recognition unit. A finger motion pattern generation unit that measures time-series data of positions and generates a finger motion pattern; the finger motion pattern generated by the finger motion pattern generation unit; and the finger motion stored in the storage unit The finger movement pattern matching unit that matches the pattern, and the finger movement pattern that matches the pattern as a result of the matching by the finger movement pattern matching unit is selected from the storage unit, and the communication associated with the selected finger movement pattern You may make it have a contact means determination part which determines the contact means identified by means identification information.

  The contact operation support device further includes a communication unit that communicates with an external communication device, and the communication unit externally inputs the name of the contact partner determined by the determination unit and the contact means identification information selected by the determination unit. You may make it transmit to a communication apparatus or the command which performs the contact by the contact means determined by the determination part with the contact party determined by the determination part to an external communication apparatus.

  According to the contact operation support device, a desired contact partner and desired contact means on a communication device (for example, a tablet terminal having a transparent touch panel in at least a part of the housing frame) to which the contact operation support device is applied. In addition, a desired contact partner and a desired contact means on an external communication device (for example, a mobile phone, a notebook PC, or a tablet terminal) can be easily determined.

  In order to solve the above problem, a contact operation support device according to another aspect of the present invention is a transparent touch display formed in a frame of a housing so that the other surface side can be seen from one surface side of the housing. A contact operation support apparatus that determines a contact partner and a contact means using an image seen through the touch display, and stores at least facial feature data for recognizing a person, When the storage unit storing the finger movement pattern indicating the movement pattern of the finger tracing on the touch display and the contact means identification information for identifying the contact means in association with each other, and the housing placed on the image, A recognizing unit for recognizing a finger movement on the touch display by a user and a face area on an image seen through an instruction area by the finger movement on the touch display; The face feature data of the face in the face area on the selected image is selected from the storage unit, the person identified by the selected face feature data is determined as a contact partner, and a contact means with the contact partner to be determined The contact means identification information associated with the finger movement pattern obtained from the finger movement recognized by the recognition section is selected from the storage section, and the contact means identified by the selected contact means identification information is determined. And a determining unit for performing the processing.

  According to the contact operation support device, the desired contact partner and the desired contact means can be easily determined in the same manner as the contact operation support system.

  In the contact operation support device, the recognition unit detects a contact position on the touch display, recognizes the finger operation on the touch display, and an image seen through the touch display. A face for recognizing a face area corresponding to an instruction area on the touch display by the finger movement recognized by the finger movement recognition section, which is a face area in a shot image shot by the shooting section; A facial feature data generation unit that extracts facial feature data in the facial region recognized by the facial region recognition unit and generates facial feature data; and the facial feature A face feature data collating unit that collates the facial feature data generated by the data generating unit and the facial feature data stored in the storage unit; and a result of the collation by the facial feature data collating unit; A contact partner determining unit that selects the face feature data having the same character from the storage unit and determines a person identified by the selected face feature data as a contact partner; and the touch display recognized by the finger motion recognition unit. The finger movement pattern generation unit that measures the time series data of the finger position from the finger movement above and generates a finger movement pattern, the finger movement pattern generated by the finger movement pattern generation unit, and the storage unit The finger motion pattern matching unit that matches the stored finger motion pattern, and the finger motion pattern that matches the pattern as a result of matching by the finger motion pattern matching unit is selected from the storage unit, and the selected finger motion pattern A contact means determination unit that determines a contact means identified by the contact means identification information associated with the contact means.

  The contact operation support device further includes a communication unit that communicates with an external communication device, and the communication unit externally inputs the name of the contact partner determined by the determination unit and the contact means identification information selected by the determination unit. You may make it transmit to a communication apparatus or the command which performs the contact by the contact means determined by the determination part with the contact party determined by the determination part to an external communication apparatus.

  According to the contact operation support device, a desired contact partner and a desired contact on a communication device (for example, a tablet terminal having a transparent touch display in at least a part of the housing frame) to which the contact operation support device is applied. In addition to the means, a desired contact partner and a desired contact means on an external communication device (for example, a mobile phone, a notebook PC, or a tablet terminal) can be easily determined.

  The contact operation support device is superimposed on an image seen through the touch display, the finger movement locus recognized by the recognition unit, the mark on the face of the person determined by the determination unit as a contact partner, When the storage unit stores the facial feature data and the name of the person in association with each other, the name of the person determined as the contact partner by the determination unit, or when transmitting a command to an external communication device You may make it further provide the display control part which displays on the said touch display as alerting | reporting information at least 1 of the content of a command.

  According to the contact operation support device, the user can visually check whether an intended operation (finger operation) is input to the device, whether an intended result is obtained from the device, or the like. Therefore, the user can perform the operation again when the operation content and the result content are not correct.

  In the above-described contact operation support device, the display control unit displays the notification information at a position on the image that can be seen through the touch display and the face of the person determined by the determination unit as a contact partner is not hidden. You may make it display.

  According to the contact operation support device, it is possible to prevent the face of the contact partner from being hidden by the notification information.

  In the contact operation support system and the contact operation support device, the finger motion pattern matching unit includes the finger motion pattern generated by the finger motion pattern generation unit and the finger motion stored in the storage unit. The position coordinate at each sample time during finger movement is considered as a two-dimensional vector between the two finger movement patterns of the pattern, and the match between the two finger movement patterns is determined based on the sum of error angles to be made. You may make it collate.

  According to the contact operation support system, it is possible to improve the accuracy of finger movement verification. In other words, the error angle is a measurement that does not change with the size of the finger movement, so if the mark is the same, both the large and small circles ignore the difference in size. Since the match can be determined as the same mark, the accuracy of the finger motion verification is improved.

  In the contact operation support system and the contact operation support device, the finger motion pattern matching unit includes the finger motion pattern generated by the finger motion pattern generation unit and the finger motion stored in the storage unit. DP matching may be applied to calculate the degree of difference between the two-finger movement patterns of the patterns, and the two-finger movement patterns may be collated.

  According to the contact operation support system, it is possible to improve the accuracy of finger movement verification. That is, if the marks are the same, partial expansion / contraction between different finger movement patterns and shifts in corresponding positions are allowed each time trials are performed, and stable collation can be performed, so that collation accuracy of finger movement is improved.

  In order to solve the above-described problem, a contact operation method according to another aspect of the present invention stores at least facial feature data for recognizing a person and a finger motion pattern indicating a motion pattern of a finger tracing an image, A contact operation method using a system having a storage unit that associates and stores contact means identification information for identifying a contact means, comprising: a finger motion on an image by a user; and a face area indicated by the finger motion on the image A recognition step for recognizing the face, and the face feature data of the face in the face area on the image recognized in the recognition step is selected from the storage unit, and the person identified by the selected face feature data is used as a contact partner. The finger motion obtained from the finger motion recognized in the recognition step as a contact means for determining the contact partner determination step and the contact partner determined in the contact partner determination step Said contact means identification information associated with the turn selected from the storage unit, characterized in that it comprises a contact means determination step of determining a communication means that is identified by the selected said contact means identification information.

  According to the contact operation support device, the desired contact partner and the desired contact means can be easily determined in the same manner as the contact operation support system.

  According to the present invention, as described above, a desired contact partner and a desired contact means can be easily determined. Further, according to the present invention, even when there are a plurality of subjects, it is possible to determine the access partner and the access means without any problem.

1 is a schematic external view of a contact operation support device 1 according to a first embodiment of the present invention. It is explanatory drawing explaining an example of the usage method of the contact operation assistance apparatus. It is a schematic sectional drawing of the contact operation assistance apparatus 1. It is a schematic block diagram of the contact operation assistance apparatus 1 grade | etc.,. It is explanatory drawing explaining the function structure of the contact operation assistance apparatus. 4 is an example of information stored in a storage unit 90. It is explanatory drawing explaining the function of the contact operation assistance apparatus. It is explanatory drawing explaining the function of the contact operation assistance apparatus. It is explanatory drawing explaining the function of the contact operation assistance apparatus. It is explanatory drawing explaining the function of the contact operation assistance apparatus. It is explanatory drawing explaining the calculation method of the difference degree by DP matching. It is a flowchart which shows an example of the flow of a process of the contact operation assistance apparatus 1. It is a schematic external view of the contact operation assistance apparatus 2 by the 2nd Embodiment of this invention. It is a schematic block diagram of the contact operation assistance apparatus 2 grade | etc.,. It is explanatory drawing explaining the function structure of the contact operation assistance apparatus 2. FIG. It is explanatory drawing explaining an example of the usage method of the contact operation assistance system. It is explanatory drawing explaining the function structure of the contact operation assistance system 3. FIG. It is explanatory drawing for demonstrating a prior art.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a schematic external view of a contact operation support device 1 according to the first embodiment of the present invention. FIG. 2 is an explanatory diagram illustrating an example of a method for using the contact operation support device 1. FIG. 3 is a schematic cross-sectional view of the contact operation support device 1.

  As shown in FIG. 1A, the contact operation support device 1 includes a transparent touch display (hereinafter referred to as a transmissive type) formed in a frame of a housing so that the other surface side can be seen from one surface side of the housing. (Referred to as a touch display) and two cameras. The transmissive touch display has a display function and a detection function.

  The display function of the transmissive touch display is realized by applying, for example, an electroluminescence (EL) element. The transmissive touch display displays information as a display function according to control of a display control unit 27 (described later). The detection function of the transmissive touch display is realized by applying, for example, a resistance film method, a capacitance method, or the like. As a detection function, the transmissive touch display accepts designation of an arbitrary position on the upper surface (the surface visible in FIG. 1) and detects two-dimensional coordinates. For example, the transmissive touch display detects the two-dimensional coordinates of the contact position of the user's fingertip of the contact operation support device 1.

  That is, the transmissive touch display is obtained by adding a transmissive touch panel (transparent touch panel) having a detection function to the surface of a transmissive display having a display function (see FIG. 3 described later). As described above, since the transmissive touch display is transparent, as shown in FIG. 2, when the contact operation support device 1 is placed on a photograph, the user of the contact operation support device 1 passes through the transmissive touch display. You can see photos.

  The two cameras are provided at positions where information that can be visually recognized on the transmissive touch display (transparent visible information that can be seen through the transmissive touch display, and visible visible information that is displayed on the transmissive touch display) can be captured. In the example of FIG. 1A, each camera is provided on the housing frame so as to face the transmissive touch display side (inside). In each camera, the shooting area of the transmissive touch display shot by one camera and the shooting area on the transmissive touch display shot by the other camera do not overlap as much as possible. It is provided so as to be in the shooting area of any camera as much as possible.

  FIG. 3A is a schematic cross-sectional view of the contact operation support device 1 taken along the alternate long and short dash line D shown in FIG. In FIG. 3A, the contact operation support device 1 is placed on the photograph P. As shown in FIG. 3A, the camera in FIG. 1A includes a first mirror (curve mirror), a second mirror, a lens optical system, and a sensor. The same applies to the right camera in FIG.

  The first mirror has a curvature so that it can receive a wide range of transmitted visible information light flux. The second mirror is provided at a position for receiving the light beam reflected by the first mirror. The second mirror may be a plane mirror or a curved mirror. The lens optical system is provided at a position for receiving the light beam reflected by the second mirror. The lens optical system includes a single or a plurality of condenser lenses. The sensor is provided at a position for receiving the light beam collected by the lens optical system. An example of the sensor is, for example, a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor.

  FIG. 3B is a schematic cross-sectional view of another configuration. In FIG. 3A, the contact operation support device 1 is placed on the photograph P. The camera in FIG. 1A may have a configuration including a wide-angle lens optical system and a sensor, as shown in FIG. The same applies to the right camera in FIG. The sensor of FIG. 3B is the same as the sensor of FIG. The wide-angle lens optical system is a lens optical system having a wider angle of view (viewing angle). For example, a wide-angle lens optical system is a lens optical system with an angle of view exceeding 90 degrees. The wide-angle lens optical system includes a single or a plurality of condenser lenses.

  The respective images (hereinafter referred to as partial images) taken by the two cameras may be joined together by an image processing unit (not shown) and combined into one image. For example, when a partial area of each partial image overlaps, the image processing unit connects a non-overlapping area in both partial images and an overlapping area in one partial image. Note that the image processing unit may perform image processing for correcting aberrations in the partial images and synthesize the corrected partial images.

  Note that since the lens position of the camera is fixed in the housing, the coordinates of the detection area of the detection function of the transmissive touch display and the coordinates of the shooting area by the camera uniquely correspond. For example, the upper left of both areas is defined as (0, 0), the upper right is (1, 0), the lower left is (0, 1), and the lower right is (1, 1). Also good.

  Note that the number of cameras provided in the contact operation support device 1 is not limited to two. The contact operation support device 1 may include one camera, or may include three or more cameras. However, it is desirable that the entire transmissive touch display can be photographed as much as possible.

  FIG. 1B is another configuration example of the contact operation support device 1. As shown in FIG. 1 (b), the contact operation support device 1 has one camera so that one end faces the transmissive touch display side (diagonally below) on the other end of the support portion fixed to the housing frame. You may make it provide. The support unit may be detachable from the housing frame of the contact operation support device 1. Moreover, the support part may be able to bend and stretch. Moreover, the support part may be foldable. The transmissive touch display in FIG. 1B is the same as the transmissive touch display in FIG. The transmissive touch display in FIG. 1C will be described later.

  FIG. 4 is a schematic configuration diagram of the contact operation support device 1 and the like. As shown in FIG. 4, the contact operation support apparatus 1 includes a transmissive display (described above), a camera (described above), a CPU that controls the entire device, a memory, a storage, and a communication interface, each of which is connected by a system bus. Yes. The contact operation support device 1 communicates with an external communication device 9 or the like via a communication interface. An example of the external communication device 9 is a mobile phone, a PC, or a set-top box (STB). Note that the memory of the contact operation support device 1 is a work area that temporarily stores facial feature data generated by a facial feature data generation unit 21 described later, a finger motion pattern generated by a finger motion pattern generation unit 24 described later, and the like. is there.

  As shown in FIG. 4, the communication device 9 includes a storage unit (storage unit), a communication interface, a display unit, and a CPU that controls the entire device, and each is connected by a system bus. The communication device 9 communicates with other communication devices, the contact operation support device 1 and the like through a communication interface. The accumulation unit accumulates (stores) an address book and the like. The display unit displays information stored in the storage unit, information received via the communication interface, and the like.

  FIG. 5 is an explanatory diagram illustrating a functional configuration of the contact operation support device 1. FIG. 6 is an example of information stored in the storage unit 90. 7-10 is explanatory drawing explaining the function of the contact operation assistance apparatus 1. FIG.

  As shown in FIG. 5, the contact operation support device 1 includes a recognition unit 10, a determination unit 20, a communication unit 30, a display unit 40, and a storage unit 90 when viewed from the functional aspect. The recognition unit 10 includes a photographing unit 11, a finger motion recognition unit 12, and a face area recognition unit 13. The determination unit 20 includes a face feature data generation unit 21, a face feature data collation unit 22, a contact partner determination unit 23, a finger movement pattern generation unit 24, a finger movement pattern collation unit 25, a contact means determination unit 26, and a display control unit 27. Have.

  As indicated by broken lines, the photographing unit 11 corresponds to the function of the camera shown in FIG. 4, and the finger motion recognition unit 12 corresponds to the detection function of the transmissive touch display shown in FIG. Unit 13, determination unit 20 (face feature data generation unit 21, face feature data verification unit 22, contact partner determination unit 23, finger movement pattern generation unit 24, finger movement pattern verification unit 25, contact means determination unit 26, display control unit 27) corresponds to the function of the CPU shown in FIG. 4, the communication unit 30 corresponds to the function of the communication interface shown in FIG. 4, and the display unit 40 corresponds to the display function of the transmissive touch display shown in FIG. The storage unit 90 corresponds to the storage function shown in FIG.

  The storage unit 90 stores at least face feature data for recognizing a person, and associates a finger movement pattern indicating a finger movement pattern traced on the display unit 40 with contact means identification information for identifying a contact means. Remember. For example, the storage unit 90 stores face feature data in association with a person's name as shown in FIG. 6 (a), and communicates in association with a finger movement pattern as shown in FIG. 6 (b). Means identification information is stored.

  “Name” in FIG. 6A is an example of person identification information, and includes a nickname and the like in addition to the name. The same applies to a “name” in FIG. The figure in FIG. 6B is a visual representation of finger movement patterns for convenience of explanation. That is, according to FIG. 6B, the storage unit 90 stores the contact means identification information “R01” in association with the finger motion pattern for drawing a rectangle, and the contact means in association with the finger motion pattern for drawing a star. Identification information “R02” is stored. In addition to the above, the finger motion pattern stored in the storage unit 90, that is, the finger motion pattern to be defined may be a finger motion pattern for drawing a circle, a finger motion pattern for drawing a triangle, or the like. Also, a finger movement pattern may be defined in consideration of how to draw. For example, a finger motion pattern that draws a circle clockwise and a finger motion pattern that draws a circle counterclockwise may be defined separately.

  In addition to the above, the storage unit 90 may store a contact unit in association with the contact unit identification information as shown in FIG. In addition to the above, the storage unit 90 may store one or more contact means in association with the name, as shown in FIG. In FIGS. 6C and 6D, the cellular phone and the fixed phone may be distinguished with respect to the phone. In addition, regarding mail, mobile phone mail and PC mail may be distinguished.

  The recognizing unit 10 (finger motion recognizing unit 12) recognizes a finger motion on the display unit 40 by the user when the casing is placed on the image. For example, as illustrated in FIG. 7A, when the user draws a quadrangle with a finger on the display unit 40, the recognition unit 10 recognizes a finger motion (finger motion Y <b> 1) that draws the quadrangle. As shown in FIG. 7B, when the user draws a star with a finger on the display unit 40, the recognition unit 10 recognizes a finger motion (finger motion Y2) for drawing a star.

  In addition, the recognition unit 10 (face area recognition unit 12) uses an image captured by the imaging unit 11 when the casing is placed on the image, and transmits the instruction region by the finger operation on the display unit 40. Recognize the face area on the image. For example, the recognizing unit 10 recognizes a face area (a face area of a female face) on a photograph that can be seen through an instruction area S1 (an area within a square of the finger action Y1) by the finger action Y1 of FIG. To do. In addition, for example, the recognition unit 10 on the photograph seen through the indication area S2 (the area in the rectangle substantially circumscribed by the star mark of the finger action Y2; the area in the dotted line) by the finger action Y2 in FIG. Recognize face area (face area of female face). Details of the recognition unit 10 will be described later.

  The determination unit 20 (the face feature data generation unit 21, the face feature data collation unit 22, and the contact partner determination unit 23) selects from the storage unit 90 the face feature data of the face in the face area on the image recognized by the recognition unit 10. Then, the person identified by the selected face feature data is determined as the contact partner. For example, the deciding unit 20 uses the facial feature data of the female face in the face area recognized by the recognizing unit 10 from the designated area S1 in FIG. 7A, or the recognizing unit 10 designates the designated area S2 in FIG. From the storage unit 90, the facial feature data of the female face in the face area recognized from the above is selected. For example, when the face feature data “ABCD...” In FIG. 6A is selected from the storage unit 90, the determination unit 20 uses “Emiko Mobile” associated with the face feature data “ABCD. decide.

  In addition, the determination unit 20 (finger movement pattern generation unit 24, finger movement pattern matching unit 25, contact means determination unit 26) is a finger obtained from the finger movement recognized by the recognition unit 10 as a contact means with the contact partner to be determined. The contact means identification information associated with the operation pattern is selected from the storage unit 90, and the contact means identified by the selected contact means identification information is determined. For example, the determination unit 20 selects the contact means identification information “R01” associated with the finger movement pattern “finger movement pattern for drawing a rectangle” obtained from the finger movement Y1 in FIG. The contact means identified by the contact means identification information “R01” is determined. Further, for example, the determination unit 20 stores the contact means identification information “R02” associated with the finger movement pattern “finger movement pattern for drawing an asterisk” obtained from the finger movement Y2 of FIG. The contact means selected is determined by the contact means identification information “R02”. Details of the determination unit 20 will be described later.

  The display control unit 27 superimposes on the image seen through the display unit 40, and the finger movement locus recognized by the recognition unit 10 or the mark on the face of the person determined by the determination unit 20 as a contact partner At least one is displayed on the display unit 40 as notification information. When the storage unit 90 stores the facial feature data and the name of the person in association with each other as the notification information as shown in FIG. 6A, the display control unit 27 determines the person determined by the determination unit 20 as the contact partner. May be displayed on the display unit 40. When the communication unit 30 transmits a command to the external communication device 9, the display control unit 27 may display the content of the command on the display unit 40.

  For example, as shown in FIG. 8, the display control unit 27 displays the notification information at a position on the image that can be seen through the display unit 40 so that the face of the person determined by the determination unit 20 as a contact partner is not hidden. It may be displayed. Note that the notification information (command content) in FIG. 8A represents a call instruction to Emiko Mobile (or a call button display instruction immediately before calling Emiko Mobile). The notification information (command content) in FIG. 8B represents a new mail creation screen display command in which Emiko Mobile is designated as the transmission destination.

  The communication unit 30 transmits the name of the contact partner determined by the determination unit 20 and the contact means identification information selected by the determination unit 20 to the external communication device 9. Note that the communication unit 30 may transmit to the external communication device 9 a command to execute communication by the contact means determined by the determination unit 20 with the contact partner determined by the determination unit 20 instead of or in addition to the above. Good.

  For example, as illustrated in FIG. 9, when the determination unit 10 determines that the contact partner is Emiko Mobile and the contact means identification information is “R01”, the communication unit 30 sets the contact partner name “Emiko Mobile” and the contact information. The means identification information “R01” is transmitted to the communication device 9. The communication device 9 that has received the name “Emiko Mobile” and the contact means identification information “R01” from the contact operation support device 1 automatically displays a screen as shown in FIG. 9 on the display unit, for example. For example, as illustrated in FIG. 10, when the determination unit 10 determines that the contact partner is Emiko Mobile and the communication means identification information is “R02”, the communication unit 30 determines the name “Emiko Mobile” of the contact partner. The communication means identification information “R02” is transmitted to the communication device 9. The communication device 9 that has received the name “Emiko Mobile” and the contact means identification information “R02” from the contact operation support device 1 automatically displays a screen as shown in FIG. 10 on the display unit, for example.

  As described above, the contact operation support device 1 uses the image seen through the display unit 40 (transparent visible information seen through the transparent touch display) to determine the contact partner and the contact means. Specifically, the contact operation support device 1 contacts the contact partner if the user designates the face of the desired contact partner with the finger operation corresponding to the desired contact means on the image that has passed through the display unit 40. Determine the means. Hereinafter, determination of a contact partner and a contact means will be described in detail.

(Determining contact person)
The face area recognition unit 13 acquires a captured image from the imaging unit 11 (or an image processing unit (not shown) that processes a captured image of the imaging unit 11). The face area recognition unit 13 recognizes a face area in the captured image. Specifically, the face area recognition unit 13 recognizes a face area in an instruction area by a finger motion. For example, the face area recognition unit 13 recognizes a face area of a female face on a photograph that can be seen through the instruction area S1 of FIG. In other words, the face area recognition unit 13 recognizes a face area in the captured image captured by the imaging unit 11 and corresponding to the instruction area on the display unit 40 by the finger motion recognized by the finger motion recognition unit 12. To do.

  More specifically, the face area recognizing unit 13 performs face detection for the entire area of the captured image stored in the memory (see FIG. 4) after shooting by the shooting unit 11. The face area recognition unit 13 stores the face existing area in the memory as a result of the face detection. For example, the face area recognition unit 13 stores the minimum rectangular area including the face in the memory. After that, when there is a finger movement of the display unit 40 by the user, the face area recognition unit 13 specifies an existing area that is closest to the indicated position by the finger action among the existing areas of the face stored in the memory, It is recognized as a face area in the instruction area by the finger operation. The face area recognition unit 13 reduces the amount of processing required for face detection by performing face detection only in the vicinity of the position indicated by the finger movement after the user performs a finger movement on the display unit 40. May be.

  The technique described in Reference Document 1 below can be applied to face detection.

  [Reference 1] Paul Viola, Michael Jones, "Robust Real-time Object Detection", Second International Workshop on Statistical and Computational Theories of Vision-Modeling, Learning, Computing, and Sampling, 2001.

  In Reference 1, a large number of images including a face are prepared in advance (a predetermined number is prepared), a luminance signal in a region including the face is converted into a Haar-like feature value, The comparison of the luminance signal common to the face image, such as a small luminance signal and a high cheek luminance signal for the eyebrows, is learned by a machine learning method called Adaboost, and a common judgment rule is described for the face image Generated Adaboost classifiers. When detecting a face in a newly input image, a specific rectangle is moved in the raster scan order from left to right and from top to bottom, and the luminance signal in each destination rectangle is Haar-like feature quantity. Whether or not a person's face is included is determined based on whether or not the determination rule described in the Adaboost classifier generated in advance is satisfied.

  The face feature data generation unit 21 extracts face feature amounts in the face area recognized by the face area recognition unit 13 to generate face feature data. For example, the face feature data generation unit 21 generates face feature data by extracting the feature amount of the female face in FIG.

  Note that the technique described in Reference 2 (Japanese Patent Laid-Open No. 2009-212490) can be applied to the generation of facial feature data. Reference 2 invented a technique for comparing images based on adjacent relative bit strings (LBP) that are resistant to variations in illumination and characteristics of photographing equipment, and performing image recognition based on the similarity score. It is.

  In Reference Document 2, an eye-nose point is specified in the detected face image, and the face image is normalized and cut out from the photograph so that the reference according to the position of both eyes and the distance between the eyes and nose is constant. Pay attention to each pixel in the normalized face image one after another, and for each target pixel, the relative value of the relative feature value between the target pixel and a plurality of peripheral pixels adjacent to the target pixel It is converted into LBP (Local Binary Pattern) arranged according to the position. LBP is a texture analysis technique, and is mainly used for area division or the like. This LBP is similar to edge extraction filter (Laplacian filter, Canny filter, etc.) processing in the field of image processing, and the output value is determined based on the relative value of the feature quantity between the pixel of interest and the surrounding pixels. Although two face images obtained by photographing the face of the same person under different illuminations have a luminance difference or a contrast difference according to the illumination variation, the influence of the illumination variation is suppressed in the LBP image. When the LBP of all pixels is obtained, the LBP of the reference face image and the LBP of each referenced face image are compared, and a similarity score is calculated based on the similarity between the two.

  The face feature data collation unit 22 collates the face feature data generated by the face feature data generation unit 21 with the face feature data stored in the storage unit 90. That is, the face feature data collating unit 22 collates the face on the captured image instructed by the user with the faces of a plurality of persons who have previously stored face feature data.

  The contact partner determining unit 23 selects face feature data whose features match as a result of the matching by the face feature data matching unit 22 from the storage unit 90, and determines a person identified by the face feature data as a contact partner. For example, when the facial feature data “ABCD...” Generated by the facial feature data generation unit 21 is generated, the facial feature data “ABCD...” Generated by the facial feature data generation unit 21 and the facial feature of FIG. Since the collation result indicates that the features of the data “ABCD...” Match, the contact partner determination unit 23 selects the face feature data “ABCD...” In FIG. "Emiko Mobile" associated with "" is determined as the contact partner. That is, the contact partner determination unit 23 determines a person whose face matches the face on the captured image instructed by the user as the contact partner.

(Determination of contact method)
The finger movement recognition unit 12 detects the contact position on the display unit 40 and recognizes the finger movement on the display unit 40. In other words, the finger motion recognition unit 12 detects a contact position on the display unit 40 as a finger motion on the display unit 40. For example, the finger movement recognition unit 12 recognizes the finger movement Y1 in FIG. That is, the finger movement recognition unit 12 detects the contact position when the finger movement Y1 of FIG.

  The finger movement pattern generation unit 24 measures the time series data of the finger position from the finger movement on the display unit 40 recognized by the finger movement recognition unit 12 and generates a finger movement pattern. For example, the finger movement pattern generation unit 24 measures the time series data of the position of the finger movement Y1 in FIG. 7A and generates a finger movement pattern (finger movement drawing a square).

  More specifically, the finger movement pattern generation unit 24 performs re-sampling so that the time series data of the finger position measured from the start to the end of finger tracing becomes a constant series length N, and the finger movement pattern generation unit 24 performs finger movement. A pattern a = (a [1], a [2],..., A [N]) is generated. Note that a [t] = (a_x [t], a_y [t]) is the coordinates of the finger position at the sample time t of the finger motion pattern a.

  The finger movement pattern generation unit 24 may convert the coordinates of the finger movement pattern a as follows in order to improve the collation accuracy by a finger movement pattern collation unit 25 described later. Now, assume that the center of gravity of the finger movement pattern a is <a> = (<a_x>, <a_y>). The finger motion pattern generation unit 24 translates the entire finger motion pattern a by the center of gravity <a> to obtain a new finger motion pattern a ′. That is, a ′ [t] = a [t] − <a>, t = 1, 2,. In other words, the finger motion pattern generation unit 24 performs coordinate conversion of the finger motion pattern a so that the center of gravity becomes 0 to obtain a new finger motion pattern a ′.

  The finger motion pattern matching unit 25 matches the finger motion pattern generated by the finger motion pattern generation unit 24 with the finger motion pattern stored in the storage unit 90. That is, the finger motion pattern matching unit 25 matches the finger motion pattern by the user with a plurality of finger motion patterns stored in advance.

  For example, the finger movement pattern collation unit 25 performs a finger movement between the finger movement patterns generated by the finger movement pattern generation unit 24 and the finger movement patterns stored in the storage unit 90. The position coordinates at each sample time are regarded as a two-dimensional vector, a match between the two-finger movement patterns is determined based on the sum of error angles formed, and the two-finger movement patterns are collated.

The finger movement pattern generated by the finger movement pattern generation unit 24 is a, and the finger movement patterns stored in the storage unit 90 are b1, b2, b3,. The finger motion pattern bj stored in the storage unit 90 may be a finger motion pattern that can be drawn with a single stroke, for example, b1 is a finger motion pattern when a circle is traced with a finger, and b2 is a ☆ mark. desirable.
Also, the coordinates of the finger position at the sample time t of the finger movement pattern bj are bj [t] = (bj_x [t], bj_y [t]), and bj = (bj [1], bj [2], bj [3 ],..., Bj [N]).

  The technique described in Reference Document 3 (Japanese Patent Application Laid-Open No. 2008-129704) can be applied to determine the match between the two series of finger movement patterns a and finger movement patterns b.

  In Reference Document 3, the degree of difference D (a, b) is calculated as an index for determining the match between the finger motion pattern a and the finger motion pattern b. The position coordinate a [s] of the finger motion pattern a at the sample time s and the position coordinate b [t] of the finger motion pattern b at the sample time t are regarded as two-dimensional vectors, and a [s] is obtained by the following equation (1). And b [t] to calculate the error angle θ [s, t], and the sum of the error angles θ [s, t] is defined as the difference D (a, b) between the finger motion patterns a and b.

  When the directions of a [s] and b [t] are the same, the error angle indicates a small value close to 0. Conversely, when the directions of a [s] and b [t] are different, the error angle shows a large value. Accordingly, the degree of difference D (a, b) is smaller and closer to 0 indicates that the degree of coincidence between the finger movement patterns a and b is higher.

  Further, since the error angle θ [s, t] is a measurement amount that is invariant to the sizes of a [s] and b [t], for example, the size of the size of the large circle mark and the small circle mark is small. You can ignore the difference and determine the match as the same circle. Even though they are intended to be drawn with the same size, the size may actually differ, so that the coincidence determination using the error angle, which is a measurement amount that does not change in size, is effective.

  On the other hand, when using a mark that is difficult to draw exactly the same every time and tends to partially expand or contract each time it is drawn (for example, a ☆ mark), depending on the degree of expansion or contraction, the degree of difference There is a possibility that the value becomes large and the mark is not determined as the intended mark. In such a case, DP matching is applied to the calculation of the degree of difference between the finger movement patterns, and the partial expansion and contraction and the shift of the corresponding position are absorbed and stabilized.

  That is, the finger movement pattern matching unit 25 calculates the difference between the finger movement patterns generated by the finger movement pattern generation unit 24 and the finger movement patterns stored in the storage unit 90. DP matching may be applied to collate both-finger motion patterns.

  When DP matching is applied to the calculation of the difference between the finger movement patterns, the difference D (a, b) is calculated recursively using the following formula (2) to the following formula (6).

  FIG. 11 is an explanatory diagram for explaining a method of calculating the degree of difference D (a, b) by DP matching. In FIG. 11, they are expressed as as = a [s] and bt = b [t]. The row direction of the table corresponds to the finger movement pattern b stored in the storage unit 90, the column direction corresponds to the newly input finger movement pattern a, and the coordinates of the finger position at the sample time t are in the t-th row and the s-th column. An error angle θ [s, t] between b [t] and the finger position coordinate a [s] at the sample time s is obtained from the above equation (1), and the difference degree D [s, t is obtained from the above equation (3). t] is calculated recursively. The difference degree D [s, t] is calculated for all of the N × N frames from s = 1, t = 1, and the final difference degree D (a, b) is obtained. The difference degree D (a, b) finally obtained means that the smaller the value, the higher the degree of coincidence between the finger movement patterns a and b. Conversely, the larger the value, the lower the matching degree. Therefore, a threshold value T for collation determination is set, and the finger motion pattern indicating the minimum difference degree D (a, bj) among the P finger motion patterns bj stored in the storage unit 90 is smaller than the threshold value T. If the degree D (a, bj) is indicated, it is determined that the newly input finger action pattern a matches the finger action pattern bj.

  The contact means determination unit 26 selects, from the storage unit 90, a finger action pattern that matches the pattern as a result of the matching by the finger action pattern matching unit 25, and is identified by the contact means identification information associated with the selected finger action pattern. Determine the communication method. For example, when the finger movement recognition unit 12 recognizes the finger movement Y1 of FIG. 7A, the finger movement pattern “finger drawing a rectangle” generated by the finger movement pattern generation unit 24 and the finger movement pattern of FIG. Since the collation result indicates that the characteristics of the finger movement pattern “finger-drawing a square” match, the contact means determination unit 26 selects the finger movement pattern “finger-drawing a square” in FIG. The contact means identified by the contact means identification information “R01” associated with the selected finger action pattern “finger-drawing a rectangle” is determined. That is, the contact means determination unit 26 determines the contact means identified by the contact means identification information associated with the finger action pattern that matches the finger action pattern by the user.

  Hereinafter, the flow of processing of the contact operation support apparatus 1 will be described. FIG. 12 is a flowchart illustrating an example of a process flow of the contact operation support device 1. For convenience of explanation, human actions are shown in step S1.

  In FIG. 11, the user places the contact operation support apparatus 1 on the photo album as shown in FIG. 2 (step S1). The imaging unit 11 captures an area including a photograph under the contact operation support device 1 (step S2). The captured image is temporarily stored in a memory (see FIG. 4). The face area recognition unit 13 recognizes a face area on the captured image (step S3). That is, the face area recognition unit 13 performs face detection on the entire area of the captured image stored in the memory and stores the face existing area (for example, a rectangular area) in the memory.

  When the user designates the face of the desired contact partner with the finger motion corresponding to the desired contact means on the image seen through the display unit 40, the finger motion recognition unit 12 recognizes the finger motion on the display unit 40. (Step S4). The face area recognition unit 13 recognizes a face area in the instruction area on the display unit 40 by the finger movement recognized by the finger movement recognition unit 12 (step S5). For example, the face area recognizing unit 13 identifies the existing area closest to the position indicated by the finger movement from the existing areas of the face stored in the memory, and recognizes it as the face area in the specified area. The face feature data generation unit 21 extracts face feature amounts and generates face feature data (step S6). The face feature data matching unit 22 determines whether the generated face feature data matches the face feature data of a plurality of persons stored in the storage unit 90, and the face on the captured image instructed by the user and the face feature in advance. The faces of a plurality of persons storing data are collated (step S7). As a result of the collation, the contact partner determination unit 23 determines a person whose face matches the face on the captured image designated by the user as a contact partner (step S8).

  The finger movement pattern generation unit 24 measures the time series data of the finger position from the finger movement on the display unit 40 recognized by the finger movement recognition unit 12, and generates a finger movement pattern (step S9). The generated finger movement pattern is temporarily stored in a memory (see FIG. 4). The finger movement pattern collation unit 25 collates the finger movement pattern generated by the finger movement pattern generation unit 24 with the finger movement pattern stored in the storage unit 90 (Step S10). The contact means determination unit 26 determines the contact means identified by the contact means identification information associated with the finger action pattern that matches the finger action pattern by the user as a result of the collation (step S11).

  The display control unit 27 displays notification information (for example, a trajectory of finger movement, a mark on the face of a person determined as a contact partner, the name of the person determined as a contact partner, and the content of a command transmitted in the next step). 40 (step S12). The communication unit 30 transmits to the communication device 9 a command for executing contact with the contact partner determined in step S8 by the contact means determined in step S11. Then, the flowchart of FIG. 12 ends.

  In step S <b> 13, the communication unit 30 may transmit the name of the contact partner and contact means identification information to the communication device 9 instead of the above command. The display control unit 27 may display the name of the person determined as the contact partner on the display unit 40 after the contact partner is determined (following step S8).

  In the flowchart of FIG. 12, the face area recognition unit 13 performs face detection on the entire area of the captured image stored in the memory before the finger movement (step S3). An area that is closest to the position indicated by the finger movement is identified and recognized as a face area in the pointing area (step S5), but the face movement is not performed before the finger movement is detected. It may be recognized as a face area within the designated area by performing face detection only for the vicinity of the designated position by the finger movement for the first time later.

  In the flowchart of FIG. 12, the contact operation support apparatus 1 determines the contact means after determining the contact partner, but may determine the contact partner after determining the contact means. That is, in FIG. 12, the order of determining the contact partner (step S6 to step S8) and the procedure of determining the contact means (step S9 to step S11) may be interchanged.

  As described above, according to the contact operation support device 1 according to the first embodiment, it is possible to easily determine a desired contact partner and a desired contact means. That is, the first step and the second step described in the assignment can be performed at that time. For example, when getting information about an acquaintance or contacting an acquaintance, you can trace the printed photo with your finger, identify the person you want to access by recognizing the person traced with your finger, Can be realized.

[Second Embodiment]
Hereinafter, a second embodiment of the present invention will be described with reference to the drawings. FIG. 13 is a schematic external view of a contact operation support device 2 according to the second embodiment of the present invention.

  As shown in FIG. 13A, the contact operation support device 2 is a transparent touch panel (hereinafter referred to as a transmissive touch panel) formed in the frame of the housing so that the other surface side can be seen from one surface side of the housing. And two cameras. The transmissive touch panel is realized by applying, for example, a resistance film method, a capacitance method, or the like. The transmissive touch panel receives designation of an arbitrary position on the upper surface (the surface visible in FIG. 13) and detects two-dimensional coordinates. For example, the transmissive touch panel detects the two-dimensional coordinates of the contact position of the fingertip of the user of the contact operation support device 2.

  The two cameras included in the contact operation support apparatus 2 illustrated in FIG. 13A are the same as the two cameras included in the contact operation support apparatus 1 illustrated in FIG. That is, in terms of appearance, the contact operation support device 1 shown in FIG. 1A is provided with a transmissive touch display, whereas the contact operation support device 2 shown in FIG. 13A is provided with a transmissive touch panel. .

  Since the transmissive touch panel is transparent, like the contact operation support device 1 shown in FIG. 2, when the contact operation support device 2 is placed on the photograph, the user of the contact operation support device 2 passes through the transparent touch panel. You can see photos.

  FIG. 13B is another configuration example of the contact operation support device 2. A contact operation support device 2 shown in FIG. 13B is different from the contact operation support device 1 shown in FIG. 1B in that it includes a transmissive touch display.

FIG. 13C is another configuration example of the contact operation support device 2. The contact operation support device 2 shown in FIG. 13 (c) is similar to the contact operation support device 2 shown in FIG. 13 (b), and further has a transmissive touch panel at the other end of the support portion fixed to the housing frame. The projector is provided with one projector that projects an image. In addition, the support part for projectors may be detachable from the housing frame of the contact operation support device 2, bendable / extendable, or foldable, similar to the support part for the camera. Also good. Moreover, you may share a support part with a camera and a projector. That is, a camera and a projector may be attached to one support part.
Unless otherwise specified, the contact operation support device 2 does not include a projector as shown in FIGS. 13 (a) and 13 (b).

FIG. 14 is a schematic configuration diagram of the contact operation support device 2 and the like. As shown in FIG. 14, the contact operation support device 2 includes a transparent touch panel (described above), a camera (described above), a CPU that controls the entire device, a memory, a storage, and a communication interface, each of which is connected by a system bus. Yes. The contact operation support device 2 communicates with the communication device 9 and the like illustrated in FIG. 14 through a communication interface. That is, the configuration of the contact operation support device 2 is different in that the contact operation support device 1 shown in FIG.
However, when a projector is provided as shown in FIG. 13C, the contact operation support apparatus 2 further includes a projector as an output system.

  FIG. 15 is an explanatory diagram illustrating the functional configuration of the contact operation support device 2. As shown in FIG. 15, the contact operation support device 2 includes a recognition unit 110, a determination unit 120, a communication unit 130, and a storage unit 190 when viewed from the functional aspect. The recognition unit 110 includes a photographing unit 111, a finger motion recognition unit 112, and a face area recognition unit 113. The determination unit 120 includes a face feature data generation unit 121, a face feature data collation unit 122, a contact partner determination unit 123, a finger action pattern generation unit 124, a finger action pattern collation unit 125, and a contact means determination unit 126.

  As indicated by a broken line, the photographing unit 111 corresponds to the function of the camera shown in FIG. 14, the finger motion recognition unit 112 corresponds to the transmission type touch panel shown in FIG. The unit 120 (the face feature data generation unit 121, the face feature data collation unit 122, the contact partner determination unit 123, the finger motion pattern generation unit 124, the finger motion pattern collation unit 125, and the contact means determination unit 126) is the CPU shown in FIG. The communication unit 130 corresponds to the function of the communication interface shown in FIG. 14, and the storage unit 190 corresponds to the function of the storage shown in FIG.

  However, when a projector is provided as shown in FIG. 13C, the contact operation support device 2 corresponds to a display unit (not shown) corresponding to a transmissive touch panel that is a projection surface by the projector and a function of a CPU. And a display control unit (not shown) for controlling the projector.

  Note that the functions of the recognition unit 110 (the photographing unit 111, the finger motion recognition unit 112, and the face area recognition unit 113) included in the contact operation support device 2 are the same as the recognition unit 10 (the image capturing unit 11, finger operation) included in the contact operation support device 1. Since it is the same as the recognition unit 12 and the face area recognition unit 13), the description is omitted. The determination unit 120 (the face feature data generation unit 121, the face feature data verification unit 122, the contact partner determination unit 123, the finger motion pattern generation unit 124, the finger motion pattern verification unit 125, the contact means determination unit 126 provided in the contact operation support device 2 ) Functions of the determination unit 20 (face feature data generation unit 21, face feature data verification unit 22, contact partner determination unit 23, finger action pattern generation unit 24, finger action pattern verification unit 25, Since it is the same as that of the contact means determination unit 26), the description is omitted. The same applies to the communication unit 130 and the storage unit 190.

  As described above, according to the contact operation support device 2 according to the second embodiment, as in the contact operation support device 1 according to the first embodiment, it is possible to easily determine a desired contact partner and a desired contact means. Further, when a projector is provided as shown in FIG. 13C, whether or not the intended finger movement is input by the notification information as in the case of the contact operation support device 1 according to the first embodiment, the intended result. It is possible to make the user visually confirm whether or not is obtained from the apparatus.

[Third Embodiment]
Hereinafter, a third embodiment of the present invention will be described with reference to the drawings. FIG. 16 is an explanatory diagram illustrating an example of a method for using the contact operation support system 3. FIG. 17 is an explanatory diagram illustrating the functional configuration of the contact operation support system 3.

  The contact operation support system 3 includes a camera 3-1 and a computer 3-2 as shown in FIG. The installation location of the camera 3-1 is not particularly limited. For example, it may be indoors or outdoors. In FIG. 16, the number of cameras is one, but the contact operation support system 3 may be composed of two or more cameras.

  The contact operation support system 3 includes a recognition unit 210, a determination unit 220, a communication unit 230, and a storage unit 290 as shown in FIG. The recognition unit 210 includes a photographing unit 211, a finger motion recognition unit 212, and a face area recognition unit 213. The determination unit 220 includes a face feature data generation unit 221, a face feature data verification unit 222, a contact partner determination unit 223, a finger motion pattern generation unit 224, a finger motion pattern verification unit 225, and a contact means determination unit 226.

  As indicated by a broken line, the photographing unit 211 corresponds to the function of the camera 3-1 illustrated in FIG. 16, and includes a finger motion recognition unit 212, a face area recognition unit 213, a face feature data generation unit 221, and face feature data. The collation unit 222, the contact partner determination unit 223, the finger movement pattern generation unit 224, the finger movement pattern collation unit 225, the contact means determination unit 226, the communication unit 230, and the storage unit 290 are added to the functions of the computer 3-2 illustrated in FIG. Equivalent to.

  The imaging unit 211 images the surroundings. For example, the photographing unit 211 photographs a finger operation by the user on a photo album as shown in FIG. The captured image is transmitted to the computer by wireless (or wired).

The finger motion recognition unit 212 recognizes the finger motion on the image by the user from the captured image captured by the imaging unit 211. The face area recognizing unit 213 recognizes a face area in the captured image captured by the image capturing unit 211 and designated by the finger motion recognized by the finger motion recognizing unit 212.
That is, the recognizing unit 210 (the photographing unit 211, the finger motion recognizing unit 212, and the face area recognizing unit 213) recognizes the finger motion on the image by the user and the face region instructed by the finger motion.

  Similar to the storage unit 90 of the contact operation support device 1, the storage unit 290 stores at least facial feature data for recognizing a person, and identifies a finger motion pattern indicating a finger motion pattern tracing the image and a contact means. And associated contact means identification information. For example, the storage unit 290 stores information as illustrated in FIG.

The determination unit 220 (the face feature data generation unit 221, the face feature data collation unit 222, and the contact partner determination unit 223) selects from the storage unit the facial feature data of the face in the face area on the image recognized by the recognition unit 210. The person identified by the selected face feature data is determined as the contact partner.
Specifically, the face feature data generation unit 221 extracts the facial feature amount in the face area recognized by the face area recognition unit 213, similarly to the face feature data generation unit 21 included in the contact operation support device 1. Generate feature data. The face feature data matching unit 222 uses the face feature data generated by the face feature data generation unit 211 and the face feature data stored in the storage unit 290 in the same manner as the face feature data matching unit 22 included in the contact operation support device 1. Collate. The contact partner determination unit 223 selects, from the storage unit 290, the face feature data whose features match as a result of the collation by the face feature data collation unit 222, as with the contact partner determination unit 23 included in the contact operation support device 1. A person identified by the facial feature data is determined as a contact partner.

Also, the determination unit 220 (finger movement pattern generation unit 224, finger movement pattern collation unit 225, and contact means determination unit 226) identifies the contact means associated with the finger movement pattern obtained from the finger movement recognized by the recognition unit 210. Information is selected from the storage unit 290, and the contact means identified by the selected contact means identification information is determined.
Specifically, the finger movement pattern generation unit 224 is similar to the finger movement pattern generation unit 24 included in the contact operation support device 1, and the time-series data of the finger position from the finger movement on the image recognized by the finger movement recognition unit 212. Is measured to generate a finger movement pattern. The finger movement pattern matching unit 225 is similar to the finger movement pattern matching unit 25 included in the contact operation support device 1, and the finger movement pattern generated by the finger movement pattern generation unit 224 and the finger movement pattern stored in the storage unit 290. To match. Similar to the contact means determination unit 26 provided in the contact operation support device 1, the contact means determination unit 226 selects, from the storage unit 290, the finger action pattern that matches the pattern as a result of the matching by the finger action pattern matching unit 225. The contact means identified by the contact means identification information associated with the finger movement pattern is determined.

  The communication unit 230 transmits the name of the contact partner determined by the determination unit 220 and the contact means identification information selected by the determination unit to the external communication device 9 as in the communication unit 30 included in the contact operation support device 1, or Then, a command for executing contact by the contact means determined by the determination unit 220 with the contact partner determined by the determination unit 220 is transmitted to the external communication device 9.

  As described above, according to the contact operation support system 3 according to the third embodiment, as in the contact operation support device 1 according to the first embodiment, it is possible to easily determine a desired contact partner and a desired contact means.

  In the above description, the contact operation support system 3 of the third embodiment has been described as being configured by two devices, the camera 3-1 and the computer 220, but may be configured by one or more devices. .

  In the above description, in the contact operation support device 1 of the first embodiment, the determination unit 20 determines the contact partner and the contact means, and the communication unit 30 determines the determined contact partner and the contact means such as a telephone call and an email. Although it has been described that the transmission is performed to the communication device 9 having the function, the communication operation support device 1 itself may have functions such as telephone and e-mail, similarly to the communication device 9. That is, the contact operation support device 1 may communicate with the contact partner determined by the contact operation support device 1 using the contact means determined by the contact operation support device 1 itself. In other words, the contact operation support device 1 may determine the contact partner and the contact means for itself instead of determining the contact partner and the contact means for another device. The same applies to the contact operation support device 2 of the second embodiment and the computer 3-2 in the contact operation support system of the third embodiment.

  Note that, as described above, when the contact operation support device 1 (same as the contact operation support device 2 and the computer 3-2) has functions such as telephone and e-mail, the contact partner and contact means determined by itself are changed. It is not necessary to transmit to the communication device 9 which is a device. For example, the contact operation support device 1 (such as the contact operation support device 2 and the computer 3-2) having functions such as telephone and e-mail transmits a contact partner and a contact means to a predetermined communication device 9, or Whether the contact means is used by itself may be switched according to the setting of the user.

  If the contact operation support device 1 has functions such as telephone and e-mail, it may have an appearance as shown in FIG. 1C instead of the appearance as shown in FIG. That is, the contact operation support device 1 in FIG. 1C is a modification of the contact operation support device 1 in FIG. 1A, but a transparent touch is applied to a part of the frame (right side in FIG. 1C). It has a display, and a part in the frame (left side in FIG. 1C) has a normal touch display. The same applies to the contact operation support device 2 of FIG.

  A program for executing each process of the contact operation support devices 1 and 2 and the contact operation support system 3 according to the embodiment of the present invention is recorded on a computer-readable recording medium and recorded on the recording medium. The above-described various processes relating to the respective processes of the contact operation support devices 1 and 2 and the contact operation support system 3 may be performed by causing the computer system to read and execute the program. Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

1, 2, 3 ... Contact operation support device 9 ... Communication devices 10, 110, 210 ... Recognition units 11, 111, 211 ... Shooting units 12, 112, 212 ... Finger motion recognition units 13, 113, 213 ... Face region recognition unit 20, 120, 220, determination unit 21, 121, 221, face feature data generation unit 22, 122, 222, face feature data collation unit 23, 123, 223, contact partner determination unit 24, 124, 224, operation pattern generation unit 25, 125, 225 ... finger movement pattern matching units 26, 126, 226 ... communication means determination unit 27 ... display control units 30, 130, 230 ... communication unit 40 ... display unit

Claims (16)

A contact operation support system for determining a contact person and a contact means using an image,
A storage unit that stores at least facial feature data for recognizing a person, and stores a finger movement pattern indicating a movement pattern of a finger tracing an image in association with contact means identification information for identifying a contact means;
A recognition unit for recognizing a finger motion on an image by a user and a face area instructed by the finger motion on the image;
The face feature data of the face in the face area on the image recognized by the recognition unit is selected from the storage unit, and the person identified by the selected face feature data is determined as a contact partner, and the contact partner to be determined As the contact means, the contact means identification information associated with the finger movement pattern obtained from the finger movement recognized by the recognition section is selected from the storage section, and is identified by the selected contact means identification information. A contact operation support system comprising: a determination unit that determines a contact means. The recognition unit
A photographing unit for photographing an image and a user;
A finger motion recognition unit for recognizing the finger motion on an image by a user from a captured image captured by the imaging unit;
A face area in a photographed image photographed by the photographing section, and a face area recognition section for recognizing a face area instructed by the finger movement recognized by the finger movement recognition section,
The determination unit
A facial feature data generation unit that extracts facial feature amounts in the facial region recognized by the facial region recognition unit and generates facial feature data;
A face feature data collation unit that collates the face feature data generated by the face feature data generation unit with the face feature data stored in the storage unit;
As a result of the collation by the face feature data collation unit, the face feature data with matching features is selected from the storage unit, and a contact partner determination unit that determines a person identified by the selected face feature data as a contact partner;
A finger motion pattern generation unit that measures time series data of a finger position from a finger motion on an image recognized by the finger motion recognition unit and generates a finger motion pattern;
A finger motion pattern matching unit that matches the finger motion pattern generated by the finger motion pattern generation unit and the finger motion pattern stored in the storage unit;
As a result of the collation by the finger movement pattern collation unit, the finger movement pattern that matches the pattern is selected from the storage unit, and a contact unit identified by the contact unit identification information associated with the selected finger movement pattern is determined. The contact operation support system according to claim 1, further comprising: The contact operation support system includes:
A communication unit that communicates with an external communication device;
The communication unit is
The name of the contact partner determined by the determination unit and the contact means identification information selected by the determination unit are transmitted to an external communication device, or the contact determined by the determination unit with the contact partner determined by the determination unit The communication operation support system according to claim 1 or 2, wherein a command for executing communication by means is transmitted to an external communication device. A transparent touch panel formed in a frame of the housing so that the other surface side can be seen from one surface side of the housing, and using an image seen through the touch panel, A contact operation support device for determining
A storage unit that stores at least facial feature data for recognizing a person, and stores a finger movement pattern indicating a movement pattern of a finger tracing the touch panel in association with contact means identification information for identifying a contact means;
A recognition unit that recognizes a finger movement on the touch panel by a user and a face area on the image that can be seen through an instruction area by the finger movement on the touch panel when the casing is placed on the image When,
The face feature data of the face in the face area on the image recognized by the recognition unit is selected from the storage unit, and the person identified by the selected face feature data is determined as a contact partner, and the contact partner to be determined As the contact means, the contact means identification information associated with the finger movement pattern obtained from the finger movement recognized by the recognition section is selected from the storage section, and is identified by the selected contact means identification information. And a determination unit that determines a communication means. The recognition unit
A finger movement recognition unit that detects a contact position on the touch panel and recognizes the finger movement on the touch panel;
A photographing unit for photographing an image seen through the touch panel;
A face area recognition unit for recognizing a face area in a photographed image photographed by the photographing unit and corresponding to an instruction area on the touch panel by the finger motion recognized by the finger motion recognition unit. ,
The determination unit
A facial feature data generation unit that extracts facial feature amounts in the facial region recognized by the facial region recognition unit and generates facial feature data;
A face feature data collation unit that collates the face feature data generated by the face feature data generation unit with the face feature data stored in the storage unit;
As a result of the collation by the face feature data collation unit, the face feature data with matching features is selected from the storage unit, and a contact partner determination unit that determines a person identified by the selected face feature data as a contact partner;
A finger motion pattern generation unit that measures time series data of a finger position from the finger motion on the touch panel recognized by the finger motion recognition unit and generates a finger motion pattern;
A finger motion pattern matching unit that matches the finger motion pattern generated by the finger motion pattern generation unit and the finger motion pattern stored in the storage unit;
As a result of the collation by the finger movement pattern collation unit, the finger movement pattern that matches the pattern is selected from the storage unit, and a contact unit identified by the contact unit identification information associated with the selected finger movement pattern is determined. The contact operation support device according to claim 4, further comprising: a contact means determination unit that performs communication. The contact operation support device includes:
A communication unit that communicates with an external communication device;
The communication unit is
The name of the contact partner determined by the determination unit and the contact means identification information selected by the determination unit are transmitted to an external communication device, or the contact determined by the determination unit with the contact partner determined by the determination unit The communication operation support apparatus according to claim 4 or 5, wherein a command for executing communication by means is transmitted to an external communication device. It has a transparent touch display formed in the frame of the housing so that the other surface side can be seen from one surface side of the housing, and communicates with a contact partner using an image seen through the touch display A contact operation support device for determining means,
A storage unit that stores at least facial feature data for recognizing a person, and stores a finger movement pattern indicating a movement pattern of a finger tracing on the touch display in association with contact means identification information for identifying a contact means; ,
When the housing is placed on the image, the user recognizes the finger movement on the touch display and the face area on the image seen through the instruction area by the finger movement on the touch display. A recognition unit;
The face feature data of the face in the face area on the image recognized by the recognition unit is selected from the storage unit, and the person identified by the selected face feature data is determined as a contact partner, and the contact partner to be determined As the contact means, the contact means identification information associated with the finger movement pattern obtained from the finger movement recognized by the recognition section is selected from the storage section, and is identified by the selected contact means identification information. And a determination unit that determines a communication means. The recognition unit
A finger movement recognition unit that detects a contact position on the touch display and recognizes the finger movement on the touch display;
A photographing unit for photographing an image seen through the touch display;
A face area recognition unit for recognizing a face area in a photographed image captured by the photographing unit and corresponding to an instruction area on the touch display by the finger motion recognized by the finger motion recognition unit. And
The determination unit
A facial feature data generation unit that extracts facial feature amounts in the facial region recognized by the facial region recognition unit and generates facial feature data;
A face feature data collation unit that collates the face feature data generated by the face feature data generation unit with the face feature data stored in the storage unit;
As a result of the collation by the face feature data collation unit, the face feature data with matching features is selected from the storage unit, and a contact partner determination unit that determines a person identified by the selected face feature data as a contact partner;
A finger motion pattern generation unit that measures time series data of a finger position from the finger motion on the touch display recognized by the finger motion recognition unit and generates a finger motion pattern;
A finger motion pattern matching unit that matches the finger motion pattern generated by the finger motion pattern generation unit and the finger motion pattern stored in the storage unit;
As a result of the collation by the finger movement pattern collation unit, the finger movement pattern that matches the pattern is selected from the storage unit, and a contact unit identified by the contact unit identification information associated with the selected finger movement pattern is determined. The contact operation support apparatus according to claim 7, further comprising: The contact operation support device includes:
A communication unit that communicates with an external communication device;
The communication unit is
The name of the contact partner determined by the determination unit and the contact means identification information selected by the determination unit are transmitted to an external communication device, or the contact determined by the determination unit with the contact partner determined by the determination unit The communication operation support apparatus according to claim 7 or 8, wherein a command for executing communication by means is transmitted to an external communication device. The finger movement trajectory recognized by the recognition unit, superimposed on an image seen through the touch display, a mark on a person's face determined by the determination unit as a contact partner, and the storage unit the facial feature data And the person's name in association with each other and stored, or at least one of the person's name determined by the determination unit as the contact partner, or the content of the instruction when transmitting a command to an external communication device The contact operation support apparatus according to any one of claims 7 to 9 , further comprising a display control unit configured to display one on the touch display as notification information. The display control unit
The notification information is displayed on a face on an image seen through the touch display, at a position where a face of a person determined by the determination unit as a contact partner is not hidden. Contact operation support device. The finger movement pattern matching unit
Between the finger movement pattern generated by the finger movement pattern generation unit and the finger movement pattern stored in the storage unit, position coordinates at each sample time during finger movement are 2 The contact operation support system according to claim 2 , wherein the contact operation support system is regarded as a dimension vector, determines a match between the two-finger movement patterns based on a sum of error angles formed, and collates the two-finger movement patterns. The finger movement pattern matching unit
DP matching is applied to calculate the degree of difference between the finger motion patterns generated by the finger motion pattern generation unit and the finger motion patterns stored in the storage unit. The contact operation support system according to claim 2 , wherein operation patterns are collated. The finger movement pattern matching unit
Between the finger movement pattern generated by the finger movement pattern generation unit and the finger movement pattern stored in the storage unit, position coordinates at each sample time during finger movement are 2 The contact operation support device according to claim 5 or 8 , characterized in that it is regarded as a dimension vector, a match between both finger motion patterns is determined based on a sum of error angles formed, and the both finger motion patterns are collated. . The finger movement pattern matching unit
DP matching is applied to calculate the degree of difference between the finger motion patterns generated by the finger motion pattern generation unit and the finger motion patterns stored in the storage unit. The contact operation support apparatus according to claim 5, wherein the operation patterns are collated. A system having a storage unit that stores at least face feature data for recognizing a person, and stores a finger movement pattern indicating a finger movement pattern tracing an image in association with contact means identification information for identifying a contact means A contact operation method to be used,
A recognition step for recognizing a finger movement on an image by a user and a face area instructed by the finger movement on the image;
A contact partner determining step of selecting the face feature data of the face in the face area on the image recognized in the recognition step from the storage unit, and determining a person identified by the selected face feature data as a contact partner;
As the contact means with the contact partner determined in the contact partner determination step, select the contact means identification information associated with the finger movement pattern obtained from the finger movement recognized in the recognition step from the storage unit, A contact means determining step for determining a contact means identified by the selected contact means identification information.

Images