JP2019184625A - Recognition support system, recognition support device and recognition support server - Google Patents

Abstract

To provide a recognition support system, a recognition support device, a recognition support server.MEANS FOR SOLVING THE PROBLEM: The recognition support system is configured to extract voice print data from an acquired voice and store the voice print data as reference data, then extract name data formed of language data and associate the name data with the voice print data and store the same, then compare voice print data extracted from a newly acquired voice with the reference data to determine a person who uttered the voice. The system picks up a voice of a conversation partner before the system's eyes by a voice microphone. The system acquires a voice from telephone conversation by a telephone. The system extracts a self introduction part from a beginning part of the voice, then extracts a name part as language information from the voice. The system reads the name data from a calling card. The system cross-checks the name data read by a character reading part and the name data formed of the language data from the voice. The system extracts face feature data from the acquired face image, associates the face feature data with the name data as reference data for determining a person, and stores the same. The system cross-checks the determination of a person who uttered the voice and determination of a person in the face image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a recognition support system, a recognition support device, and a recognition support server.

  In recent years, various proposals have been made for identification and authentication of people. For example, the eyeglass device is provided with a line-of-sight sensor, a photographing unit, and a display unit that displays personal information, and the detection result of the line-of-sight or the face detection state in the image photographed by the photographing unit satisfies the search start condition. When there is no corresponding record for the detected face, a new record is created and stored, and if there is a corresponding record, the person information to be displayed from the corresponding record It has been proposed to take it out and display it on a display means. (Patent Document 1)

  On the other hand, a voiced voice receiving unit that receives voiced voice that is voiced by the voiced logic from the user terminal and voiced by the voiced logic, and a voice of the user registered in advance are voiced by the voiced logic. There has also been proposed an authentication server having a voiceprint matching unit that performs voiceprint authentication by matching the received voice and the received changed voice. (Patent Document 2)

  However, there are still issues to be considered regarding the identification, authentication, and utilization of people.

JP 2010-061265 A Japanese Patent Laid-Open No. 2006-136299

  In view of the above, the problem to be solved by the present invention is to propose a cognitive support system, a cognitive support device, a cognitive support server, and the like that utilize human identification and authentication.

  In order to solve the above problems, the present invention provides a voice acquisition unit, and extracts voiceprint data from the voice acquired by the voice acquisition unit and stores it as reference data for personal identification, and name data based on language data from the voice A recognition support data unit for extracting and storing the voiceprint data in association with the voiceprint data, and comparing the voiceprint data extracted from the voice newly acquired by the voice acquisition unit with the reference data. There is provided a cognitive support system characterized by having a control unit that performs determination.

  According to a specific feature of the present invention, the voice acquisition unit is a microphone that picks up the voice of a conversation partner in front of you. According to another specific feature, the voice acquisition unit is a telephone function unit that receives a voice of a call partner.

  According to another specific feature of the present invention, the recognition support data part extracts a self-introduction part as language information from the beginning part of the voice, and further extracts a name part as language information from the part. And stored as name data in association with the voiceprint data.

  According to another specific feature of the present invention, the recognition support system includes a character reading unit that reads text data from a business card, and the recognition support data unit reads the name data read by the character reading unit. Store in association with voiceprint data.

  According to a more specific feature of the present invention, the name data read by the character reading unit and the name data based on the language data from the speech are cross-checked, and if both do not match, one of them is adopted.

  According to another specific feature of the present invention, the recognition support system includes a face image acquisition unit, and the recognition support data unit extracts face feature data from the face image acquired by the face image acquisition unit. Then, the reference data is stored in association with the name data as personal reference data, and the control unit compares the facial feature data extracted from the face image newly acquired by the face image acquisition unit with the reference data. The newly acquired face image is identified.

  According to a more specific feature of the present invention, the control unit cross-checks the voice personality and the facial image personality, and adopts one of them if they do not match.

  As described above, according to the present invention, it is possible to propose a cognitive support system, a cognitive support device, a cognitive support server, and the like that utilize human identification and authentication. As a result, it is possible to provide cognitive support when the partner's name is forgotten, and to prevent the user from losing confidence and contribute to maintaining sociality.

It is the drawing substitute photograph which showed the block diagram which shows the whole recognition assistance system regarding the Example of this invention. FIG. 3 is a drawing-substituting photograph showing a reference data structure and a table showing an example of data stored in the voiceprint database, face database, and OCR database of the personal server in the embodiment of FIG.

  FIG. 1 is a block diagram showing an entire cognitive support system according to an embodiment of the present invention, and the system includes a hearing aid and glasses-type cognitive support device 2, a mobile phone 4, and a human identification server 6. The hearing aid combined glasses-type recognition support device 2 is controlled by the support device control unit 8. The storage unit 10 stores a program for operating the support device control unit 8 and also stores data such as face image data from the camera 12 and audio data from the microphone 14. The hearing aid combined glasses-type cognitive support device 2 also controls display by the in-field superimposition display unit 18 and sound output by the stereo earphone, respectively, based on the display data and the stereo audio data received by the support device short-range communication unit 16.

  The hearing aid combined glasses-type recognition support apparatus 2 first functions as normal glasses having the binocular lens 22, and the in-field superimposed display unit 18 superimposes and displays display data in the field of view through the binocular lens 22. Further, the hearing aid combined glasses-type cognitive support device 2 also functions as a hearing aid, and outputs to the stereo earphone 20 audio data such as the voice of the conversation partner picked up by the microphone 14.

  The hearing aid combined glasses-type cognitive support device 2 further displays the name of the conversation partner and the like determined by face detection based on the camera 12 on the in-field superimposed display unit 18. For this purpose, the camera 12 is arranged in the hearing aid combined spectacle-type recognition support device 2 so that the face of the conversation partner naturally enters the imaging range when the user's head is pointed at the conversation partner in front of the eyes. Further, the hearing aid combined glasses-type cognitive support device 2 outputs the name of the conversation partner in front of the human eye determined by voiceprint detection based on the microphone 14 from the one-side channel of the stereo earphone 20. The personal identification based on face detection and the personal identification based on voiceprint detection are cross-checked. If they are different, a more reliable personality is adopted according to an estimation algorithm based on the cognitive support application 30 described later. With the above configuration, cognitive support is provided for patients with dementia and the like when the name of the person they are facing cannot be remembered. Forgetfulness of the partner's name is a phenomenon that occurs with aging without dementia, but the hearing aid combined spectacle-type cognitive support device 2 prevents the loss of self-confidence of the user with the above-mentioned support, sociability This contributes to the maintenance of

  Because of the above-described recognition support function, the hearing aid combined use glasses-type recognition support device 2 is linked to the mobile phone 4 and the personal identification server 6. The face image data from the camera 12 and the voice data from the microphone 14 stored in the storage unit 10 are read from the storage unit 10 and transmitted from the support device short-range communication unit 16 to the mobile phone short-range communication unit 24. . As these short-range communication units, those capable of wireless LAN or infrared communication are adopted. The mobile phone control unit 26 stores the received face image data and audio data in the storage unit 28. These stored data are the reference data stored in the recognition support data unit 32 by the function of the mobile phone control unit 26 based on the processing program of the recognition support application (hereinafter referred to as “cognition support application”) 30. It is verified and the conversation partner is identified. The data such as the name of the conversation partner based on the personality is transmitted from the mobile phone short-range communication unit 24 to the support device short-range communication unit 16 and displayed on the in-field superimposition display unit 18 as described above. Sound is output from one side channel of the earphone 20. The data such as the name of the conversation partner is also displayed on the display unit 34.

  The cellular phone control unit 26 is originally for controlling the entire cellular phone 4 including the telephone function unit 36 and operates according to a program stored in the storage unit 28. The operation unit 38 and the display unit 34 are also originally provided for operation and display related to the function of the telephone function unit 36, but are also used for the above-described recognition support. The camera 37 originally provided in the mobile phone 4 can be used for videophone calls in combination with a microphone in the telephone function unit 36. The camera 37 and the microphone in the telephone function unit 36 will also be described later. In addition, it is utilized for recognition support based on the recognition support application 30.

  The global positioning system 40 (hereinafter referred to as “GPS”) inherent to the mobile phone 4 is configured to support the recognition of the current position of the user by the function of the mobile phone control unit 26 based on the processing program of the recognition support application 30. It is also used for cognitive support regarding behavior at home and at home.

  An optical character reader 39 (hereinafter referred to as “OCR”) of the mobile phone 4 reads the name of the other party from the image of the other party's business card, converts it into text data, and recognizes it by associating it with a person identification by face detection and a person identification by voiceprint detection. It is stored in the data part 32. For this purpose, when the user's head is pointed at a business card received from the other party, the camera 12 captures an image of the other party's name that falls within the imaging range and stores it in the storage unit 10, and reads this to assist The information is transmitted from the device short-range communication unit 16 to the mobile phone short-range communication unit 24.

  The mobile phone 4 can communicate with the personal server 6 via the Internet 41 by the telephone function unit 36. The personal server 6 includes a server control unit 42, a voiceprint database 44, a face database 46, an OCR database 48, and an input / output interface 50, and is linked to other mobile phones and other cognitive support devices. The voice print data, face data, and OCR data of the same person acquired at various occasions are accumulated and shared under privacy strict conditions, thereby improving the accuracy of reference data for identification. The data structure of the reference data stored in the recognition support data unit 32 of the mobile phone 4 is the same as the reference data stored in the voiceprint database 44, the face database 46, and the OCR database 48. Reference data relating to a person introduced to the user of the telephone 4 and acquired at another opportunity is provided from the personal server 6 to the recognition support data section of the mobile phone 4. Conversely, the reference data acquired by the hearing aid combined glasses-type cognitive support device 2 is transmitted from the mobile phone 4 to the personal identification server 6 from the cognitive support data unit 32 and receives self-introduction from the same person. Provided to mobile phones. Details of the personal server 6 will be described later.

  The recognition support application 30 and the recognition support data unit 32 in the mobile phone 4 function by utilizing not only the camera 12 and the microphone 14 of the hearing aid and glasses-type recognition support device 2 but also the telephone function unit 36 and the camera 37 of the mobile phone. . First, the telephone function unit 36 receives the other party's voice through a call, but the recognition support application 30 and the recognition support data unit 32 also perform cognitive support by the telephone function unit 36 using the voice pattern of the other party. Furthermore, the user's face data imaged by the camera 37 is provided to the personal identification server 6 as reference data for cognitive support.

  Next, acquisition of reference data for identification will be described. First, for face data, when the user's head is pointed at the first meeting partner, the camera 12 captures the face of the conversation partner, and the storage unit 10, the support device short-range communication unit 16, and the mobile phone short-range communication unit 24. Is stored together with the extraction of face feature data in the recognition support data section. Furthermore, the voice of the first meeting partner at the same opportunity is picked up by the microphone 14, and similarly, the voiceprint data is extracted to the recognition support data unit via the storage unit 10, the support device short-range communication unit 16, and the mobile phone short-range communication unit 24. At the same time, it is stored as audio data.

  As for who the face data and voiceprint data acquired in the above way are, first of all, use the other party's voice in the self-introduction, and if the business card is handed over during the self-introduction Use information. First, the use of speech will be described. The recognition support application 30 extracts a self-introduction portion as language information from the beginning portion of the speech data stored in the recognition support data portion 32, and further, a name portion as language information from the extracted portion. To extract. The name data recognized as linguistic information in this way is associated with face data and voiceprint data and stored in the recognition support data section. In this way, the voice data is utilized as having the dual significance of personal information called voiceprint and language information called name. The voice data acquisition uses the voice of the other party who receives the voice of the conversation partner at the time of picking up with the microphone 14 of the hearing aid and glasses-type cognitive support device 2 as described above and the normal call of the mobile phone 4 as described above. It is possible to do.

  Further, when the business card is handed over, when the user's head is pointed at the business card received from the other party as described above, the camera 12 captures this and converts it into text data with the OCR 39 of the mobile phone 4. It is associated with face data and voiceprint data as name data and stored in the recognition support data section. When a business card is handed over, there may be cases where the name is simply "I am this kind of person" and the name is not given, and the above-described text conversion using the OCR 39 is useful.

  If there is both voice name receipt and business card reception, the name data based on the language information of the voice data and the name data based on the OCR 39 of the mobile phone 4 are cross-checked. In this case, the name data is more likely according to the estimation algorithm based on 30, information from the business card is given priority unless it is unclear and difficult to read.

  Next, the reunion will be explained. When you meet again, you may not bother to name yourself, but if you continue to be unable to remember the name of the other person, you tend to lose confidence and avoid meeting people. In the case of a phone call, if you can clearly recognize the voice and face of the other party but the name does not appear, you tend to avoid taking the call. As support in these cases, a reunion's voice is picked up by the microphone 14 of the hearing aid and glasses-type recognition support device 2, and the mobile phone control unit is communicated with the support device short-range communication unit 16 and the mobile phone short-range communication unit 24. 26. Then, by the function of the mobile phone control unit 26 based on the processing program of the recognition support application 30 as described above, collation with the voiceprint reference data stored in the recognition support data unit 32 is performed. When the two match, the data such as the name associated with the voice print data is transmitted from the mobile phone short-range communication unit 24 to the support device short-range communication unit 16, and the name is visually displayed on the in-field superimposed display unit 18. At the same time, the name is output by voice from one channel of the stereo earphone 20.

  At the time of reunion, the same is done for face data. That is, when the user's head is pointed at the conversation partner, the rehearsal's face is imaged by the camera 12 of the hearing aid and glasses-type cognitive support device 2, and the support device short-range communication unit 16 and the mobile phone short-range communication unit 24. Is sent to the mobile phone control unit 26 by communication. As described above, collation with the face reference data stored in the recognition support data unit 32 is performed by the function of the mobile phone control unit 26 based on the processing program of the recognition support application 30. If the two match, the data such as the name associated with the face data is transmitted from the mobile phone short-range communication unit 24 to the support device short-range communication unit 16, and the name is visually displayed on the in-field superimposition display unit 18. At the same time, the name is output by voice from one channel of the stereo earphone 20. Note that, as described above, the personality based on the voiceprint data and the personality based on the face data are cross-checked, and in the unlikely event that one of them is correct, it is estimated that either one is correct.

  In the above recognition support, when a name or the like is visually displayed on the in-field superimposition display unit 18, display data is superimposed and displayed in the field of view through the binocular lens 22. Although it is low, it is desirable to display names and the like that do not obstruct the normal visual field at the edge of the lower visual field. On the other hand, when a name or the like is conveyed by sound from the stereo earphone 20, it may be difficult to hear both of them due to overlapping with the voice of the conversation partner. Therefore, as described above, the name is transmitted only from one side channel of the stereo earphone 20, thereby facilitating discrimination from the voice of the conversation partner in front of the eyes coming from both ears. In the case of use as a hearing aid, when a name is output from one channel of the stereo earphone 20, the enhanced voice output of the conversation partner is output only from the other channel. Further, instead of outputting the name from the one side channel of the stereo earphone 20, it may be detected that the voice of the conversation partner is interrupted and output of the name is started in the blank time zone. Alternatively, the output of the name from one channel of the stereo earphone 20 may be used in combination with the output of the name in a blank time period in which the voice of the conversation partner is interrupted.

  FIG. 2 shows data structures and data examples of the voiceprint reference data 52, the face reference data 54, and the OCR reference data 56 stored in the voiceprint database 44, the face database 46, and the OCR database, respectively, in the personal identification server 6. The server control unit 42 shown in FIG. 1 performs data management as described below in cooperation with a large number of mobile phones based on data having such a data structure. The data structure of each reference data in FIG. 2 includes, for example, “No.”, “Personal ID”, “Acquirer ID”, “Acquisition Date / Time”, and “Voiceprint Data” in the voiceprint reference data 52. For example, the data “No. 1” indicates reference “voice print data” for identifying a person to whom the ID “521378” is assigned. The actual name of the person corresponding to the personal ID is registered in the personal server 6 but is not disclosed. Further, the ID “521378” in the voiceprint reference data 52 is name data based on language information extracted from the voice data from which “voiceprint 1” is extracted, and includes a possibility of language recognition error.

  The data of “No. 1” was provided to the personal server 6 as being acquired by the person assigned with the ID “38295” at the date “12:56” of “2018/03/30”. It is a thing. Therefore, if the voiceprint data in the reference data matches the voiceprint of the voice acquired from the other party who is actually talking, it can be determined that the other party is a person to whom the ID “521378” is assigned.

  The data of the voiceprint reference data 52 is from the personal server 6 to the recognition support data section 32 of his / her mobile phone 4 even if it is voiceprint data acquired by another person on condition that he / she is a person who has introduced himself / herself. Downloaded as reference data. In other words, the recognition support data unit 32 includes data downloaded from the personal server 6 under the above conditions, in addition to the voiceprint reference data acquired by the hearing aid and glasses-type recognition support device 2 and the mobile phone 4. This is updated daily.

  Specifically, the data of “No. 3” in the voiceprint reference data 52 is provided from another person who is given an ID of “41537” who has received a self-introduction from a person who has been given an ID of “521378”. Has been. Therefore, the person who is given the ID “521378” is given the person who is given the ID “38295” in the data “No. 1” and the ID “41537” in the data “No. 3”. The self-introduction to both of the persons who have been made and their own “voice print 1” and “voice print 3” are shared as reference data to both parties, so even if their names are revealed to both by the voice print There is no state. Therefore, the data of “No. 1” and the data of “No. 3” are downloaded to the cognitive support data section in each mobile phone so that they are shared by the persons with IDs of “38295” and “41537”, respectively. . In this way, voice print data of the same person acquired by another person is shared at different occasions on the condition that he / she is introduced to himself / herself, and the cross-check improves voice print authentication accuracy and efficiency. It is done. Note that data sharing is performed only by ID, and as described above, the relationship between the acquirer ID and the name of another person is not disclosed. There will be no leakage of privacy information.

  The data structure of the face reference data 54 is the same as that of the voiceprint reference data 52 except that the reference data portion is a feature image of face data. The “personal ID” in the face reference data 54 is also name data based on the language information extracted from the speech data in principle when the “face data” is extracted, as in the voiceprint reference data 52. The possibility of recognition error is included.

  Here, when the voiceprint reference data 52 and the face reference data 54 are compared, only “personal ID” and “acquirer ID” in the data of “No. 1” and “No. 2” are interchanged. The “acquisition date” is the same. This indicates that the data is uploaded to the personal server 6 on the same occasion when both parties meet. Both faces, voiceprints, and language information are successfully acquired on the occasion.

  Further, since the data corresponding to “No. 4” of the voiceprint reference data 52 has not been uploaded as the face reference data 54, it is estimated that this data is obtained by a voice-only telephone. On the other hand, since the data corresponding to “No. 4” of the face reference data 54 has not been uploaded as the voiceprint reference data 52, the data of “No. 4” of the face reference data 54 is a meeting without hearing by a hearing impaired person or the like. It is presumed that it was acquired at In this case, the “personal ID” in “No. 4” of the face reference data 54 is based on cross-checking with OCR data (including name transfer by writing as well as business cards) described later, or on the mobile phone 4 Presumed to be manual input by the operation unit 38.

  The data structure of the OCR reference data 56 is the same as that of the voiceprint reference data 52 and the face reference data 54 except that the reference data portion is “text” of the OCR data. However, unlike these, except in the case of OCR errors due to unclear business card characters or obfuscated characters, the “personal ID” in the OCR data is more reliable than the voiceprint reference data 52 and the face reference data 54 in the sense that it is direct data. Is expensive. Note that data corresponding to “No. 2” data of the voiceprint reference data 52 and the face reference data 54 considered to correspond to the same opportunity is not uploaded as the OCR reference data 56 at this opportunity. Therefore, at this time, it is presumed that the business card from the person to whom the ID “3829595” was given was not presented to the person to whom the ID “521378” was given. However, according to the data uploaded as “No. 3” in the OCR reference data 56, the business card from the person assigned with the ID “38295” is presented to another person assigned with the ID “41537”. . Since the person to whom the ID “521378” has been given has received a self-introduction from the person to whom the ID “382295” has been given, there is no objection to this person even if the data that can identify his / her name through a business card is transmitted. Is in a state. Therefore, the data of “No. 3” in the OCR reference data 56 is downloaded to the recognition support data portion of the mobile phone of the person assigned with the ID “521378”. In this way, on the condition that you are a person who introduced yourself, even if there are different types, the same person's reference data obtained by another person is shared at different occasions, and the different types of reference data The cross check makes it possible to improve the accuracy and efficiency of support.

  The various features of the present invention are not limited to the above-described embodiments, but can be widely applied to other implementations. For example, although the embodiment of the present invention is configured by the hearing aid and glasses-type cognitive support device 2, the mobile phone 4, and the personal identification server 6, for example, based on other recognition support devices that are not limited to the hearing aid and glasses type. You may implement. Further, if attention is paid to application to the telephone function, the hearing aid and glasses-type recognition support device 2 may be omitted. In this case, for example, an OCR reading input of a business card is performed by the camera 37 of the mobile phone 4. The recognition support application 30 can be downloaded to the mobile phone 4 as a kind of application in the smartphone.

  The present invention can be applied to a cognitive support system, a cognitive support device, a cognitive support server, and the like utilizing identification and authentication of a person. For example, a dementia patient assists when the partner's name is forgotten, and the user Can prevent the loss of confidence and contribute to the maintenance of sociability.

14, 36 Voice acquisition unit 32 Cognitive support data unit 26, 30 Control unit
39 Character reading unit 12, 37 Face image acquisition unit

The present invention, recognition support system utilizing specific or authentication, etc. of a person, recognition support system, and can be applied to recognition support server or the like, for example, dementia performs assistance in such case where forgotten opponent's name, using Can contribute to the maintenance of sociality by preventing the loss of confidence.

Claims (7)

  Extracting voiceprint data from the voice acquired by the voice acquisition unit and the voice acquisition unit and storing it as reference data for personal identification, extracting name data based on language data from the voice and storing it in association with the voiceprint data A recognition support data unit, and a control unit configured to identify the newly acquired voice by comparing voice print data extracted from the voice newly acquired by the voice acquisition unit with the reference data. A cognitive support system.   The recognition support system according to claim 1, wherein the voice acquisition unit is a microphone that picks up a voice of a conversation partner in front of the eyes or a telephone function unit that receives a voice of a call partner.   The recognition support data part extracts a self-introduction part as language information from the beginning part of the voice, further extracts a name part as language information from among the parts, and stores it as name data in association with the voiceprint data The cognitive support system according to claim 1 or 2.   4. A character reading unit for reading text data from a business card, wherein the recognition support data unit stores name data read by the character reading unit in association with the voiceprint data. The cognitive support system according to any of the above.   5. The cognitive support system according to claim 4, wherein name data read by the character reading unit and name data based on language data from the voice are cross-checked, and if both do not match, one of them is adopted.   A facial image acquisition unit, wherein the recognition support data unit extracts facial feature data from the facial image acquired by the facial image acquisition unit and stores it in association with the name data as reference data for identification; The said control part performs the personalization of the said newly acquired face image by comparing the face feature data extracted from the face image newly acquired by the said face image acquisition part with the said reference data. Item 6. The cognitive support system according to any one of Items 1 to 5.   The cognitive support system according to claim 6, wherein the control unit cross-checks the personality of the voice and the personality of the face image, and adopts one of them when they do not match.