EP4176395A1 - Electronic system and method for enabling payment of a good or service by means of voice commands - Google Patents
Electronic system and method for enabling payment of a good or service by means of voice commandsInfo
- Publication number
- EP4176395A1 EP4176395A1 EP21742497.7A EP21742497A EP4176395A1 EP 4176395 A1 EP4176395 A1 EP 4176395A1 EP 21742497 A EP21742497 A EP 21742497A EP 4176395 A1 EP4176395 A1 EP 4176395A1
- Authority
- EP
- European Patent Office
- Prior art keywords
- voice
- server device
- message
- service
- subject
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 50
- 238000012790 confirmation Methods 0.000 claims description 113
- 239000013598 vector Substances 0.000 claims description 79
- 230000006870 function Effects 0.000 claims description 56
- 238000012545 processing Methods 0.000 claims description 20
- 230000005540 biological transmission Effects 0.000 claims description 3
- YTAHJIFKAKIKAV-XNMGPUDCSA-N [(1R)-3-morpholin-4-yl-1-phenylpropyl] N-[(3S)-2-oxo-5-phenyl-1,3-dihydro-1,4-benzodiazepin-3-yl]carbamate Chemical compound O=C1[C@H](N=C(C2=C(N1)C=CC=C2)C1=CC=CC=C1)NC(O[C@H](CCN1CCOCC1)C1=CC=CC=C1)=O YTAHJIFKAKIKAV-XNMGPUDCSA-N 0.000 claims 3
- 238000004422 calculation algorithm Methods 0.000 description 26
- 238000003058 natural language processing Methods 0.000 description 20
- 238000012795 verification Methods 0.000 description 10
- 238000006243 chemical reaction Methods 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 235000013550 pizza Nutrition 0.000 description 5
- 238000004590 computer program Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000001172 regenerating effect Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 241000238558 Eucarida Species 0.000 description 2
- 230000004913 activation Effects 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 2
- 238000013475 authorization Methods 0.000 description 2
- 230000008901 benefit Effects 0.000 description 2
- 241001430696 Protis Species 0.000 description 1
- 230000004931 aggregating effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- BFPSDSIWYFKGBC-UHFFFAOYSA-N chlorotrianisene Chemical compound C1=CC(OC)=CC=C1C(Cl)=C(C=1C=CC(OC)=CC=1)C1=CC=C(OC)C=C1 BFPSDSIWYFKGBC-UHFFFAOYSA-N 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 235000013305 food Nutrition 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 230000005236 sound signal Effects 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000000844 transformation Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 230000001755 vocal effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
- G06Q20/386—Payment protocols; Details thereof using messaging services or messaging apps
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/02—Reservations, e.g. for tickets, services or events
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/083—Shipping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/08—Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
- G06Q10/087—Inventory or stock management, e.g. order filling, procurement or balancing against orders
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/30—Payment architectures, schemes or protocols characterised by the use of specific devices or networks
- G06Q20/32—Payment architectures, schemes or protocols characterised by the use of specific devices or networks using wireless devices
- G06Q20/325—Payment architectures, schemes or protocols characterised by the use of specific devices or networks using wireless devices using wireless networks
- G06Q20/3255—Payment architectures, schemes or protocols characterised by the use of specific devices or networks using wireless devices using wireless networks using mobile network messaging services for payment, e.g. SMS
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
- G06Q20/40—Authorisation, e.g. identification of payer or payee, verification of customer or shop credentials; Review and approval of payers, e.g. check credit lines or negative lists
- G06Q20/401—Transaction verification
- G06Q20/4014—Identity check for transactions
- G06Q20/40145—Biometric identity checks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q20/00—Payment architectures, schemes or protocols
- G06Q20/38—Payment protocols; Details thereof
- G06Q20/42—Confirmation, e.g. check or permission by the legal debtor of payment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/04—Billing or invoicing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q30/00—Commerce
- G06Q30/06—Buying, selling or leasing transactions
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L2015/223—Execution procedure of a spoken command
Definitions
- the present invention relates to the field of voice assistants.
- the present invention relates to a system and method for enabling payment of a good or service with a medium-high security level, by means of the use of voice commands from the subject requesting the delivery of the service to be paid or the purchase of a good.
- voice assistants allow a subject to use voice commands in order to obtain various types of information, such as the weather forecast, the results of a football match, request the playback of music tracks, the translation of a sentence in a certain language, etc.
- Said voice assistants can be realized by means of dedicated eiectronic devices (also known as “smart speakers”), such as Amazon Alexa and Google Home.
- voice assistants can be realized using software programs, such as Google Assistant for smartphones, or tablets which use the Android operating system, or Siri for iPhones or iPads.
- Digital payment systems such as PayPal, Amazon Pay or Apple Pay are known, which have the advantage of allowing the payment for a good or service purchased online simply by clicking on a button indicating the payment by means of one of the systems indicated above, without requiring entering the data of a credit card and exploiting an account which was previously created by the user.
- the present invention relates to a method and system for enabling payment of a good or service by means of voice commands of a subject requesting the purchase of a good or service, wherein the method and system are defined in the accompanying claims 1 and 8, respectively, and by the preferred embodiments thereof described in the dependent claims 2 to 6 and 9-10
- the Applicant has perceived that the system and method for enabling payment of a good or service in accordance with the present invention allow the use of voice and/or face profiles to enable payment for a good or service with a medium-high level of security, while respecting the requirements of the protection of personal data (in particular the GDPR regulation, EU Regulation no. 679/2016) and possibly also those of the EIDAS regulation (Electronic Identification Authentication and Signature), EU Regulation no. 910/2014.
- the basic idea is to use two or more server devices to provide the information necessary to enable payment of the good or service, in which each of the two servers stores a respective different portion of a reference voice and/or face profile (or feature vector) of the subject requesting the good or service (depending on the security level for the requested good or service), thus the reference voice and/or face profile (or the reference feature vector associated with the voice signal) is recomposed on a separate electronic device, which enables payment of the requested good or service as a function of fhe comparison befween the reference voice and/or face profile and a sample voice and/or face profile acquired in real time or as a function of the comparison between a reference feature vector and a sample feature vector generated in real time: thereby the profile (or feature vector) and the information of the subject are not normally available to any of the elements which contribute to realizing the purchase and payment transaction of the good or service, but are available only in the short time when the payment of the requested good or service is enabled.
- Figure 1 shows a block diagram of an electronic system for enabling payment of a good or service by means of voice commands according to a first embodiment of the invention
- Figures 2A-2D show a time trend of the messages exchanged between the different components of the system according to the first embodiment of the invention
- Figure 3 shows a block diagram of an electronic system for enabling payment of a good or service by means of voice commands according to a second embodiment of the invention
- Figures 4A-4C show a time trend of the messages exchanged among the different components of the system of the second embodiment of the invention
- Figure 5 shows a block diagram of the electronic system tor enabling payment of a good or service by means of voice commands according to a third embodiment of the invention
- Figures 6A-6D show a time trend of the messages exchanged among the different components of the system according to the third embodiment of the invention.
- FIG. 1 a block diagram of an electronic system 1 for enabling payment of a good or service by means of voice commands is shown according to the first embodiment of the invention.
- the good or service is requested by the subject 7 and can be, for example: a booking of a medical examination; the purchase of a book in digital format; the purchase of a financial product; the purchase of food; the booking of a holiday.
- the electronic system 1 comprises: a voice assistant 2; an electronic device 8, for example a mobile type; an electronic human language processor 3; an application 4 for delivering services to be paid; a proti!e decoding and payment enabling server device 5; a profiling server device 6; a payment server device 9.
- the electronic system 1 further comprises a service aggregator 11 and at least one external service provider 12, which will be explained in more detail below.
- the set of the electronic human language processor 3, the application for delivering services to be paid 4, the profile decoding and paymenf enabling server device 5 and possibly the service aggregator 11 constitutes a voice platform based on Artificial Intelligence and on NLU (Natural Language Understanding) and NLP (Natural Language Processing) techniques, which perform an analysis of the spoken human language and understand the sense of the spoken language.
- NLU Natural Language Understanding
- NLP Natural Language Processing
- the electronic processor 3, the application for delivering services to be paid 4 and the profile decoding and payment enabling server device 5 are included within a medium-long distance telecommunications network 10, for example the Internet network which uses the TCP/IP protocol, with a client-server architecture and use of Web Services
- the profiling server device 6, the payment server device 9 and the service aggregator 11 can be positioned outside the network 10 or therein.
- a subject 7 uses the electronic system 1 to request enabling to the payment for a certain good or service using only voice commands: the subject 7 (which is supposed to have already been previously identified by means of a profiling procedure by means of the profiling server device 6 ⁇ first requests the deiivery of a good or service to be paid, then receives confirmation of the availability and cost of the requested service, then is identified by means of the comparison of biometric profiles of the voice and/or face type (and possibly also use of OTP), finally is enabled to make the payment for the requested good or service and subsequently the subject 7 receives the requested service delivered directly by the profile decoding and payment enabling server device 5 or by the external service provider 12 connected to the profile decoding and payment enabling server device 5 by means of the service aggregator 11 and by means of the activation of a web service.
- the voice assistant 2 has the function of interpreting human language and dialoguing therewith.
- the voice assistant 2 can be a dedicated electronic device (smart speaker), such as Amazon Echo/ Echo Dot (with Alexa) or Google Home.
- the voice assistant 2 is a software application (i.e., a software program) performed by means of a processor of the electronic device 8 (typically mobile), such as the Google Assistant application for smartphones, or tablets which use the Android operating system, or the Siri application for iPhones or iPads, or the Gorfana voice assistant for personal computers with Windows operating system: in this case the voice assistant and the device 8 are impiemeted in a single electronic component (for example, a smartphone or IPhone or a personal computer).
- a software application i.e., a software program
- the electronic device 8 typically mobile
- the voice assistant and the device 8 are impiemeted in a single electronic component (for example, a smartphone or IPhone or a personal computer).
- the voice assistant 2 is a dedicated electronic voice assistant device, this comprises a speaker, a microphone and a processing unit which executes an appropriate program capable of interpreting human language and communicating therewith (for example, the Alexa program developed by Amazon); furthermore, the electronic voice assistant device 2 comprises a suitable transceiver for exchanging audio messages with the electronic processor 3, through the telecommunications network 10.
- the voice assistant 2 is instead a software application running on the processor of the electronic device 8 (for example, a smartphone or tablet), the microphone and speaker (and possibly a camera) integrated in the electronic device 8 itself are used and the transceiver integrated in the electronic device 8 itself is used to exchange audio messages with the human language electronic processor 3.
- the voice assistant 2 is configured to receive from the subject 7 a voice signal indicative of a request for delivering a good or service to be paid and is configured to transmit towards the electronic human language processor 3 an audio message indicative of the request for delivering the good or service.
- the voice assistant 2 is configured to receive from the human language processor 3 an audio message indicative of a confirmation or rejection of the availability of the requested good or service, then the voice assistant 2 is configured to generate towards the subject 7 a voice signal indicative of said confirmation or rejection of the availability of the requested good or service.
- the voice assistant 2 is such as to receive the audio message indicative of the confirmation of the availability of the requested good or service, the voice assistant 2 is configured to receive from the subject 7 a voice signal indicative of the confirmation of the wish to pay for the requested good or service, then the voice assistant 2 is configured to transmit towards the human language electronic processor 3 an audio message indicative of said confirmation of the wish to pay for the requested good or service.
- the voice assistant 2 is configured to receive from the human language processor 3 an audio message indicative of a defined phrase, therefore the voice assistant 2 is configured to generate towards the subject 7 a voice signal Indicative of said defined phrase.
- the voice assistant 2 is configured to receive from the subject 7 a sample voice signal representative of the voice of the subject 7, then the voice assistant 2 is configured to forward the sample voice signal towards the electronic device.
- the voice assistant 2 is configured to receive from the human language processor 3 an audio message indicative of a confirmation or rejection of the payment for the requested good or service, then the voice assistant 2 is configured to generate towards the subject 7 a voice signal indicative of an authorization or a rejection of the payment for the requested good or service.
- the electronic device 8 belongs to the subject 7 and can be a fixed type (for example, a personal computer) or a mobile type (for example, a smartphone, tablet or laptop computer).
- the electronic device 8 comprises a speaker and a microphone and, preferably, a camera. It is assumed that the subject 7 has already been identified (by means of the profiling server device 6) in a previous profiling procedure in a secure condition, in which persona! data of the subject has been acquired, such as his/her name, surname, telephone number, identity card and a reference voice and/or face profile of the subject 7.
- a unique identifier of the subject 7 was associated with the electronic device 8.
- the electronic device 8 is a smartphone provided with a SIM card and thus the unique identifier of the subject 7 is the phone number associated with the SIM card.
- reference voice profile means a reference profile of the digital identity of the subject 7 generated as a function of the voice signal representative of the voice of the subject 7, wherein said reference voice profile has been previously acquired from the subject 7 in a profiling procedure by means of the profiling server device 6 and in secure conditions, and in which said reference voice profile has been stored at least in part in a memory of the profiling server device 6: the reference voice profile has therefore been previously verified and is considered reliable.
- one or more images representative of the face of the subject 7 are acquired, thus generating a reference voice/face profile as a function of the voice signal and of the image of the face of the subject 7, wherein said reference voice/face profile has been previously acquired in the profiling procedure by means of the profiling server device 6 and in secure conditions, and wherein said reference voice/face profile has also been stored at least in part in the memory of the profiling server device 6.
- a video recording is acquired in which at least the face of the subject 7 is framed and in which the subject says a defined phrase aloud, thus generating the reference voice/face profile.
- a medium security level wherein only the voice profile of the subject 7 (i.e., a single authentication factor) is used to perform an online verification of his/her identity, in order to enable the payment for the requested good or service
- a medium-high level of security level 2
- a double authentication factor is used to perform an online verification of his/her identify, in order to enable the payment for the requested good or service
- the electronic system 1 uses a high level of security (level 3) to perform an online verification of the identity of the subject 7, wherein a triple authentication factor is used which comprises the voice profile of the subject 7, the face profile of the subject 7 and the OTP code valid only once for a limited time (OTP): thereby the triple factor E!DAS regulation is respected.
- level 3 a high level of security
- OTP limited time
- the reference voice profile is implemented with an alphanumeric code generated by means of a suitable hash algorithm which receives in input a digital audio track representative of the voice of the subject and generates in output (by means of said hash algorithm) an alphanumeric code (also indicated with a fingerprint), i.e., a string of alphanumeric characters.
- a suitable hash algorithm which receives in input a digital audio track representative of the voice of the subject and generates in output (by means of said hash algorithm) an alphanumeric code (also indicated with a fingerprint), i.e., a string of alphanumeric characters.
- the reference voice/face profile is implemented for example with an alphanumeric code generated by means of a suitable hash algorithm which receives in input a digital audio track of the voice of the subject 7 and data representative of the image of the face of the subject 7 and generates in output (by means of said hash algorithm) an alphanumeric code (fingerprint), i.e., a string of alphanumeric characters.
- a suitable hash algorithm which receives in input a digital audio track of the voice of the subject 7 and data representative of the image of the face of the subject 7 and generates in output (by means of said hash algorithm) an alphanumeric code (fingerprint), i.e., a string of alphanumeric characters.
- the reference voice profile of the subject 7 is divided into a plurality of portions greater than or equal to two and it is stored into a data structure of a b!ockchain.
- the reference voice and/or face profile is divided info two portions, wherein a first portion is stored in a memory 6-1 associated with the profiling server device 6 and the second portion is stored in a memory 9-1 associated with the payment server device 9: this allows maximum security to be ensured through a so-called "double helix" mechanism in which the two reference profiles (for example, two alphanumeric codes generated with a hash algorithm) present in the memory 6-1 and in the memory 9-1 are recomposed on the profile decoding server device 5 (through a random algorithm) to reconstruct the complete reference profile (for example, an alphanumeric code generated with a hash algorithm) of the voice or voice/face type.
- the two reference profiles for example, two alphanumeric codes generated with a hash algorithm
- the complete reference profile for example, an alphanumeric code generated with a hash algorithm
- the first and/or second portion of the reference voice (or voice/face) profile are stored in a data structure of a blockchain.
- reference voice/face profile which also includes biometric data associated with a image representative of the face of the subject 7, or of a part thereof comprising at least the eyes, nose and mouth.
- a photo of the face of the subject 7 is taken (by means of a camera of an electronic device 8, for example the front camera of a smartphone) and an image of the face of the subject 7 is acquired therefrom, then a reference voice/face profile is generated (by means of a random algorithm) as a function of the digital audio track of the voice of the subject 7 and of the acquired image of the face of the subject 7; the reference voice/face profile is divided into two portions, of which a first portion is stored in the memory 6-1 associated with the profiling server device 6, while a second portion is stored in the memory 9-1 associated with the payment server device 9.
- the electronic device 8 is configured to receive from the profiling server device 6 a request to acquire a sample voice profile and/or a sample voice/face biometric profile of the subject 7, wherein said request can be a voice call or a text message or a multimedia message (i.e., of an audio or audio-video type) or an email message.
- sample voice profile means a sample digital identity profile of the subject 7 associated with the sample digital audio track of the voice of the subject 7, wherein said sample digital audio track is generated by means of the conversion from analog to digital of the voice of the subject 7 acquired in real time by means of the microphone integrated in the electronic device 8.
- sample voice/face profile means data associated with a combination of the sample digital audio track of the voice of the subject and biometric data associated with the face of the subject 7, the latter generated by means of an image of the face of the subject 7 acquired with a camera integrated in the electronic device 8.
- a video recording is acquired in real time in which at least the face of the subject 7 is framed and in which the subject says a defined phrase aloud, thus generating the sample voice/face profile.
- the sample voice profile is implemented with an alphanumeric code generated by means of a suitable hash algorithm which receives in input a sample digital audio track representative of the voice of the subject 7 and possibly also data representative of the face profile of the subject 7 and generates in output (by means of said hash algorithm) an alphanumeric code (digital fingerprint), i.e., a string of alphanumeric characters.
- a suitable hash algorithm which receives in input a sample digital audio track representative of the voice of the subject 7 and possibly also data representative of the face profile of the subject 7 and generates in output (by means of said hash algorithm) an alphanumeric code (digital fingerprint), i.e., a string of alphanumeric characters.
- the sample digital audio track is acquired by addressing to the subject 7 (by means of the voice assistant 2) one or more defined phrases (i.e., known in advance) and acquiring (by means of the same voice assistant 2) one or more corresponding responses from the subject 7 wherein said responses constitute the voice of the subject 7 which is converted from analog to digital, generating a sample digital audio track, which is used to generate the sample voice profile, then said sample digital audio track will then be forwarded to the profile decoding and payment enabling server device 5, crossing the electronic processor 3, the application for delivering services to be paid 4 and the profiling server device 6.
- the subject 7 is asked (by means of the voice assistant 2) to say one or more of the following defined phrases: ⁇ am name and surname and this is my voice";
- more than one phrase is exchanged between the subject 7 and the voice assistant 2, in order to acquire the voice signal of the voice of the subject 7 and generate the sample digital audio track, such as the following sequence of phrases: subject 7: ⁇ am [NAME] and [SURNAME] and this is my voice", where the [NAME] and [SURNAME] is the actual name and surname of the subject 7;
- the voice signal representative of the voice of the subject 7 is acquired by means of the electronic device 8 which receives a text message (for example, an SMS) transmitted by the profiling server device 6, wherein said text message contains an alphanumeric code, then the subject 7 is asked to read the alphanumeric code of the text message received at the electronic device 8.
- a text message for example, an SMS
- the face profile is instead acquired by asking the subject 7 to take a seifie (i.e., a photograph of himself/herself) by means of an audio, audio-video, SMS message transmitted from the profiling server device 8 to the electronic device 8.
- a seifie i.e., a photograph of himself/herself
- the electronic device 8 is further configured to transmit to the profiling server device 6 an audio message representative of the sample voice profile of the subject 7, or an audio-video message representative of the sample voice/face profile of the subject 7.
- the voice assistant 2 is configured to receive a voice signal (i.e., an acoustic wave) generated by the subject 7 indicative of a request for delivering a good or service and to convert (by means of the microphone 2-2) said voice signal into an audio message indicative of a request for delivering the good or service, such as a service which requires a medium (level 1) or medium-high (level 2) level of security.
- a voice signal i.e., an acoustic wave
- the voice assistant 2 is configured to transmit towards the human language electronic processor 3 an audio message indicative of the request for delivering a good or service.
- the voice assistant 2 is further configured to receive from the human language processor 3 an audio message carrying a confirmation or a rejection of payment for the requested good or service, then the electronic voice assistant device 2 is configured to generate to the subject 7 (by means of the speaker of the voice assistant 2 or by means of the speaker incorporated in the electronic device 8) a voice signal (i.e., an acoustic wave) carrying said confirmation or rejection of payment for the requested good or service.
- a voice signal i.e., an acoustic wave
- the human language processor 3 has the function of receiving from the voice assistant 2 an audio message carrying the voice of the subject 7 and of performing voice recognition functions of the voice of the subject 7 himself/herself, in particular the analysis and understanding of the language of the subject 7, thus performing a conversion of the received audio message into a text string representative of the content of the audio message.
- the electronic processor 3 is capable of independently sending answers to questions from the subject 7 in audio format.
- the electronic processor 3 also has the function of performing a conversion of a text string into an audio signal.
- the electronic processor 3 is based on artificial intelligence, which allows to analyse the spoken human language and to understand the sense of the spoken language, by means of techniques known with NLU (Natural Language Understanding) and NLP (Natural Language Processing): in this case the electronic processor 3 is an NLU/NLP server device.
- NLU Natural Language Understanding
- NLP Natural Language Processing
- the electronic processor 3 is the cloud platform of Amazon Alexa or of Google Assistant.
- the electronic processor 3 is configured to receive from the voice assistant 2 an audio message representative of a request for delivering a good or service, then the electronic processor 3 is configured to transmit towards the application for delivering services to be paid 4 an audio message representative of said request for delivering the good or service.
- the electronic processor 3 is configured to receive from the application for delivering services to be paid 4 a text message carrying a confirmation or a rejection of the payment for the requested good or service, is configured to perform a conversion of the text message received into an audio message indicative of the confirmation or rejection of the payment for the requested good or service, then the electronic processor 3 is configured to transmit towards the voice assistant 2 said audio message indicative of the confirmation or rejection of the requested good or service.
- the voice assistant 2 is an Amazon device (Amazon Echo, Amazon Echo Dot)
- the electronic processor 3 is the set of web services provided by the cloud computing platform known as Amazon Web Service (AWS), see the website aws.amazon.com.
- AWS Amazon Web Service
- the application for delivering services to be paid 4 is a software program (also referred to as "Voice pay") which has the function of requesting the availability of a good or service to be paid, requesting the payment for the requested good or service and confirming or denying the payment for the requested good or service.
- VoIP pay a software program which has the function of requesting the availability of a good or service to be paid, requesting the payment for the requested good or service and confirming or denying the payment for the requested good or service.
- a particular defined phrase is associated (or configured), by means of which the corresponding application for delivering services to be paid 4 is activated.
- the application for delivering services to be paid 4 is associated with the purchase of a pizza and is activated by the subject 7 who says (towards the voice assistant 2) the phrase "Voice Pay, I would like to buy a pizza", then the Pizza application 4 is activated.
- Another example is the booking of a medical examination: in this case the subject 7 says the phrase "Voice Pay, I would like to book a medical examination", then the Medical application 4 is activated.
- the application for delivering services to be paid 4 is configured to receive a request for enabling payment from a subject requesting payment for the purchase of a good or service (in particular, a service which requires a medium or medium-high level of security), by means of the exchange of audio and text messages with the electronic processor 3 and with the profile decoding and payment enabling server device 5.
- the application for delivering services to be paid 4 is installed in the cloud where the electronic processor 3 is present.
- the application for delivering services to be paid 4 is configured to receive from the electronic processor 3 a text message carrying a request for delivery of a good or service with a certain type of payment, then the application for delivering services to be paid 4 forwards said request for delivery of the good or service to the profile decoding and payment enabling server device 5.
- the application for delivering services to be paid 4 is configured to transmit towards the electronic processor 3 a text message carrying a confirmation or rejection of payment for the requested good or service.
- the application for delivering services to be paid 4 is a new skill that allows to request the availability of a good or service to be paid, request payment tor the requested good or service (for example, the purchase of a book or the payment for a medical examination) and confirm payment for the good or service, using only voice commands.
- the profile decoding and payment enabling server device 5 is an electronic device which comprises a transceiver for exchanging data with the application for delivering services to be paid 4, with the profiling server device 6 and possibly with the service aggregator 11 , as will be explained in more detail below.
- the profile decoding and payment enabling server device 5 further comprises a processing unit (for example, a microprocessor) running a software program to perform the functions which will be illustrated below.
- the profile decoding and payment enabling server device 5 has the function of decoding the two portions of the reference voice profile (or decoding the two portions of the reference voice/face profile) and composing the two portions of the reference voice profile (or composing the two portions of the reference voice/face profile) by means of a random algorithm, so as to generate the reference voice (or voice/face) profile.
- random algorithm means that each time the reference profile is generated as a function of the first and second portion of the reference profile, the hash function used to generate the alphanumeric code associated with the first portion of fhe reference profile and the alphanumeric code associated with the second portion of the reference profile is changed, thus generating two different alphanumeric codes each time, as long as the final result (e., the alphanumeric code associated with the reconstructed reference profile) always has the same value for the same biometric reference information.
- the profile decoding and payment enabling server device 5 has the function of confirming or rejecting the identity of the subject 7 and the function of enabling or rejecting (as if it were a traffic light) the payment for the requested good or service by the subject 7 using voice commands, by means of the comparison between the sample voice and/or face profile of the subject 7 (generated in real time by means of the electronic device 8) and the reference voice and/or face profile.
- the profile decoding and payment enabling server device 5 is configured to receive from the application for delivering services to be paid 4 a message indicative of a request for availability of the good or service and it is configured to forward it to the profile decoding and payment enabling server device 5; furthermore, the profile decoding and payment enabling server device 5 is configured to receive from the service aggregator 11 a message indicative of a confirmation of availability of the requested good or service and indicative of a request for payment for the requested good or service and the related cost, or receive a message indicative of a rejection of availability of the requested service or good, then the profile decoding and payment enabling server device 5 is configured to forward said message to the application for delivering services to be paid 4.
- the profile decoding and payment enabling server device 5 is configured to receive from the application for delivering services to be paid 4 a message indicative of a payment request for the requested good or service, then the profile decoding and payment enabling server device 5 is configured to transmit to the profiling server device 6 a message indicative of a request for enabling payment for the requested good or service.
- the profile decoding and payment enabling server device 5 is configured to receive from the profiling server device 6 the sample digital audio track representative of the voice of the subject 7 (and possibly an image representative of at least part of the face of the subject 7), together with the first and second portion of the reference voice profile, then the profile decoding and payment enabling server device 5 is configured to generate the sample digital audio profile representative of the voice of the subject 7 (or generate the voice/face profile as a function of the sample digital audio track of the voice of the subject 7 and the image representative of at least part of the face of the subject 7), finally the profile decoding and payment enabling server device 5 is configured to recompose the first and second portion of the reference voice profile, generating the reference voice profile therefrom.
- the profile decoding and payment enabling server device 5 is configured to perform a comparison between the sample voice (or voice/face ⁇ profile of the subject 7 (generated in real time by means of the electronic device 8) and the reference voice (or voice/face) profile (obtained by recomposing the two portions), in order to perform the recognition of the subject 7 and therefore enable or reject the payment.
- Said enabling of payment is obtained by verifying whether the sample voice (or voice/face) profile is compatible with the reference voice (or voice/face) profile, or by verifying if both profiles belong to the same person (i.e., the subject 7), and also by verifying if the subject 7 is authorized to make the payment for the requested good or service.
- the profile decoding and payment enabling server device 5 is configured to transmit to the application for delivering services to be paid 4 (and possibly to the service aggregator 11) a message indicative of a confirmation of payment for the requested good or service if the payment is not enabled, the profile decoding and payment enabling server device 5 is configured to transmit to the application for delivering services to be paid 4 (and possibly to the service aggregator 11) a message indicative of a rejection of payment for the requested good or service.
- the profile decoding and payment enabling server device 5 verifies that the sample biometric voice (or voice/face) profile is compatible with the reference biometric voice (or voice/face) profile of the subject 7 (previously acquired and stored partly in the profiling server device 6 and partly in the payment server device 9) and also verifies that the subject 7 is authorized to make the payment for the requested good or service, the profile decoding and payment enabling server device 5 authorizes (as if it were a traffic light) to make the payment for the requested good or service by the subject 7, thus allowing the application for delivering services to be paid 4 to receive the confirmation of the payment of the requested good or service, and in turn allowing the electronic processor 3 to produce said confirmation on the voice assistant 2 in the form of vocal sounds.
- the profile decoding and payment enabling server device 5 is separate from the profiling server device 6, thus increasing the level of payment security, since the device which enables payment (i.e., 5) is separate from the device (i.e., 6) which possesses (at least in part) the personal data of the subject 7, in the form of a voice or voice/face profile, and further the device which enables payment (i.e., 5) possesses the algorithm to regenerate the reference voice (or voice/face) profile, but does not possess the two portions of the reference voice (or voice/face) profile.
- the service aggregator 11 has the function of aggregating the services provided by a plurality of external providers, by means of connecting with one or more external service providers 12.
- the service aggregator 11 is a software application running on a processing unit of a server device, which performs API (Application Programming interface) calls to the external service provider 12, in order to know the availability and cost of a particular requested good or service.
- API Application Programming interface
- the service aggregator 11 is then configured to receive (from the profile decoding and payment enabling server device 5) a message indicative of a request for availability of a requested good or service to be paid, then the service aggregator 11 is configured to transmit towards the external service provider 12 a message indicative of a request for availability of the requested good or service to be paid, by means of a call to an Application Programming interface (API) of the external service provider 12.
- API Application Programming interface
- the service aggregator 11 is configured to receive from the external service provider 12 a message indicative of a confirmation of availability of the requested good or service and indicative of a request for payment for the good or service and the respective cost, or indicative of a lack of availability of the requested good or service.
- the external service provider 12 has the function of verifying the availability of a particular requested good or service to be paid and the respective cost.
- the external service provider 12 comprises a catalogue of the goods or services available, such as: a list of the medical services which are delivered by a particular healthcare facility and the respective cost; a list of books which can be purchased and the respective cost; a list of types of pizza which can be purchased for home delivery and fhe respective cost.
- the external service provider 12 is a software application running on a processing unit of a server device, which exposes an Application Programming Interface (API) having the function of indicative of the availability or non-availability of a certain requested good or service to be paid.
- API Application Programming Interface
- the external service provider 12 is configured to receive from the service aggregator 11 a message indicative of a request for availability of the requested good or service to be paid, is configured to verify the availability of the requested good or service by means of access to a catalogue and the corresponding cost (and any other details associated with the requested good or service), and is configured to transmit towards the service aggregator 11 a message indicative of a confirmation of availability of the requested good or service and indicative of a request for payment for the good or service and the respective cost, or indicative of a lack of availability of the requested good or service.
- first external service provider associated with a list of medical services which can be delivered by a particular healthcare facility
- second external service provider associated with a catalogue of books which can be purchased in electronic or paper format
- the presence of the separate service aggregator 11 is not essential, i.e., the functions performed by the service aggregator 11 can be integrated within the profile decoding and payment enabling server device 5, which is thus further configured to aggregate the services delivered by one or more external service providers similar to 12.
- the presence of the separate external service provider 12 is not essential, i.e., the functions performed by the external service provider 12 can be integrated within the profile decoding and payment enabling server device 5, which is thus further configured to verify the availability of the requested good or service in a catalogue of goods or services directly associated with the profile decoding and payment enabling server 5.
- the profiling server device 6 has the function of profiling the subject 7 during a profiling procedure (prior to the normal operation step of the electronic system 1) which occurs in a condition of maximum security, during which personal data of the subject (such as his/her name, surname, telephone number, identity card), the reference voice profile of the subject 7 and possibly the reference face profile of the subject 7 are acquired.
- the profiling server device 6 is configured to manage payments other than the standard payment system: for example, the payment system used for Alexa is Amazon Pay, while the payment system used by the system in question and present on the payment server device 9 is PayPal.
- the profiling server device 6 is connected to a non-volatile memory 8-1 (internal or external) configured to store a first portion of the reference voice and/or face profile of the subject 7.
- the profiling server device 6 is configured to receive from the profile decoding and payment enabling server device 5 a signal carrying a request to enable payment for the requested good or service using a first payment system and is configured to verify if the subject 7 is authorized to make the payment with the first payment system requested.
- the profiling server device 6 is configured to transmit towards the electronic device 8 a message indicative of a request to acquire the sample voice (or voice/face) profile of the subject 7, which can be carried by means of, alternatively: a voice call from the profiling server device 6 to the electronic device 8, using the telephone number of the subject 7 which was previously acquired in the previous step of profiling the subject 7; an audio message transmitted from the profiling server device 6 to the electronic device 8 through the telecommunications network 10, such as a Whatsapp message; a multimedia message (i.e., audio-video) transmitted from the profiling server device 6 to the electronic device 8 through the telecommunications network 10, such as a Whatsapp message; a text message transmitted from the profiling server device 6 to the electronic device 8 through the telecommunications network 10, such as a Short Message Service (SMS).
- SMS Short Message Service
- the profiling server device 6 is configured to receive from the electronic device 8 a message indicative of a sample voice (or voice/face) profile and is configured to transmit to the payment server device 9 a message indicative of a request for a second portion of the reference voice (or voice/face) profile.
- the profiling server device 6 is configured to receive from the payment server device 9 a message carrying a second portion of the reference voice (or voice/face) profile, is configured to read from the memory 6-2 the first portion of the reference voice (or voice/face) profile, and is configured to transmit towards the profile decoding and payment enabling server device 5 a message carrying a sample voice (or voice/face) profile of the subject 7, together with a first and second portion of the reference voice (or voice/face) profile of the subject 7.
- the electronic processor 3 is implemented with an NLU/NLP cloud computing platform, the profiling server device 6 is inside the cloud where the electronic processor 3 is present and is connected through Web Services.
- the profiling server device 8 is connected to a database distributed in a blockchain, where the first portion of the reference voice (or voice/face) profile of the subject 7 is stored.
- the payment server device 9 is an electronic device which comprises a transceiver for exchanging data with the profiling server device 6 and further comprises a processing unit (for example, a microprocessor) running a software program to perform the functions which will be illustrated below.
- a processing unit for example, a microprocessor
- the payment server device 9 is a payment gateway which is located at a third party which manages payment systems, tor example at a financial institution or a bank.
- the payment server device 9 has the function of managing the payment for the requested good or service.
- the payment server device 9 is connected to a non-volatile memory 9-1 (internal or external) configured to store a second portion of the reference voice profile of the subject 7.
- the payment server device 9 is configured to receive from the profiling server device 6 a message indicative of a request for the second portion of the reference voice (or voice/face) profile of the subject 7, is configured to read from the memory 9-1 the second portion of the reterence voice (or voice/face) profile and is configured to transmit towards the profiling server device 6 a message carrying said second portion of the reference voice (or voice/face) profile.
- the payment server device 9 is configured to receive (from the profile decoding and payment enabling server device 5) a message indicative of a confirmation of the identity of the subject 7 indicative of the fact that the subject 7 has been successfully identified by means of his/her voice (or voice/face) profile and indicative of a request for confirmation of payment for the requested good or service, then the payment server device 9 is configured to transmit to the profiling server device 6 a message indicative of a confirmation of payment for the requested good or service; alternatively, the payment server device 9 is configured to receive a message indicative of a rejection of verification of the identity of the subject 7 indicative of the fact that the subject 7 has not been successfully identified, then the payment server device 9 is configured to transmit to the protiling server device 6 a message indicative of rejection of payment for the requested good or service.
- the payment server device 9 is connected to a database distributed in a blockchain, where the second portion of the reference voice (or voice/face) profile of the subject 7 is stored. It should be noted that for simplicity’s sake in Figure 1 only one payment server device 9 has been shown, but more generally there may be two or more payment server devices similar to 9 and connected to the profiling server device 6.
- the tracking of the purchase and payment transactions of the requested good or service is carried out, for the purpose of any disputes.
- said tracking comprises recording a voice message of the subject 7, when he/she says the confirmation of fhe wish to pay for the requested good or service; therefore a voice message representafive of the confirmation of the wish to pay for the requested service is stored, for example in a memory associated with the profile decoding and payment enabling server device 5.
- the geographical position (i.e., geolocation) of the subject 7 is stored in the same memory, when the subject expresses the confirmation of the wish to pay for the requested good or service: said geographical position can be expressed by means of global coordinates (e.g., of the GPS type), by means of the estimated position by means of fhe radio mobile network in which the electronic device 8 of fhe mobile type (smartphone) is located or by means of the network address (typically IP address) uniquely associated with the electronic device 8 of the subject 7.
- the date and time in which the subject 7 expresses the confirmation of the wish to pay for the requested good or service is further stored
- FIGS 2A--2B show the trend over time of the text and audio messages exchanged between the voice assistant 2, the electronic device 8, the human language electronic processor 3, the application for delivering services to be paid 4, the profile decoding and payment enabling server device 5, the profiling server device 6 and the payment server device 9 of the electronic system 1 .
- the electronic device 8 is a smartphone; it is used a reference and sample profile of the voice type; the voice assistant 2 is a software application installed on the smartphone and it is activated by voice saying a defined activation word; it is used a platform based on Al, NLP and NLU technologies to implement the electronic processor 3 (indicated with NLP/NLU electronic processor 3); the service requested by the subject 7 is the purchase of a medical examination in a catalogue on the external service provider 12; it is presente the service aggregator 11 ; the application for delivering services to be paid 4 is a new skill dedicated to booking medical examinations to be paid, it is connected to the voice assistant 2 and it is indicated below with "Medical skill"; the profile decoding and payment enabling server device 5 has the function of authorizing the payment and thus the delivering of the service; the profiling server device 6 is an external application server which manages the onboarding and profiling of the customer and manages the reference and sample profiles
- a profiling of the subject 7 has already been carried out (by means of the profiling server device 6), in particular the telephone number of the subject 7 has already been acquired and has been stored in a non-volatile memory of the profiling server device 6; moreover at the instant tO the reference voice profile of the subject 7 has already been acquired and partly stored in the memory 6-1 of the profiling server device 6 and partly in the memory 9-1 of the payment server device 9.
- the subject 7 asks the voice assistant 2 a request for delivering a service to be paid.
- the subject 7 generates the following voice message (i.e., a sound):
- the voice assistant 2 receives (by means of the microphone integrated in the smartphone 8) said voice message at the instant t1 and subsequently transmits towards the electronic processor NLP/NLU 3 an audio message indicative of a request for delivering a service to be paid, in particular the purchase of a medical examination at the XYZ clinic.
- the NLP/NLU electronic processor 3 receives the audio message indicative of the request for delivering the service to be paid (medical examination) and at the instant t4 converts the audio message into a text message, then at the instant t5 the NLP/NLU electronic processor 3 transmits towards the Medical skill 4 said text message indicative of the request for delivering the service to be paid (medical examination).
- the Medical skill 4 receives said text message indicative of the request for delivering the service to be paid (medical examination), then at the instant t7 (subsequent to t6) the Medical skill 4 transmits towards the profile decoding and payment enabling server device 5 a message indicative of a request for availability of the requested service to be paid (medical examination).
- the profile decoding and payment enabling server device 5 receives said message indicative of the request for availability of the requested service to be paid ⁇ medical examination) and at the instant t9 forwards it to the service aggregator 11.
- the service aggregator 11 receives said message indicative of the request for availability of the requested service to be paid (medical examination) or at the instant t11 forwards it to the external service provider 12
- the external service provider 12 verifies the availability of the requested service to be paid and the respective cost; in particular, it is successfully verified that the requested medical examination is available at the healthcare facility XYZ and also the cost of the medical examination and any additional information (e.g., the date and time of the medical examination, the name of the doctor, etc.).
- the external service provider 12 transmits towards the service aggregator 11 a message indicative ot a confirmation of availability of the requested service to be paid (medical examination) and indicative of a request for payment for the cost of the requested service (i.e., the cost of the medical examination): said message is then received and forwarded by the service aggregator 11 , by the profile decoding and payment enabling server device 5 and by the Medical skill 4, up to the instant t16 to the NLP/NLU electronic processor 3.
- the NLP/NLU 3 electronic processor converts the text message into an audio message, then the NLP/NLU electronic processor 3 transmits towards the voice assistant 2 an audio message indicative of the availability of the requested service to be paid (medical examination) and indicative of the request for payment for the cost of the requested service (cost of the medical examination).
- the voice assistant 2 receives said audio message indicative of the availability of the requested service to be paid (medical examination) and indicative of the request for paymenf for the cost of the requested service (cost of the medical examination), then the voice assistant 2 generates (by means of a speaker of the smartphone 8) towards the subject 7 a voice message (i.e., a sound) indicative of the availability of the requested service to be paid (medical examination) and indicative of the request for payment for the cost of the requested service (cost of the medical examination), such as the following voice message:
- the subject 7 receives said voice message, then at the instant 120 the subject 7 emits with his/her voice a sound saying a phrase indicative of a confirmation of the wish to pay for the requested service (medical examination) at the indicated cost, such as the following phrase: ⁇ confirm the payment for the medical examination at the cost of 50 euros".
- a voice message representative of the confirmation of the wish to make the payment for the requested service is stored, and possibly the geographical position of the smartphone 8 of the subject 7 and/or the date/time is also stored.
- the voice assistant 2 receives (by means of the microphone of the smartphone 8) the sound representative of the confirmation of the wish to pay for the requested service at the indicated cost and subsequently transmits towards the NLP/NLU electronic processor 3 an audio message indicative of said confirmation of the wish to pay for the requested service (medical examination) at the indicated cost.
- the NLP/NLU electronic processor 3 receives the audio message indicative of said confirmation of the wish to pay for the requested service (medical examination) at the indicated cost and converts the audio message into a text message, then at the instant t23 the NLP/NLU electronic processor 3 transmits towards the Medical skill 4 said text message indicative of the confirmation of the wish to pay for the requested service (medical examination) at the indicated cost.
- the Medical skill 4 transmits towards the profile decoding and payment enabling server device 5 a message indicative of a request for payment for the requested service (medical examination).
- the profile decoding and payment enabling server device 5 receives the text message indicative of the request for payment for the requested service (medical examination), then at the instant t27 (subsequent to t26) the profile decoding and payment enabling server device 5 transmits towards the profiling server device 6 a message indicative of a request to enable the payment for the service.
- the voice message representative of the confirmation of the wish to pay for the requested service is stored (in a memory of the profile decoding and payment enabling server device 5), and possibly the geographical position of the smartphone 8 of the subject 7 and/or the date/time is also stored.
- the profiling server device 6 receives said message indicative of the request to enable the payment for the service (digital book) and at the instant t29 activates a procedure for verifying the identity of the subject 7, by means of fhe use of the voice profile.
- the profiling server device 8 transmits towards the smartphone 8 a request for acquiring the sample voice profile of the subject 7, wherein said request for acquiring fhe sample voice profile is supposed to be implemented by means of a voice call or a text message from the profiling server device 6 to the smartphone 8 of the subject 7, using the telephone number associated with the SIM card fitted in the smartphone 8 and acquired by the subject 7 in the previous profiling step.
- the electronic device 8 receives the message of the request to acquire the sample voice profile of the subject 7, in particular by means of a voice call or a text message, then the smartphone 8 transmits to the voice assistant 2 an audio message indicative of a request to say a defined phrase, such as the following phrase in the example considered for the medical examination:
- the voice assistant 2 receives the audio message indicative of the request to say the defined phrase, then generates (by means of the speaker of the smartphone 8) towards the subject 7 a voice message (i.e., a sound) indicative of the request to say the defined phrase.
- the subject 7 receives said voice message indicative of the request to say the defined phrase and at the instant t40 the subject 7 emits with the voice a sound saying the requested phrase (i.e., ⁇ am Name and Surname and this is y voice"), which will be used to generate the sample voice profile of the subject 7.
- a sound saying the requested phrase i.e., ⁇ am Name and Surname and this is y voice
- a defined phrase is not used to acquire the sample voice signal representative of the voice of the subject 7, but at the instant t30 the profiling server device 8 transmits towards the smartphone 8 a text message containing an alphanumeric code (i.e., a PIN) and the subject 7 is asked to read aloud the value of the alphanumeric code received, therefore at the instant 133 the subject 7 receives the following voice message in the example considered for the medical examination: "Hello, do you want to pay the cost of 50 euros for the medical examination at the facility XYZ with VoicePay? The subject must answer with the word YES or NO, then say the code contained in the message received".
- an alphanumeric code i.e., a PIN
- the voice assistant 2 acquires (by means of the microphone of the smartphone 8) the voice signal representative of the defined phrase (or the value of the alphanumeric code) said by the subject 7, then an analog to digital conversion of the acquired voice signal is carried out and a digital audio track representative of the voice of the subject 7 is generated therefrom, then said digital audio track sample is forwarded by the voice assistant 2 to the smartphone 8.
- the smartphone 8 receives the sample digital audio track representative of the voice and transmits towards the profiling server device 6 an audio message carrying the sample digital audio track representative of the voice of the subject 7, wherein said audio track is for example a wave, mp3 or ogg format.
- the profiling server device 6 receives the audio message of the sample digital audio track representative of the voice of the subject 7 and temporarily stores it in the memory 6-1 , then the profiling server device 6 transmits towards the payment server device 9 a message indicative of a request for a second portion of the reference voice profile of the subject 7.
- the payment server device 9 receives the message indicative of the request for the second portion of the reference voice profile, reads from the memory 9-1 thereof the second portion of the reference voice profile and at the instant ⁇ 50 (subsequent to t46) transmits towards the profiling server device 6 a message carrying the second portion of the reference voice profile (for example, an alphanumeric code generated with a hash algorithm).
- the profiling server device 6 receives the message carrying the second portion of the reference voice profile, reads from the memory 6-1 thereof the first portion of the reference voice profile and subsequently the profiling server device 6 transmits towards the profile decoding and payment enabling server device 5 a message carrying the digital audio track representative of the voice of the subject 7, together with the first and second portion of the voice profile of the subject 7.
- the profile decoding and payment enabling server device 5 receives the message carrying the sample digital audio track representative of the voice of the subject 7 and generates therefrom the sample voice profile; for example, the processing unit of the profile decoding and payment enabling server device 5 generates a sample alphanumeric code with a hash algorithm as a function of the sample digital audio track representative of the voice of the subject 7 which previously said the defined phrase.
- the profile decoding and payment enabling server device 5 receives the first and second portion of the reference voice profile, then the profile decoding and payment enabling server device 5 decodes the first and second portion of the reference voice profile and recomposes the first and second portion of the reference voice profile, regenerating the reference voice profile therefrom; in particular, a random hash algorithm is used to generate a reference alphanumeric code associated with the reference voice profile, as a function of the two reference alphanumeric codes associated with the first and second portion of the reference voice profile.
- random algorithm means that each time the reference voice profile is generated as a function of the first and second portion of the reference voice profile, the hash function used to generate the alphanumeric code associated with the first portion of the reference voice profile and the alphanumeric code associated with the second portion of the reference voice profile is changed, thus generating two different alphanumeric codes each time, as long as the final result (i.e., the alphanumeric code associated with the reconstructed reference voice profile) always has the same value for the same reference biometric voice information.
- fhe profile decoding and payment enabling server device 5 generates in clear the personal data of the subject 7 in order to verify the identity thereof, but the profile decoding and payment enabling server device 5 does not store the reference voice (or voice/face) profile of the subject 7, which is instead stored partly in the memory 6-2 associated with the profiling server device 6 and partly in the memory 9-1 associated with fhe payment server device 9.
- the identity of the subject 7 is also in dear to the payment server device 9, which must know the identity of the person requesting the payment, in order to verify if he/she is present in the CRM thereof.
- the profile decoding and payment enabling server device 5 compares the sample voice profile and the reference voice profile, in order to verify if they are compatible with each other (i.e., if both belong to the same person, i.e., the subject 7); in particular, a comparison is performed between the sample alphanumeric code (associated with the sample voice profile with a hash function) and the reference alphanumeric code (associated with the reference voice profile with the hash function).
- the profile decoding and payment enabling server device 5 detects (by means of the processing unit thereof) that the sample voice profile is compatible with the reference voice profile: in this case, at the instant t53 the profile decoding and payment enabling server device 5 transmits towards the profiling server device 6 a message indicative of a confirmation of the identity of the subject 7 and indicative of a request for confirmation of the payment for the requested service, then said message is received by the profiling server device 6 and is forwarded to the payment server device 9.
- the payment server device 9 receives (from the profile decoding and payment enabling server device 5) in dear the personal data of the subject 7, in order to verify (for example in a CRM) whether the subject is authorized to pay for the requested good or service.
- the payment server device 9 performs a verificationfion if the subject 7 is authorized to make the payment for the requested service, in particular if the subject 7 is authorized to pay 50 euros to book the medical examination at the healthcare facility XYZ.
- the payment server device 9 uses a Paypal or Amazon Pay or Apple Pay payment system, which can be the same or different from the payment system used by the profile decoding and payment enabling server device 5.
- the payment server device 9 transmits towards the profiling server device 6 a message indicative of a confirmation of payment for the requested service (for example, a booking confirmation and payment for the medical examination), then said message is forwarded from the profiling server device 8 to the profile decoding and payment enabling server device 5.
- the profile decoding and payment enabling server device 5 receives the message indicative of the confirmation of payment for the requested service, then at the instant t63 the profile decoding and payment enabling server device 5 transmits towards the service aggregator 11 a message indicative of the confirmation of payment for the requested service, then said message is forwarded to the external service provider 12, wherein the requested service is actually delivered at the instant t66.
- the external service provider 12 sends the smartphone 8 a link at which the subject 7 receives confirmation of payment and receipt of the purchased service.
- the profile decoding and payment enabling server device 5 transmits towards the application for delivering services to be paid 4 a message indicative of a confirmation of the payment for the requested good or service (medical examination).
- the algorithm used to regenerate the reference voice profile is changed as a function of the first and second portion of the reference voice profile: therefore in the case of a new request for delivering a good or service to be paid by the same subject 7, another algorithm will be used to regenerate the reference profile, always obtaining the same reference voice profile.
- the application for delivering services to be paid 4 receives the message indicative of the confirmation of payment for the requested service (medical examination) and transmits a text message indicative of the confirmation of payment for the requested service to the NLP/NLU electronic processor 3.
- the NLP/NLU electronic processor 3 receives the text message indicative of the confirmation of payment for the requested service (medical examination) and converts the text message into an audio format, then at the instant t74 the NLP/NLU electronic processor 3 transmits towards the voice assistant 2 an audio message indicative of the confirmation of payment for the requested service (medical examination).
- the voice assistant 2 receives said audio message indicative of the confirmation of the payment for the requested service (medical examination), then the voice assistant 2 generates (by means of the speaker integrated in the smartphone 8) a voice signal (i.e., a sound) indicative of the confirmation of the payment for the requested service (digital book), such as the following phrase:
- the reference voice profile has been divided into two portions stored in respective network server devices, but more generally the reference voice profile can be divided into two or more distinct portions stored in two or more corresponding network server devices.
- the tace profile of the subject 7 is used in addition to the voice profile, in order to enable or deny the payment for the requested service: in this way the security level of the verification step of the identity of the subject 7 is increased.
- an image representative of the face (or a part thereof) of the subject 7 was acquired (by means of a camera of the smartphone 8), then this image is used by the profiling server device 6 to generate a reference face profile of the subject 7, in addition to the reference voice profile, hence it will be indicated later with reference voice/face profile: said reference voice/face profile will be stored partly in the memory 6-1 of the profiling server device 6 and partly in the memory 9-1 of the payment server device 9.
- the profiling server device 6 transmits towards the electronic device 8 a message indicative of a request to acquire a sample face profile, in addition to the request to acquire the sample voice profile, which will be indicated with sample voice/face profile;
- the smartphone 8 receives the message requesting acquisition of the sample voice/face profile of the subject 7;
- the voice assistant 2 receives the audio message indicative of the request to say the defined phrase and to take a photo (seifie) of the face of the subject 7, such as: "Hello, do you want to pay the cost of 50 euros for the medical examination at the facility XYZ with VoicePay?
- the subject must answer with the word YES or NO, followed by the phrase I AM NAME and SURNAME and THIS IS MY VOICE, furthermore the subject must take a seifie"; at the instant t43, the smartphone 8 acquires ⁇ by means of the front camera) a seifie of the face of the subject 7 and transmits towards the profiling server device 6 a multimedia message (i.e., audio-video) carrying the sample digital audio track representative of the voice of the subject 7 and carries the image representative of the face of the subject 7; at the instant t52, the profile decoding and payment enabling server device 5 compares the sample voice/face profile and the reference voice/face profile, in order to verify if they are compatible with each other.
- a multimedia message i.e., audio-video
- an OTP code sent by the profiling server device 6 to the smartphone 8 is also used (in addition to the voice profile), in order to verify the identity of the subject 7 using a double authentication factor thus increasing security, in the case where for example it is not possible compare the sample and reference voice profiles, due to noise during the acquisition of the sample voice signal representative of the voice of the subject 7.
- the sample face profile (instead of the sample voice profile), in combination with the OTP code, is used in order to verify the identity of the subject 7 using a double authentication factor.
- a voice signature is used in the electronic system 1 , which is added after the authorization granted to make the payment for the requested good or service (i.e , subsequent to the instant t77).
- the requested service to be paid is the purchase of an insurance policy for a motor vehicle with a limited duration (a few hours), which requires an electronic signature of the subject 7: the driver of the motor vehicle purchases the policy while driving the vehicle, using only voice commands and virtually signing the policy with the voice signature.
- Another example is the purchase of a financial product, which requires an electronic signature of the purchase contract, which is implemented with the voice signature.
- the voice signature is implemented by storing, during a profiling procedure prior to normal operation, a reference digital audio track representative of the voice of the subject 7 associated with two or more defined words; during the normal operation step, the sample voice signal associated with said two or more defined words is acquired in real time, in order to compare the sample voice proiiie with respect to the reference voice profile of the two or more defined words.
- the voice assistant 2 generates (by means of the speaker of the smartphone 8 ⁇ towards the subject 7 a voice message (i.e., a sound) indicative of a request to confirm the payment for the good or service with a voice signature;
- the subject 7 receives said voice message indicative of the request for confirmation with voice signature and awaits instructions:
- the voice assistant 2 generates (by means of the speaker of the smartphone 8) towards the subject 7 a voice message (i.e., a sound) indicative of a request to say one or more defined words (i.e., known in advance), based on the levei of security requested;
- the subject 7 receives said voice message indicative of the request to say one or more words, then the subject 7 emits a sound with the voice saying the requested words;
- the profile decoding and payment enabling server device receives the sample digital audio track representative of the requested words and compares the sample digital audio track and the previously stored reference digital audio track representative of the same requested words; if the comparison between the sample and reference digital audio track is positive, the electronic system 1 approves the payment transaction;
- an OTP code is sent from the profiling server device 6 to the smartphone 8; if the comparison of the OTP code is positive, the electronic system 1 approves the payment transaction;
- the electronic system 1 rejects the payment transaction.
- the voice of the subject 7 is recorded during the acquisition step of the voice signature, and possibly the storage of the geographical position of the electronic device 8 of the subject 7 and/or date/time.
- an integrated voice assistant 2 has been considered as software installed in the electronic device 8 of the mobile type, but the invention is also applicable in the case where the voice assistant 2 is installed on a personal computer or on an IOT (Internet-of-things) device or in the case where the voice assistant 2 is a dedicated electronic device separate from the electronic device 8 typically of the mobile type (smartphone or tablet). It should be noted that the invention is also applicable to enable payment for a good or service using two different payment systems, i.e., the profile decoding and payment enabling server device 5 uses a payment system which is different from that used by the payment server device 9.
- the profile decoding and payment enabling server device 5 uses Apple Pay
- the payment server device 9 uses the Amazon Pay or Paypal payment system.
- FIG. 3 shows a block diagram of an electronic system 101 for enabling payment of a good or service by means of voice commands according to a second embodiment of the invention.
- the electronic system 101 of Figure 3 differs from the electronic system 1 of Figure 1 in that it comprises a first authentication server device 56 in place of the profiling server device 6, because the first authentication server device 56 performs some functionalities different with respect to the profiling server device 6.
- the electronic system 101 comprises an operative server device 105 in place of the server device 5, because the operative server device 105 performs some functionalities different with respect to the server device 5.
- the electronic system 101 of Figure 3 differs from the electronic system 1 of Figure 1 in that it further comprises a second authentication server device 106, in addition to the first authentication server device 56: therefore in this case the second portion of the reference voice profile is stored in the second authentication server device 106, instead of in the payment server device 9.
- the operative server device 105 is connected, through the telecommunications network 10, to both the first authentication server device 56, and to the second authentication server device 106, as well as to the payment server device 9.
- the operation of the electronic system 1 of the second embodiment is the same as the operation of the electronic system 1 of the first embodiment up to the instant t19, i.e., the operation shown in Figure 2A is also applicable to the second embodiment, with the difference that in Figure 4A there is the operative server device 105 in place of the server device 5 of Figure 2A.
- the operation of the second embodiment continues as shown in Figures 4B-4D and differs from the operation of the first embodiment for the following differences: - the verification of the identity of the subject 7 is performed in the operative server device 105 instead of in the profiling server device 6, that is the latter is functionally incorporated within the operative server device 105;
- the operative server device 105 it is performed a comparison between the sample voice profile and the reference voice profile acquired in a previous profiling procedure in a security condition.
- the operation between the instant t120 ⁇ subsequent to t19) and the instant t126 is the same as that illustrated between the instants t20 and t26 for the first embodiment of Figure 2B.
- the operative server device 105 receives the text message indicative of the request for payment for the requested service (medical examination) and at the instant t127 (subsequent to t126) a procedure for verifying the identity of the subject 7 is activated in the operative server device 105, by means of the use of the voice profile, similar to that illustrated for the instant t28 for the first embodiment of the invention.
- the operative server device 105 transmits towards the smartphone 8 an acquisition request for the sample voice profile of the subject 7, wherein said acquisition request for the sample voice profile is supposed to be implemented by means of a voice call or a text message from the operative server device 105 to the smartphone 8 of the subject 7, using the telephone number associated with the SIM mounted into the smartphone 8 and acquired by the subject 7 in the previous profiling step.
- the electronic device 8 receives the message of the acquisition request for the sample voice profile of the subject 7, in particular by means of a voice call or a text message, then the smartphone 8 transmits to the voice assistant 2 an audio message indicative of a request to say a defined phrase.
- the voice assistant 2 receives the audio message indicative of the request to say the defined phrase, then generates (by means of the speaker of the smartphone 8) towards the subject 7 a voice message (i.e., a sound) indicative of the request to say the defined phrase.
- the subject 7 receives said voice message indicative of the request to say the defined phrase and at the instant t240 the subject 7 emits with the voice a sound saying the requested phrase, which will be used to generate the sample vector feature associated with the subject 7.
- the subject 7 emits with the voice a sound by saying a defined phrase or by reading aloud the value of an alphanumeric code received, then at the instant t141 the voice assistant 2 acquires (by means of the microphone of the smartphone 8) the sound representative of the defined phrase (or the value of the alphanumeric code) said by the subject 7, then an analog-to-digital conversion of the acquired voice signal is performed and a sample digital audio track representative of the voice of the subject is generated therefrom, then said sample digital audio track is forwarded by the voice assistant 2 to the smartphone 8.
- the smartphone 8 receives the sample digital audio track representative of the voice of the subject 7 and transmits towards the operative server device 105 an audio message carrying the sample digital audio track representative of the voice of the subject 7.
- the operative server device 105 receives the audio message representative of the sample digital audio track representative of the voice of the subject 7 and temporarily stores it in an internal memory or a memory associated therewith, then the operative server device 105 transmits towards the tirst authentication server device 56 a message indicative of a request tor a first portion of the reference voice profile of the subject 7.
- the first authentication server device 56 receives the message indicative of the request for the first portion of the reference voice profile, reads from the memory 56-1 thereof the first portion of the reference voice profile and at the instant t148 (subsequent to tl 47) transmits towards the operative server device 105 a message carrying the first portion of the reference voice profile (e.g., an alphanumeric code generated with a hash function).
- a message carrying the first portion of the reference voice profile e.g., an alphanumeric code generated with a hash function.
- the operative server device 105 receives the message carrying the first portion of the reference voice profile and temporarily stores it in an internal memory or a memory associated therewith.
- the operative server device 105 transmits towards the second authentication server device 106 a message indicative of a request for a second portion of the reference voice profile of the subject 7.
- the second authentication server device 106 receives the message indicative of the request for the second portion of the reference voice profile, reads from the memory 106-1 thereof the second portion of the reference voice profile and at the Instant 1152 (subsequent to 1151) transmits towards the operative server device 105 a message carrying the second portion of the reference voice profile (e.g., an alphanumeric code generated with a hash function).
- a message carrying the second portion of the reference voice profile e.g., an alphanumeric code generated with a hash function.
- the operative server device 105 receives the message carrying the second portion of the reference voice profile and temporarily stores it in an internal memory or a memory associated therewith.
- Figure 4A shows that the first portion of the reference voice profile is first requested and then the second portion of the reference voice profile, but it is also possible to reverse the two requests (i.e., the second portion of the reference voice profile is first requested and then the first portion of the reference voice profile ) or the two requests are performed simultaneously.
- Figure 4A shows for simplicity's sake that the first portion of the reference voice profile is first received by the operative server device 105 and then the request for the second portion of the reference voice profile is transmitted, but it is also possible to transmit the request for the second portion of the reference voice profile before the first portion of the reference voice profile is received by the operative server device 105.
- the operative server device 105 At the instant tt 54 (subsequent to t153) the operative server device 105 generates a sample voice profile as a function of the sample audio track representative of the voice of the subject 7.
- sample voice profile may alternatively be generated at the previous instant t145 in which it is received at the operative server device 105.
- the operative server device 105 decodes the first and second portion of the reference voice profile and recomposes the first and second portion of the reference voice profile, regenerating the reference voice profile therefrom; in particular, a random algorithm is used to generate a reference alphanumeric code associated with the reference voice profile with a hash function, as a function of the two reference alphanumeric codes associated with the first and second portion of the reference voice profile with the hash function.
- the operative server device 105 performs a comparison between the sample voice profile and the reference voice profile, in order to verify whether they are compatible with each other (i.e., if they both belong to the same person, i.e., the subject 7); in particular, a comparison is performed between the sample alphanumeric code (associated with the sample voice profile) and the reference alphanumeric code (associated with the reference voice profile).
- the operative server device 105 detects (by means of the processing unit thereof) that the sample voice profile is compatible with the reference voice profile: in this case, at the instant t155, the operative server device 105 transmits towards the payment server device 9 a message indicative of a request tor confirmation of payment for the requested service.
- the payment server device 9 receives the message indicative of the request for confirmation of payment for the requested service and at the instant t157 the payment server device 9 verifies whether the subject 7 is authorized to make the payment for the requested service.
- the operative server device 105 receives the message indicative of the confirmation of the payment for the requested service, then at the instant tf 63 the operative server device 105 transmits towards the service aggregator 11 a message indicative of the confirmation of the payment for the requested service, then said message is forwarded to the external service provider 12, wherein the requested service is actually delivered at the instant t166.
- the reference voice profile has been divided into two portions stored into respective network server devices 56, 106, but more generally the reference voice profile can be divided into two or more distinct portions stored into two or more corresponding network server devices.
- FIG. 5 shows a block diagram of an electronic system 201 for enabling payment of a good or service by means of voice commands according to a third embodiment of the invention.
- the electronic system 201 of Figure 5 differs from the electronic system 101 of Figure 3 in that the smartphone 8 is configured to generate a "sample feature vector" as a function of an analog voice signal (i.e., an analog audio track) representative of the voice of the subject 7 and in that it is performed a comparison between the "sample feature vector" and a "reference feature vector” (instead of a comparison between a sample voice profile and a reference voice profile), wherein said comparison is performed in the smartphone 8 (instead of the comparison of the voice profiles In the server device 5): in this way if is avoided to transmit the sample digital audio track representative of the voice of fhe subjecf 7 from the smartphone 8 to the telecommunications network 10, because two portions of the "reference feature vector" are transmitted which are anonymous (i.e , which are not easily associated with a particular user), thus Increasing the security of the transmitted data with respect to the possible interception of the data in transit by malicious persons.
- an analog voice signal i.e., an analog audio track
- sample feature vector means a binary code representative of the digital identity of the subject 7 and uniquely associated with the analog voice signal acquired in real time from the subject 7 and representative of the voice of the subject 7.
- the sample feature vector is generated by means of an algorithm which encodes in binary the distinctive features of the voice of the subject 7, such as one or more of the following features of the voice of the subject 7: the voiceprint of the subject 7; height (pitch) of the voice of the subject 7; intensity (loudness) of the voice of the subject 7; frequency of the voice of the subject 7; bandwidth of the voice of the subject 7; clarity of the voice of the subject 7, i.e., the power of the acoustic signal at high frequencies; number of times the acoustic signal representative of the voice of the subject 7 crosses the null value; spectral model of the acoustic signal representative of the voice of the subject 7; spectrogram of the acoustic signal representative of the voice of the subject 7.
- the "reference feature vector" is a binary code representative of the digital identity of the subject 7 and uniquely associated with the analog voice signal acquired in real time from the subject 7 and representative of the voice of the subject 7, wherein said reference feature vector has been previously acquired by the subject 7 in a profiling procedure by means of the operative server device 105 and in secure conditions, and wherein said reference feature vector has been partly stored into a memory 56-1 associated with the first authentication server device 56 and partly into a memory 106-1 associated with the second authentication server device 106: the reference feature vector has thus been previously verified and it is considered reliable.
- the coding algorithm which generates the sample or reference feature vector can be implemented with a deterministic procedure or with a model obtained with machine learning techniques; the models used can be both statistical and neural, such as recurring networks, convolutional networks, autoencoding models (autoencoder).
- the models can have as input both audio files (as is the case for wav2vec models), and features extracted with techniques such as STFT or Mel-spectrogram.
- the comparison in the electronic device 8 between the "sample feature vector" and the “reference feature vector” is then performed by caiculafing the similarity between the sample feature vector and the reference feature vector, wherein said similarity between the two vectors is calculated, for example, by means of the heuristic technique of cosine similarity or by means of the Euclidean distance.
- the similarity index has small values if the sample feature vector and the reference feature vector belong to different persons; the similarity index has high values if the sample feature vector and the reference feature vector belong to the same person.
- the operation of the electronic system 1 of the third embodiment is the same as the operation of the electronic system 1 of the first and second embodiments up to the instant t19 of Figure 6A, i.e., the operation shown in Figure 2A is also applicable to the third embodiment of Figure 6A, with the difference that in Figure 6A there is the operative server device 105 in place of the server device 5 of Figure 2A.
- the verification of the identity of the subject 7 is performed in the operative server device 105 instead of in the profiling server device 6, i.e., the latter is functionally incorporated within the operative server device 105;
- the smartphone 8 is configured to generate a "sample feature vector" as a function of an analog voice signal (i.e., an analog audio track) acquired by the subject 7 and representative of the voice of the subject 7;
- an analog voice signal i.e., an analog audio track
- the operative server device 105 receives the text message indicative of the request for payment tor the requested service (medical examination) and at the instant 1227 (subsequent to 1226) a procedure for verifying the identity of the subject 7 is activated in the operative server device 105, by means of the use of the voice profile, similar to that illustrated for the instant t28 for the first embodiment of the invention.
- the operative server device 105 transmits towards the smartphone 8 an acquisition request for the sample voice profile of the subject 7, wherein said acquisition request for the sample voice profile is supposed as having been implemented by means of a voice call or a text message from the operative server device 105 to the smartphone 8 of the subject 7, using the telephone number associated with the SIM fitted in the smartphone 8 and acquired by the subject 7 in the previous profiling step.
- the electronic device 8 receives the message of the acquisition request for the sample voice profile of the subject 7, in particular by means of a voice call or a text message, then the smartphone 8 transmits to the voice assistant 2 an audio message indicative of a request to say a defined phrase.
- the voice assistant 2 receives the audio message indicative of the request to say the defined phrase, then generates (by means of the speaker of the smartphone 8) towards the subject 7 a voice message (i.e., a sound) indicative of the request to say the defined phrase.
- the subject 7 receives said voice message indicative of the request to say the defined phrase and at the instant t24Q the subject 7 emits with the voice a sound saying the requested phrase, which will be used to generate the sample vector feature associated with the subject 7.
- the subject 7 emits with the voice a sound by saying a defined phrase or by reading aloud the value of an alphanumeric code received, then at the instant t241 the voice assistant 2 acquires (by means of the microphone of the smartphone 8) the voice signal representative of the defined phrase (or the value of the alphanumeric code) said by the subject 7, then an analog to digital conversion of the acquired voice signal is performed and a sample digital audio track representative of the voice of the subject 7 is generated therefrom, then said sample digital audio track is forwarded by the voice assistant 2 to the smartphone 8.
- the smartphone 8 receives the sample digital audio track representative of the voice and at the instant t244 the processing unit of the smartphone 8 generates in real time a sample vector feature as a function of the sample digital audio track representative of the voice of the subject 7
- the smartphone transmits towards the first authentication server device 58 a message indicative of a request for a first portion of a reference feature vector of the subject 7.
- the first authentication server device 56 receives the message indicative of the request for the first portion of the reference feature vector, reads from the memory 56-1 thereof the first portion of the reference feature vector and at the instant t247 (subsequent to 1246) transmits towards the smartphone 8 a message carrying the first portion of the reference feature vector.
- the smartphone 8 receives a message carrying the first portion of the reference feature vector and temporariiy stores it in an internal memory or a memory associated therewith.
- the smartphone 8 transmits towards the second authentication server device 106 a message indicative of a request for a second portion of a reference feature vector of the subject 7.
- the second authentication server device 106 receives the message indicative of the request for the second portion of the reference feature vector, reads from the memory 106-1 thereof the second portion of the reference feature vector and at the instant 1251 (subsequent to t250) transmits towards the smartphone 8 a message carrying the second portion of the reference feature vector.
- the smartphone 8 receives a message carrying the second portion of the reference feature vector and temporarily stores it in an internal memory or a memory associated therewith.
- the smartphone 8 decodes the first and second portion of the reference feature vector and recomposes the first and second portion of the reference feature vector, regenerating the reference feature vector therefrom.
- the smartphone 8 compares the similarity between the sample feature vector and the reference feature vector, in order to verity whether they belong to the same person, Le., the subject 7; in particular, a similarity index is calculated and this is compared with a similarity threshold value.
- the smartphone 8 detects (by means of the processing unit thereof) that the value of the calculated similarity index is greater than the similarity threshold value: in this case, at the instant 1254 the smartphone 8 transmits towards the payment server device 9 a message indicative of a request for confirmation of the payment for the requested service.
- the payment server device 9 receives the message indicative of the request for confirmafion of payment for the requested service and at the instant t256 the payment server device 9 verifies whether the subject 7 is authorized to make the payment for the requested service.
- the smartphone 8 receives the message indicative of the confirmation of the payment for the requested service, then at the instant 1259 the smartphone 8 transmits towards the service aggregator 11 a message indicative of the confirmation of the payment for the requested service, then said message is forwarded to the external service provider 12, wherein the requested service is actually delivered at the instant t166.
- the reference voice profile has been divided into two portions stored in respective network server devices 56, 106, but more generally the reference voice profile can be divided into two or more distinct portions stored into two or more corresponding network server devices.
- a public- private key digital signature ⁇ asymmetric encryption is used to verify the authenticity and integrity of the messages transmitted by the subject 7 requesting the good or service, Le., to verify that the sender of the message is really who he/she claims to be (i.e., the subject 7) and that the message has not been altered along the path from the sender to the recipient.
- the public and private keys are generated in advance under secure conditions, wherein the public key is stored into a respective memory associated with the first authentication server device 56 and the second authentication server 106, while the private key is stored only into the operative server device 105 and is known only thereto.
- a configuration parameter indicative of a defined hash algorithm (e.g., SHA256) is stored in a respective memory associated with the operative server device 105, the first authentication server device 56 and the second authentication server 106.
- a user identifier uniquely associated with the subject 7 is stored in the smartphone 8, wherein said user identifier represents a unique signed key used to sign and encrypt the messages exchanged between the smartphone 8 and the first authentication server device 56 and the messages exchanged between the smartphone 8 and the second authentication server device 106.
- the operation of the variant of the second embodiment is modified as follows: at the instant t143, the smartphone 8 transmits towards the operative server device 105 an audio message carrying the digital audio track representative of the voice of the subject 7, together with the user identifier; at the instant t145, the operative server device 105 receives the audio message carrying the sample digital audio track and the user identitier associated with the subject 7, then the operative server device 105 generates (by means of the processing unit thereof) a digital fingerprint (message digest) of the user identifier based on a defined hash algorithm, thereby generating a string of alphanumeric characters (i.e., an alphanumeric code), then an encryption of the generated alphanumeric code is performed using a private key so as to generate a new alphanumeric code representing the digital signature of the user identifier, and finally the operative server device 105 transmits towards the first authentication server device 56 the message indicative of a request for a first portion of the reference voice profile of the subject 7, together with the user identifier;
- the operation of the variant of the third embodiment is modified as follows: at the instant t243, the smartphone 8 receives the audio message carrying the sample digital audio track representative of the voice of the subject 7, then the smartphone 8 generates (by means of the processing unit thereof) a digital fingerprint (message digest) of the user identifier based on a defined hash algorithm, thereby generating a string of alphanumeric characters (i.e., an alphanumeric code), then an encryption of the alphanumeric code generated is performed using a private key so as to generate a new alphanumeric code representing the digital signature of the user identifier, and finally the smartphone 8 transmits towards the first authentication server device 56 the message indicative of a request for a first portion of the reference voice profile of the subject 7, together with the user identifier and the digital signature of the user identifier; the operation at the instant t246 is the same as that illustrated at the instant t147 for the variant of the second embodiment; the operation at the instant t247 is the same as that illustrated at the instant t2
- one or more images representative of the face of the subject 7 are further acquired, in addition to the voice signal representative ot the voice of the subject 7, both in the profiling procedure and in real time, thus generating a reference voice/face profile and a sample voice/face profile.
- one or more images representative of the face of the subject 7 are further acquired, in addition to generating the feature vector of the subject 7, both in the profiling procedure and in real time, thus generating a reference feature vector and a reference face profile and generating a sample feature vector and a sample face profile.
- a video recording is acquired in which at least the face of the subject 7 is framed and in which he/she says a defined phrase aloud, thus generating the reference face profile together with the reference feature vector; similarly, a video recording is acquired in real time in which at least the face of the subject 7 is framed and in which he/she says a defined phrase aloud, thus generating the sample face profile together with the sample feature vector.
- the method is implemented in part by means of a suitable software program run on an electronic processor ⁇ for example, a microprocessor or an 10T device or an chicken) of the voice assistant 2 or which implements the voice assistant 2, in part by means of a suitable software program run on an electronic processor (for example, a microprocessor) of the electronic device 8, in part by means of a software program which implements the application for delivering services to be paid 4, in part by means of a suitable software program run on an electronic processor (for example, a microprocessor) of the profile decoding and payment enabling server device 5, in part by means of a suitable software program run on an electronic processor (for example, a microprocessor) of the profiling server device 6 and in part by means of a suitable software program run on an electronic processor (for example, a microprocessor) of the payment server device 9.
- a suitable software program run on an electronic processor ⁇ for example, a microprocess
- the method for enabling payment comprises, alternatively, the same steps indicated:
- the software program of the application for delivering services to be paid 4 performs some steps of the method for enabling payment for a good or service illustrated above of the first, second or third embodiment or of the variants of the second or third embodiment. It is also an object of the present invention a computer program comprising software code portions run on an electronic processor of the profile decoding and payment enabling server device 5 of the first embodiment, or run on a computer of the operative server device 105 of the second embodiment.
- Non-transitory computer-readable storage medium having a program comprising software code portions run on a computer of the profile decoding and payment enabling server device 5 of the first embodiment, or run on a computer of the operative server device 105 of the second embodiment.
- the software program of the server device 5 or the operative server device 105 performs some steps of the method for enabling payment of a good or service illustrated above respectively for the first or second embodiment.
- the software program of the profiling server device 6 runs some steps of the method for enabling payment of a good or service illustrated above.
- the software program of the payment server device 9 runs some steps of the method for enabling payment of a good or service illustrated above.
- the software program of the electronic device 8 runs some steps of fhe method for enabling payment of a good or service illustrated above for the third embodiment or for fhe related variant.
- the invention in the three embodiments and related variants indicated above is applicable not only to enable the payment for a good or service by means of voice commands, but more generally can be used to control an electro-mechanical actuator by means of voice commands, for example to control the opening of an access door, the opening of an automatic gate, the ignition of a motor vehicle.
- the invention differs from the three embodiments illustrated above in that: there is no application for delivering services to be paid 4; the server device 5 or 105 is replaced by a profile decoding and control enabling server device; the payment server device 9 is replaced by an electro-mechanical actuator.
- the subject 7 says aloud an actuation command to be executed by means of the eiectro-mecbanica!
- this command is processed by means of the human language electronic processor 3, which is capable of performing a sound-to-text conversion of the actuation command and then the actuation command is extracted, which is used to command the electro-mechanical actuator (in the example, the opening of the automatic gate).
- the control method of the actuator comprises: a) receiving, at a voice assistant 2, a voice message indicative of a request to control the actuator and transmitting, towards an electronic human language processor 3, a first audio message indicative of the request to control the actuator: b) receiving, at the human language processor 3, said audio message indicative of the request to control the actuator and transmitting, towards a profile decoding and control enabling server device, a second message indicative of an availability request of the actuator; c) receiving, at the profile decoding and control enabling server device 5, a third message indicative of a confirmation of availability of the requested actuator and forwarding the third message to the human language electronic processor 3; d) receiving, at the human language processor 3, the third message and transmitting, towards the voice assistant 2, an audio message indicative of the availability of the actuator; e) receiving, at the voice assistant 2, said audio message and generating a voice message indicative of an availability of the actuator; f) receiving, at the voice assistant 2, a voice message indicative of a confirmation of the wish to control the actuator and transmitting, towards the electronic human language processor 3,
- the control system comprises a voice assistant 2, a human language electronic processor 3 connected to the voice assistant 2, a profile decoding and control enabling server device 5 connected to the human language electronic processor 3, a profiling server device 6 connected to the profile decoding and control enabling server device, an electro-mechanical actuator connected to the profiling server device 6 and an electronic device 8, wherein the voice assistant 2 is configured to: receive a voice message indicative of a request to control the actuator and transmit, towards an electronic human language processor 3, an audio message indicative of the request to control the actuator; receive an audio message indicative of an availability of the actuator and generate a voice message indicative of the availability of the actuator and indicative of a request for confirmation of the will to control the actuator; receive a voice message indicative of a confirmation of the wish to control the actuator and transmitting, towards the electronic human language processor 3, an audio message indicative of the confirmation of the wish to control the actuator; receive an audio message indicative of a request to say a phrase and generate a voice message indicative of fhe request to say the phrase; acquire a sound representative of the requested phrase and generating therefrom
Landscapes
- Business, Economics & Management (AREA)
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Strategic Management (AREA)
- Accounting & Taxation (AREA)
- General Business, Economics & Management (AREA)
- Economics (AREA)
- Development Economics (AREA)
- Finance (AREA)
- Marketing (AREA)
- Human Resources & Organizations (AREA)
- Tourism & Hospitality (AREA)
- Entrepreneurship & Innovation (AREA)
- Quality & Reliability (AREA)
- Operations Research (AREA)
- Multimedia (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- General Health & Medical Sciences (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computer Security & Cryptography (AREA)
- Computational Linguistics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Acoustics & Sound (AREA)
- Educational Administration (AREA)
- Game Theory and Decision Science (AREA)
- Telephonic Communication Services (AREA)
- Control Of Vending Devices And Auxiliary Devices For Vending Devices (AREA)
Abstract
Description
Claims
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
IT102020000015973A IT202000015973A1 (en) | 2020-07-02 | 2020-07-02 | ELECTRONIC SYSTEM AND METHOD FOR ENABLING PAYMENT FOR A GOOD OR SERVICE BY MEANS OF VOICE COMMANDS |
PCT/IB2021/055428 WO2022003474A1 (en) | 2020-07-02 | 2021-06-21 | Electronic system and method for enabling payment of a good or service by means of voice commands |
Publications (3)
Publication Number | Publication Date |
---|---|
EP4176395A1 true EP4176395A1 (en) | 2023-05-10 |
EP4176395C0 EP4176395C0 (en) | 2024-01-31 |
EP4176395B1 EP4176395B1 (en) | 2024-01-31 |
Family
ID=72561871
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP21742497.7A Active EP4176395B1 (en) | 2020-07-02 | 2021-06-21 | Electronic system and method for enabling payment of a good or service by means of voice commands |
Country Status (5)
Country | Link |
---|---|
US (1) | US20230259928A1 (en) |
EP (1) | EP4176395B1 (en) |
ES (1) | ES2973988T3 (en) |
IT (1) | IT202000015973A1 (en) |
WO (1) | WO2022003474A1 (en) |
Families Citing this family (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR102520513B1 (en) * | 2021-11-16 | 2023-04-11 | 주식회사 딥이티 | Apparatus and method for face recognition using user terminal |
US20230238000A1 (en) * | 2022-01-27 | 2023-07-27 | Ford Global Technologies, Llc | Anonymizing speech data |
Family Cites Families (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20030229588A1 (en) * | 2002-06-05 | 2003-12-11 | Pitney Bowes Incorporated | Voice enabled electronic bill presentment and payment system |
US8554674B1 (en) * | 2006-10-31 | 2013-10-08 | United Services Automobile Association (Usaa) | Transfer caller into speech make-a-payment transaction |
US9092781B2 (en) * | 2007-06-27 | 2015-07-28 | Verizon Patent And Licensing Inc. | Methods and systems for secure voice-authenticated electronic payment |
US8271285B2 (en) * | 2007-08-02 | 2012-09-18 | International Business Machines Corporation | Using speaker identification and verification speech processing technologies to activate and deactivate a payment card |
US8489507B1 (en) * | 2012-03-28 | 2013-07-16 | Ebay Inc. | Alternative payment method for online transactions using interactive voice response |
US10192219B2 (en) * | 2014-01-09 | 2019-01-29 | Capital One Services, Llc | Voice recognition to authenticate a mobile payment |
US20180108001A1 (en) * | 2014-03-24 | 2018-04-19 | Thomas Jason Taylor | Voice triggered transactions |
CA2982196C (en) * | 2015-04-10 | 2022-07-19 | Huawei Technologies Co., Ltd. | Speech recognition method, speech wakeup apparatus, speech recognition apparatus, and terminal |
US10769630B2 (en) * | 2016-05-11 | 2020-09-08 | Mastercard International Incorporated | Mobile person to person voice payment |
US20170337558A1 (en) * | 2016-05-19 | 2017-11-23 | Mastercard International Incorporated | Method and system for voice authenticated distribution of payment credentials |
US20180357645A1 (en) * | 2017-06-09 | 2018-12-13 | Walmart Apollo, Llc | Voice activated payment |
US10810574B1 (en) * | 2017-06-29 | 2020-10-20 | Square, Inc. | Electronic audible payment messaging |
KR20190102509A (en) * | 2018-02-26 | 2019-09-04 | 삼성전자주식회사 | Method and system for performing voice commands |
US11176543B2 (en) * | 2018-09-22 | 2021-11-16 | Mastercard International Incorporated | Voice currency token based electronic payment transactions |
US11538012B2 (en) * | 2019-02-11 | 2022-12-27 | Mastercard International Incorporated | Systems and methods for generating a shared payment via voice-activated computing devices |
US11341500B2 (en) * | 2020-03-13 | 2022-05-24 | Mastercard International Incorporated | Inaudible voice payment |
-
2020
- 2020-07-02 IT IT102020000015973A patent/IT202000015973A1/en unknown
-
2021
- 2021-06-21 EP EP21742497.7A patent/EP4176395B1/en active Active
- 2021-06-21 ES ES21742497T patent/ES2973988T3/en active Active
- 2021-06-21 WO PCT/IB2021/055428 patent/WO2022003474A1/en active Application Filing
- 2021-06-21 US US18/003,742 patent/US20230259928A1/en active Pending
Also Published As
Publication number | Publication date |
---|---|
WO2022003474A1 (en) | 2022-01-06 |
ES2973988T3 (en) | 2024-06-25 |
EP4176395C0 (en) | 2024-01-31 |
US20230259928A1 (en) | 2023-08-17 |
IT202000015973A1 (en) | 2022-01-02 |
EP4176395B1 (en) | 2024-01-31 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11032421B2 (en) | Systems for transitioning telephony-based and in-person servicing interactions to and from an artificial intelligence (AI) chat session | |
TWI703465B (en) | Core body method and device | |
US11010803B2 (en) | Identity verification and authentication | |
US20210377264A1 (en) | Out-of-Band Biometric Enrollment and Verification Using Interactive Messaging | |
US20220400109A1 (en) | Centralized gateway server for providing access to services | |
US20200169552A1 (en) | Using an audio interface device to authenticate another device | |
US8095372B2 (en) | Digital process and arrangement for authenticating a user of a database | |
US11539526B2 (en) | Method and apparatus for managing user authentication in a blockchain network | |
US10665238B1 (en) | Alert through voice assistant | |
EP4176395B1 (en) | Electronic system and method for enabling payment of a good or service by means of voice commands | |
US8954317B1 (en) | Method and apparatus of processing user text input information | |
US11580505B2 (en) | Methods for facilitating funds disbursements and devices thereof | |
US11729624B2 (en) | Techniques for call authentication | |
CN111353925A (en) | Block chain-based fraud prevention system and method | |
US10270771B1 (en) | Mid-session live user authentication | |
EP3241177A1 (en) | Out-of-band biometric enrollment and verification using interactive messaging | |
CN111695905B (en) | Payment method, device, computing equipment and storage medium | |
US10488940B2 (en) | Input commands via visual cues | |
WO2014172502A1 (en) | Integrated interactive messaging and biometric enrollment, verification, and identification system | |
US11924378B2 (en) | Systems for transitioning telephony-based and in-person servicing interactions to and from an artificial intelligence (AI) chat session | |
US10924485B2 (en) | Electronic signing authorization system | |
CN1655501A (en) | Identification apparatus and method employing biological statistic data |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: UNKNOWN |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE |
|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20230201 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
DAV | Request for validation of the european patent (deleted) | ||
DAX | Request for extension of the european patent (deleted) | ||
INTG | Intention to grant announced |
Effective date: 20230817 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602021009064 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
U01 | Request for unitary effect filed |
Effective date: 20240228 |
|
U07 | Unitary effect registered |
Designated state(s): AT BE BG DE DK EE FI FR IT LT LU LV MT NL PT SE SI Effective date: 20240306 |
|
REG | Reference to a national code |
Ref country code: ES Ref legal event code: FG2A Ref document number: 2973988 Country of ref document: ES Kind code of ref document: T3 Effective date: 20240625 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240531 |
|
U20 | Renewal fee paid [unitary effect] |
Year of fee payment: 4 Effective date: 20240529 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: IE Payment date: 20240618 Year of fee payment: 4 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240501 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240131 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240430 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240430 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240430 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240531 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240131 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240501 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240131 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20240131 |