EP4728424A1

EP4728424A1 - Method and system for generating a fingerprint, and method and system for determining a resemblance between a first and second text

Info

Publication number: EP4728424A1
Application number: EP24732018.7A
Authority: EP
Inventors: Eric Decoux; Philippe THEVOZ
Original assignee: SICPA Holding SA
Current assignee: SICPA Holding SA
Priority date: 2023-06-19
Filing date: 2024-06-13
Publication date: 2026-04-22
Also published as: WO2024260849A1

Abstract

A computer-implemented method for generating a fingerprint of a first text, comprising: segmenting the first text into first sentences; for each first sentence, removing noise from the first sentence according to a noise removal rule, thereby obtaining a first simplified sentence, the noise removal rule applied to the first sentence defining a selection of words of the first sentence to be kept and a format into which the selection of words is converted to obtain the first simplified sentence; hashing each first simplified sentence to obtain a first hash associated therewith; associating first metadata with each first hash, the first metadata including a first timestamp indicating a time of generation of the corresponding first sentence or of performing at least part of the computer-implemented method; and storing the fingerprint of the first text in a database, the fingerprint being formed by each first hash and by the first metadata

Description

METHOD AND SYSTEM FOR GENERATING A FINGERPRINT, AND METHOD AND SYSTEM FOR DETERMINING A RESEMBLANCE BETWEEN A FIRST AND SECOND TEXT

TECHNICAL FIELD

The present invention relates to the technical field of verifying whether a text has been copied or plagiarized. In detail, the present invention relates to a computer-implemented method for generating a fingerprint of a first text, to a computer-implemented method for determining a resemblance between a first text and a second text, a computer program product, a database storing the fingerprint of the first text, a fingerprint generation system, and a text resemblance determination system.

BACKGROUND ART

Many people such as teachers face the problem of not knowing whether a submitted text has indeed been written by the person who pretends to be the author. This is problematic, as the teacher may award a grade on a work that has been unallowable copied somewhere.

With the advent of artificial intelligence (Al) systems allowing to generate texts (e.g. ChatGPT by OpenAI), it becomes even more important to determine whether a given text has indeed been written by the claimed author, or whether it has been written by an Al system.

The paper "Copy Detection Mechanisms for Digital Documents" by S. Brin et al., (SIGMOD '95: Proceedings of the 1995 ACM SIGMOD international conference on Management of data, June 1995, Pages 398-409) describes a copy detection service for detecting exact copies and documents that overlap in significant ways. To this end, when an author creates a new work, he or she registers it at a server. As the documents are registered, they are broken into small units, such as sentences. Each sentence is hashed and a pointer to it is stored in a large hash table. When a document is to be checked, it is also broken into sentences. For each sentence, the hash table is probed to see if that particular sentence has been seen before. If the document and a previously registered document share more than some threshold number of sentences, then a violation is flagged.

If a fraudster introduces a small modification in the copied text, the above solution does not recognize any violation. It is therefore an objective of the invention to provide a more robust solution for determining a resemblance between two texts. SUMMARY OF THE INVENTION

According to a first aspect, a computer-implemented method for generating a fingerprint of a first text is provided. The method comprises: segmenting the first text into first sentences; for each first sentence, removing noise from the first sentence according to a noise removal rule, thereby obtaining a first simplified sentence, the noise removal rule applied to the first sentence defining a selection of words of the first sentence to be kept and a format into which the selection of words is converted to obtain the first simplified sentence; hashing each first simplified sentence to obtain a first hash associated therewith; associating first metadata with each first hash, the first metadata including a first timestamp indicating a time of generation of the corresponding first sentence or of performing at least part of the computer-implemented method; and storing the fingerprint of the first text in a database, the fingerprint being jointly formed by each first hash and by the corresponding first metadata.

Storing the fingerprint of the first text allows storing an information relating to the content of the first text without directly disclosing the content of the first text. The privacy of the first text can be preserved as only the first hashes are stored. This fingerprint of the first text can be used to compare it with a fingerprint of a second text, thereby determining a resemblance between the texts, without having to consider the content of the texts directly. Storing the fingerprint of the first text instead of the entire first text is advantageous as it requires less storage space. Preferably, the fingerprints of multiple texts, of all available texts and/or of all texts created by an Al (forming first texts) are generated based on the method according to the first aspect and stored in the database. This allows storing an encoded (hashed) version of all these first texts, for example for detecting resemblances between texts.

Since each first hash is calculated based on a first simplified sentence, the solution is robust against small modifications of a text by a fraudster. Indeed, such small modifications would not change the simplified sentence and the corresponding hash. This will be described in more detail below, in particular in view of the second aspect.

As defined herein, the expression "computer-implemented method" designated a method that can be entirely performed by a computer, in particular by running a corresponding program on the computer. The first text can be any text written by an author, such as an article, a book, a blog post or the like. The first text may be written by an Al, such as ChatGPT, in which case the Al is considered as the author (also referred to as "issuer"). The first text is preferably provided digitally. In case that the first text is provided as a printed document, the method may include scanning the first text and running an optical character recognition (OCR) on it. The first text may also be a record or transcription of an audio track (such as a song, radio, podcast or the like).

The step of segmenting the first text into first sentences can include breaking or splitting the first text into separate parts, each part forming a sentence (first sentence). A sentence can be defined as a string of words expressing a complete thought. A sentence can start with a predefined starter element (such as a capital letter) and end with a predefined end element (such as a full stop). In order to perform segmenting of the first text, an OCR may previously be run over the first text to recognize characters thereof (in case that the first text is provided on a printed document) or a tool for extracting text from a digital file (such as a PDF or DOC file) containing the first text can be used. A word can be defined as a single distinct meaningful element of speech or writing, used with others (or sometimes alone) to form a sentence and typically shown with a space on either side when written or printed. The term "word" as used herein may also correspond to a sub-word (i.e. a portion of a word) or a token (i.e. common sequences of characters found in text).

Each first sentence is processed individually. First, each first sentence is simplified using the noise removal rule. One same noise removal rule is applied to each first sentence. The noise removal rule is defined in advance. The noise removal rule can define which words of the first sentence are to be kept (corresponding to the selection of words of the first sentence) and which ones are to be discarded. Further, the noise removal rule can define a format of the selection of words, such that the selection of words is put into the defined format. Examples for the specific rules defining which words of the first sentence are kept as well as for the format are given further below. Preferably, after applying the noise removal rule, the resulting first simplified sentence comprises only the meaningful words of the original first sentence, said words being shortened to only include their stem, and reordered alphabetically. One first hash is calculated for each first simplified sentence. Preferably, it is impossible to go back from the first hash to the corresponding first (simplified) sentence, ensuring that the privacy of the first text is preserved. The first hash may be a cryptographic hash (described below), or any type of a one-way function.

The first timestamp can indicate a date and/or time at which the text has been created, submitted for generating the fingerprint, or at which the first hash or the fingerprint is generated according to the method of the first aspect. The accuracy of the first timestamp can be one minute, one second, or less. The first timestamp can be in the format date-hour-minute-second. The first timestamp can be expressed in unix time. Associating the first metadata including the first timestamp with the first hash can correspond to storing the first metadata and the first hash jointly, providing a pointer to the first hash in the first metadata and/or providing a pointer to the first metadata in the first hash. The first timestamp can be part of the same file as the first hash or provided in a separate file. In particular, the first timestamp can be written in plaintext at the end or beginning of the first hash. Providing a first timestamp with the first hash allows a temporal specification of the first hash, which is particularly useful when comparing the content of different texts using their fingerprint, as a copied text is necessarily more ancient than a copying text.

The database can be any type of storage means (including a centralized database, a distributed database, a blockchain, an append-only database and the like). Each fingerprint can form an entry in the database. The fingerprint of the first text is defined as consisting of the first hashes and their first metadata. Multiple fingerprints (one for each first sentence) can be associated with the first text. Preferably, the fingerprints of multiple texts, of all available texts and/or of all texts generated by an issuer (such as an Al or a journal, for example) are generated based on the method according to the first aspect and stored in the database. The fingerprint can also be designated as an invisible fingerprint because it does not require any addition to the first text itself. Storing the fingerprint instead of the first text not only allows to maintain confidentiality, it also allows to reduce a required storage space because the fingerprint requires less storage space than the first text itself.

This invisible fingerprint can allow proving who is the author of a first text and/or when it has been written or timestamped (depending on the content of the first metadata). Accordingly, an author or issuer could provide the proof, verifiable by anyone, that a given text has well been produced by said author or issuer. In such case, the first metadata should contain the issuer/author name. To verify that a text indeed corresponds to a first text, a verifier checks that upon segmenting, simplifying using the noise removal rule and hashing the sentences of the first text to be verified, the same hashes as for the original first text are obtained.

According to a second aspect, a computer-implemented method for determining a resemblance between a first text and a second text using a fingerprint associated with the first text which is stored in a database is provided, the fingerprint including multiple first hashes (and optionally corresponding first metadata). The method comprises: segmenting the second text into second sentences; for at least one second sentence, removing noise from the second sentence according to a noise removal rule applied on the second sentence, thereby obtaining a second simplified sentence, the noise-removal rule applied on the second sentence defining a selection of words of the second sentence to be kept and a format into which the selection of words is converted to obtain the second simplified sentence; hashing the second simplified sentence to obtain a second hash associated therewith; comparing the second hash with at least some of the first hashes stored in the database to determine whether the second hash and one of the first hashes used for the comparison are identical; and outputting an identical hash information indicating a resemblance between the first text and the second text if it is determined that the second hash and one of the first hashes used for the comparison are identical. The method according to the second aspect defines how it can be verified whether a second text is a copy of a first text, the fingerprint of which is stored in a database. The first text may correspond to the first text described in view of the first aspect and its fingerprint may be generated as described in view of the first aspect.

The second text can be any type of text , such as an article, a book, a blog post, a transcript of an audio track, or the like. The author of the second text might be unknown. In particular, it is desired to determine an author of the second text. It can be determined whether an indicated author of the second text is indeed the genuine author of the second text or not. For example, the method of the second aspect allows determining whether the second text has been written by an Al, such as ChatGPT, or by a human. The second text is preferably provided digitally. In case that the second text is provided as a printed document, the method may include scanning the second text and running an optical character recognition (OCR) on it. The second text may be identical to the first text. In this case, the described method allows checking that the second text has indeed been written or issued to the pretending author or issuer.

In the method of the second aspect, the second text is processed in a nearly identical manner to the first text in the first aspect. In detail, the steps of segmenting, removing noise and hashing the second simplified sentence according to the second aspect may be identical to the steps of segmenting, removing noise and hashing the first simplified sentence according to the first aspect, but applied on the second text and at least one second sentence instead. Preferably, the noise removal rule applied on the first sentence is identical with the noise removal rule applied on the second sentence to ensure that the first and second simplified sentences are in an identical format. Where not described otherwise, all features described with reference to the first text (including first sentence, first simplified sentence, first hash, first metadata, first timestamp and the like) also apply to the second text (including second sentence, second simplified sentence, second hash, second metadata, second timestamp and the like). The determination and storage of second metadata and a second timestamp is optional for the second text.

In order to determine the resemblance between the first and second texts, the second hash is compared with one or multiple first hashes previously stored in the database. Accordingly, it can be determined whether the second hash corresponds to one of the first hashes used for the comparison. If this is the case, an identical hash information is output, for example through a display. The identical hash information can indicate that there is a resemblance between the compared first and second texts, which can be an indicator of plagiarism. In detail, when it is determined that the second hash corresponds to one of the first hashes used for the comparison, it is likely that the second sentence corresponding to the second hash has been copied (which is an indicator for plagiarism). When it is determined that the second hash does not correspond to one of the first hashes used for the comparison, it is likely that the second sentence corresponding to the second hash is an original (or at least that it has not been written by any issuer of first texts which have been registered in the corresponding database of first text hashes).

If several second sentences have a positive result (i.e. their second hash is identical to a first hash) and if the first timestamps of the identical first hashes are the same or close from each other, this means that these different second sentences are not only coming from the same issuer, but from the same original text.

As the solution is acting on each sentence, independently from the others, a suspicion about the origin of the second text can be detected even if a single first sentence of the first text has been used.

Optionally, the step of outputting an identical hash information further includes an outputting of the first timestamp and/or of the first metadata associated with the first hash which is found to be identical to the second hash.

The noise removal rule is predefined such that minor changes to a sentence lead to a same simplified sentence (after application of the noise removal rule) as the original sentence. Accordingly, the first simplified sentence and the second simplified sentence can be identical even if the corresponding original first and second sentences were not entirely identical, but diverged only through the use of synonyms or order of words, for example. In turn, two not entirely identical first and second sentences can lead to identical first and second hashes. This renders the solution robust against small modifications of a text by a fraudster. In particular, a fraud can be detected even if the fraudster changed part of the second text, removed some sentences, inverted the order of some sentences and/or combined sentences coming from two different original texts.

For convenience, the expressions "text", "sentence", "simplified sentence", "hash", "metadata", "timestamp", and the like can be used to respectively designate either the first or second text, sentence, simplified sentence, hash, metadata, timestamp, and the like, when no distinction between first and second of these entities needs to be made.

According to an embodiment, in the method of the second aspect, the fingerprint of the first text is generated according to the method of the first aspect.

According to a further embodiment, the noise removal rule of the first and/or of the second aspect replaces each word of the selection of words to be kept by a code corresponding to said word so that the first and/or second simplified sentence is formed by a juxtaposition of codes, each code being defined through a prestored table associating the code with one or more corresponding predefined words, wherein the one or more corresponding predefined words have similar meanings, are synonyms and/or are translations of a same word in different languages.

When a fraudster performs slight modifications to a copied text in order to hide the fact that the text is copied, he standardly replaces a word by a synonym, and/or translates all or parts of the text. In order to account for these small modifications and still detect that the fraudster has copied a preexisting text, the noise removal rule is preferably defined such that synonyms and/or translations of a same word are treated as identical words, leading to a same simplified sentence and consequently a same hash. In this regard, all words that are considered as having a same or similar meaning, or as being translations of a same word in different languages, are provided in the prestored table and associated with a code. The code can be a word, a predefined letter combination, a predefined number, and/or a predefined combination of letters and numbers. When a first and/or second sentence is run through the noise removal rule, each word to be kept is replaced by its corresponding code as defined by the prestored table. Using this code system renders the determination of a resemblance between texts even more resistant because it accounts for the use of synonyms, words with similar meanings and/or translations of a same word. The prestored table can include at least parts of the WordNet (such as WordNet 3.0) database.

According to a further embodiment, the noise removal rule defines, as the selection of words of the first or second sentence to be kept: that pronouns and adverbs are ignored; that proper nouns are ignored; that stop-words (which may be defined in a stop-word list and may include words such as "a", "the", "this", "who" and the like) are ignored; and/or that words that do not belong to a reference dictionary or the prestored table are ignored (or a weight is associated with each word and only words with a weight above a predetermined threshold are kept).

Accordingly, the noise removal rule allows keeping only a selection of words that are "meaningful" and allow distinguishing a sentence from another sentence. As such, common terms can be ignored. Selecting the noise removal rule accordingly improved the robustness of the solution for determining a resemblance between a first and second text, as it allows for modifications by a fraudster to be detected.

According to a further embodiment, the noise removal rule defines, as the format into which the selection of words is converted to obtain the first or second simplified sentence, that: all characters are put in lower-case or in upper-case; all verbs are replaced by a corresponding infinitive or a predefined verb form; all common nouns and adjectives are put in the masculine singular or into another predefined form; all words are replaced by their stem using a stemming algorithm; all words are replaced by their lemma using a lemmatization algorithm; all special characters are replaced by an equivalent simpler character (for example letter with accents are replaced by their equivalent without accents); all spelling mistakes are corrected; all punctuation, multiple spaces, tabs, bullet points, and/or carriage returns are ignored; all numbers are ignored; a single space separates each word; the words are placed in alphabetical order or in another predefined order; words which allow multiple spellings (such as "authorize" and "authorise") are replaced by a selected word; and/or repetitions of identical words in a same sentence are ignored.

Accordingly, the noise removal rule allows putting the selected words into a format that does not account for small formulation or stylistic modifications of a sentence. For example, through the alphabetic sorting, the inversion of two words will still lead to identical hashes. As such, equivalent or similar sentences lead to the same simplified sentence. Selecting the noise removal rule accordingly improves the robustness of the solution for determining a resemblance between a first and second text, as it allows for modifications by a fraudster to be detected.

The noise removal rule may further split long sentences in shorter ones, according to rules allowing to keep the meaning (i.e. not splitting the sentence arbitrarily). It could be possible to determine and/or store both the hash of the original sentence as well as hashes of the resulting "subsentences", thereby allowing to detect a resemblance between texts even if a fraudster split long sentences into multiple portions.

Due to the way a sentence is processed, any small discrepancy that a fraudster could introduce will not fool the verification. In order to avoid a detection, a fraudster would need to replace nouns or verbs in each sentence. Although this is theoretically possible, this could be quite long and tedious.

According to a further embodiment, the hashing of the first sentence and/or of the second sentence uses a cryptographic hash algorithm on the first sentence and/or second sentence. Examples for such cryptographic hash algorithms include SHA-1 , SHA-2, SHA-256, or the like. The hashing algorithms used to calculate the first and the second hashes are identical. In theory, any cryptrographic hash algorithm can be used. Preferably however, the obtained hash is sufficiently robust to avoid any decoding thereof (for confidentiality reasons in applications in which confidentiality is required). Further, the hash is preferably large enough to prevent any collisions (i.e. an identical hash being calculated for entirely different simplified sentences). At the same time, the hash is preferably short enough to encode short sentences and such that it does not require too much storage space. Using a SHA-256 appears a good compromise between these aspects.

According to a further embodiment, the first and/or second texts are texts generated by an artificial intelligence (Al). Each text generated by the artificial intelligence can be processed as the first text. Before performing the verification according to the second aspect, it might be unknown whether the second text is generated by Al or not. The first and/or second texts may alternatively be texts published in a journal, in particular by a specific editor.

Texts generated by Al include, for example, texts fully and/or partly generated by ChatGPT (by OpenAI, for example GPT-3 or GPT-4) or other conversational chatbots. Preferably, a fingerprint of each (first) text generated by the Al is stored in the database, so that it can be verified whether a text has been generated or copied from an Al-generated text. This is particularly useful for teachers to check whether the work submitted by a student indeed comes from the student or not. It is further useful to check that a quote in a news article, attributed to a given author, has not instead been written by an Al system, thereby allowing to fight against fake news. In both of these cases, one in particular should not find any sentence with the corresponding hash in the database.

The method according to the first aspect, the second aspect, or an embodiment thereof, can be performed by the Al provider. In other words, the Al provider may hold the database in which all fingerprints of the generated first texts are stored and/or may perform the resemblance determination between first and second texts.

According to a further embodiment, the step of comparing the second hash with at least some of the first hashes stored in the database includes: comparing the second hash with all of the first hashes stored in the database; or comparing the second hash with only the first hashes stored in the database which are associated with a first timestamp belonging to a predefined time range.

If the second hash is compared with all of the first hashes stored in the database, a resemblance between the second text and any first text can be determined. The reliability of plagiarism detection can thereby be increased. There are however many cases in which no comparison with all first hashes is required. This is for example the case when a teacher gave a homework on a given date, in which case any first texts generated prior to this date do not need to be considered in the case in which the first texts are Al generated. As another example, if the second hash comprises a second timestamp (obtained along the same lines as the first timestamp but for the second hash), only first hashes with a first timestamp earlier than the second timestamp need to be considered. In both of these cases, the second hash may be compared only with the first hashes stored in the database which are associated with a first timestamp belonging to a predefined time range. Reducing the number of first hashes to perform the comparison with allows increasing a speed of determining a resemblance between a first and a second text, while decreasing a required processing power for said comparison.

According to a further embodiment, the method according to the second aspect or any embodiment thereof further comprises: comparing the second hash with at least some of the first hashes stored in the database to determine how many first hashes stored in the database and used for the comparison are identical with the second hash; and outputting a frequency information indicating how many first hashes stored in the database and used for the comparison are identical with the second hash.

When determining the resemblance between two texts, it is important to distinguish between the case in which a sentence determined as highly resembling another sentence is a "standard" sentence or not. Standard or common sentences are sentences which frequently appear in texts (such as "the weather is nice", "the children play football" or the like), and which are not indicative of plagiarism. By counting how many first hashes are identical with a given second hash, an indicator for how standard a sentence is can be obtained.

According to a further embodiment, the method according to the second aspect or any embodiment thereof further comprises: performing the steps of removing noise from the second sentence and hashing the second simplified sentence on all second sentences of the second text, thereby obtaining multiple second hashes; calculating a measure of text resemblance indicating a proportion of the second hashes which are identical with the first hashes of a single first text; and outputting the determined measure of text resemblance. Second hashes can be obtained for all second sentences of the second text by performing the same process as the one described above. For a single first text, a comparison between all second hashes and all first hashes of said first text is performed. This gives an indication about how close not only one second sentence but the entire second text is to the first text. The measure of text resemblance is a score indicating a proportion (such as a percentage) of the second hashes which are identical with first hashes of the first text. The higher the measure of text resemblance, the more likely it is that the second text is a copy of the first text. The measure of text resemblance may additionally take the first timestamps of the first hashes into account, as there might be no other indicator that several first sentences are part of a same first text.

According to a further embodiment, the step of outputting the identical hash information comprises: displaying the second sentence with a marker or a highlight of a colour that indicates the identical hash information, the frequency information and/or the measure of text resemblance.

The result of the comparison between the second text and one or more first texts can be provided visually to a user, to allow rapid evaluation of the result. For example, a color scheme can be used to mark (highlight) sentences of the second text in accordance with the determined identical hash information, frequency information and/or measure of text resemblance. For example, second sentences for which an identical first hash has been found can be underlined. Additionally, the second sentence can be marked in colours depending on a frequency of occurrence of the identical first hash (the colours can be blue, orange and red, blue indicating a highest frequency range and red indicating a lowest frequency range).

According to a further embodiment, the first metadata further includes an information about an origin of the first text and/or about an author of the first text. The first metadata associated with a first sentence may further include an order of said first sentence within the first text. Optionally, the method comprises outputting the first metadata if it is determined that the second hash and one of the first hashes used for the comparison are identical. Second metadata may optionally be provided with information corresponding to that of the first metadata.

According to a third aspect, a computer program product stored on a machine-readable media and comprising machine readable instructions for executing the method according to the first or second aspect or any embodiment thereof is provided. All features described in view of the first or second aspect or any embodiment thereof also hold for the computer program product of the third aspect.

According to a fourth aspect, a database storing the fingerprints of the first text generated according to the method of the first aspect or any embodiment thereof is provided, wherein the database in particular is an append-only database.

All features described in view of the first aspect or any embodiment thereof also hold for the database of the fourth aspect. Providing the database as an append-only database (such as a blockchain) is advantageous in that data (such as the first hashes) stored therein is immutable, thereby increasing the reliability of the resemblance determination. Further, the append-only database allows bringing an undisputable proof that the second text is having second sentences coming from a first text from a specific issuer. In addition, any attempt to hack the append-only database will be immediately visible.

According to a fifth aspect, a fingerprint generation system for generating a fingerprint of a first text, in particular by executing the method according to the first aspect or an embodiment thereof, is provided. The fingerprint generation system includes: a unit for segmenting the first text into first sentences; a unit for, for each first sentence, removing noise from the first sentence according to a noise removal rule, thereby obtaining a first simplified sentence, the noise removal rule defining a selection of words of the first sentence to be kept and a format into which the selection of words is converted to obtain the first simplified sentence; a unit for hashing each first simplified sentence to obtain a first hash associated therewith; a unit for associating first metadata with each first hash, the first metadata including a first timestamp indicating a time of generation of the corresponding first sentence, or of performing at least part of the computer-implemented method; and a database configured to store the fingerprint of the first text in a database, the fingerprint being jointly formed by each first hash and by the corresponding first metadata.

All features described in view of the first aspect or any embodiment thereof also hold for the fingerprint generation system of the fifth aspect. All units of the fingerprint generation system can be implemented in hardware and/or software.

According to a sixth aspect, a text resemblance determination system for determining a resemblance between a first text and a second text using a fingerprint associated with the first text which is stored in a database, in particular by executing the method according to the second aspect or an embodiment thereof, is provided, the fingerprint including multiple first hashes and the corresponding first metadata. The text resemblance determination system includes: a unit for segmenting the second text into second sentences; a unit for, for at least one second sentence, removing noise from the second sentence according to a noise removal rule applied on the second sentence, thereby obtaining a second simplified sentence, the noise-removal rule applied on the second sentence defining a selection of words of the second sentence to be kept and a format into which the selection of words is converted to obtain the second simplified sentence; a unit for hashing the second simplified sentence to obtain a second hash associated therewith; a unit for comparing the second hash with at least some of the first hashes stored in the database to determine whether the second hash and one of the first hashes used for the comparison are identical; and a unit for outputting an identical hash information indication a resemblance between the first text and the second text if it is determined that the second hash and one of the first hashes used for the comparison are identical.

All features described in view of the first and second aspect or any embodiment thereof also hold for the text resemblance determination system of the sixth aspect. All units of the text resemblance determination system can be implemented in hardware and/or software.

The present invention will be described more fully hereinafter with reference to the accompanying figures in which like numerals represent like element throughout the different figures, and in which prominent aspects and features of the invention are illustrated.

BRIEF DESCRIPTION OF THE FIGURES

Fig. 1 shows a method for generating a fingerprint of a first text;

Fig. 2 shows a method for determining a resemblance between a first text and a second text; and

Fig. 3 shows an example of a computer displaying a second text and its resemblance to a first text.

DETAILED DESCRIPTION

In the following example, the first and the second texts as well as any processed version of these texts are in English for the sake of the example. It is understood that for the described example to make sense and for the corresponding hashes to make sense, the example needs to remain in English, even in case that the remaining description of the present patent application is translated into another language. The described example is for illustrative purposes only and the described processes can in theory also be applied on different languages, alphabets and/or writing systems with the same effects of generating a fingerprint of a first text and/or of determining a resemblance between a first and a second sentence.

Fig. 1 shows a method of generating a fingerprint of a first text according to an embodiment of the invention. The method of Fig. 1 includes method steps S1 to S5, which are performed in this order by a computer device. In the described example, the first text (original text) is:

"Artificial intelligence (Al) refers to the simulation of human intelligence in machines that are programmed to perform tasks that typically require human cognition, such as perception, reasoning, learning, and problem-solving.

Al algorithms can be trained on large datasets and can recognize patterns, make predictions, and optimize outcomes.

Al has the potential to revolutionize industries and create new opportunities, but it also raises ethical and social issues related to privacy, bias, and job displacement."

This first text has been generated by an Al (ChatGPT) on February 17, 2023 at 14h 58min 05s.

In step S1 of Fig. 1 , the above first text is segmented into first sentences. Here, three first sentences are obtained from this segmentation. The first of these first sentences is: "Artificial intelligence (Al) refers to the simulation of human intelligence in machines that are programmed to perform tasks that typically require human cognition, such as perception, reasoning, learning, and problem-solving.". The second of these first sentences is: "Al algorithms can be trained on large datasets and can recognize patterns, make predictions, and optimize outcomes.". The third of these first sentences is: "Al has the potential to revolutionize industries and create new opportunities, but it also raises ethical and social issues related to privacy, bias, and job displacement.".

After step S1 , each sentence is processed individually. Namely, in a step S2 of Fig. 1 , for each sentence, noise is removed from the first sentence according to a noise removal rule. In detail, the noise removal rule includes rules which process the first sentences in three individual steps (i) to (iii). In a step (i), the noise removal rule processes the first sentence such as to: put all characters into lower-case, replace all verbs by their infinitive form, replace all common nouns and adjectives by their masculine singular form, replace synonyms by a selected word, ignore acronyms and ignore punctuation and parentheses. In a step (ii), the noise removal rule orders the words obtained in step (i) alphabetically. In a step (iii), the noise removal rule suppresses any repetitions of a same word. The result of step (iii) forms a first simplified sentence.

Applying the noise removal rule on the above-defined first sentences gives the following results. For the first one of the first sentences (i.e. "Artificial intelligence (Al) refers to the simulation of human intelligence in machines that are programmed to perform tasks that typically require human cognition, such as perception, reasoning, learning, and problem-solving."), the three steps (i) to (iii) of the noise removal rule processing lead to the following:

Step (i) results in: "artificial intelligence refer simulation human intelligence machine program perform task require human cognition perception reason learn problem solve".

Step (ii) results in "artificial cognition human human intelligence intelligence learn machine perception perform problem program reason refer require simulation solve task". Step (iii) results in "artificial cognition human intelligence learn machine perception perform problem program reason refer require simulation solve task", which forms the first of the first simplified sentences.

For the second one of the first sentences (i.e. " Al algorithms can be trained on large datasets and can recognize patterns, make predictions, and optimize outcomes."), the three steps (i) to (iii) of the noise removal rule processing lead to the following:

Step (i) results in: "algorithm train large dataset recognize pattern make prediction optimize outcome".

Step (ii) results in "algorithm dataset large make optimize outcome pattern prediction recognize train". Step (iii) results in "algorithm dataset large make optimize outcome pattern prediction recognize train", which forms the second of the first simplified sentences.

For the third of the first sentences (i.e. " Al has the potential to revolutionize industries and create new opportunities, but it also raises ethical and social issues related to privacy, bias, and job displacement."), the three steps (i) to (iii) of the noise removal rule processing lead to the following:

Step (i) results in: "have potential revolution industry create new opportunity raise ethic social issue relate privacy bias job displacement".

Step (ii) results in "bias create displacement ethic have industry issue job new opportunity potential privacy raise relate revolution social".

Step (iii) results in "bias create displacement ethic have industry issue job new opportunity potential privacy raise relate revolution social", which forms the second of the first simplified sentences.

In a subsequent step S3, each first simplified sentence obtained in step S2 is processed individually to calculate a hash thereof. In detail, in step S3, a hexadecimal SHA-256 hash of each first simplified sentence is determined.

For the first of the first simplified sentences, the obtained first hash is the following: "c15cd8126a84454c09c08d488ac8f41 e1 b11 aca605172b800e6c846361 e2b5a2".

For the second of the first simplified sentences, the obtained first hash is the following: "22105b44b50ab486a073f665fbf60089a2a86c75e16c475fd0fe67d24bbe1cdf". For the third of the first simplified sentences, the obtained first hash is the following: " 6fb20cb3ee3e97788c3aca859521 c8519ae390c72b2730e6eadc0f62ded24f11 ".

In a step S4 of Fig. 1 , a first timestamp, which forms first metadata, is added in plaintext at the end of each first hash. The first timestamp indicates a time at which the corresponding first sentence has been generated. Here, since all three first sentences are part of the same first text, their first timestamp is identical. The first timestamp here is: "20230217-145805", which indicates a generation of the first text on February 17, 2023 ("20230217") at 14:58:05 (14 hours, 58 minutes, 05 seconds, i.e. "145805").

As a result of step S4, the following first hashes including first timestamps are obtained: For the first of the first sentences:

"c15cd8126a84454c09c08d488ac8f41 e1 b11 aca605172b800e6c846361 e2b5a2-20230217- 145805".

For the second of the first sentences:

"22105b44b50ab486a073f665fbf60089a2a86c75e16c475fd0fe67d24bbe1 cdf-20230217-145805".

For the third of the first sentences:

"6fb20cb3ee3e97788c3aca859521 c8519ae390c72b2730e6eadc0f62ded24f11 -20230217-145805".

In a step S5, each first hash with its first timestamp (i.e. the result of step S4) is stored in a database. The database is an immutable blockchain in the present example (but in theory, any other database can be used). The database stores the first hashes and the first timestamps of all first sentences generated by the Al. The database is managed by the issuer of the first text. In the database of the issuer, the original first text shown in the example above will be “encoded” as the three following lines (hashes + timestamp), the "..." indicating previous and subsequent entries for previous and subsequent first texts.

II c15cd8126a84454c09c08d488ac8f41 e1 b11 aca605172b800e6c846361 e2b5a2-20230217-145805 22105b44b50ab486a073f665fbf60089a2a86c75e16c475fd0fe67d24bbe1 cdf-20230217-145805 6fb20cb3ee3e97788c3aca859521 c8519ae390c72b2730e6eadc0f62ded24f11 -20230217-145805

II

Metadata could additionally be added to these three lines (e.g. with the name or ID of the issuer). This database allows any verifier to find back that the given text is coming from a given issuer, and possibly to have access to the timestamp and any potential associated metadata. The steps S2 to S5 described above are performed for each first sentence of the first text individually. For example, step S2 is performed for all first sentences, then step S3 is performed for all first sentences, etc. Alternatively, S2 to S5 are performed in this order for the first of the first sentences and subsequently, S2 to S5 are performed in this order for the second of the first sentences, and so on for each first sentence, one after another.

Accordingly, all texts generated by the Al are tracked through the database, without however storing their content. This allows verifying whether a further text is copied from one of the Al-generated texts or whether it is an original text. The above-described noise removal rules allow to account for small modifications in the copied sentences by a fraudster. Namely, even when a fraudster performs minor modifications to the copied text, the resulting simplified sentence and the resulting hash will still be identical.

The process for verifying how similar two text are and thereby obtaining an indication regarding whether one of the texts is a copy of the other is described based on Fig. 2. In detail, Fig. 2 shows a method for determining a resemblance between a first text and a second text. A fingerprint of the first text is preferably obtained and stored as described in the method of Fig. 1 . The first text is a "legitimate" text, which has indeed been issued by the pretending issuer, namely the Al. The second text is a text whose origin needs to be checked. In particular, the second text may be an original work from an author (such as a journalist or student), or it may be copied from one or multiple of the first texts issued by the Al. The required verification can be done using the method of Fig. 2.

In the present example, the second text reads "Artificial Intelligence corresponds to the simulation of human intelligence in machines which are programmed to perform tasks that typically require human cognition, such as perception, reasoning, learning, and problem-solving. Al is sometimes opposed to intelligence displayed by humans or other animals." One notes that the first sentence of the second text has been copied from the first text, but with some small modifications. Namely, "refers to" has been replaced by "corresponds to", "that are programmed" has been replaced by "which are programmed", and the order of the items in the last list of the sentence has been modified. The second sentence of the second text has not been copied from the first text.

In detail, according to the method of Fig. 2, the second text (to be verified) is processed along the same lines as the first text. Namely, in a step S6, the second text is decomposed (segmented) into second sentences, thereby obtaining the following two second sentences. The first of the second sentences is "Artificial intelligence corresponds to the simulation of human intelligence in machines which are programmed to perform tasks that typically require human cognition, such as learning, reasoning, perception, and problem-solving.", and the second of the second sentences is "Al is sometimes opposed to intelligence displayed by humans or other animals.". In a subsequent step S7 of Fig. 2, the same noise removal rule as in step S2 of Fig. 1 is applied to the second sentences obtained in step S6. Accordingly, the following second simplified sentences are obtained.

The first of the second simplified sentences reads: "artificial cognition human intelligence learn machine perception perform problem program reason refer require simulation solve task". We note that this second simplified sentence is identical to the first simplified sentence, although the original first of the second sentences is not entirely identical with the first of the first sentences described in view of Fig. 1. However, the noise removal rule, in particular step (i) thereof, recognizes the word "corresponds" as corresponding to the word "refers" and thus replaced therewith. Further, the noise removal rule removes words such as "that" and "which", which hence cannot have any distinguishing functionality. Since the noise removal rule sorts all kept terms alphabetically, the modification of the order of the words in the copied sentence does not affect the generated simplified sentence.

The second of the second simplified sentences reads: "animal display human intelligence oppose", which is different from any of the first simplified sentences.

In a step S8, a second hash (also a SHA-256) of each of the second simplified sentences is calculated. For the first of the second simplified sentences, the calculated second hash is "c15cd8126a84454c09c08d488ac8f41 e1 b11 aca605172b800e6c846361 e2b5a2", which is identical with one of the calculated first hashes described above. For the second of the second simplified sentences, the calculated second hash is

"4c35f27574d69e9379f1558e07bc2f22b4fe702ebdb0f67841969b54282dca5d", which is different from any of the calculated first hashes described above.

In a subsequent step S9, the second hashes are individually considered and compared with all the stored first hashes. As a result of step S9, it is obtained that the calculated second hash of the first of the second sentences is identical with one of the first hashes. Accordingly, an identical hash information is output to a human verifier in step S10. This corresponds to displaying the second text with a red highlight on the first of the second sentences, indicating that an identical hash has been found. This can be an indication of a fraud and the human verifier can accordingly take actions, if necessary. Next to the highlight, metadata (such as the first timestamp) of the identical first hash is displayed. As a further result of step S9, it is obtained that the calculated second hash of the second of the second sentences is identical with none of the first hashes in the database. Accordingly, no identical hash information is output for said sentence. It is also possible to instead highlight the second of the second sentences in green, for example, indicating that it has not been copied.

The steps S7 to S9 described above can be performed for each second sentence individually. For example, step S7 is performed for all second sentences, then step S8 is performed for all second sentences, etc. Alternatively, steps S7 to S9 are performed in this order for the first of the second sentences and subsequently, S7 to S9 are performed in this order for the second of the second sentences, and so on for each second sentence, one after another.

The resulting display is illustrated in Fig. 3. Fig. 3 shows the computer 1 on which the abovedescribed methods of Fig. 1 and 2 are executed. A display 2 of the computer 1 displays the second text 10, including two second sentences 3, 4. The first of the two sentences 3 is highlighted by a red highlight 5 indicating that an identical first hash was found in the database. The second of the two sentences 6 is highlighted by a green highlight which indicates that no identical first hash was found in the database. When the user moves his computer mouse over the second sentence 3, a po-up window 8 appears on the display 2, which shows all the metadata available for the first hash identified as identical, said metadata at least including a first timestamp 9 of the first hash.

The above disclosed subject-matter is to be considered illustrative, and not restrictive, and serves to provide a better understanding of the invention defined by the independent claims. For example, only a specific sentence of the second text may be considered for verification purposes. Further, the noise removal rules can be adapted as required for the purpose of the verification, in particular such as to allow more or less modifications in the text to still lead to a same hash. It is for example possible to encode each word of the first/second sentence by a code (for example by a number), the same code being used for synonymous word, words with similar meaning and/or translations. It is also possible to additionally calculate and display a frequency at which a first hash which is identical to a second hash is found.

REFERENCE NUMERALS

1 computer

2 display

3 first of the second sentences

4 second of the second sentences

5 red highlight

6 green highlight

7 mouse

8 pop-up window

9 timestamp

10 second text

Claims

1. A computer-implemented method for generating a fingerprint of a first text, the method comprising: segmenting (S1) the first text into first sentences; for each first sentence, removing (S2) noise from the first sentence according to a noise removal rule, thereby obtaining a first simplified sentence, the noise removal rule applied to the first sentence defining a selection of words of the first sentence to be kept and a format into which the selection of words is converted to obtain the first simplified sentence; hashing (S3) each first simplified sentence to obtain a first hash associated therewith; associating (S4) first metadata with each first hash, the first metadata including a first timestamp indicating a time of generation of the corresponding first sentence, or of performing at least part of the computer-implemented method; and storing (S5) the fingerprint of the first text in a database, the fingerprint being jointly formed by each first hash and by the corresponding first metadata.

2. A computer-implemented method for determining a resemblance between a first text and a second text using a fingerprint associated with the first text which is stored in a database, the fingerprint including multiple first hashes, the method comprising: segmenting (S6) the second text into second sentences; for at least one second sentence, removing (S7) noise from the second sentence according to a noise removal rule applied on the second sentence, thereby obtaining a second simplified sentence, the noise-removal rule applied on the second sentence defining a selection of words of the second sentence to be kept and a format into which the selection of words is converted to obtain the second simplified sentence; hashing (S8) the second simplified sentence to obtain a second hash associated therewith; comparing (S9) the second hash with at least some of the first hashes stored in the database to determine whether the second hash and one of the first hashes used for the comparison are identical; and outputting (S10) an identical hash information indicating a resemblance between the first text and the second text if it is determined that the second hash and one of the first hashes used for the comparison are identical.

3. The method according to claim 2, wherein the fingerprint of the first text is generated according to the method of claim 1 .

4. The method according to any one of claims 1 to 3, wherein the noise removal rule replaces each word of the selection of words to be kept by a code corresponding to said word so that the first and/or second simplified sentence is formed by a juxtaposition of codes, each code being defined through a prestored table associating the code with one or more corresponding predefined words, wherein the one or more corresponding predefined words have similar meanings, are synonyms and/or are translations of a same word in different languages.

5. The method according to any one of claims 1 to 4, wherein the noise removal rule defines: as the selection of words of the first or second sentence to be kept: that pronouns and adverbs are ignored; that proper nouns are ignored; that stop-words are ignored; and/or that words that do not belong to a reference dictionary or the prestored table are ignored; and/or as the format into which the selection of words is converted to obtain the first or second simplified sentence, that: all characters are put in lower-case or in upper-case; all verbs are replaced by a corresponding infinitive or a predefined verb form; all common nouns and adjectives are put in the masculine singular or into another predefined form; all words are replaced by their stem using a stemming algorithm; all words are replaced by their lemma using a lemmatization algorithm; all special characters are replaced by an equivalent simpler character; all spelling mistakes are corrected; all punctuation, multiple spaces, tabs, bullet points, and/or carriage returns are ignored; all numbers are ignored; a single space separates each word; the words are placed in alphabetical order or in another predefined order; words which allow multiple spellings are replaced by a selected word; and/or repetitions of identical words in a same sentence are ignored.

6. The method according to any one of claims 1 to 5, wherein the hashing (S3, S8) of the first sentence and/or of the second sentence uses a cryptographic hash algorithm, in particular a SHA-1 , a SHA-2, a SHA-256, on the first sentence and/or second sentence.

7. The method according to any one of claims 1 to 6, wherein the first and/or second texts are texts generated by an artificial intelligence, and in particular wherein each text generated by the artificial intelligence is processed as the first text.

8. The method according to any one of claims 2 to 7, wherein the step of comparing (S9) the second hash with at least some of the first hashes stored in the database includes: comparing the second hash with all of the first hashes stored in the database; or comparing the second hash with only the first hashes stored in the database which are associated with a first timestamp belonging to a predefined time range.

9. The method according to any one of claims 2 to 8, further comprising: comparing the second hash with at least some of the first hashes stored in the database to determine how many first hashes stored in the database and used for the comparison are identical with the second hash; and outputting a frequency information indicating how many first hashes stored in the database and used for the comparison are identical with the second hash.

10. The method according to any one of claims 2 to 9, further comprising: performing the steps of removing noise from the second sentence and hashing the second simplified sentence on all second sentences of the second text, thereby obtaining multiple second hashes; calculating a measure of text resemblance indicating a proportion of the second hashes which are identical with the first hashes of a single first text; and outputting the determined measure of text resemblance.

11. The method according to any one of claims 2 to 10, wherein the step of outputting the identical hash information comprises: displaying the second sentence with a marker or a highlight of a colour that indicates the identical hash information, the frequency information and/or the measure of text resemblance.

12. A computer program product stored on a machine-readable media and comprising machine readable instructions for executing the method according to any one of claim 1 to 11 .

13. A database storing the fingerprints of the first text generated according to the method of claim 1 , wherein the database in particular is an append-only database.

14. A fingerprint generation system for generating a fingerprint of a first text, in particular by executing the method according to claim 1 , the fingerprint generation system including: a unit for segmenting the first text into first sentences; a unit for, for each first sentence, removing noise from the first sentence according to a noise removal rule, thereby obtaining a first simplified sentence, the noise removal rule defining a selection of words of the first sentence to be kept and a format into which the selection of words is converted to obtain the first simplified sentence; a unit for hashing each first simplified sentence to obtain a first hash associated therewith; a unit for associating first metadata with each first hash, the first metadata including a first timestamp indicating a time of generation of the corresponding first sentence or of performing at least part of the computer-implemented method; and a database configured to store the fingerprint of the first text in a database, the fingerprint being jointly formed by each first hash and by the corresponding first metadata.

15. A text resemblance determination system for determining a resemblance between a first text and a second text using a fingerprint associated with the first text which is stored in a database, in particular by executing the method according to any one of claims 2 to 11 , the fingerprint including multiple first hashes and the corresponding first metadata, the text resemblance determination system including: a unit for segmenting the second text into second sentences; a unit for, for at least one second sentence, removing noise from the second sentence according to a noise removal rule applied on the second sentence, thereby obtaining a second simplified sentence, the noise-removal rule applied on the second sentence defining a selection of words of the second sentence to be kept and a format into which the selection of words is converted to obtain the second simplified sentence; a unit for hashing the second simplified sentence to obtain a second hash associated therewith; a unit for comparing the second hash with at least some of the first hashes stored in the database to determine whether the second hash and one of the first hashes used for the comparison are identical; and a unit for outputting an identical hash information indication a resemblance between the first text and the second text if it is determined that the second hash and one of the first hashes used for the comparison are identical.