NL2010445C2 - Method, software and user device for authenticating an item. - Google Patents

Description

Title: Method, software and user device for authenticating an item

The invention relates to a method of authenticating an item, a software program and a user device comprising such software program.

Counterfeiting of items is observed in many different fields. For example, imitations of products that are offered for sale to the public, are sometimes not or almost not distinguishable for a potential client from a genuine product. This problem may occur in the field of pharmaceuticals, food products, spare parts, as well as documents, and many other items. As a result, a potential client that intends to acquire an item and expecting certain characteristics in terms of quality, safety, reliability, origin, etc., may be get into an undesirable situation when e.g. acquiring an imitation of the item. Also, grey import of items, i.e. the offering for sale of an item which originates from the claimed manufacturer and/or brand owner, but which was intended for sale in another market, may result in an offering for sale of items that may not comply with local requirements, such as legal requirements, quality, product documentation in an appropriate language, warranty, after sales service, etc. Generally, US2012/0187185 discloses a system and method for detecting counterfeiting. A readable indicia, such as a Quick Response (QR) code is scanned by e.g. a smartphone to obtain data stored in the readable indicia, and the data is used to detect counterfeiting, and to track and authenticate documents.

A problem associated with the system and method as disclosed in US2012/0187185 is that any misreading of the code, e.g. a scanning of a wrong product, a scanning of a wrong part of a product, a misreading due to reflection, stray light, or other optical effects, a user scanning another object instead (e.g. a photographic image of the product) etc. may in use result in a large amount of messages of a code not being recognized, i.e. in a large amount of warnings of item not being authenticated.

The invention intends to provide an improved detection of an optically readable code. In order to achieve this goal, according to an aspect of the invention there is provided a method of authenticating an item, comprising: - capturing a first image of the item by a user device; - starting a process for detecting a code pattern in the first image; - comparing the code pattern, if detected in the first image, with a list of at least one authenticated code pattern; - generating an authentication status message in case the code pattern if detected is not included in the list of authenticated code patterns; - in case no code pattern has been detected: performing an image recognition on the first image and establishing whether at least part of the item can be recognized by image recognition, and - deciding about a further code pattern detection and/or further image recognition based on an outcome of the image recognition.

An image of the item is captured by a user device, such as a mobile telephone comprising a camera, a smartphone comprising a camera, a scanner, etc. The image may be formed by an image file, such as a picture file, an infrared picture file, etc. and may be stored in a cache or other memory of the user device. When the image has been captured, a code pattern is searched for in the image. In case a code pattern is detected, the code pattern may be compared with a list of authenticated code patterns (the list may be formed either way, namely by an “inclusive” list of authenticated code patterns or by an “exclusive” list of non-authenticated code patterns or in any other suitable form). The list may store the code patterns for example in the form of a binary number, alphanumeric code, ASCI code, or any other code. The comparison of the code pattern with the list may be performed by deriving a code from the code pattern and comparing such code with the list. If the code is found to be correct, a notification is provided to a user of the user device, and if not, an authentication status message may be generated and e.g. transmitted to a logging database for storage thereof. In case however no code pattern is detected, several scenario’s may be distinguished. First, it is established, by applying image (pattern) recognition to the image, if at least part of the item can be recognized. Thereby, it can be verified if what is scanned is indeed an item as intended, or something else. Any suitable image recognition technique may be applied, for example searching for a contour, an edge, a shape, a logo, a label, etc. Also, it can be established if, in case indeed an item as intended is scanned, if an incorrect part of the item is scanned. Based on the outcome of the image recognition, it is decided whether or not to perform a further code pattern recognition and/or a further code pattern detection. Thereby, by means of the image recognition, a distinction can be made between an item as intended and something else, so that - in case it appears that something else was scanned (i.e. in case the item was not recognized) - the process may be stopped thus avoiding occupying data network communication resources, performing logging based on incorrect image capturing (e.g. capturing of an image of something that does not seem to be an item to be authenticated), etc. The generating and/or logging an authentication status message may be omitted, thus reducing false alarms. The first image may be a single image, a plurality of images or a sequence of image data forming video data thus enabling to perform the method even when the item moves in respect of the user device or vice versa. The list of authenticated code patterns may comprise a list of codes, one or more ranges of codes, etc.

In an embodiment, the method further comprises refraining from generating and logging the authentication status message in case no part of the item has been recognized by the image recognition, so as to avoid or reduce a filling of a logging database with data from images of apparently something else than the item to be authenticated. Similarly, the authentication of the item may be aborted in case no part of the item has been recognized by the image recognition.

In an embodiment, the method further comprises, if at least part of the item is recognized by the image recognition, - providing instructions to a user for capturing a second image of the item; - starting a process for detecting a code pattern in the second image, and - comparing the code pattern, if detected in the second image, with the list of at least one authenticated code pattern.

In case the image (pattern) recognition establishes that the image is indeed an image of an item, however the code pattern could not be recognized, instructions may be provided to a user for capturing a second image of the item. Based on the results of the pattern recognition, the user may be instructed to for example scan another part of the item, scan from another angle, scan from another distance, etc. Again, using the second pattern, it can be established if the code pattern can be recognized, and if recognized, it can be determined if the detected code pattern is comprised in the list of authenticated code patterns. If not, the authentication status message may be generated similarly to above. Thus, a probability of detecting the code pattern may be increased by capturing the second image, as the code pattern may be better readable from the second image, or at least providing a second chance to detect the code pattern. The instructions to the user on scanning may comprise a guide to a correct scanning distance, an alert if the device is moved during the scanning process based on position measurement as performed by the user device (e.g. using GPS-technology of smartphone), usage of a built-in spotlight of the smartphone or other user device amongst others may be given to minimalize the influence of the user in the scanning process.

The authentication status message may also be generated in case at least part of the item has been recognized in the image recognition while no code pattern has been detected in the second image, so that, in case the pattern recognition as performed establishes that the image is taken from an item as intended, while no code pattern can be distinguished in the first and second images, a situation of a high likelihood of scanning a counterfeit item may have been encountered, so that a reliability of detecting a counterfeit item may be increased.

In an embodiment, the method further comprises, in case the image recognition detected the (first or second) image to have low quality, providing instructions to a user for capturing a second image of the item; starting a process for detecting a code pattern in the second image, and comparing the code pattern, if detected in the second image, with the list of at least one authenticated code pattern.

Thus, in case no code pattern was detected while the image recognition detected that the image is low quality (e.g. un-sharp, to dark, over-illuminated, etc.) corresponding instructions for obtaining an improved image may be provided to the user, and the detecting of the pattern may be performed using such improved image. Alternatively, in the second image, instead of or in addition to code pattern detection, image recognition may be performed, for example to verify if at least part of the item can be recognized in the second image, thus being able to establish again if the image is taken from an item to be authenticated or something else..

In an embodiment, the authentication status message comprises at least one of the first and the second image. Thereby, when authentication status messages are inspected (for example after having been collected on a server), the authentication status messages may be further assessed, for example by an operator who further distinguishes images of intended items from other images, thus enabling to further reduce false alarms.

In an embodiment, the authentication status message comprises the code pattern if detected, so that for example duplicated code patterns or code patterns that have been intended for sale in another market, may be derived also.

In an embodiment, the method comprises transmitting by the user device the authentication status message to a logging database, thus enabling to collect results from a plurality of user devices in a central database. As only the authentication status message is logged, false results such as a user capturing an image of something else other than an item, improper image capturing, etc. may be refrained from being stored in the database, as the method may omit the transmission by the user device of the authentication status message to a logging database in case of a negative result of the image recognition,

In an embodiment, the list of authenticated code patterns is stored in the user device, preferably in encrypted form, enabling to have steps of pattern recognition, and verification of the code in the list, done by the user device, so as to avoid delays in response to the user by having to send data via a network (e.g. the internet or mobile telephone network) to a server and back for performing such operations.

In an embodiment, the method further comprises: capturing, by the user device, location information on the location of the optical detection device, and wherein the authentication status message comprises the location information. A more detailed analysis may be made when location information is stored. For example, a scan in a public environment (such as a shop) may be taken into account as a potential counterfeit offer for sale, while a scan in a private environment (such as a user at home) may be taken into account using other criteria: for example a warning for grey import may be omitted as the user may have legitimately purchased the item elsewhere and brought it to his home.

In order to distinguish between public and private locations, the method may further comprise: deriving, from the captured location information and a location database containing information about a location being public or private, if the captured location information relates to a public location or a private location, and displaying by the optical detection device a warning message in case the captured location information relates to a public location.

In an embodiment, the at least part of the item to be recognized comprises at least one of a shape, a label and a logo of the item. Thus, an effective recognition may be provided by searching, using pattern recognition, for an easily recognizable characteristic of the item, such as its shape, its label or logo. Also, having recognized the item or at least part thereof, an expected location of the code pattern on the item may be determined from a model of the item, and an instruction may be provided to the user so as to take the second image to include an image of the code pattern.

The code pattern as provided on the item may be a unique code pattern of each individual item. Such unique code may be generated in a controlled environment and may be distributed in all kind of printed matters of the product, the product range, the printing company, the agency, brand owner or any other stakeholder. The comparison of the code in the code pattern with a list of authenticated code patterns may be performed in a controlled environment, such as a central server. Codes may be generated based on unique series reserved for particular brand owners or manufacturers and the positive reading of these codes may be restricted to their (e.g. branded) apps or other user device software. These apps or other user device software may be distributed through controlled channels as the Apple App Store, the Google Play store or any other controlled distribution platform.

The distribution of the code pattern as added to all kinds of printed materials may be done via the printing workflow channel e.g. the agency - the lithographer - the printing company amongst others that may be involved in a controlled environment. The physical insertion of the code may be performed either in the controlled area of an issuer of the codes, or in a controlled area of an origin of the item, such as a brand owner or manufacturer, thus linking the code specific to a specified package artwork (e.g. a brand logo). Afterwards the artwork with the included code may be sent via a controlled channel and/or encrypted to the agency - the lithographer - the printing company. No admission may be available to the coding software itself by the agency - the lithographer - the printing company.

According to another aspect of the invention, there is provided a software program comprising program instructions for, when loaded into a memory of a data processing device, make the data processing device to perform the method according to the invention. The software program may be comprised in (e.g. loaded in) a user device, such as a mobile telephone, a smartphone, a notebook computer or a tablet computer.

Generally, a preferably unique, preferably invisible code is applied to the item, e.g. by printing technology, the item being e.g. a package or any other printing matter.

In printing technology, it is known to add a machine readable code to a pattern printed on a substrate. The machine readable code may for example be readable by a camera, an optical scanner, a smartphone, a tablet device or other optical detection device, etc. Generally, an aim may be to provide that the code, as patterned into the colour pattern of the printed matter, is to some extent concealed to the human eye, i.e. may not be readily recognized by a human observer.

In printing technology, generally use is made of patterns of different colours, which are each printed onto a substrate. The different patterns together form a resulting pattern. The patterns may for example be printed in the colours cyan, magenta and yellow. Combinations (i.e. mixes) of these colours provides for other colours as may be required:

For example, a combination of yellow and magenta results in red, a combination of magenta and cyan results in blue, and a combination of cyan and yellow results in green. Generally, colours in a colour spectrum visible to the human eye may be generated by a suitable mix of cyan, yellow and magenta. It is to be understood that other colours may be applied also. In addition to the colours, further contrast may be added by the addition of a black pattern. The colours cyan, magenta, yellow and black are generally abbreviated as CMYK, whereby K represents the black pattern (contrast or kontrast). It is noted that in printing technology, generally colours are used that are opposite to the colours used in electronic display and television technology, namely Red, Green and Blue (RGB). This is because is electronic display and television technology, displayed colours add intensity: displaying full intensity red green and blue simultaneously at a television set or electronic display results in white, while in printing, the printing of full intensity generally results in black. Printing therefore generally tends to use a subtractive colour scheme such as CMY, instead of an additive colour scheme, such as RGB.

When making use of a printing colour scheme, such as CMYK, while an optical detection device that is used to read the code in the printed pattern uses an RGB like colour scheme (thus being able to detect e.g. red green and blue), the code pattern in yellow is detected by red and green detection of the detection device, the code pattern in magenta is detected by red and blue detection of the detection device , and the code pattern in cyan detected by blue and green detection of the detection device.

Instead of or in addition to the use of a colour scheme of a “standard” colour palette, in some applications specific colours (also referred to as spot colours) are used, for example in printing of packages of consumer products. These colours provide brighter colours then the mixed colours of CMYK. In Europe, spot colours are mostly defined as Pantone-colours. Pantone is a trademark of a company that provides list of colours and the recipes for commercial printers how to mix these colours out of 12 inks. Other provides of so called colour books are HKS, Toyo, AN PA and Focoltone. A difference to CMYK is that spot colours are mixed as an ink to achieve a certain colour, not in print. Also the amount of printed inks can be up to 12 inks when creating a package.

Generally, when applying a hidden code pattern in the colour patterns to be printed, the code pattern may only be applied with a low strength, i.e. with a low modulation intensity for several reasons, such as ensuring that the code remains invisible to the human eye, preventing saturation effects (saturation is to be understood as a merging of neighboring printed dots, which merging may occur at higher intensity). As a result, the visual readability of the code is low, the machine-readability of the code in the printed matter still remains.

The colour patterns may be cyan, magenta and yellow respectively. Any other suitable colour pattern may be applied likewise.

The code pattern may comprise any code pattern. Preferably, the code pattern comprises a two dimensional code pattern so as to bring a large amount of information into the (e.g. two dimensionally printed) printed matter. Any type of code pattern may be applied, e.g. the code pattern comprising one of a dot-code, a bar-code and a QR code.

As an example, use may be made of Steganografy: In digital images, small changes in colour strengt hand colour shift will hardly be noticed by a human observer. As a result, systematic (relatively small) changes may be applied in the digital image file, which may be hardly noticeable or not noticeable to a human observer. As another example a pattern of small (for example yellow) dots may be applied in the image file. Yellow is preferred because of a low sensitivity of the human eye for the colour yellow.

In order to make the code pattern even less visible, the code pattern may (in a spatial frequency domain) be a spread spectrum code pattern, thus the information coded in the code being spread over a spatial frequency range, thereby for example making use of coding techniques similar as code division multiple access. A detection key may be applied in order to retrieve the code from the spatial pattern(s). A spread spectrum techniques generally result in low amplitude coding exhibiting a noise like appearance, the code may easily be made invisible to the human observer.

Information carrying fields of the code pattern (e.g. dots) may be positioned next to each other. Alternatively, in order to make the code less distinct tot e human eye, information carrying fields of the code pattern may mutually spaced apart, whereby parts of the original patterns are presented in between the information carrying fields.

The code pattern may be applied once to the coloured patterns, or multiple patterns may be applied at a mutual spatial distance, so as to allow detection of the code pattern even when only a part of the printed matter is optically detected by an observing camera.

Further features, advantages, and effects of the invention will be explained in connection with the enclosed drawing, illustrating a non limiting embodiment of the invention, in combination with the below description, wherein:

Figure 1 depicts a block schematic view of a system in which the method according to the invention may be practiced, and

Figure 2 depicts a flow diagram of a method in accordance with an aspect of the invention.

Figure 1 depicts a schematic view of a system in which the method according to the invention may be employed, and depicts a user device UD, such as in this example a smartphone. The user device may be formed by any image acquisition device, such as a photo camera, a notebook computer, a palmtop computer, a tablet computer, an optical scanner, etc. The user device UD is connected to a server SE via a data communication network, such in the internet, and may be performed making use of e.g. a wireless network connection, a wired connection, etc. The server may be provided with a list of authenticated codes LST, a database DB for storage of authentication status messages. The user device may make use of specific software, such as an app, a browser application, etc. in order to performed part of the steps of the method as described below. The user device UD is provided with an image capturing device such as a camera, a scanner, etc. for capturing an image of the item. The user device may further be provided with a position sensor that measures a geographic location of the user device, such as a satellite navigation (e.g. GPS) sensor, a mobile telecommunication network related sensor (which determines a position of the mobile device from signals of base stations as received by the mobile device), etc.

In accordance with Figure 2, an example of the method of authenticating an item, in accordance with an aspect of the invention starts with a user capturing an image of the item by a user device, such as a smartphone (step 200). Then the smartphone, e.g. an app or other software application in the smartphone attempts to detect a code pattern in the image (step 201). If a code pattern is detected, as checked in step 202, the app compares in step 203 the code pattern, if detected in the image, with a list of at least one authenticated code pattern (i.e. a list of code patterns that are considered genuine). The app may thereto download the list from the server SE, the list may be comprised in the app and may be encrypted or may send the code pattern to the server in order to perform the comparison by the server, the server may then send the result back to the user device. If the code pattern cannot be authenticated, in step 204, an authentication status message is generated that indicates that the code pattern as detected on the item cannot be authenticated. In case no code pattern has been detected in step 202, the app uses pattern (i.e. image) recognition in step 205 to establish if in the image at least part of the item can be recognized or not. In step 206, it is checked if at least part of the item is recognized in the image, and based thereon, it is decided whether or not to proceed. If the image was not recognized, the process may stop at 207, Thus, in case no code pattern is detected and no item is recognized, the generation (and logging) of an authentication status message is omitted, thus reducing a likelihood of logging of false or irrelevant authentication status messages.

In case at least part of the item is recognized in the image, the app provides instructions in step 208 to a user for capturing a second image of the item. The user may be instructed to take the second image from another distance, another perspective, using other illumination conditions, being directed at an other part of the item, etc., so as to increase a probability to detect the code pattern. In step 209, the app attempts to detect a code pattern in the second image, and in step 210, the code pattern, if detected in the second image, is compared with the list of at least one authenticated code pattern. If the code pattern has not been detected, in step 211, the authentication status message is generated (and e.g. transmitted to be logged in the logging database DB) in step 212 in case no code pattern has been detected in the second image. Thus, an authentication status message is generated that may signal that the item has been detected by pattern recognition, however that no code pattern has been detected, hence this combination of facts resulting in a high likelihood of the item being counterfeit. In case a code pattern has been detected, similarly to what has been described with reference to steps 202-204, if a code pattern is detected, as checked in step 211, the app compares in step 213 the code pattern, if detected in the image, with a list of at least one authenticated code pattern. If the code pattern cannot be authenticated, in step 214, an authentication status message is generated that indicates that the code pattern as detected on the item cannot be authenticated. The status messages may be transmitted to the logging database DB for logging.

The method as described may for example allow consumers instead of professionals to check on counterfeiting by scanning items, such as packages and other printed matters with a controlled code pattern, such as a watermark, and will be further explained below. Items, such as packages and other printed matters will be scanned by consumers in their demand for product information, sales promotions or whatever reason and implicit or explicit check if the package is genuine or grey import. The advantage for the brand owner or manufacturer is the large scale of real-time checking enriched with geographical data with low costs.

Some scenarios that may occur when making use of the method are described below.

If a printed matter e.g. package that is provided with a controlled watermark is scanned by a consumer three options occur: 1 The watermark is recognized by the app and the package is genuine and not grey imported 2 The watermark is recognized by the app and the package is genuine but is grey imported 3 The watermark is not recognized by the app

Ad 1 The code is recognized in the app and the package or other printed matter and is checked by the software in the app or by software that is on a controlled server the app is connecting to. The package or printed matter is recognized as genuine and as localized to the area the consumer is scanning. The user (e.g. a consumer) will receive a positive message that the package or other printed matter is genuine in the app on the smartphone or tablet or any other device that is used for scanning.

Ad 2 The code is recognized in the app and the package or other printed matter and is checked by the software in the app or by software that is on a controlled server the app is connecting to. The package or printed matter is recognized as genuine and but not as localized to the area the consumer is scanning. If the consumer accepts the app to collect location-data, in the app or online at a server is checked if the location is private versus public. If the location is private, the app shows that the product is genuine. If the location is public, the app also shows that the product is genuine but an extra check is done if this location is a retail environment. If so, the brand owner or the manufacturer is addressed to visit this location to investigate grey import. The customer receives the message that the product is genuine.

Ad3 The code is not recognized by the app. This might have three reasons: 3.1 The consumer does not scan an appropriate package or printed matter.

3.2 The consumer does not correctly scan an appropriate package or printed matter.

3.3 The consumer did scan an appropriate product but the package or printed matter itself is not genuine

Ad 3.1 The consumer is asked to rescan the package or printed matter. A photograph is taken while rescanning and sent to the server. The app checks with image recognition if the photograph contains an appropriate package by recognizing the shape, the label and/or the logo. If the checking result is negative, the consumer is warned that not an appropriate package or printed matter is scanned and an instruction is given how to scan again. This procedure is at most done twice, if still no recognition has occurred, the procedure is stopped and the app has to be restarted. A predefined amount of attempts within a defined time space can be built in to prevent unwanted abuse of resources.

Ad 3.2 The consumer is asked to rescan the package or printed matter. A photograph is taken while rescanning and sent to the server. The app checks with image recognition if the photograph contains an appropriate package by recognizing the shape, the label and/or the logo. If the checking result is positive, the consumer is helped with additional instructions how to scan the product. If the watermark in the product is recognized, step 1 or 2 as described above will be done in the app. If the watermark recognition is negative, the consumer receives a warning the product might be counterfeited and that no responsibility in any means can be taken by the supplier, the brand owner or the manufacturer. This procedure is at most done twice, if still no recognition has occurred, the procedure is stopped and the app has to be restarted. A predefined amount of attempts within a defined time space can be built in to prevent unwanted abuse of resources.

Ad 3.3 The consumer is asked to rescan the package or printed matter. A photograph is taken while rescanning and sent to the server. The app checks with image recognition if the photograph contains an appropriate package by recognizing the shape, the label and/or the logo. If the checking result is positive, the consumer is helped with additional instructions how to scan the product. If watermark recognition is then negative, the consumer receives a warning the product might be counterfeited and that no responsibility in any means can be taken by the supplier, the brand owner or the manufacturer. This procedure is at most done twice, if still no recognition has occurred, the procedure is stopped and the app has to be restarted. A predefined amount of attempts within a defined time space can be built in to prevent unwanted abuse of resources.

Both 3.2 and 3.3: The consumer is asked by the app, if location is not admitted to be sent, to fill in the location, and always asked for the retailer and product information. The consumer can be granted for the cooperation by a reward.

Claims (16)

A method for authenticating an item, comprising: - recording a first image of the item with a user device; - starting a process for detecting a code pattern in the first image; - comparing the code pattern, when detected in the first image, with a list of at least one authenticated code pattern; - generating an authentication status message in case the code pattern when detected is not included in the list of authenticated code patterns; - if no code pattern is detected: performing an image recognition on the first image and determining whether at least a portion of the item with image recognition can be recognized, and - deciding on a further code pattern detection or further image recognition based on an outcome of the image recognition. The method of claim 1, further comprising refraining from generating and logging the authentication status message in case no portion of the item is recognized by the image recognition. The method of claim 1 or 2, further comprising canceling the authentication of the item in case no portion of the item is recognized by the image recognition. The method of any one of the preceding claims, further comprising, when at least portion of the item is recognized by the image recognition, providing instructions to a user for recording a second image of the item; starting a process for detecting a code pattern in the second image, and - comparing the code pattern, when detected in the second image, with the list of at least one authenticated code pattern. The method of claim 4, further comprising: - generating the authentication status message in case at least portion of the item is recognized in the image recognition while no code pattern is detected in the second image. A method according to any of the preceding claims, further comprising, in case the image recognition has detected that the image has a low quality, - providing instructions to a user for taking a second image of the item; - starting a process for detecting a code pattern in the second image, and - comparing the code pattern, when detected in the second image, with the list of at least one authenticated code pattern. The method of any one of the preceding claims, wherein the authentication status message comprises at least one from the first and the second image. The method of any one of the preceding claims, wherein the authentication status message comprises the code pattern when detected. A method according to any one of the preceding claims, comprising sending the authentication status message to a logging database by the user device. Method according to any of the preceding claims, wherein the list of authenticated code patterns is stored in the user device, preferably in encrypted form. Method according to any of the preceding claims, further comprising: - recording, by the user device, location information at the location of the optical detection device, and wherein the authentication status message comprises the location information. The method of claim 11, further comprising: - extracting information from the recorded location information and a location database including whether a location is public or private, whether the recorded location information relates to a public location or a private location, and displaying, by optical detection device, a warning message in case the recorded location information relates to a public location. The method of any one of the preceding claims, wherein the at least portion of the item to be recognized by image recognition comprises at least one of a shape, a label, and a logo of the item. A software program comprising program instructions for, when loaded into a memory of a data processing device, causing the data processing device to perform the method according to any of the preceding claims. A user device comprising the software program according to claim 14. A user device according to claim 15, comprising at least one of a mobile phone, a smartphone, a notebook computer and a tablet computer.