EP3348334B1

EP3348334B1 - Recognition systems and method for address data on physical mail

Info

Publication number: EP3348334B1
Application number: EP17151822.8A
Authority: EP
Inventors: Ekkehard HAHN; Simon FLIß; Ricardo Rivera
Original assignee: Mail Professionals GmbH
Current assignee: Mail Professionals GmbH
Priority date: 2017-01-17
Filing date: 2017-01-17
Publication date: 2020-03-04
Anticipated expiration: 2037-01-17
Also published as: WO2018134216A1; EP3348334A1

Description

The invention relates to an apparatus for recognizing address information on physical mail, the apparatus comprising a camera, a recognition surface for mail to be placed onto it, and a housing, to a process for recognizing address information on physical mail, and to a system for distributing physical mail to a recipient.
Recognizing address information on physical mail for processing physical mail has been a challenge in the past. Some solutions of the prior art detect the address information field in physical mail by detecting the reflection of the window in envelopes. However, it is these reflections that hinder accurate recognition of the address information.
CN 103763451 A discloses a solution using a point light source shining on an envelope at a certain angle of about 45°, wherein the bright spot which is the reflection of the point light source is eliminated by the technology disclosed in this reference.
EP 12 57 954 B1 discloses a method for distributing mail.
DE 699 31 086 T2 discloses an apparatus that helps to sort mail for distribution.
DE 198 01 246 A1 discloses a method for detecting windows for address information in envelopes by detecting a reflection of the window from different angles.
US 2001/0019619 A1 discloses a solution using two point light sources shining on an envelope at a certain angle of about 45° and causing reflections.
WO 2009/095602 A1 discloses a system for acquiring images for identifying signs on postal packages.
FR 2 926 899 A1 discloses a device for acquiring images of postal items comprising fluorescent and contrast marks.
FR 2 895 820 A1 discloses a postal parcel image acquisition device for use in postal sorting machine that has LEDs including light radiations that illuminate address information visible by contrast and with fluorescent mark in zones of window, respectively.
The problem of the present application is to provide a technology to overcome the shortcomings of the prior art.
In a first embodiment of the invention, this problem is solved by an apparatus, especially for recognizing address information on physical mail, comprising a camera, a light source, a recognition surface for mail to be placed onto it, and a housing for blocking light entering an interior of the housing,

a. wherein the housing is arranged at least partly around: the camera and the recognition surface, and wherein the housing has at least one opening to place mail on the recognition surface,
b. wherein the camera is positioned opposite to the recognition surface,
c. wherein the camera is directed to the recognition surface for imaging,

Another advantage of the housing lies in the avoidance of stress being caused in the workforce due to extreme light exposure.

Camera

Preferably, optical axis of the camera is arranged relative to the recognition surface at an angle in a range of from 80° to 100°.
Preferably, the camera has a focal length in a range of from 2 to 10 mm. This rather short focal length ensures that the camera can capture the whole recognition surface and at the same time be as close as possible to the recognition surface in order to avoid any extra stray light or reflections entering the camera. If the focal length is too long, the distance of the camera to the recognition surface must be increased, which in turn may result in a higher probability of additional stray light or reflections entering the optical path into a lens system (for example an objective) of the camera on its way from the recognition surface to the lens system of the camera. This has been found to be one of the important features when it comes to optimizing recognition quality.
Preferably, the aperture of the camera is a in a range of from 1.5 to 4. Below this range, the camera may not reliably reproduce sharp images for varying thickness of pieces of physical mail. Above this range, it may be the case that not enough light enters the camera in order to yield an image with the type of quality required for reliable OCR processing.
Preferably, the distance between an outermost lens of a lens system of the camera and the recognition surface is in a range of from 15 cm to 100 cm, more preferably in a range of from 20 cm to 50 cm. The lens system may be an objective. It was found that this distance had the advantage of being able to, on one hand, capture an image of the whole piece of physical mail and, on the other hand, still provide sufficiently uniform illumination allowing for the achievement of an optimum of image quality.
Preferably, the outermost lens of the lens system of the camera has a diameter of 30 to 70 mm.
Preferably, the camera has a rolling shutter. This has been shown to improve recognition results.
Preferably, the camera is a digital camera. Preferably, the resolution of an image sensor of the camera is in a range from 5 to 500 megapixels (one megapixel = one million pixels). For the particular purpose of the present invention, a digital camera has the clear advantage of being able to easily automate the recognition of address information. Below said range, the image quality may be insufficient for accurate recognition of address information. Above this range, computing the image information and extracting address information may be inefficient.
Preferably, the camera and the lens system of the camera are selected such that the camera captures at least the whole area of the recognition surface.
Preferably, the outermost lens of the lens system of the camera has a polarizing filter attached to it. It has been found that reflections and unwanted bright spots in the captured image can be further minimized this way.
Preferably, the camera does not generate a black & white image. More preferably, the camera generates a color image. This has the advantage that indicators such as priority mail can be recognized by the system according to the present invention based on their shape and color.
Preferably, the camera has a fixed focus. In other words, the lens system in the form of the objective is set to a specific focus. This has been found to make the overall performance much more robust and more tolerant to handling errors. Preferably, the focus of the camera is set to a distance in a range of 0.5 to 5 mm from the recognition surface. This has the advantage that the surface of the mail to be recognized - depending on the thickness of the piece of mail - can have varying distance from the recognition surface.
Preferably, the angle of the optical axis of the camera to the recognition surface is in a range of 85° to 95°, most preferably 90°. It has been found that angles outside the preferred range may result in unwanted reflections and other optical errors in the captured image.

Light Source

Preferably, the illumination uniformity u of the light source is in a range of 0.8 to 1, more preferably in a range of 0.9 to 1, and most preferably in a range of 0.97 to 1. Illumination uniformity is typically defined as $u = E_{\min} / E_{average},$
wherein u is the illumination uniformity, E_min is the minimal illuminance and E_average is the area weighted average illuminance. Illuminance is the total luminous flux incident on a surface measured in lux (Ix) or lumens per square meter (cd·sr·m^-2). It has been found that one key to eliminating unwanted reflections on the physical mail is uniform lighting incident on the physical mail.
Preferably, the light source is not a point light. Preferably, the light source is not a flash light. As can be seen in the prior art, point lights such as spot lights or flash lights generate unwanted bright spots in the resulting image captured by the camera that need to be removed by complicated and inefficient techniques. Preferably, the light emitting surface of the light source is planar. More preferably, the light source is a light panel, even more preferably an LED light panel, and most preferably an organic LED light panel. Especially with organic LEDs (or OLEDs), very uniform area light sources such as panel lights can be achieved. As mentioned before, it has been found that uniform lighting is a key factor to efficient recognition of address information on physical mail.
Preferably, the light source is an LED light. LED light has the advantage of less heat development. Compared to the aforementioned solutions of the prior art, the inventors found that they do not need complicated heat dissipation or ventilation solutions.
Preferably, the light source is arranged around the outermost lens of the lens system of the camera; more preferably the outermost lens of the lens system of the camera is completely encircled by the light source. Most preferably, the light source is a light panel with a hole in the center that fits the size of the lens of the camera.
Preferably, the smallest distance of the light source from the recognition surface deviates less than 10%, more preferably less than 5%, from the distance of the outermost camera lens of the lens system from the recognition surface. Said distance has been found to further eliminate bright spots and unwanted reflections in the captured image of the camera.
Preferably, the center of the lens of the lens system of the camera, deviates less than 10 mm, more preferably less than 2 mm, from a straight line extending perpendicular from the center of the light source. Preferably, the center of the outermost lens of the lens system of the camera, deviates less than 10 mm, more preferably less than 2 mm, from the center of the light emitting surface of the light source.
Preferably, the area of the light emitting surface of the light source is in a range of 200 to 1500 cm².
Preferably, the light emitting surface of the light source has a polarizing filter attached to it. It has been found that reflections and unwanted bright spots in the captured image can be further minimized this way. More preferably, this polarizing filter is a foil.
More preferably, the optical angle between the polarizing filter attached to the camera lens and the polarizing filter attached to the light source is in a range of 10° to 80°, more preferably in a range of 20° to 70°.
Preferably, the light source emits collimated light. The inventors have found that this is an important factor for eliminating unwanted reflections and bright spots in the image captured by the camera.
Preferably, the angle between the optical axis of the camera and a straight line through an intersection point, where this optical axis intersects the recognition surface, and any point of the light emitting surface of the light source is in a range of 0.1° to 40°, more preferably 0.3° to 30°. The inventors have found that - especially in combination with the above mentioned collimation and/or the polarizing filters and/or anodized side walls - this is a very important feature for achieving optimal image quality for address information recognition.

Recognition Surface

Preferably,

a. the center of the outermost lens of the lens system of the camera, and
b. the center of the light emitting surface of the light source,

Preferably, the area of the recognition surface is in a range of 200 to 1500 cm².
Preferably, the area of the recognition surface is larger than the area of the light emitting surface of the light source. More preferably, the area of the recognition surface is 2% to 20% larger than the area of the light emitting surface of the light source. Preferably, the light source points toward the recognition surface. The light emitting surface of the light source is arranged in a first plane. The recognition surface is arranged in a second plane. Preferably, the first plane is arranged relative to the second plane at an angle of less than 1°.
This way the light source and the recognition surface are more or less parallel, which has been found to be another key factor for minimizing unwanted reflections and bright spots in the image captured by the camera.
Preferably, the distance between the light emitting surface of the light source and the recognition surface is in a range of 15 cm to 100 cm, more preferably in a range of 20 cm to 50 cm. It has been found that this distance has the advantage of, on one hand, enabling the camera to capture an image of the whole piece of physical mail and, on the other hand, still providing sufficient even lighting allowing for the achievement of an optimum of image quality.

Housing

Preferably, the light source is located within the housing. Preferably, the light source is positioned within the top 10% of the height of the housing. This way the light source is located near the top of the housing.
Preferably, the inner surface of the housing has in antireflective surface. Preferably, the inner surface of the housing is black. Preferably, the inner surface of the housing has an arithmetic average surface roughness R_a in a range of 0.2 to 2 µm as measured by VSI (vertical scanning interferometry). Preferably, the inner surface of the housing is made of an anodized metal, especially aluminum. Preferably, at least the inner surface of the side walls of the housing between the light source and the recognition surface exhibit one of more features as mentioned in this paragraph. Most preferably, the inner surface of the side walls of the housing between the light source and the recognition surface is black and made of anodized aluminum. In contrast to solutions of the prior art, it has been found that eliminating any reflection of the light traveling from the light source to the recognition surface is detrimental to the quality of the captured image.
Preferably the side walls of the housing extend at least from the recognition surface to the light source.
Preferably, the side walls of the housing have a hole or opening for placing physical mail onto the recognition surface. The opening may preferably be a slit formed by the side walls of the housing, the front plate as well as the recognition surface. This hole may be closeable by a door or flap. Preferably, the area of the hole in the side walls is in a range of 2 to 7% of the area of the side walls. Preferably, the area of the hole is in a range of 100 to 500 cm². Below these ranges, the hole may be too small to fit all typical sizes of physical mail onto the recognition surface. Above these ranges, too much stray light may enter the inside of the apparatus and deteriorate the image quality of the camera.
Preferably, the hole has a height in a range of 2 to 15 cm.
Preferably, the hole has at least one straight edge, wherein that edge is adjacent to one edge of the recognition surface. The advantage is that mail will not be lost by falling into a gap between the hole and the recognition surface.
Preferably, the side walls have no further opening or hole between the recognition surface and the outermost camera lens except for the above mentioned hole between the recognition surface and the camera lens.
Preferably, the distance between the recognition surface and the side walls of the housing is in a range of 0 to 2 mm. The advantage resulting from this is less physical mail getting lost by falling into the gap between the housing and the recognition surface.
Preferably, the housing has a touchscreen attached to the outside of the housing. This makes control of the apparatus and the whole system much easier.

Other Embodiments

The problem according to the present invention is solved in another embodiment by a process for recognizing address information on physical mail comprising the step of recognizing the address information using an apparatus according to the present invention.
Preferably, location of the address information on the physical mail is part of this process. More preferably, location of the address information on the physical mail is not accomplished by finding reflective parts of the surface of the physical mail that could indicate a window in an envelope. The goal of the present invention is to eliminate any source of reflection in the captured image. More preferably, location of the address information on the physical mail is accomplished by OCR (optical character recognition) in the image captured by the camera and subsequent search for indicators in a block of text such as typical personal names, streets, postal codes, city names, country names, and then defining a block of text containing one or more of these indicators as the location of the address information. This has the particular advantage over the solutions of the prior art that this method can also deal with printed address information on paper as compared to address information behind e.g. a vinyl window in an envelope.
Preferably, an image of the physical mail is captured and preferably stored.
After capturing the image information, preferably the piece of physical mail is identified in the image and preferably an image of just the piece of physical mail is extracted. Preferably this is achieved by finding the largest closed contour on the captured image and this contour is cut out as the image of just the piece of physical mail.
Preferably, this image of just the piece of physical mail is rotated around the center of the image.
Preferably, the location of the address information (e.g. the address window) is determined in the image of just the piece of physical mail.
Preferably, the located address information is cut out as an image of the address information.
Preferably, the image of the address information is processed with text recognition to obtain the text of the address information.
The problem according to the present invention is solved in another embodiment by a system for distributing physical mail to a recipient, characterized in that the address of the recipient on the physical mail is recognized using an apparatus according to the present invention and then the mail is delivered to the recipient physically or electronically (e.g. after opening the physical mail and scanning).
Preferably, according to the present invention, the physical mail is not transported automatically to the recognition surface.
Preferably, the physical mail is recognized using an apparatus according to the present invention when it arrives at the mail distributing facility. The mail distributing facility according to the present invention can be a company with a plurality of potential mail recipients (e.g. very large companies with hundreds or thousands of employees at one location) or a mail distribution center of a logistics company (e.g. a postal company).
Preferably, based on rules that can be assigned in the system, certain actions can be taken after recognizing the physical mail with the apparatus according to the present invention. These actions are not limited to, but can be selected from, the group of rush delivery, courier delivery, rerouting, opening, digitalizing, destruction, pre-filling of datasets or forms based on the recognized information, printing of labels or any sensible combination thereof.
Preferably, all such actions are documented.
Preferably, these rules can be determined by the recipient. Preferably, the recipient is presented with these options as a menu of selectable items or buttons in a computer program, a web application, a smartphone application or an intranet application. Preferably, the system recognizes confidential physical mail (e.g. by finding words like "confidential" on the envelope) and overrules any rules established by the recipient and delivers the physical mail to the recipient without opening or scanning the mail.
Preferably, a receiver cost center is associated with the recipient.
To ensure compliance with privacy laws, the recipient can preferably refuse that mail may be processed other than for regular delivery. Alternatively, the recipient can preferably actively allow the system to conduct one or more of the aforementioned actions.
Preferably, the recipient can determine whether the aforementioned actions are to apply to only one single piece of physical mail, a selection of pieces of physical mail or all physical mail.
Preferably, after the recognition step and before the pieces of physical mail are distributed to the final recipient, the pieces of mail are put into a receptacle that is provided for one recipient or a group of recipients. That receptacle can preferably be a mailbox that preferably has a scanner for the machine readable label.
The system can preferably be configured to reroute the mail from the originally intended recipient indicated on the physical mail to another destination, e.g. the home office location of the recipient or, if the recipient has moved, to the new location of the recipient.
As an example, a recipient may define in the rules to open all mail, digitalize the mail, have it sent to the recipient via email, except for invoices, which should be sent to the accounting department and automatically entered into the ERP system. In another scenario, the recipient could determine that all mail should be opened, digitalized and forwarded to the recipient via email, except for parcels, which should be forwarded to the home office location of the recipient.
Preferably, the system assigns a unique identifier to each individual piece of physical mail.
Preferably, a machine readable label is attached to the physical mail after the recognition step. Preferably, the machine readable label comprises a QR code. Preferably, the QR code encodes information about the recipient. Alternatively or in addition to the QR code, machine readable label comprises machine readable text consisting of 5 to 30 characters. Preferably, this machine readable text encodes information about the recipient. Preferably, the QR code and/or the machine readable text encode a unique identifier for each individual recognized piece of physical mail.
Preferably, after the label has been attached to the physical mail, the physical mail is not processed by an external postal service. Preferably, after the label has been attached to the physical mail, the physical mail is only processed within the target organization or at the address of the recipient.
Preferably, an image of the physical mail is stored as an image file during, before or after the recognition step. Preferably, this image is then made available to the recipient via electronic means (e.g. via the intranet, a web application or a smartphone application).
Preferably, the system always assigns a recipient to the physical mail. If the recipient cannot be clearly identified automatically, preferably the recipient is searched for until found. Preferably, the information about the recipient is always stored in a database record together with the unique identifier.
Preferably, the system creates an audit trail. Preferably, this audit trail comprises one or more of the following pieces of information: the unique identifier, each step performed with time information, recipient information, sender information, rules (e.g. as defined by the recipient).
There is also a computer program product, stored on a computer readable medium, comprising parts of code, and adapted to perform steps of the process according to the present invention when the entire computer program product has been loaded into the memory of a computer.

Brief description of the figures

Preferred embodiments and advantages of the invention are illustrated with the accompanying figures and the following description of the figures. It is to be understood that the following description relates to preferred features that do not necessarily limit the scope of the invention.

Fig. 1: Sectional view of the apparatus
Fig. 2: Perspective view of the apparatus
Fig. 3: Process flow of the system for distributing physical mail

Description of these preferred embodiments

Description will now be directed to preferred embodiments of the present invention with reference to the attached figures.
Fig. 1 shows a sectional view of the apparatus 1 according to the present invention. The cuboid shaped housing 9 can be made of aluminum with a wall thickness of about 1 - 3 mm. The inside of the housing 9 can be anodized to yield a black surface with a certain surface roughness to absorb stray light inside the housing. The recognition surface 7 can be made of aluminum with a wall thickness of about 1 - 3 mm. The recognition surface can be square shaped and have dimensions of e.g. 30 to 50 cm for each side. Centered above the recognition surface 7 at a distance of about 40 cm is a camera 3 (e.g. Camera TBN 5.4 10MP by THE IMAGING SOURCE EUROPE GMBH) pointing straight down at the recognition surface 7. Arranged around an objective 13 of the camera 3 is a planar LED light 5 (e.g. LGCB3020 of Lumimax®) with about the same dimension as the recognition surface 7. The planar LED light 5 is parallel to the recognition surface 7 and has a round hole just for the objective 13. A polarizing foil is mounted in front of the planar LED light 5 and another polarizing filter is mounted on the outermost lens of the objective 13.
Fig. 2 shows a perspective view of the apparatus 1 where the housing 9 comprises a front cover plate of the housing 15. Through the slit 17 formed by the shorter front cover plate of the housing 15, the recognition surface 7 is visible. The slit 17 is high enough to allow for smaller packages to be inserted into the apparatus 1.
Fig. 3 shows the process flow of the system for distributing physical mail. When starting the processing of the incoming mail, a piece of incoming physical mail is checked for being internal mail or not. If it is internal mail, this is marked automatically or in an input mask of the computer program product according to the invention. Then the piece of physical mail is placed onto the recognition surface of the apparatus according to the present invention. A database record for this particular piece of physical mail may then be generated by the computer program product. With the help of the camera of the apparatus, the computer program product or the user (with a monitor showing the live camera image) checks whether there is a number such as a tracking number or a number of a piece of registered mail on the piece of physical mail. If this is the case, a property in the database record for this piece of mail is modified accordingly (e.g. as a piece of registered mail or a trackable parcel). After that, the computer program product tries to recognize the number with the help of the apparatus or a user may enter the number. The number may then also be stored in the database record.
As a next step, the camera captures an image of the piece of mail. The computer program product identifies the location of the address information and performs optical character recognition (OCR) on the address information. The resulting information is displayed to the user of the apparatus. If the information is found to be incorrect, the image of the piece of mail is re-taken and the last step is performed again.
Once the address information of the recipient has been captured successfully, the computer program product checks whether the recipient is contained in a database of all possible recipients at the particular location where this process is performed.
If the recipient is not recognized in the database, the computer program products checks whether a similar recipient (e.g. with a very similar name or the same room number) is present in the database and will display a list of possible recipients. If the correct recipient is present in the list, the user confirms this recipient and the process will continue.
If the correct recipient is not present in the list or if the computer program product has not been able to identify similar possible recipients, the recipient is entered into the database manually as a new recipient. If this recipient is traceable within the present location (e.g. office location, i.e. if the recipient is known to be in this location), the recipient is added to the above mentioned list, then possibly confirmed by the user and the process will continue.
If the new user is not traceable, a manual search process is activated and a search label is printed and affixed to the piece of mail. This search label comprises a code (e.g. a barcode or a QR code that corresponds to the database record).
After confirmation by the user, the regular processing of incoming physical mail continues with printing a label onto the physical mail. This label comprises a barcode or a QR code that corresponds to the database record, the recipient information in human readable form and, if the user has defined any particular action such as scanning the mail, information on such predefined action to be taken. Then the mail is passed on to the mail sorting in the location where this process is performed. This concludes the processing of the incoming mail.

List of reference signs

1: apparatus
3: camera
5: light source
7: recognition surface
9: housing
11: light emitting surface of the light source
13: outermost lens of an objective of the camera
15: front cover plate of the housing
17: opening, hole, slit between the front cover plate of the housing and the recognition surface

Claims

Apparatus (1) comprising a camera (3), a light source (5), a recognition surface (7) for mail to be placed onto it, and a housing (9) for blocking light entering an interior of the housing (9),
a. wherein the housing (9) is arranged at least partly around: the camera (3) and the recognition surface (7), and wherein the housing has at least one opening (17) to place mail on the recognition surface (7),

b. wherein the camera (3) is positioned opposite to the recognition surface (7),

c. wherein the camera (3) is directed to the recognition surface (7) for imaging,

d. wherein the camera (3) has an optical axis, the optical axis being arranged relative to the recognition surface (7) at an angle in a range of from 10° to 170°, and

e. wherein
i. the light emitting surface (11) of the light source (5) is arranged in a first plane,

ii. the recognition surface (7) is arranged in a second plane, characterised characterised in that

iii. the first plane is arranged relative to the second plane at an angle of less than 5°.
Apparatus (1) according to claim 1, characterized in that the camera (3) has a fixed focus.
Apparatus (1) according to claim 1 or 2, characterized in that
(i) the light source (5) has an illumination uniformity, and that

(ii) the illumination uniformity of the light source (5) is in at least one of the following ranges: from 0.8 to 1, from 0.9 to 1, and from 0.97 to 1.
Apparatus (1) according to any of claims 1 to 3, characterized in that the light source (5) comprises a planar light emitting surface (11).
Apparatus (1) according to any of claims 1 to 4, characterized in that the camera (3) comprises a lens system having an outermost lens (13) and that the apparatus (1) comprises at least one of the following:
(i) the light source (5) is arranged around the outermost lens (13) of the lens system of the camera (3),

(ii) the outermost lens (13) of the lens system of the camera (3) is completely encircled by the light source (5).
Apparatus (1) according to claim 5, characterized in that a center of the outermost lens (13) of the camera (3) is arranged at a distance from a center of the light emitting surface (11) of the light source (5), wherein the distance is at least one of the following: less than 10 mm and less than 2 mm.
Apparatus (1) according to any of claims 1 to 6, characterized in that
(i) the optical axis of the camera (3) intersects the recognition surface (7) at an intersection point, and in that

(ii) a straight line connecting the intersection point and a point of the light emitting surface (11) of the light source (5) is arranged relative to the optical axis at a further angle being in a range of at least one of the following: from 0.1° to 40° and from 0.3° to 30°.
Apparatus (1) according to any of claims 1 to 7, characterized in that
(i) the recognition surface (7) has a first surface area ,

(ii) the light emitting surface (11) of the light source (5) has a second surface area, and that

(iii) the first surface area is larger than the second surface area.
Apparatus (1) according to any of claims 1 to 8, characterized in that
(i) the light emitting surface (11) of the light source (5) is arranged in a first plane,

(ii) the recognition surface (7) is arranged in a second plane, and in that

(iii) the first plane is arranged relative to the second plane at an angle of less than 1°.
Apparatus (1) according to any of claims 1 to 9, characterized in that the inner surface of the housing is made of an anodized metal.
Apparatus (1) according to any of claims 1 to 10, characterized in that the light source emits collimated light.
Apparatus (1) according to any of claims 1 to 11, characterized in that the light emitting surface of the light source has a polarizing filter attached to it.
Process for recognizing address information on physical mail comprising the step of recognizing the address information using an apparatus (1) according to any one of claims 1 to 12.
System for distributing physical mail to a recipient, characterized in that
(i) an address of the recipient on the physical mail is recognized using an apparatus (1) according to any one of claims 1 to 12, and in that

(ii) the mail is physically delivered to the recipient or information comprised in the physical mail is electronically delivered to the recipient after the address of the recipient has been recognized.