Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T1/00—General purpose image data processing
  • G06T1/0021—Image watermarking
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/20—Image preprocessing
  • G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
  • G06V10/225—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition based on a marking or identifier characterising the area

Definitions

• the present invention concerns a method and system for image capture and facilitated annotation. Moreover the present invention deals with a computer program product suitable for performing the steps of the method presented hereinafter.
• Document WO2012/019163 discloses a method wherein a still or video image is captured by a cell phone and a rectangular portion corresponding to a video display size is retrieved and then hidden marks within such a portion are identified.
• Document WO201 1/088386 discloses a way to augment reality of a captured image including hidden data that have been previously included by a content authoring system.
• US2007/0024527 discloses a method wherein an image, after having been captured by a cell phone and having been made subject to a hidden content inclusion, is sent with such a hidden content to a remote unit.
• WO201 1/1 56475 discloses a method wherein a hidden content regarding the location is included within an image. Such an image can be captured by a cell phone. The document further discloses overlaying location dependent information onto the captured image once displayed by the cell phone.
• images are commonly captured by a user with a hand-held camera, such as a mobile phone with a camera, and compared with reference images stored in a collection. In case of match, i.e., when the captured image corresponds to one of the previously stored reference images, annotating elements associated with this reference image are retrieved and superimposed over the captured image, resulting in an annotated image.
• the user's user device Since the user's user device does not know which of the captured images are associated with annotations, it needs to send all the captured images to the remote server, which will then try to match each captured and transmitted image with each reference image in its collection. In case of video images, this results in the useless transmission of a video stream where many frames don't match any reference image and cannot be annotated, thus wasting bandwidth. Moreover, this also results in an increased workload for the remote server which needs to process a large number of irrelevant images that it receives with reference images.
• the workload of the remote server is even more increased if the conditions of capture of the received images are different from the conditions of capture of the reference images. For example, if the image is captured from a different distance and/or angle, the server might need to crop, rotate, and/or otherwise distort the received image in order to make the registration and verify a possible match. Other adjustments might be needed to compensate different zooming factors, brightnesses, contrasts, saturations, white balances, etc in the captured and in the reference images. Recognizing in real time a large number of received captured images requires however an enormous processing power. Moreover, the server does not know in advance how the captured image should be registered, and might need different adjustments - for example, by cropping or resizing in different possible ways the captured image.
• these aims are achieved by way of a method comprising obtaining captured data representing an image with an image sensor; retrieving steganographic marks hidden in said captured data; image-processing said captured data based on said hidden marks, so as to generate image-processed data representing a modified image;
• the hidden mark may be easy to detect, for example by a module within the user's device.
• the user's device can sort out those images which include one of those hidden mark and are therefore likely to match a reference image, and send only those images to the remote server, resulting in reduced amount of data to transmit.
• the steganographic marks are hidden in the image, they are not visible without some special image processing and are therefore easy to integrate in an existing still or video image.
• the claimed method has the further advantage that the data representing the captured images are processed using information included in the hidden marks, resulting in image-processed data which can be easier to match with data representing reference images.
• the hidden marks might be used for determining how the captured image should be resized, cropped, rotated, or otherwise transformed in order to match the corresponding reference image.
• the hidden marks define at least one border of the image which has to be captured. [0017] In one embodiment, the hidden marks can be used to correct geometrical distortions or white balance, contrast, saturation and/or brightness of the image.
• the hidden marks only need to indicate data required for stabilising or otherwise image-processing the image.
• the annotations themselves depend on the visible content of the image, as determined with computer vision and feature recognition methods. Therefore, the hidden marks can be made tiny and easy to hide. It is possible however to hide some contents used for determining the annotations in the hidden marks as well.
• successive images such as successive still images or successive frames in a video
• the process comprises trying to match only those captured images that contain hidden marks with said reference images.
• Data representing images without any hidden mark are not sent to the server and not further processed, avoiding a waste of bandwidth and computational resources.
• these aims are also pursued by way of a user device, comprising an image sensor suitable for capturing data representing at least one image and a processor configured to retrieve hidden marks in said captured image and locally generating a processed image based on said hidden marks retrieval and matching annotations processed remotely.
• the user device comprises a display suitable for showing images and the processor are configured to aid a user in the hidden marks retrieval and image capture by making aiding signs appear onto the display.
• the aiding signs are made appear onto the display when an automatic full identification of said hidden marks fails and/or when device detects that the user could move its camera, or modify the zooming factor, or change the capture condition in any other possible way to improve the hidden marks retrieval and/or the registration of the captured image.
• the user device user device locally detects hidden marks, without any need to transmit those images to a remote server to detect hidden marks.
• the hidden marks can be detected locally and without any data transmission. Detection of a hidden mark can be used to decide whether or not the captured image is associated with some annotations, and should be processed locally or in a remote server to retrieve those annotations.
• the annotation process can be done within the device, for example using some reference models stored in the device or retrieved from a server, or in a remote server storing reference models and to which the captured image, or a processed version of the captured image, is sent.
• the decision to process the image locally or in the server may depend on the hidden mark.
• the device may try to find a matching reference image locally, and send the captured image, or an image-processed version of the captured image, to a remote server when it fails to find a matching reference image in its local memory.
• At least one user device with an image sensor suitable for capturing data representing images and with a processor arranged for identifying hidden marks in said data;
• a remote server suitable for receiving and transmitting data to said at least one user device, to process at least a part of a captured image transmitted from said user device and to send annotations or annotated images to said user device.
• the system allows the user device to display annotated images based on data received from an external server that unburdens the user device from the computational effort of an image matching process.
• the identification of hidden marks is performed in at least one of the following domains: frequency domain, time domain and/or space domain.
• Another advantage of the above described features is to reduce the amount of data which is transmitted between the user's device and the remote server.
• Fig. 1 shows a general view of a system for image capturing and annotation.
• Fig.2 shows a detail of a part of the system of figure 1 .
• Fig.3 shows a flow chart containing parts of the algorithm performed by the system.
• Fig.4 shows a flow chart describing a hidden mark identification method choosing.
• Fig. 5 shows an example of use of the method and system.
• the system for image capturing and annotation comprises at least one remote server 20 as well as a user device 10, that can be in a non limiting extent a hand-held device such as for example a personal cell phone, a smartphone, a tablet, a computer, a camera with a communication interface, a helmet with a camera, goggles with camera, contact lenses, or other devices embarked on a person and comprising a sensor for capturing visible, infrared and/or ultraviolet light, as well as a display.
• the remote server 20 and the at least one user device 10 are suitable for exchanging data representing images and other data. Even though in the present description we refer to a remote server, it can be equivalently substituted by any remote processing system capable of performing image processing and data transceiving, including for example a cloud server or a plurality of servers.
• the user device 10 comprises a display 1 1 and an image sensor 12 capable of capturing still and/or video images (including for instance, slides, presentation, portraits, advertisements, spots).
• the user device comprises also a processor 13 capable of processing data
• a transceiver 15 such as a WIFI and/or cellular interface is available for connecting the device 10 to a remote server 20 or cloud server over a network such as the Internet, and for wirelessly exchanging data with the remote server 20.
• This transceiver will be adapted according to the actual category of user device 10 used, and will be designed so as to comply if possible with known data transceiving standards.
• the program code stored in memory 14 and executed by the processor 13 in user's device 10 could include for example an application software, or app, that can be downloaded and installed by the user in the user's device 10.
• the program code could also include part of the operating code of the device 10.
• the program code could also include code
• the program code may be stored as a computer program product in a tangible apparatus readable medium, such as a Flash memory, a hard disk, or any type of permanent or semi- permanent memory.
• the server 20 includes a storage 21 with a database such as a SQL database, a set of XML documents, a set of images, etc , for storing a collection of reference images, such as 2D and/or 3D reference images, and/or one or a plurality of global models, and a processor, such as a microprocessor with computer code for causing the microprocessor to perform the operations needed in the annotation method.
• a database such as a SQL database, a set of XML documents, a set of images, etc
• a collection of reference images such as 2D and/or 3D reference images, and/or one or a plurality of global models
• a processor such as a microprocessor with computer code for causing the microprocessor to perform the operations needed in the annotation method.
• At least some of the still images or video frames that are captured by the image sensor 12 comprise some hidden marks 30 also indicated in the present description with the synonym steganographic marks, which can be used to image-process the captured image.
• Those hidden marks can be extracted by a suitable software module executed by the processor 13 in the user's device 10 processing an image while the human eye cannot notice them.
• a lot of information can be stored in steganographic marks without making the reader or user recognise a difference with an image without those marks.
• some marks detectable by the system 1 could represent alignments marks indicating corners 30a-30d of an image 61 , such as a still or video image projected on a screen, on a monitor, or printed on paper.
• Some marks could also represent a test pattern indicating for example other parameters of the image, such as a reference grey level for adjusting the brightness and/or white balance, lines for correcting geometric distortions, etc.
• Some hidden marks may also be included just to indicate that the image is associated with annotations available from a remote server 20. For example, some hidden marks may include a hidden hyperlink.
• the marks can be included in a plurality of subsequent frames that the human eye or brain averages; or included as unnoticeable variations of the image in the time and/or spatial and/or frequency domain.
• Some hidden marks may be revealed only through a Fourier transformation of the image data, such as a DFT (Discrete Fourier Transform) or DCT (Direct Cosine Transform) methods. Those methods of hidding marks can be mixed up together so as to provide hidden marks more robust to detection.
• DFT Discrete Fourier Transform
• DCT Direct Cosine Transform
• a DCT technique is used for hiding the marks into the image, and for retrieving the marks.
• each image is cut into a plurality of blocks.
• a Discrete Cosine Transform is then applied independently on each block.
• three different kinds of data are hidden into three blocks so that those three blocks can be used as geometric insights in order to retrieve the image plane later on.
• LSB least significant bit
• the hidden marks can also be "extended hidden marks". Let's assume the case of a newspaper with an advertisement; in this case, it could be difficult to recognise steganographic signs correctly, due to the potential small dimension of the image, lack of texture of the image, and the poor quality associated thereto.
• the extended hidden mark can be hidden not only in the image to capture itself, but also, at least in part, in its surroundings, including textured areas surrounding the image to be captured. In this way, the hidden data contained in the image to be captured is only a part of the full hidden data which actually includes the above named extended hidden marks.
• the system performs a method that leads to an eased image focusing, deblurring, stabilization, cropping and/or front- planning with the possibility to annotate data on the images captured by the image sensor 12 while reducing as much as possible the amount of data which is exchanged between the user device 10 and the server 20.
• the detailed operations of the algorithm will be discussed in detail hereinafter.
• Various steps of the algorithm can conveniently be executed by a computer program stored in a memory in the user device 10 or in the server 20.
• the first step 100 of the algorithm performed by the system is to capture data representing an image, either a still image, one frame of a video, or a complete video, by way of the user device 10 through the image sensor 12.
• the captured data is stored in the memory 14 of the user device 10.
• step 1 10 consists in the detection of hidden marks 30 in the captured data.
• a module executed by the processor 13 of the user device 10 processes the captured image so as to extract hidden marks (if available), therefore decoding the data which is contained therein.
• the steganographic marks on the image can be extracted according to a plurality of different methods. This means that the module executed by the processor 12 of the user device 10 can choose among a plurality of methods in order to retrieve the hidden mark recognition automatically.
• the first step for choosing the hidden marks recognition method is a condition identification step 2000; said condition identification can depend on focusing distance of the image sensor 12, and or a scene/mode either selected by the user or automatically detected by the processor 13 of the user device 10, and/or location of the user device detected with a positioning sensor, and/or radio signals received by the user device, for example radio signals received from one object in the scene.
• the at least one condition retrieved is therefore used to automatically select a hidden mark identification method 3000 among a plurality of identification methods which could by executed by program code stored within the memory of the user device.
• a further step is therefore to apply (block 4000) the chosen identification method to the captured image in order to extract the data contained within the steganographic marks.
• the user device 10 can identify other steganographic marks in data represented the captured image(s), which could be used in order to better realize the matching and registration process.
• steganographic marks associated with an image could indicate a date, a time, geolocalization, image type, identification and in general any kind of information that can be stored in the image as a steganographic message. This additional information can be retrieved in the user device 10, or possibly in the remote server 20, in order to help the process to recognize the image, and/or to change the annotations depending on those marks.
• the processor 13 of the user device 10 detect only a part of the whole set of hidden marks.
• the program executed by processor 13 is configured to display an aiding sign on the display of the user device, or in the viewfinder, or an audible signal, to prompt the user to move or rotate his device in a given direction in order to capture all the hidden marks and hopefully the whole image to annotate (block 130, "display control signs").
• the user may also be prompted to refocus, adjust brightness, or make other adjustments in order to improve the capture of the hidden marks and of the whole image.
• the adjustment in focus, brightness and/or colour can also be electronically controlled, whereas the setting value depends on the detected hidden mark.
• hidden marks A logo, a video, an advertising may be displayed or otherwise rendered to the user, to indicate that hidden marks have been detected so that the user knows that he is currently capturing an image of an object which has hidden marks and which is therefore probably associated with annotations. Therefore, the user knows that the recognition process already started learn something useful about the scene.
• the hidden marks encode some information about the type of object to be recognized (e.g. 2D object, 3D object, face, building, etc)
• an animation or illustration related to the detected type could be presented to the user.
• Signals for helping the users can be displayed or otherwise presented during the complete duration of acquisition, to help the user in pointing the user device 10 constantly in the same position, without moving the device away from the image to be recognised.
• the message that can be provided to the user can be for instance a device vibration, a video transmitted by the processor to the display, a message overimposed to the image, graphic symbols such as lines, arrows etc for indicating if the camera is oriented correctly or in which direction it should be moved.
• step 122 the data captured during step 100 is image-processed by a software module in the user device 10 so as to generate an image- processed data corresponding to a modified image which can be for example and in a non limiting extent, more focused, realigned, resized or cropped, zoomed, adapted in colour tones, colours converted, denoised, deblurred, front-planed or rotated.
• this image processing depends on the hidden marks detected during step 1 10. For example, if the hidden marks indicate the corners of a rectangular image to annotate, the image processing performed during step 122 may include cropping all areas outside of those corners, so as to generate a modified image data including only the area of interest.
• the image processing may also include correction of geometric or optical deformations due for example to the orientation of the camera generating trapezoids instead of rectangles, and/or due to the lens of the camera. If the hidden mark include or otherwise allow to compute a reference grey or other reference colours, the image processing may compute a colour corrected image data, for example an image where the white balance, the saturation, the contrast, and/or the brightness, etc are corrected.
• an image description based on the processed image data is computed by a suitable software module in the user device 10.
• the image description may include for example a list of descriptors of local features of the image, for example features with a high entropy and reliably identifying an image. Different types of features may be used, as known in computer vision technology. A description of those features is then computed. Since the descriptors are preferably computed after image processing at step 122, more reliable detectors will be computed, avoiding for example descriptors for areas outside of the zone of interest.
• step 140 the list of descriptors is sent to the server 20, possibly with additional data related to the localization of the user as indicated by a GPS receiver, by signals received from captured objects, choices entered by the user, and/or data retrieved from the captured data, including already decoded steganographic marks.
• the server 20 then tries to find a matching reference image in a collection of reference images, including 2D or 3D models. The selection of reference images to which the captured data should be compared could be restrained by using the additional
• a first algorithm may be used for matching faces
• a different algorithm may be used for matching captured data with a 2D reference image
• a third algorithm may be used for matching captured data with a 3D model, etc.
• the choice of the best algorithm may be based, at least in part, on information retrieved from the detected steganographic mark.
• one steganographic mark on an element in a scene could indicate that a 3D model of the scene is available, and be used by the server for selecting a suitable algorithm based on 3D models.
• the server 20 performs the registration for aligning the captured image with the matching reference image or model, and determining the position of the camera 10 in the coordinate system of the reference image.
• step 160 a list of annotations associated with the matching reference image is retrieved from storage 21 by a program module executed by the server 20, along with a position or associated feature of the corresponding image. Those annotations are sent to the user device 10. Alternatively, an annotated image is sent.
• step 170 the annotations are added to the previously image- processed data, resulting in data representing a processed image 53 with annotations 530, which can be in the form of text, video, images, sounds, or hyperlinks.
• annotations 530 can be in the form of text, video, images, sounds, or hyperlinks.
• the modified and annotated image is then displayed on display 1 1 .
• the image(s) that are captured by the user device are as much as possible automatically transformed so as to appear as if they were taken in a co-planar direction, without noise and with the right colours, with a correct zoom and without rotation or unnecessary parts.
• the image(s) that are captured by the user device are as much as possible automatically transformed so as to appear as if they were taken in a co-planar direction, without noise and with the right colours, with a correct zoom and without rotation or unnecessary parts.
• an image 61 is projected against a wall by a beamer 60.
• the image could be for example a still image such as a photograph, a text file, a presentation, a movie, etc. It includes some hidden marks steganographically hidden into the image.
• a user captures the presentation -either a still picture or a video - with his user device 10.
• the image of the presentation 61 in the captured image 52 is not rectangular, but includes various geometrical deformations such as keystone or tombstone deformations.
• Other optical deformations may be due to the lens of the beamer 60, to the lens of the camera in device 10, to a surface of projection which is not flat or not perpendicular to the line of projection from the beamer, etc.
• the captured image might also have some white-balance and other colour problems.
• the program executed by the processor in the user's device detects the steganographic marks hidden into the data representing the captured image 52, and use those marks as instructions for image-processing this data, thus generating a modified data representing an image which might be stabilized, cropped, geometrically adjusted, or otherwise improved. For example, if the hidden marks indicate the four corners of a rectangular surface, the program might crop all area outside of those four corners, and redress the inner part to make it rectangular.
• the program executed in user device has detected those hidden marks, it also knows that the captured image is associated with annotations.
• the annotations or annotated image data are then retrieved either from a local memory in the user device 10, or from a remote server 20, using the previously described method.
• An annotated and modified image 53 is then generated and displayed on display 1 1 of the user device 10.
• Some hidden marks might also be associated with particular elements of a complete image. For example, an identification of person or object can be hidden as steganographic mark associated to this element, and used by the program in the user device 10 for tracking this person or object on successive frames of a video.
• any suitable means capable of performing the operations such as various hardware and/or software component(s), circuits, and/or module(s).
• any operations described in the application may be performed by corresponding functional means capable of performing the operations.
• the various means, logical blocks, and modules may include various hardware and/or software component(s) and/or module(s), including, but not limited to a circuit, an application specific integrate circuit (ASIC), or a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field
• FPGA programmable gate array signal
• PLD programmable logic device
• a general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller or state machine.
• a processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
• a server may be implemented as a single machine, as a set of machine, as a virtual server, or as a cloud server.
• the term "annotation” encompasses a wide variety of possible elements, including for example text, still images, video images, logos, image layers, sounds and/or other elements that could be superimposed or otherwise added to an image.
• determining encompasses a wide variety of actions. For example, “determining” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining, estimating and the like. Also, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining” may include resolving, selecting, choosing, establishing and the like.
• Capturing an image of a scene involves using a digital camera for measuring the brightness of light that reaches the image sensor of the camera.
• a software module may reside in any form of storage medium that is known in the art. Some examples of storage media that may be used include random access memory (RAM), read only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM and so forth.
• RAM random access memory
• ROM read only memory
• flash memory EPROM memory
• EEPROM memory EEPROM memory
• registers a hard disk, a removable disk, a CD-ROM and so forth.
• a software module may comprise a single instruction, or many instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media.
• a software module may consist of an executable program, a portion or routine or library used in a complete program, a plurality of interconnected programs, an "apps" executed by many smartphones, tablets or computers, a widget, a Flash application, a portion of HTML code, etc.
• a storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor.
• a database may be implemented as any structured collection of data, including a SQL database, a set of XML documents, a semantical database, or set of information available over an IP network, or any other suitable structure.
• certain aspects may comprise a computer program product for performing the operations presented herein.
• a computer program product may comprise a computer readable medium having instructions stored (and/or encoded) thereon, the instructions being executable by one or more processors to perform the operations described herein.
• the computer program product may include packaging material.

Applications Claiming Priority (1)

Publications (1)

Family

ID=47146348

Family Applications (1)

Country Status (5)

Families Citing this family (5)

Family Cites Families (13)

• 2012
  • 2012-10-16 JP JP2015537150A patent/JP2016500959A/ja active Pending
  • 2012-10-16 WO PCT/EP2012/070515 patent/WO2014060025A1/en active Application Filing
  • 2012-10-16 EP EP12783544.5A patent/EP2909792A1/en not_active Withdrawn
  • 2012-10-16 CN CN201280076419.4A patent/CN104871179A/zh active Pending
  • 2012-10-16 KR KR1020157009172A patent/KR20150082204A/ko not_active Application Discontinuation

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events