Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
  • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
  • G06F16/587—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using geographical or spatial information, e.g. location
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/79—Processing of colour television signals in connection with recording
  • H04N9/80—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
  • H04N9/82—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only
  • H04N9/8205—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback the individual colour picture signal components being recorded simultaneously only involving the multiplexing of an additional signal and the colour video signal
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
  • G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
  • G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• • G—PHYSICS
  • G11—INFORMATION STORAGE
  • G11B—INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER
  • G11B27/00—Editing; Indexing; Addressing; Timing or synchronising; Monitoring; Measuring tape travel
  • G11B27/10—Indexing; Addressing; Timing or synchronising; Measuring tape travel
  • G11B27/19—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier
  • G11B27/28—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording
  • G11B27/32—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording on separate auxiliary tracks of the same or an auxiliary record carrier
  • G11B27/322—Indexing; Addressing; Timing or synchronising; Measuring tape travel by using information detectable on the record carrier by using information signals recorded by the same method as the main recording on separate auxiliary tracks of the same or an auxiliary record carrier used signal is digitally coded
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
  • H04N1/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
  • H04N1/32101—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
  • H04N1/32128—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title attached to the image data, e.g. file header, transmitted message header, information on the same page or in the same computer file as the image
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N2201/00—Indexing scheme relating to scanning, transmission or reproduction of documents or the like, and to details thereof
  • H04N2201/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
  • H04N2201/3201—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
  • H04N2201/3212—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title of data relating to a job, e.g. communication, capture or filing of an image
  • H04N2201/3214—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title of data relating to a job, e.g. communication, capture or filing of an image of a date
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N2201/00—Indexing scheme relating to scanning, transmission or reproduction of documents or the like, and to details thereof
  • H04N2201/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
  • H04N2201/3201—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
  • H04N2201/3212—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title of data relating to a job, e.g. communication, capture or filing of an image
  • H04N2201/3215—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title of data relating to a job, e.g. communication, capture or filing of an image of a time or duration
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N2201/00—Indexing scheme relating to scanning, transmission or reproduction of documents or the like, and to details thereof
  • H04N2201/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
  • H04N2201/3201—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
  • H04N2201/3225—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title of data relating to an image, a page or a document
  • H04N2201/3226—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title of data relating to an image, a page or a document of identification information or the like, e.g. ID code, index, title, part of an image, reduced-size image
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N2201/00—Indexing scheme relating to scanning, transmission or reproduction of documents or the like, and to details thereof
  • H04N2201/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
  • H04N2201/3201—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
  • H04N2201/3225—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title of data relating to an image, a page or a document
  • H04N2201/3249—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title of data relating to an image, a page or a document data relating to a linked page or object, e.g. hyperlink
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N2201/00—Indexing scheme relating to scanning, transmission or reproduction of documents or the like, and to details thereof
  • H04N2201/32—Circuits or arrangements for control or supervision between transmitter and receiver or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device
  • H04N2201/3201—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title
  • H04N2201/3225—Display, printing, storage or transmission of additional information, e.g. ID code, date and time or title of data relating to an image, a page or a document
  • H04N2201/3253—Position information, e.g. geographical position at time of capture, GPS data

Definitions

• the present invention relates to a method, apparatuses and a system for cross-object information retrieval and summarization; especially for finding the relevant objects of a specific object like a photographic image or video based on analyzing multidimensional object contexts and at least partly automatic generation of object summarization from the relevant objects.
• finding the relevant objects of an object by multidimensional object context similarity computing may be carried out, the result of which is also called hyper-object-link.
• object summarization from the relevant objects may be created, and a fusion of the object summaries is used to create a smart annotation for the specific object like a picture or video, which is also called hyper-object- note.
• hyper-object- note which is also called hyper-object- note.
• Object notes may relate to different kinds of data objects including various media objects like image, photo, audio, video, music, books, papers and any other useful objects.
• a method comprising automatically obtaining picture description information from a picture source, wherein the picture description information having been formed at least partly automatically, obtaining annotation information from at least two object sources, wherein the at least two object sources are different from the picture source, automatically fusing the annotation information from the at least two object sources to form fused annotation information, and attaching the fused annotation information to the picture to create an object note for the picture.
• the method further comprises forming a hyper-object-link between the picture and at least one object source, wherein the hyper-object-link comprises a link to an object in the object source and attaching the link to the object to the picture to create an object note for a picture.
• the method further comprises forming relevance information by automatically analyzing information from the two sources against information from the picture source, and obtaining the annotation information from the at least two sources based on the relevance information.
• the method further comprises forming the relevance information by determining a correlation between the picture and the at least two sources by determining their similarity using at least one of the group of time information, location information, event information and person information and forming a weighted similarity indicator by using the at least one of the group of time information, location information, event information and person information.
• the at least two sources are one two or more of email messages, short messages, multimedia messages, instant messages, calendar entries, contact cards, blog entries, wiki entries and social network service entries.
• the method further comprises clustering pictures based on the annotation information from the at least two sources.
• the method further comprises receiving filter information or source selection information from the user for restricting the data from the at least two sources.
• forming the fused annotation information comprises selecting the content for annotation from source content, filtering the selected content to reduce irrelevant and redundant information, and enhancing the cohesion and coherence of the content.
• forming the fused annotation information comprises generating summarization of the content through natural language processing.
• an apparatus comprising at least one processor, memory including computer program code, the memory and the computer program code configured to, with the at least one processor, cause the apparatus to obtain picture description information from a picture source, wherein the picture description information having been formed at least partly automatically, to obtain annotation information from at least two object sources, wherein the at least two object sources are different from the picture source, to automatically fuse the annotation information from the at least two object sources to form fused annotation information, and to attach the fused annotation information to the picture to create an object note for the picture.
• the apparatus further comprises computer program code configured to, with the at least one processor, cause the apparatus to form a hyper-object-link between the picture and at least one object source, wherein the hyper-object-link comprises a link to an object in the object source, and to attach the link to the object to the picture to create an object note for a picture.
• the apparatus further comprises computer program code configured to, with the at least one processor, cause the apparatus to form relevance information by automatically analyzing information from the two sources against information from the picture source, and to obtain the annotation information from the at [east two sources based on the relevance information.
• the apparatus further comprises computer program code configured to, with the at least one processor, cause the apparatus to form the relevance information by determining a correlation between the picture and the at least two sources by determining their similarity using at least one of the group of time information, location information, event information and person information and to form a weighted similarity indicator by using the at least one of the group of time information, location information, event information and person information.
• the at least two sources comprise at least two of the group of email messages, short messages, multimedia messages, instant messages, calendar entries, contact cards, blog entries, wiki entries and social network service entries.
• the apparatus further comprises computer program code configured to, with the at least one processor, cause the apparatus to cluster pictures based on the annotation information from the at least two sources.
• the apparatus further comprises computer program code configured to, with the at least one processor, cause the apparatus to receive filter information or source selection information from the user for restricting the data from the at least two sources.
• the apparatus further comprises computer program code configured to, with the at least one processor, cause the apparatus to selecting the content for annotation from source content, to filter the selected content to reduce irrelevant and redundant information, and to enhance the cohesion and coherence of the content.
• the apparatus further comprises computer program code configured to, with the at least one processor, cause the apparatus to generate summarization of the content through natural language processing.
• a computer program product stored on a computer readable medium and executable in a data processing device, wherein the computer program product comprises a computer program code section for obtaining picture description information from a picture source, wherein the picture description information having been formed at least partly automatically, a computer program code section for obtaining annotation information from at least two object sources, wherein the at least two object sources are different from the picture source, a computer program code section for automatically fusing the annotation information from the at least two object sources to form fused annotation information, and a computer program code section for attaching the fused annotation information to the picture to create an object note for the picture.
• a computer program product stored on a computer readable medium and executable in a data processing device, wherein the computer program product comprises computer program code sections for carrying out the method according to embodiments of the first aspect.
• an apparatus comprising means for obtaining picture description information from a picture source, wherein the picture description information having been formed at least partly automatically, means for obtaining annotation information from at least two object sources, wherein the at least two object sources are different from the picture source, means for automatically fusing the annotation information from the at least two object sources to form fused annotation information, and means for attaching the fused annotation information to the picture to create an object note for the picture.
• a network service providing to a user picture description information from a picture source, wherein the picture description information having been formed at least partly automatically, annotation information from at least two object sources, wherein the at least two object sources are different from said picture source, fused annotation information by automatically fusing said annotation information from said at least two object sources, and an object note for said picture by attaching said fused annotation information to said picture.
• a picture signal embodied on a carrier media, the signal comprising picture description information from a picture source, wherein the picture description information having been formed at least partly automatically, annotation information from at least two object sources, wherein the at least two object sources are different from said picture source, fused annotation information by automatically fusing said annotation information from said at least two object sources, and an object note for said picture by attaching said fused annotation information to said picture.
• Fig. 1 shows a method for annotating pictures according to an example embodiment
• Fig. 3 shows an overview of an example arrangement for annotating pictures
• Fig. 4 illustrates a way for linking Information from relevant objects to a picture according to an example embodiment
• Fig. 5 illustrates a way for forming an annotation for a picture using relevant objects according to an example embodiment
• Fig. 6 shows a method for annotating pictures according to an example embodiment
• Fig. 7 illustrates an architecture for annotating pictures according to an example embodiment
• Fig. 8 shows an architecture for context collection according to an example embodiment
• Fig. 9 illustrates a storage structure of picture contexts in a context database according to an example embodiment
• Fig. 10 illustrates a clustering table for information on events and persons according to an example embodiment
• Fig. 12 shows a process for correlation based linking of objects to a picture according to an example embodiment
• Fig. 13 shows an implementation of forming an annotation for a picture according to an example embodiment.
• These digital objects may include pictures and videos taken with a camera, calendar entries, short messages (SMS) or multimedia messages (MMS), instant messaging (IM) and chat, social network services like Twitter and Facebook, emails, contact cards, blog entries, audio recordings, music, books, papers and many more.
• SMS short messages
• MMS multimedia messages
• IM instant messaging
• chat social network services like Twitter and Facebook
• emails contact cards
• blog entries audio recordings
• Finding the relevant objects when browsing a photo is not, available in existing photo browsing applications. It has also been realized here that it may be beneficial and appreciated by users to get an overview of the activity that the photo or video records when browsing the photo or video later.
• an object such as an image or video and related objects with relevant contexts
• object abstraction by integrating and summarizing pieces of the context information extracted from the linked objects.
• the relevant objects of a specific object are found by computing a multidimensional object contexts similarity measure.
• a hyper object note is created by abstracting object summarization from the relevant objects.
• Such an approach may offer advantages, for example, most relevant content concerning an object may be found automatically through association analysis of contexts essentially without user intervention, and the most important context information may be extracted to summarize and integrate an object abstraction to give user the most meaningful information at the first sight of the object.
• the relevant objects to a specific object may be found and ranked by cross-object correlation computing.
• a hyper-object-note may be abstracted as an object summarization from the relevant objects found.
• the generation of the hyper object note may be done through the following mechanisms.
• the most relevant and the most representative objects may be directly used as object notes, for example, by using top-1 relevant objects such as email, SMS or other messages, calendar entry, contact card, blog entry, Wiki page, in the annotation.
• Natural Language Processing (NLP) may be used to abstract a text note from the relevant objects.
• Fig. 1 shows a method for annotating images according to an example embodiment.
• image or photo description information is obtained for an image (or video) from an image or photo source such as a camera, a collection of images in memory or a photo album.
• the image or photo description information may have been formed at least partly automatically for example by adding a time, location (Global Positioning System or GPS coordinates) and user information to the image
• annotation information from different sources is obtained, for example by retrieving email, SMS and multimedia messages and calendar and contact information. These may be different from the image source and they may have at least partly different internal representation of information.
• the use of different sources of information may make the hyper-object-note richer and more natural.
• the obtained annotation information from the different sources is fused form the hyper-object-note.
• the hyper-object- note may contain text, images, sound, and different objects and it may contain links to the source objects or objects related to the source.
• the hyper-object-note is then attached to the image or photo to get an image with a hyper-object-note. This attaching may happen in various ways, e.g. by embedding the information into the image file, by constructing a new file where the image file and the hyper-object-note reside, by forming metadata that links the image and the hyper-object- note information, by linking the image and the hyper-object-note by file naming and so on.
• Fig. 2a displays a setup of devices, servers and networks that contain elements for annotating images residing on one or more devices.
• the different devices are connected via a fixed network 210 such as the internet or a local area network, or a mobile communication network 220 such as the Global System for Mobile communications (GSM) network, 3 rd Generation (3G) network, 3.5 th Generation (3.5G) network, 4 th Generation (4G) network, Wireiess Local Area Network (WLAN), Bluetooth, or other contemporary and future networks.
• GSM Global System for Mobile communications
• 3G 3 rd Generation
• 3.5G 3.5 th Generation
• 4G 4 th Generation
• WLAN Wireless Local Area Network
• Bluetooth wireless local Area Network
• the networks comprise network elements such as routers and switches to handle data (not shown), and communication interfaces such as the base stations 230 and 231 in order for providing access for the different devices to the network, and the base stations are themselves connected to the mobile network via a fixed connection 276 or a wireless connection 277.
• a server 240 for creating a hyper-object-note for an image or a photo and connected to the fixed network 210
• a server 241 for storing image data and connected to either the fixed network 210 or the mobile network 220
• a server 242 for creating a hyper-object-note for an image or a photo and connected to the mobile network 220.
• computing devices 290 connected to the networks 210 and/or 220 that are there for storing data and providing access to the data via e.g. a web server interface or data storage interface or such. These devices are e.g. the computers 290 that make up the internet with the communication elements residing in 210.
• the various devices are connected to the networks 210 and 220 via communication connections such as a fixed connection 270, 271 , 272 and 280 to the internet, a wireless connection 273 to the internet, a fixed connection 275 to the mobile network, and a wireless connection 278, 279 and 282 to the mobile network.
• the connections 271-282 are implemented by means of communication interfaces at the respective ends of the communication connection.
• the server 240 contains memory 245, one or more processors 246, 247, and computer program code 248 residing in the memory 245 for implementing annotation functionality.
• the different servers 241 , 242, 290 contain at least these same elements for employing functionality relevant to each server.
• the end-user device 251 contains memory 252, at least one processor 253 and 256, and computer program code 254 residing in the memory 252 for implementing the annotation functionality.
• the end-user device may also have at least one camera 255 for taking pictures.
• the end-user device may also contain one, two or more microphones 257 and 258 for capturing sound.
• the different end-user devices 250, 260 may contain at least these same elements for employing functionality relevant to each device.
• Some end-user devices may be equipped with a digital camera enabling taking digital pictures, and one or more microphones enabling audio recording during, before, or after taking a picture or a photo.
• the servers and the end-user devices comprise means for obtaining information from various sources, such as memory means, bus or other internal data connection means, or wired and/or wireless network connections for obtaining information from external sources.
• the devices and servers also contain means for processing the information such as memory, circuitry and processors. These means may be electric or optical or other suitable means. It needs to be understood that different embodiments allow different parts to be carried out in different elements.
• the creation of the hyper-object-note for an image or a photo may be carried out entirely in one user device like 250, 251 or 260, or the image annotation may be entirely carried out in one server device 240, 241 , 242 or 290, or the creation of the hyper-object-note for an image or a photo may be carried out across multiple user devices 250, 251 , 260 or across multiple network devices 240, 241 , 242, 290, or across user devices 250, 251 , 260 and network devices 240, 241 , 242, 290.
• the creation of the hyper-object-note for an image or a photo can be implemented as a software component residing on one device or distributed across several devices, as mentioned above.
• the creation of the hyper-object-note for an image or a photo may also be a service where the user accesses the service through an interface e.g. using a browser.
• Fig. 3 shows an overview of an example arrangement for creating a smart note (hyper-object-note) for pictures.
• photo or a picture is a rich context contained object 310 comprising a picture 312 and associated hyper-object-notes 314.
• the contexts of a picture 321 may describe different attributes, such as time information, location information, event information, emotions of people, content information (metadata), environment circumstances like temperature and humidity, equipment parameters and so on.
• These contexts may exist not only in photos 321 , but also in other objects, such as short messages or other messages like multimedia messages ( S) or instant messages 322, calendar entries 323, social network systems 324, contact cards 325, email messages 326, blog entries 327 and others, e.g.
• pictures can be clustered into different albums 350 to manage user photos, such as a location album 351 arranged by location, an SNS album 352 based on social contexts, a time album 353 arranged by time, a topic album 354 grouped by topic, and an emotion album 355 based on detected emotions.
• Such albums may be called multidimensional photo albums 350. It is also possible to perform a picture query from an external source 360, e.g.
• the process of establishing the hyper-object-iink 340 between photo 312, 321 and other objects may operate as follows. Since the photo and other objects share common contexts, there is a natural relationship between them. First, contexts of photo are collected from various resources. For example, the time the picture was taken, the camera model, the photographing parameters and other such information may be extracted from EXIF information (Exchangeable Image File Format information) which is attached with the image file; GPS coordinates may be collected from an internal source through an application programming interface (API); humidity, temperature and noise grade may be collected from sensor data external or internal to the device. The collected contexts constitute the raw data.
• EXIF information Exchangeable Image File Format information
• GPS coordinates may be collected from an internal source through an application programming interface (API)
• humidity, temperature and noise grade may be collected from sensor data external or internal to the device.
• the collected contexts constitute the raw data.
• Context modelling and data mining allows the relations between photo and other objects hidden in these collected raw data to be revealed. With the result of these operations, the association between photo and relevant objects may be established essentially automatically or with little help from the user to create hyper-object-link.
• the related objects may be interconnected through context relations in the described photo-centric mode. It needs to be understood that other media such as video, sound and others may be used in place and in addition to pictures or photos.
• the forming of the links and the creation of the object note may happen on a single device, or on a plurality of devices. The forming may happen as a network service provided by a service provider at least one network address. The user may access the network service e.g. to browse, organize and search for pictures.
• the service may then provide means for linking different objects to the picture as described earlier and later in this text.
• the service may then allow and enable the creation of a hyper-object-note.
• These functions of the service may provide the user with a picture signal that can be embodied on a computer readable medium, where the picture signal contains links to different objects and/or summarization of different objects.
• This data attached to the picture signal may have been created by fusing information from various objects and by creating a hyper-object-note.
• FIG. 4 illustrates a way for linking information from relevant objects to a picture according to an example embodiment to create hyper-object- link.
• the user selects 410 a photo stored in the mobile phone, and takes it as a query entry to submit to the system.
• the system After collecting the context data 420 embedded In the image file or extracted from an internal API or external sensor data, the system begins to model these raw data and does data mining to find the potential relationship between photo and other related objects 430 like calendar notes 432, SMS messages or other messages like multimedia messages or instant messages 434, email messages 436, blogs 438 and social media services and the phone book (contact cards) 439.
• a linkage graph is built by correlation computing 440. Two linkage modes may be provided based on this linkage graph. In the simple mode, a background program computes the correlation coefficient (score) of the related objects and may also rank them with a score automatically. This may result in top-N, e.g.
• a user may select a related object to acquire detailed information.
• Such an approach may allow to create a summary or to otherwise show the content 454 of the objects in relation to the picture 452 in order to form a hyper-object-note 450.
• the user can add his own constraint conditions to limit the (ink scope or to raise the degree of association.
• a user selects a photo to the system; the photo is about a relay race. After computing, the system finds four most relevant objects to this photo (email, SMS, calendar, blog). if the user wants to find relevant objects according to his own desire, he could add restrictions.
• Fig. 5 illustrates a way for forming a hyper-object-note for a picture or a photo using relevant objects according to an example embodiment
• Lin is a student of Beijing University of Posts and Telecommunications, and he likes athletics very much. A sports meeting will be held in his university on 2009-7-23. Lin and his classmates Lee, Zhang and Tom will take part in the men's 4*100 meters relay race; Lee, Zhang and Tom may have entries in the phone book 560, as well as a friend Jack. Lin is looking forward the competition and he adds an event 550 on his mobile phone 510 to record this event and remind himself to get plenty of exercise for the match. On the race day, at 10:00 am Lin and his partners stood at the starting line.
• Lin ran the first leg of the relay, and he ran very fast. Andy ran the second leg, Zhang ran the third leg and Lee ran the anchor leg. At the anchor leg, they already had a big lead against overmatched teams. At the finishing line, Lin took a photo 520 with Lee, he wanted to record the emotional moment with his camera phone. At last they won the race. They all felt very happy; at 10:23 Lin received a greeting message 540 from his friend Jack, which said "Congratulations to your victory in the sports meeting! After the match, Lin uploaded this photo from his mobile phone to server; he wanted to share his happiness with his friends and classmates.
• the taken time is extracted first; through this, the photo may be linked to the calendar and the subject, time interval, attendees and location may be found. Through the subject and time interval, the SMS and emails with the relevant subject or time are linked to the photo.
• the comprehension based hyper-object-note is more like a short essay
• the extraction based hyper-object-note is more like a catalogue listing the key contents and providing links to them.
• the two hyper-object-note styles may be also mixed and combined, and other styles may be added. Later, browsing this photo, a user could get a short summary at the first sight to help user grasp as much information as possible in a short time.
• Fig. 6 shows a method for annotating pictures according to an example embodiment.
• GPS Global Positioning System
• the pictures or photos may also be clustered based on density distribution of time and GPS coordinates.
• photo and non-photo objects are linked based on correlation of multiple contexts.
• the correlation computing is based on similarity of contexts of time, location, persons, and event, and so on.
• summary information is generated for the annotation from the linked objects.
• Fig.5 illustrates an architecture for noting pictures according to an example embodiment to create hyper-object-notes.
• the first layer may be called "photo importing and pre-process layer" 710, where the tasks are to import photos 712 from file systems and extract the taken time and position (GPS) data from EXIF parsing 714, possibly with GPS to location conversion 720.
• GPS taken time and position
• the photos may be clustered 718 into small clusters based on time and GPS coordinates. Photos with similar time stamp and GPS coordinates may be mostly related with the same event and attendee, so after pre-clustering it may be possible to recur to the cluster's annotation repository to recommend annotations to other photos.
• the results of the first layer may be stored to a context database 716.
• the second layer may be called “Context Collection layer” 730. Time and location context may not be sufficient to provide cross-object links, and a more semantic context is collected on this layer.
• information from different related sources such as calendar 732 may be used.
• the information is extracted in 734 and associated in 736 with the photo, and user identities may also be determined for this purpose in 738.
• the layer may create so-called semi-annotation for the picture.
• the third layer may be called “Correlation Linking layer” 750.
• information from different sources such as email 754, calendar entries 756 and contact information 758 may be used.
• Email 754, calendar entries 756 and contact information 758 may be indexed in 755, 757 and 759, respectively, and provided as input to a search 760.
• search 760 Through multiple context similarity computing in 764, the correlation between entry object and other potential objects are quantified. By ranking the correlation score, most relevant objects may be selected.
• the results may be stored in a result database 768.
• the fourth layer may be called "Summarization layer" 770.
• Summarizations from the already found relevant objects may be automatically generated.
• the photo 772 (which may be the same photo as 752) and the result of the correlation linking 768 may be used as a source when the relevant documents are retrieved in 774.
• email 778, calendar entries 776 and contact information 780, as well as other information 782 may be selected at least partially in 784.
• Through content filtering in 786, and summarization in 788 a hyper- object-note for the photo is created. The implementation details of the different layer are described in the following.
• Fig. 8 illustrates the process of context collection in the second layer of Fig. 7 according to an example embodiment.
• a first method is automatic context annotation 810.
• GPS can also be translated to text location through position-location name mapping, e.g. by software in 816.
• the time and location contexts may be clustered in 818 and 820 to yield different context clusters 825.
• the extracted and processed information can then be stored in to context data store 830.
• the store 830 may be indexed in 850 to have time, location, person and event indexes 860 for fast searching.
• the context annotations may also be generated semi-automatically 840.
• user may add event information to his calendar 844.
• the event is possibly related to photo 842 due to proximity in time.
• the event can then be extracted in 846 and recommend to the user, and the user may judge whether it's related to the photo. If user confirms the event, then the event may be added as hyper-object-note to the photo.
• the context annotations may also be created manually.
• the user can annotate the person in the photo by writing down the person's name.
• the photo's environment may be annotated by hand and emotion may be tagged e.g. with the help of smilies or other icons or text descriptions of the current emotion, or by other tags.
• emotion may be tagged e.g. with the help of smilies or other icons or text descriptions of the current emotion, or by other tags.
• GPS position
• time context we also cluster the photos by GPS coordinates and time context, and the annotations of a certain cluster may be recommended to user to annotate other photos which belong to the same cluster.
• FIG. 9 illustrates a storage structure of picture contexts in a context database according to an example embodiment.
• a photo context record may include photo-id 920 to identify the photo, a date 922 and time 924 when the photo was taken, a GPS position 926, a location 928 possibly generated from the position, persons related to the photo 930, event information 932, and one or more cluster ids 934 where the photo belongs.
• Different context records 910, 912, 914 and 916 may have completely different field values, or some of the field values may be the same, and realized as references.
• Fig. 10 illustrates a clustering table for information on events and persons according to an example embodiment.
• the event clustering table may have, for example, the following fields for the records 1010, 1012, 1014: cluster id 1020 to identify the cluster, event description 1022 and event count 1024.
• the event count 1024 indicates how many times the event id is used to annotate a photo in this cluster, that is, it can be used to determine how many photos belong to this event cluster.
• the person clustering table may include the following fields for the records 1030, 1032 and 1034: the cluster id 1040, person name 1042 and person count 1044.
• the person name field 1042 may contain one or more names or other tags descriptive of persons, and the person count indicates the number of photos in the cluster.
• Figs. 11 a and 11 b show flow charts of collecting event context and person context according to an example embodiment.
• event context collection is shown.
• a photo is input to the process.
• the clusters have been previously clustered by GPS and time context.
• the user may create a new event tag manually or one may be created automatically in 1118. If there is no an event tag in the cluster, a calendar event with the proper time range is searched in 1 1 18. If there is an event in the calendar matching the time duration 1 120, then the event is extracted 1122 from the calendar and recommended to the user, and the user may accept the recommen- dation in 1 125. If there is no event with a proper range in the calendar, then a new event may be created manually by the user in 1130. Finally, the results may be added to the cluster table in 1 140.
• a photo is input in 1150.
• a cluster id is allocated for the photo in 1152.
• Fig. 12 shows a process for correlation based linking of objects to a picture according to an example embodiment on the third layer of the system architecture.
• an entry object (picture/photo) 1212 is submitted to the system.
• a query is then generated in 1214 to select the metadata of the object as the basic query term from the context database 1210.
• expansion query terms may also be selected in 1216 from the cluster the photo belongs to.
• different types of objects are indexed separately beforehand in 1230 and 1235. These objects include S Ss 1220, emails 1222, contacts 1224, calendar entries 1226 and others 1228 like MMSs, instant messages, social network posts, blog entries and such.
• candidate results 1250 are searched from the potential relevant objects in 1240.
• the correlation computing module 1260 computes and ranks the correlation degree between the entry object and the candidate objects through mu!ti-context similarity computing. Finally, the top N most relevant objects are selected and stored in a database.
• a part of creating the hyper-object-link between objects is multidimensional context correlation computing.
• concept of activity theme is defined.
• the main concept of an activity theme that a photo records can be abstracted to four key dimensions:
• Photos are clustered into clusters, using contexts tags contained in other photos which belong to the same cluster with the current linking photo to supplement the query condition.
• a mail's topic may be relevant to the activity which the photo records only if the time interval between the photo's taken time and email's send/receive time is no more than 5 days:
• the function measures the similarity between photo and email in textual coexistence. If time information is expiicitly recorded in the email's subject part or body part, and the time is between the begin time and end time of the activity that the photo records, it can be deduced that the email may have a strong relation with the photo:
• the purpose 0 f g(tag m ) is to convert the format of time to corresponding type compatible to the mail.
• Person correlation may be determined as follows. An email and photo are related if they refer to the same persons. Regularly, person's information appears in the email's sender/receiver field; in some cases, the persons' name also appears in the mail's body part. So, we calculate the correlation in such an approach: Above, t 8 p «TM refers to the persons' name that is annotated as the tag for the current photo, and to £ *-TM » - « ⁇ TM / refers to the persons' name that is annotated as the tags for the photos which belong to the same cluster with the current photo. The function f( tagp ⁇ ° » ) and f ( ta S won _ ⁇ ⁇ ⁇ : ⁇ ⁇ ) translates the person's name to email address.
• the mail may have strong relation to the photo; also if the people's names which are annotated in other photos which belong to the same cluster with the current photo, the email also has some relation to the theme:
• Location correlation may be determined as follows. If the location name appears in the email body, the correlation may exist:
• ⁇ is as the same as the preceding one in the approach of person correlation computing.
• Event correlation may be determined as follows.
• VSM Vector Space Model
• TF-IDF term frequency- inverse document frequency
• w '-> refers to the weight of the l,h term in Jih mail
• Wself refers to the weight of the event tag of the current photo itself
• w k (2 ⁇ k ⁇ m) refers tQ the weight of ⁇ he k lh event tag Qf the p hotos which belong to the same cluster with the current selected photo.
• n u stands for the number of times the term occurs in the Jih mail. The higher the value of the 'u . the more important the term is.
• Time correlation may be determined as follows.
• the time information is explicitly recorded in the time field.
• the time information is extracted and the time distance between the photo's time tag and the time extracted from the calendar is computed.
• the similarity function is defined as: (begin time ⁇ tag, im[ , ⁇ end time)
• Location correlation may be determined as follows. In many cases, the location name may be explicitly recorded in the corresponding field in the calendar. The location field is examined, and if the location name matches the location tag of the current photo, it can be deduced that the photo and the calendar event may have some correlation. tag,, equals location
• Person correlation may be determined as follows. There may be an attendee field in calendar entries to record the persons' names who will attending the activity. A comparison is made to compute the correlation: if the photo's annotated person name exists in the attendee field, a high score is given, and if the person tags of other photos which belong to the same cluster with the current photo exist in the attendee field, a lower score is given; if no person tag of the cluster exists in the field, a zero score is given. 1, tag pan,, c attendee area
• Event correlation may be determined as follows.
• a subject field in the calendar records the event's summary information.
• a comparison to compute the correlation is made: if the photo's annotated event tag exists in this field, a high score is given, and if the annotated event tags of other photos which belong to the same cluster with the current photo exists in the subject field, a lower score is given; if no event tag of the cluster exist in the field, a zero score is given.
• a picture taken of a person has the context of person, which can be used measures the correlation between the photo and a contact card. If the tag "person name" equals to the record that exists in the contact, the two are related.
• Fig. 13 shows an implementation of forming a hyper-object-note for a picture according to an example embodiment.
• the annotation is based on comprehension of the contents that are used as source.
• the creation of a hyper-object-note comprises three layers.
• the first layer may be called "Content Selection" 1330.
• the relevant objects like SMS or MMS messages or instant messages 1310, emails 1312, contact cards 1314, calendar entries 1316, blog entries 1318 and other sources 1320 are first parsed into text format by the parsers 1332.
• term frequencies in each text file are computed in 1334.
• TF-IDF model is adopted.
• Each text file is expressed as a TF-IDF vector (here, the definition and computational process of TF- IDF is as defined earlier in the text).
• TF-IDF vector the definition and computational process of TF- IDF is as defined earlier in the text.
• the importance of a term to a document increases proportionally to the number of the times a term appears in the document, and is offset by the document frequency in the corpus.
• IDF inverse document frequency
• centroid value of each sentence is computed, the key concept is to calculate the weight arithmetic sum of each centroid term contained in the sentence. Then, if the sentence is in the initial or end position, additional score is given. It is because people usually put the most meaningful sentences in the two positions. Third, top-N ranked sentences are selected as the source material to generate the summarization for annotation and hyper-object-note.
• the second layer may be called “Content Filtering” 1340. Since the selected sentences may overlap or be similar with each other in lexeme, it may be good to filter them first.
• “Stigma word filtering” 1342 may remove the redundant words which have little meanings. Since sentences may start with conjunctions like “but”, “although”, “since” and so on, or verb "say” and its derivatives, or pronouns such as “he”, “she”, and “they”. Sentences with these "stigma words” may lead to discontinuity in summarization, and therefore the score of these sentences is reduced to avoid including them into the summarization.
• the "remove redundancy” module 1344 aims to remove sentences with repetitive information. Redundancy may occur if two sentences refer to the same subject matter.
• MMR Maximum Marginal Relevancy
• the third layer may be called "Cohesion& Coherence Enhancement" 1350.
• the candidate sentences are ordered by e.g. chronological order.
• Another optional technique is to arrange sentences with topically related themes together to reduce the non-fluency in 1352.
• Other natural language processing technologies may be involved to improve the cohesion and coherence in 1354.
• a practical method is to add an introductory sentence for each selected sentence, for example to take the sentence prior to the selected sentence as its introductory sentence.
• the resulting annotation may be output and stored with the photo in 1360.
• Another way to implemented the annotation may be to gather the abstract information of "time/location/person/event" and list the most relevant object for each type of object. The aim is to find the most representative objects to supplement the photo.
• the various features of the embodiments may be implemented as a photo-centric cross-object software system for organizing, management, indexing and retrieving objects, as well as automatically generating summarization from multiple sources such as email, SMS, MMS, instant messages, calendar, contact, blog and wiki, etc. All the concepts, methods, work flow, correlation computing methods and system architectures can be extended to other objects such as music, video, and so on.
• a terminal device may comprise circuitry and electronics for handling, receiving and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the terminal device to carry out the features of an embodiment.
• a network device may comprise circuitry and electronics for handling, receiving and transmitting data, computer program code in a memory, and a processor that, when running the computer program code, causes the network device to carry out the features of an embodiment.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Library & Information Science (AREA)
• Data Mining & Analysis (AREA)
• Databases & Information Systems (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
• Information Transfer Between Computers (AREA)

Priority Applications (5)

Applications Claiming Priority (1)

Publications (1)

Family

ID=44145088

Family Applications (1)

Country Status (5)

Cited By (6)

Families Citing this family (40)

Citations (5)

Family Cites Families (20)

• 2009
  • 2009-12-10 KR KR1020127017854A patent/KR101384931B1/ko active IP Right Grant
  • 2009-12-10 EP EP09851971.3A patent/EP2510460A4/en not_active Ceased
  • 2009-12-10 US US13/509,796 patent/US10567726B2/en not_active Expired - Fee Related
  • 2009-12-10 WO PCT/CN2009/075454 patent/WO2011069291A1/en active Application Filing
  • 2009-12-10 CN CN200980162638.2A patent/CN102741835B/zh not_active Expired - Fee Related

Patent Citations (5)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events