WO2016170381A1

WO2016170381A1 - Method for enhancing media content of a picture

Info

Publication number: WO2016170381A1
Application number: PCT/IB2015/000786
Authority: WO
Inventors: Clément PERROT; Robin BARATA; David Zhang; Maxime DOMAIN; Marine DUNOGUIER
Original assignee: Prynt Corp.
Priority date: 2015-04-24
Filing date: 2015-04-24
Publication date: 2016-10-27

Abstract

The present invention relates to a method for enhancing media content of a picture (303), wherein the method comprises: filming a scene (102); taking a picture of said scene (106); storing said taken picture and a video (111), the taken picture being associated with said stored video. Said stored video corresponds to a video captured during the filming of said scene, a start or an end of said stored video being function of a time when said picture is taken.

Description

METHOD FOR ENHANCING MEDIA CONTENT OF A PICTURE

BACKGROUND OF THE INVENTION

The present invention relates to the domain of capturing a picture, especially with a mobile terminal associated with a camera or any advanced camera with processing means. The approaches described in this section could be pursued, but are not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section. Furthermore, all embodiments are not necessarily intended to solve all or even any of the problems brought forward in this section.

For now, when one wants to share a moment with friends or its family, he often takes its camera and chooses to capture this moment by taking a picture of by taking a short film: the user needs to choose between these two actions as the technologies used for these types of actions are very different. Nevertheless, this decision is irreversible.

Thus, there is a need to introduce more flexibility in the creation of media contents.

SUMMARY OF THE INVENTION

The invention relates to a method for enhancing media content of a picture, wherein the method comprises: - filming a scene;

- upon a reception an input of a user for taking a picture of said scene, taking the picture of said scene ;

- storing said taken picture and a video, the taken picture being associated with said stored video.

Said stored video corresponds to a video captured during the filming of said scene, a start or an end of said stored video being function of a time when said picture is taken.

Thus, it is possible to enhance the media content associated with a given picture by associating to that picture, for instance, a short video of the scene prior to the moment the picture have been taken. The short video may have a start prior to that moment (e.g. 5 seconds before) and an end after that moment (e.g. 2 seconds after).

The filming of the video may be transparent for the user and thus easily enhance the media content without a specific manipulation of the user.

The storage of the picture and the video may be performed locally (e.g. on a smartphone) or after a transmission to a distant equipment (e.g. a server).

Taking a picture may be performed by clicking on a button / electronic button on a smartphone for instance, but it may also be performed by choosing/selecting a frame in a set of frames of a video : thus in these cases, the input is the click or the selection of the user.

In a possible embodiment, the method may further comprise:

- during the filming of said scene, creating a plurality of successive temporary videos with last filmed events, said video being limited to a predetermined duration. The stored video may thus be the last created temporary video.

Then, it is possible to dynamically crop the filmed media so that this temporary media is at most of a predetermined duration (e.g. 5 seconds). Thus the local memory is not filled with unneeded content especially if the user takes its time to take the picture.

In addition, the method may further comprise:

- after the picture is taken, cropping a temporary video created during the filming so that the cropped video is limited to a predetermined duration. The stored video may be then the cropped video.

In this embodiment, the filmed media is cropped at the end of the process. It may enable the user to determine what video / picture he wants to store

In a possible embodiment, the method may further comprise:

- identifying in a second picture/video a match for said taken picture;

- providing the video associated with said taken picture.

The identification may be performed thanks to an image recognition algorithm (for instance based on "points of interest" recognition). The second picture/video may be received through the camera of a mobile terminal of a given user (i.e. scanning). The match may correspond to only a part of the second picture/video.

Thus, by "scanning" a printed picture with a mobile terminal, it is possible to retrieve the video associated with said picture and play it. In order to ease the identification, it is possible to identify the edges (for instance delimited by a black line) of the printed picture in the second picture/video or any other forms that could characterized a printed picture (e.g. a logo, a QR code, a watermark, etc.).

In addition, the identification of the match may comprise:

- computing a first signature of the taken picture;

- computing a second signature of a subpart of the second picture/video;

- comparing the first and second signature.

The first signature may be computed when the taken picture is stored or may be computed on the fly when needed. The signature computed is function of the image recognition algorithm used. The method may also further comprise:

- if a plurality of possible matches is identified, selecting, as the identified match, a match in said matches according to a criterion based on at least one of:

- a geographical information associated with each of the possible matches;

- a user identification information associated with each of the possible matches; - a date/time information associated with each of the possible matches;

- a social network information associated with each of the possible matches;

- a rate information associated with each of the possible matches. Therefore, it is possible to identify the most relevant match between all the identified matches.

The method may further comprise encrusting the provided video in the second image/video. Thus, this encrustation is an intuitive way to display the retrieved video.

The end of said stored video may be the time when the picture is taken.

Thus, the stored video is a video of the events occurring just before the picture is taken: this video shows the contextual environment of the picture. In addition, identifying in a second picture/video a match for said taken picture may comprise: hi searching a local cache for said match; h\l querying a distant server for returning said match.

Step h\l may be conditioned by the fact that search of step hi is unsuccessful. In another embodiment, steps hi and h\l may be performed in parallel.

Moreover, step h\l may further comprise:

- receiving from the distant server a list of related pictures and updating the local cache with said related pictures.

Another aspect of the invention relates to a system for enhancing media content of a picture, the system comprising:

- a mobile terminal comprising:

- a camera for filming a scene;

- a camera for taking a picture of said scene;

- a storage media for storing said picture and a video, the picture being associated with said stored video.

Said stored video corresponds to a video captured during the filming of said scene, a start or an end of said video being function of a time when said picture is taken.

The camera for filming and the camera for taking the picture may be the same.

The storage media may be on a distant server distinct from the mobile terminal. Then, the mobile terminal and the distant server have communication units (i.e. interface such as WiFi, GSM and/or Ethernet communication module).

Thus the mobile terminal may also a processor to manage the filming, the taking of the picture, the storage and the picture and video manipulation.

A third aspect relates to a computer program product comprising a computer readable medium, having thereon a computer program comprising program instructions. The computer program is loadable into a data-processing unit and adapted to cause the data-processing unit to carry out the method described above when the computer program is run by the data-processing unit.

Other features and advantages of the method and apparatus disclosed herein will become apparent from the following description of non-limiting embodiments, with reference to the appended drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings, in which like reference numerals refer to similar elements and in which:

- Figure 1 is a flow chart describing a possible embodiment of the present invention when the picture is taken;

- Figure 2 is a representation of flow chart describing a possible embodiment of the present invention when the video is retrieved to be encrusted; - Figure 3 is a representation of a mobile terminal with an encrustation of a video;

- Figure 4 is a possible embodiment for a device that enables the present invention. DESCRIPTION OF PREFERRED EMBODIMENTS

Figure 1 is a flow chart describing a possible embodiment of the present invention when the picture is taken.

The following flow charts are described assuming that the invention is implemented into an application installed on a smartphone. Nevertheless, other embodiments may be possible.

When the application is launched (101 ) or when a given trigger is activated in the application, the camera of the smartphone is activated and the recording of a video is started (step 102). Optionally, and during the recording, it is tested if the recorded video (that is temporarily stored on the memory 103 of the smartphone) has a length greater than a predetermined duration (e.g. 5 seconds). If its length is greater that this predetermined duration (test 104, output KO), the recorded video is cropped so that its duration is limited to the predetermined duration (step 105). Thus, only the last seconds of the recorded video is kept on the memory.

In one embodiment, when the application detects that a button is pressed by the user, a picture is taken (step 106) and the recording of the video ends. In a second embodiment, when the application detects that a button is pressed by the user, the recording of the video ends and the user is prompted to select in the frames of the recorded video the frame he wants to select for the picture. In the two possible embodiment described, it is also possible that the recording continues a couple of second after the user pressed the button.

Alternatively, the user may also select any picture from various sources (such as Facebook, Instagram, the camera roll of the mobile terminal, etc.) and select a corresponding video from various sources (for instance by filming a short video).

Once the picture is taken (or selected), it is possible to print the picture (step 107), for instance with a device attached to the smartphone as described in the application FR 15 50027.

Thus, the video is retrieved (step 108) from the memory 103 of the mobile terminal.

Optionally, if no cropping has been previously performed to limit the length of the recorded video in a previous step, and if this length is greater than a predetermined duration, the recorded video is now cropped so that its duration is limited to this predetermined duration (step 109).

In addition, it is also possible to collect (step 1 10) metadata related to the picture and video. For instance, these metadata may be:

- a location of the mobile terminal when the picture is taken (through the GPS unit of the mobile terminal or through WiFi/GSM triangulation) ; - a date/time corresponding to the moment the picture is taken ;

- an id of the user currently logged on the application ;

- an id of the picture taken ;

- a comment inputted by the user;

- an id of a filter applied on the taken picture; - a privacy setting ;

- information related to the source of the picture (e.g. camera roll, facebook, Instagram, camera, etc.)

- etc.

Once the picture, the video and optionally the metadata identified, it is possible to transmit them to a distant server (step 1 1 1 ).

Optionally, it is also possible to compute (before or after step 1 1 1 ) a signature of the picture, signature which is computed for a given image recognition algorithm (e.g. Recognize. im, Pongr, Kooaba, Cortexika, Google Goggles, IQEngine, Moodstocks, Smartsy or Vuforia). Then, the picture is stored on the memory/disk/database (e.g. the disk of the distant server) in association with the video, the metadata and the signature (if existing) (step 1 13). It is finally possible to apply some restrictions (step 1 14) to the stored files to avoid these files being public. The restrictions may be for instance "accessible only by the user who took the picture", "accessible to the family members of the user who took the picture", "accessible to the friends of the user who took the picture", "accessible to the users within a predefined distance to the location where the picture have been taken", "accessible only for N views" (with N a predetermined number), etc.

Figure 2 is a representation of flow chart describing a possible embodiment of the present invention when the video is retrieved in order to be encrusted. When the application is launched (201 ) or when a given trigger is activated in the application, the camera of the smartphone is activated and the captured image/video is dynamically analyzed (step 202) to detect a known image.

First, the local memory of the smartphone is analyzed to search for previous taken pictures from the application or other pictures in the memory of the phone (e.g. the camera roll) (step 203). A signature may be computed, if needed, in order to ease this search according to the image recognition algorithm used in reference of Figure 1 .

In order to ease the process, it is also possible (not shown on the figure) to pre- upload on the smartphone (i.e. in a local cache) some pictures (and the associated metadata / video) from a distant server (e.g. cloud servers), these pictures being identified as the most probable pictures the current user of the smartphone may scan. For instance, these pre-uploaded pictures may be the N last pictures that his friends (e.g. identified through a Facebook account) have taken (with N a predetermined number). Then, there is no need of an Internet connection to be able to scan a picture and to identify the associated video.

If the image recognition algorithm does not find any match in the local memory (test 204, output KO), the image or the computed signature of the image may be transmitted to the distant server (step 205). It is noted that steps 203 and 205 may be performed in parallel (i.e. without the test 204): the first returned result or the most accurate result is thus used. Thus, the distant server may be able to determine if a picture matches the transmitted image or signature according to the image recognition algorithm (step 206).

If more than one picture match the transmitted image or signature (test 207, output KO), the most probable / relevant picture is selected. This selection may be for instance based on at least one of these criteria: the distance between the location information of the matching pictures (in the metadata, see above) and a location of the mobile terminal which is currently requesting the identification of the image / signature : for instance, the shorter the distance is, the more probable the picture may be ; the distance between the date/time information of the matching pictures (in the metadata, see above) and current date/time : for instance, the shorter the distance is, the more probable the picture may be ; the number of printing of the matching pictures ; the relation between the users that have taken the matching pictures and the user which is currently requesting the identification of the image / signature (e.g. friends in Facebook, etc.) ; the rate of the matching pictures (i.e. the rate of correspondence between a picture and a signature) ; etc.

The selection may be a weighted function of various criteria.

If only one picture matches the transmitted image or signature (test 207, output OK), this picture is simply selected.

Then, once the picture is selected, the rights / restrictions apply to said picture are checked to ensure that the user requesting the picture is allowed (step 209) to retrieve said picture and the video associated to.

It is also possible to implement this step 209 in the selection described in step 208 to select the most probable / adequate picture that the user is allowed to retrieve. It is also possible, optionally, to update the metadata associated with the selected picture (e.g. to update the number of views or the list of users having identified this selected picture, to update the list of image in which the selected picture have been identified, to update the list of location where the selected picture have been identified, to update the list of date/time when the selected picture have been identified) (step 210). In addition, the video associated to the selected picture may also be updated if, for instance, the current user requesting an identification of said picture is the user who originally took the picture.

Then, the video associated to the selected picture may be sent to the user (step 21 1 ). In addition, the selected picture and its associated signature may also be sent to the mobile terminal so that they can be stored in the local cache of the mobile terminal. This storing may ease a future recognition step (see above step 203).

In a possible embodiment, it is also possible to send other pictures (and their signatures) related to said selected picture to the mobile terminal so that they can be stored in the local cache of the mobile terminal. Said other pictures can be related to the selected picture because:

- they have been taken in the same location than the selected picture;

- they have been taken by the same user;

- they have been taken at the same date / time; - etc.

A preview of the sent video may then be inserted / encrusted (step 212) in the image 301 captured by the smartphone 300. For instance, in relation of figure 3 (which is a representation of a mobile terminal with an encrustation of a video), an printed image 303 lying on a table 302 has been captured, and has been recognized by the recognition image process and the smartphone have added a "play" button 304 on the printed image 303, so that the user may click on that button, and the preview of the sent video replaces the actual recognized image.

If the button 304 is pressed (test 213), the video is played (step 214) in the frame of the printed image (i.e. with the deformation due to the perspective, encrusted / inserted in the frame of the printed image 303) or in "full screen" (at least on a rectangular zone of the screen of the smartphone, i.e. without perspective deformation).

Figure 4 is a possible embodiment for a device that enables the present invention.

In this embodiment, the device 400 comprise a computer, this computer comprising a memory 405 to store program instructions loadable into a circuit and adapted to cause circuit 404 to carry out the steps of the present invention when the program instructions are run by the circuit 404. The memory 405 may also store data and useful information for carrying the steps of the present invention as described above.

The circuit 404 may be for instance:

- a processor or a processing unit adapted to interpret instructions in a computer language, the processor or the processing unit may comprise, may be associated with or be attached to a memory comprising the instructions, or

- the association of a processor / processing unit and a memory, the processor or the processing unit adapted to interpret instructions in a computer language, the memory comprising said instructions, or - an electronic card wherein the steps of the invention are described within silicon, or

- a programmable electronic chip such as a FPGA chip (for « Field- Programmable Gate Array >>).

This computer comprises an input interface 403 for the reception of data used for the above method according to the invention (i.e. picture and video data from a camera 410 and other data from a receiving unit 41 1 , such as a wireless communication unit) and an output interface 406 for connecting to a transmission unit 407. The receiving unit 41 1 and the transmission unit 407 can be merged into a single communication unit.

To ease the interaction with the computer, a screen 401 and a keyboard 402 may be provided and connected to the computer circuit 404.

The computer device may be miniaturized and the device may be a tablet and/or a mobile terminal such as a smartphone.

Part of the flow charts described in Figures 1 and 2 can represent steps of an example of a computer program which may be executed by the above mentioned device.

Expressions such as "comprise", "include", "incorporate", "contain", "is" and "have" are to be construed in a non-exclusive manner when interpreting the description and its associated claims, namely construed to allow for other items or components which are not explicitly defined also to be present. Reference to the singular is also to be construed in be a reference to the plural and vice versa. A person skilled in the art will readily appreciate that various parameters disclosed in the description may be modified and that various embodiments disclosed may be combined without departing from the scope of the invention.

Claims

1 . A method for enhancing media content of a picture (303), wherein the method comprises:

- filming a scene (102); - upon a reception an input of a user for taking a picture of said scene, taking the picture of said scene (106);

- storing said taken picture and a video (1 1 1 ), the taken picture being associated with said stored video; wherein said stored video corresponds to a video captured during the filming of said scene, a start or an end of said stored video being function of a time when said picture is taken.

2. The method according to claim 1 , wherein the method further comprises:

- during the filming of said scene, creating (105) a plurality of successive temporary videos with last filmed events, said video being limited to a predetermined duration ; wherein the stored video is the last created temporary video.

3. The method according to claim 1 , wherein the method further comprises: - after the picture is taken, cropping (109) a temporary video created during the filming so that the cropped video is limited to a predetermined duration ; wherein the stored video is the cropped video.

4. The method according to one of the preceding claims, wherein the method further comprises:

- identifying (202) in a second picture/video a match for said taken picture; - providing (21 1 ) the video associated with said taken picture;

5. The method according to claim 4, wherein the identification of the match comprise:

- computing a first signature (1 12) of the taken picture;

- computing a second signature (202) of a subpart of the second picture/video;

- comparing the first and second signature (203, 206).

6. The method according to one of the claims 4 and 5, wherein the method further comprises:

- if a plurality of possible matches is identified (207), selecting (208), as the identified match, a match in said matches according to a criterion based on at least one of:

- a geographical information associated with each of the possible matches;

- a user identification information associated with each of the possible matches;

- a date/time information associated with each of the possible matches;

- a social network information associated with each of the possible matches;

- a rate information associated with each of the possible matches.

7. The method according to one of the claims 4 to 6, wherein the method further comprises:

- encrusting (212) the provided video in the second image/video.

8. The method according to one of the preceding claims, wherein the end of said stored video is the time when the picture is taken.

9. The method according to one of the preceding claims, wherein identifying (202) in a second picture/video a match for said taken picture comprises: hi searching a local cache for said match; h\l querying a distant server for returning said match.

10. The method according claim 9, wherein step h\l further comprises: - receiving from the distant server a list of related pictures and updating the local cache with said related pictures.

1 1 . A system for enhancing media content of a picture, the system comprising:

- a mobile terminal comprising:

- a camera for filming a scene;

- a camera for taking a picture of said scene;

- a storage media for storing said picture and a video, the picture being associated with said stored video; wherein said stored video corresponds to a video captured during the filming of said scene, a start or an end of said video being function of a time when said picture is taken.

12. A non-transitory computer readable storage medium, having stored thereon a computer program comprising program instructions, the computer program being loadable into a data-processing unit and adapted to cause the data-processing unit to carry out the steps of any of claims 1 to 10 when the computer program is run by the data-processing device.