NO330120B1 - Method and apparatus for receiving and modifying image data - Google Patents

Abstract

A method and device for receiving and modifying image, eg video data for film production. Image data, e.g. Video data in a compressed raw format is received from a camera. Metadata representing camera functions or settings is received from a first wireless metadata recording device such as a mobile phone. Metadata representing script data is received from a second wireless metadata recording device such as a portable computer shin. Modified image data is generated based on the received image data and the metadata received, e.g. by color-correcting the image data in dependence on the metadata.

Description

Field of the invention

The present invention relates to a method and a device for receiving and modifying image data, e.g. video data received from a digital film camera. The method and device can be particularly used in film production or television production.

Background for the invention

The filmmaking process is usually divided into five stages:

- development, including writing a manuscript; - pre-production; including preparations for filming;

- production; in which the raw elements of the film are shot ("on set"); and

- post-production, which includes film editing, soundtrack editing, adding effects, etc.; and

- sales and distribution.

The present method and device are particularly applicable for the production phase indicated above.

In existing film or television production, especially in the "on set" phase of production, the film script (the "story") is the basis for decisions made in the workflow.

On a film set, the director and the director of photography materialize the script through their visual interpretation. For around 130 years, film production has been based on analogue film. It is now a growing trend in the industry to switch to digital film recording. Film production, e.g. drama film production involves high costs in a limited shooting period. The captured material has a high economic value. While conventional analog film provides a protected, fixed medium for the original recording, digital film involves more volatile original data that must be preserved securely.

The digital file provided by a digital film camera is typically recorded in a RAW format. This format requires manipulation before it can be displayed or transformed into a format suitable for viewing. Such manipulation requires a type of metadata in order to know how the image is to be presented. During production, metadata is provided, among other things, by the camera assistant and script supervisor.

Today, metadata is created by the staff on the film set, usually written down on paper, and follows the production workflow in paper form or in digital formats that do not communicate with the digital workflow.

Products in the background technique that use manual input of the production staff's metadata are known as Iridas Speedgrade OnSet and 3Cp from Gamma & Density.

There is therefore a general need for methods and devices that can improve the overall film production process, or details of the process.

Summary of the Invention

The method and device according to the present invention are defined by the appended patent claims.

Brief description of the drawings

Embodiments of the invention will be described below with reference to the drawings. Fig. 1 is a schematic block diagram illustrating principles of a device and in a method for receiving and modifying image data. Fig. 2 is a schematic flowchart illustrating principles of a first embodiment and a method for receiving and modifying image data. Fig. 3 is a schematic flowchart illustrating principles of a second embodiment of a method for receiving and modifying image data.

Detailed description of embodiments

Fig. 1 is a schematic block diagram illustrating principles of a device and a method for receiving and modifying image data.

A device 100 for receiving and modifying image data is arranged to receive image data from a camera 110. The received image data can e.g. be video data in a compressed raw format.

The camera 110 may be a digital film camera, i.e. a professional digital video camera with very high resolution, suitable for professional film production. Other digital video cameras may also be applicable.

Video data in compressed raw format contains minimally processed data (ie with minimal loss) from the image sensor in the digital camera. Other video/image formats, including e.g. an uncompressed raw format may also be applicable.

The device 100 for receiving and modifying image data is further arranged to receive metadata from at least one metadata registration device 120, 130.

To this end, the device 100 includes input circuits 140 for receiving raw image data from the camera 110, and for receiving metadata from a

metadata recording device; and processing circuitry (not illustrated) configured to generate modified image data based on the raw image data and metadata.

The at least one metadata recording device may include a camera metadata recording device 120, intended to be operated by e.g. a camera assistant, or a manuscript metadata decoding device 130, intended to be operated by e.g. a script supervisor, or both.

The device 100 for receiving and modifying image data is further arranged to generate modified image data based on the received image data and the received metadata.

To this end, the device 100 for receiving and modifying image data includes processing circuitry (not illustrated), a memory (not illustrated), and associated input/output circuitry (not illustrated).

In one aspect, generation of the modified image data performed by the device 100 includes color correction, or color grading, of the received raw image data depending on the content of the metadata.

In one aspect, the received metadata includes camera functions or settings, indicated as "technical information" in FIG. 1.

Such metadata may include one or more elements selected from the following elements: position data (eg data representing GPS coordinates), acceleration data, eg. gyro data, time data.

In this case, the metadata recording device 120 may be a wireless terminal, and the reception of metadata may include receiving the camera functions or

- the settings via a wireless interface.

Alternatively, a wired interface or other type of interface can be used to transfer the metadata.

The camera function or camera setting metadata can be received from a metadata recording device in the form of a wireless terminal, e.g. a portable computer, a PDA, a mobile phone or the like, e.g. provided with WLAN communication capability or other wireless or wired communication capability. In this case, receiving the metadata includes receiving camera function metadata or camera setting metadata via a wireless interface such as a WLAN interface.

In a particular embodiment, a wireless terminal 120 which has built-in functions to provide at least one of the following data can be used: position data (e.g. provided by a built-in GPS receiver), acceleration data (e.g. provided by a built-in gyro sensor), and time data (e.g. provided by a built-in GPS receiver). This would be particularly useful if the wireless terminal is physically attached to the camera, since the metadata provided by the wireless terminal's built-in functions would be representative of the position and movement of the camera, and of the absolute time.

As an example, the wireless terminal 120 may be an Apple iPhone. Other types of terminals, including portable computers, PDAs, smartphones, mobile phones may be applicable.

In another aspect, the received metadata may include script metadata.

Such script metadata can identify at least one of: a day, a scene (eng.: a scene), a slate and a take (eng.: a take).

The script metadata can also be received from a metadata recording device in the form of a wireless terminal 130, e.g. a portable computer, a PDA, a mobile phone or the like, e.g. provided with WLAN communication capability or another wireless or wired communication capability. In this case, receiving metadata includes receiving script metadata via a wireless interface such as a WLAN interface.

Alternatively, a wired or other type of interface can be used to transfer the metadata.

As an example, the wireless terminal 130 may be a portable Macintosh computer OSX. Other types of terminals, including portable computers, PDAs, smartphones, mobile phones may be applicable.

In the device 100 for receiving and modifying image data, the received metadata can be stored in a database 160, e.g. together with the storage of the received image data.

Although the database 160 has been illustrated as an integral element of the device 100, it should be understood that the database 160 may be external to the device 100. Furthermore, the database 160 may be implemented either as a central database or distributed among any suitable number physical database elements.

The received metadata and the received image data can also be stored in a backup subsystem (eng.: a backup subsystem) 170, which can be equipped with a verification subsystem 172, further connected to external (or internal) storage devices such as a magnetic tape storage (indicated as " LTO tape") 174, a disk array (referred to as RAID; Redundant Array of Inexpensive Disks) 178, or another magnetic, optical or solid state storage device (referred to as "Hard Disk") 176, or any combination thereof.

The generation step, performed by the device 100, may include a de-Bayering step or demosaicing step (eng.: demosaicing step), performed on the received image data. An element representing the de-Bayering or demosaicing step is illustrated at 180. The de-Bayering or demosaicing step involves a pre-transformation of the raw data into observable form.

The generation of the modified image data, in particular color correction, or color grading, of the received raw data, can be performed based on a day, scene, slate or shot, identified by the script metadata. This color correction can be performed by the color correction element 190.

The sub-step of color correction (or color grading) based on metadata, especially script metadata, provides a useful connection of the story (eng.: the story)

(identified by script data) to the digital workflow. As a result, a user is given the opportunity to display and digitally color grade the work in a story context, in an efficient and time-saving way.

The color correction element 190 is further connected to the color profile element 210, which is further connected to the digital display 220.

In addition to the metadata received from the camera recording device 120 and the script data recording device 130, additional metadata may also be received from the camera 110.

Such additional metadata may include camera identification data, lens data, shutter data, camera time data, etc.

The device 100 for receiving and modifying image data can also be arranged to output the modified image data and the metadata, e.g. using the output rendering element (eng.: the output rendering element) 200. The modified image data and the metadata can e.g. is forwarded to a post-production facility 270. The metadata provided to the post-production facility 270 may include XML data, as illustrated at 260.

In one aspect, the image data received by the device 100 for receiving and modifying image data includes a first set of time stamps, and the metadata includes a second set of time stamps. The generation of modified images may then include comparing the first and second sets of time stamps.

In one aspect, the device 100 for receiving and modifying image data may further comprise a subprocess for encoding modified image data into multiple output formats.

The multiple output formats may include formats with different resolutions, and they may be useful for various purposes such as preview, editing, storage, transmission, etc.

The sub-process of encoding the modified image data into multiple video formats may comprise a common decompression sub-step; a common de-Bayering or demosaicing substep; and a common color correction substep; followed by separate encoding substeps. In this context, "common" step denotes a step that is performed only once in the overall coding sub-process. Each encoding substep can be adapted to a specific video format. The division of the total encoding sub-process into common pre-substeps (decompression, de-Bayering or demosaicing, and color correction), which are common to different resulting file formats, and separate encoding sub-steps, each specially adapted to the file format in question, results in a significant reduction of the overall processing -/rendering time (eng.: overall processing/rendering time), compared to the traditional approach of decompression, de-Bayering/demosaicing, color correction and encoding, where all steps are performed separately for each required output format.

The output data may be transferred to one or more of the following: the XML output 260, a network element 259 connected to the Internet, a storage element 240 such as a portable hard disk drive, a display device 230 such as an optical film display.

Fig. 2 is a schematic flow chart illustrating principles of the first exemplary embodiment of a method for receiving and modifying image data.

The method starts at step 400.

Further, in step 410, image data is received from a camera. The image data can be video data in a compressed raw format.

Further, in step 420, metadata is received from a metadata recording device, e.g. including camera functions or camera settings, or script metadata, or both.

Further, in step 430, modified image data is generated based on the received image data and the received metadata, e.g. by color correction of the image data depending on the metadata.

The method ends at step 490.

Fig. 3 is a schematic flowchart illustrating principles of a second exemplary embodiment of a method for receiving and modifying image data.

The same reference numbers have been used for the elements that also appear in fig. 2. The description of elements shown in fig. 2 also applies to the description of the same elements depicted in fig. 3.

The method starts at step 400.

Further, in step 410, image data is received from a camera, e.g. a digital film camera.

Further, in step 420, metadata is received from a metadata recording device, e.g. including camera functions or settings, or script metadata, or both.

In substep 422, metadata including camera functions or settings is received from a first wireless terminal, and the camera functions or settings are received via a wireless interface associated with the first wireless terminal.

Further, in substep 424, metadata including script data is received from a second wireless terminal. The script metadata is received via a wireless interface associated with the second wireless terminal.

Further, in step 430, modified image data is generated based on the received image data and the received metadata. The received image data may include a first set of time stamps, and the metadata includes a second set of time stamps. The generation of the modified image data may include comparing the first and second sets of time stamps.

First in the generation step 430, in substep 432, the received image data is subjected to an initial de-Bayering or demosaicing process.

Further, in substep 434, at least one of: a day, a scene, a slate, and a take is identified from the script metadata.

Furthermore, in sub-step 436, a color correction is performed on the image data depending on the metadata. The color correction is performed based on a day, scene, slate or shot identified by the script metadata.

Further, in sub-step 440, the modified image data and metadata are output and forwarded to a post-production facility. The output metadata may include XML data.

The modified image data can be encoded into multiple formats. The encoding may include a common decompression substep; a common de-Bayering or demosaicing substep; and a common color correction substep; as well as separate encoding sub-steps, each adapted for a particular video format, in the same manner as already explained with reference to FIG. 2.

The metadata can also be stored in a database. The received image data can also be stored in the database. The received image data may also be stored in a backup facility, in the same manner as has already been explained earlier in the present specification.

Various aspects of the invention can, among other things, provide at least some of the following advantages: - the overall workflow of the film production process can be made more efficient, - a user of a film production facility can gain better control over the creative process, - safety and reliability are increased, compared with previous solutions, which assemble digital raw data in film production, - the work of the camera assistant and script supervisor can be done more efficiently, since previous manual operations and tasks are effectively replaced by the use of wireless communications, - management of files resulting from a film production set is done more efficient, which makes it possible to make quick and well-considered decisions based on an early consideration of recorded material,

- digital post-processing can be done more efficiently.

The invention has been described here by way of examples. Numerous variations and alternatives will be apparent to those skilled in the art. Thus, the scope of the invention is set forth by the appended claims and their equivalents.

Claims (19)

1. Method for receiving and modifying image data, comprising: - receiving image data from a camera; - receiving metadata from a metadata recording device; - to generate modified image data based on the image data and the metadata, characterized thereby the received image data includes a first set of time stamps, the metadata includes a second set of timestamps, and the generation of modified image data includes comparing the first and second sets of time stamps. 2. Procedure in accordance with claim 1, where the generation of the modified image data includes color correction of the image data in dependence on the metadata. 3. Procedure in accordance with claim 1 or 2, where the metadata includes camera functions or camera settings. 4. Method according to claim 3, wherein the metadata recording device is a wireless terminal, and wherein the step of receiving the metadata includes receiving the camera functions or camera settings via a wireless interface. 5. Method in accordance with one of the above requirements, where the metadata includes script metadata. 6. The method of claim 5, wherein the script metadata identifies at least one of a day, a scene, a slate, and a shot, and wherein the color correction is performed based on a day, scene, slate, or shot as identified by the script metadata. 7. Method according to claim 5 or 6, wherein the metadata recording device is a wireless terminal, and wherein the step of receiving the metadata includes receiving the script metadata via a wireless interface. 8. Method in accordance with one of the above requirements, where the metadata is stored in a database. 9. A method in accordance with one of the above claims, wherein the generation step includes an initial de-Bayering or demosaicing step performed on the received image data. 10. Method in accordance with one of the above claims, further comprising receiving additional metadata from the camera. 11. Method in accordance with one of the above claims, where the camera is a digital film camera. 12. Method in accordance with one of the above claims, where the image data is video data in a compressed raw format. 13. Method in accordance with one of the above requirements, further comprising providing the modified image data and the metadata. 14. Method according to claim 13, wherein the modified image data and the metadata are delivered to a post-production facility. 15. Method according to claim 13, wherein the metadata delivered to the post-production facility comprises XML data. 16. Method in accordance with one of the above claims, further comprising encoding the modified image data into multiple video formats. 17. Method according to claim 16, wherein the coding to multiple video formats comprises: - a common decompression substep; - a common de-Bayering or demosaicing substep; and - a common color correction substep; and - separate encoding sub-steps, each adapted to a specific video format. 18. Device for receiving and modifying image data, comprising: - first input circuits for receiving image data from a camera; - other input circuits to receive metadata from a metadata recording device; and - processing circuitry to generate modified image data based on the received image data and metadata, characterized by the received image data includes a first set of time stamps, the metadata includes a second set of timestamps, and the generation of modified image data includes comparing the first and second sets of time stamps. 19. Device in accordance with claim 18, adapted to carry out a method as set forth in one of claims 1-17.