WO2015107257A1 - Method and apparatus for multiple-camera imaging - Google Patents

Method and apparatus for multiple-camera imaging Download PDF

Info

Publication number
WO2015107257A1
WO2015107257A1 PCT/FI2014/050033 FI2014050033W WO2015107257A1 WO 2015107257 A1 WO2015107257 A1 WO 2015107257A1 FI 2014050033 W FI2014050033 W FI 2014050033W WO 2015107257 A1 WO2015107257 A1 WO 2015107257A1
Authority
WO
WIPO (PCT)
Prior art keywords
camera
metadata
scene
optical
processor
Prior art date
Application number
PCT/FI2014/050033
Other languages
French (fr)
Inventor
Riku Suomela
Leo Kärkkäinen
Sampo VESA
Marja Salmimaa
Markku OIKKONEN
Original Assignee
Nokia Technologies Oy
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nokia Technologies Oy filed Critical Nokia Technologies Oy
Priority to PCT/FI2014/050033 priority Critical patent/WO2015107257A1/en
Publication of WO2015107257A1 publication Critical patent/WO2015107257A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/0007—Image acquisition
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Definitions

  • Photographing has developed with increasingly rapid pace following the advancements provided by digitized photographing and image processing. For instance, 3D and augmented reality footage are shot with two or more cameras. Moreover, different views can be enabled by providing users with multiple views of e.g. a sports event.
  • multiple-camera broadcast In real-time multiple-camera broadcast, numerous live video streams are transmitted.
  • the user can be provided e.g. thumb-nail video images representing the available video streams so that the user can easily and intuitively select any of the views based on her own preferences.
  • different video footages are recorded typically by different devices.
  • a multiple-camera recording is then assembled from the different recordings. This process can be performed e.g. manually by grouping sequences of frames and mutually aligning them with a graphical user interface, e.g. by dragging with a computer mouse.
  • an apparatus comprising: a processor configured to obtain metadata configured to assist multiple-camera imaging of a scene; and
  • an optical metadata distributor configured to distribute an optical indication of the metadata via the scene under control of the processor.
  • the scene may consist of one or more image objects that are being imaged by multiple cameras. Different cameras may have different views such that different cameras may be directed or focused to different parts of the scene.
  • the optical metadata distributor may comprise a projector configured to distribute the metadata in an automatically recognizable form.
  • the automatically recognizable form may comprise any of text; bar code; two-dimensional bar code; other symbols; and any combination thereof.
  • the optical indicator needs not to be a static, but instead the optical indicator may be dynamic i.e. changing in course of time.
  • the optical metadata distributor may comprise a pulsating light source configured to cast on the scene pulsating light that is indicative of the metadata.
  • the pulsating may be modulated by amplitude modulation; pulse width modulation; pulse-position modulation; pulse-duration modulation; and / or on-off keying.
  • the pulsating light may vary with a frequency less or equal to a frame rate of cameras used for imaging the scene.
  • the optical metadata distributor may be configured to distribute any of: visible light; infra-red light; and ultraviolet light.
  • the optical metadata distributor may be configured to distribute metadata in two or more parts.
  • One metadata part may comprise any of: a time reference; scene description; geographic location; resource locator associated to the scene; imaging parameters for guiding cameras imaging the scene; and timing information (such as interval) of subsequent metadata parts.
  • Another metadata part may comprise any of: a timing indicator; a delta-coded indication of change in the location, imaging parameters or timing information; and any combination thereof.
  • the optical metadata distributor may be controlled by the processor to indicate the metadata via a surface of the scene.
  • the apparatus may further comprise a camera.
  • the processor may be further configured to identify the surface using the camera.
  • the processor may be further configured to detect when the camera is being used for imaging and to identify the surface so that the surface is not in focus of the camera, if the distributed metadata is visible to the camera.
  • the processor may be further configured to geometrically correct the distribution of the metadata on the surface taking into account the geometry of the surface with relation to the optical metadata distributor.
  • the geometry may comprise the shape and / or angle of the surface with regard to the optical metadata distributor.
  • the processor may be further configured to detect focused portion of the scene and to identify the surface at or near the focused portion.
  • the imaging may refer to capturing of video image or a series of sub-sequent still images with at least partial overlap.
  • the metadata may comprise at least one of: geographic position of the apparatus or of the scene; resource locator associated to the scene; date; time; name of the scene; description of the scene; and camera settings.
  • the processor may be configured to automatically obtain the metadata.
  • the apparatus may comprise a positioning circuitry configured to determine the position of the apparatus. The positioning may refer to determining the position of the apparatus in three dimensions (e.g. Cartesian x-y-z dimensions) and to additionally determining the orientation of the apparatus around one or more different dimensions (e.g. Cartesian x-y-z dimensions).
  • the processor may further configured to determine or estimate position of the scene based on the position of the apparatus and estimated direction and distance to the scene.
  • the processor may further configured to determine or estimate position of the scene based on content of images captured by the camera, such as the time and position of the Sun and / or of the moon; the horizon; and orientation of walls or roofs of buildings.
  • the apparatus may further comprise a user input for receiving the metadata or a portion thereof from a user of the apparatus.
  • an apparatus comprising:
  • a memory configured to store camera captured image information of a scene
  • a processor configured to obtain metadata from the image information.
  • the apparatus may further comprise a camera configured to form the image information by imaging of the scene.
  • the processor may be further configured to control the operation of the camera based on the metadata.
  • the processor may be further configured to control, based on the metadata, at least one of: focusing of the camera; focal length of the camera; white balance of the camera; resolution of the imaging performed by the camera; field of view of the imaging performed by the camera; time base of the camera; and illumination of the scene by the camera and/or by other common or separate illumination equipment.
  • the camera and the optical metadata distributor may be configured to dynamically allocate a frequency band to be used for distributing the metadata and for detecting the metadata.
  • the dynamic allocation may be based on one or more criteria.
  • the criteria may comprise any of: visibility of an optical metadata indication in prevailing conditions; predetermined scheme of using different ranges for given types of metadata; and availability of possible frequency bands.
  • the camera may be configured to operate with visible light. Alternative and / or additionally, the camera may be configured to operate with invisible light.
  • the processor may be configured to modify the image information so as to conceal the metadata if visible in images defined by the image information.
  • the processor may be configured to attach the metadata to the image information as separate data.
  • the processor may be configured to attach the metadata to the image information by watermarking.
  • the distributing of the optical indication may comprise projecting the metadata in an automatically recognizable form.
  • the automatically recognizable form may comprise any of text; bar code; two-dimensional bar code; other symbols; and any combination thereof.
  • the optical indicator needs not to be a static, but instead the optical indicator may be dynamic i.e. changing in course of time.
  • the method may further comprise identifying using a camera a surface of the scene on which the optical indication is to be distributed.
  • the method may further comprise detecting when the camera is being used for imaging and identifying the surface so that the surface is not in focus of the camera, if the distributed metadata is visible to the camera.
  • the method may further comprise geometrically correcting the distribution of the metadata on the surface taking into account the geometry of the surface with relation to the optical metadata distributor.
  • the method may further comprise detecting focused portion of the scene and identifying the surface at or near the focused portion.
  • the method may further comprise automatically obtaining the metadata.
  • the method may further comprise determining the position of the apparatus.
  • the method may further comprise determining or estimating a position of the scene based on the position of the apparatus and estimated direction and distance to the scene.
  • the method may further comprise determining or estimating a position of the scene based on content of images captured by the camera.
  • the method may further comprise receiving the metadata or a portion thereof from a user of the apparatus.
  • the method may further comprise forming the image information by imaging of the scene with a camera.
  • the method may further comprise controlling the operation of the camera based on the metadata.
  • the controlling of the operation of the camera may comprise controlling at least one of: focusing of the camera; focal length of the camera; white balance of the camera; resolution of the imaging performed by the camera; field of view of the imaging performed by the camera; time base of the camera; and illumination of the scene by the camera and/or by other common or separate illumination equipment.
  • the method may further comprise dynamically allocating a frequency band to be used for distributing the metadata and for detecting the metadata.
  • the dynamic allocation may be based on one or more criteria.
  • the criteria may comprise any of: visibility of an optical metadata indication in prevailing conditions; predetermined scheme of using different ranges for given types of metadata; and availability of possible frequency bands.
  • the camera may be configured to operate with visible light. Alternative and / or additionally, the camera may be configured to operate with invisible light.
  • the method may further comprise modifying the image information so as to conceal the metadata if visible in images defined by the image information.
  • the method may comprise attaching the metadata to the image information as separate data.
  • the method may comprise attaching the metadata to the image information by watermarking.
  • obtaining metadata configured to assist multiple-camera imaging of a scene; and distributing an optical indication of the metadata via the scene;
  • a computer program comprising computer executable program code for:
  • the computer program may be stored in a memory medium.
  • the memory medium may be a non-transitory memory medium.
  • the memory medium may comprise a digital data storage such as a data disc or diskette, optical storage, magnetic storage, holographic storage, opto-magnetic storage, phase-change memory, resistive random access memory, magnetic random access memory, solid-electrolyte memory, ferroelectric random access memory, organic memory or polymer memory.
  • the memory medium may be formed into a device without other substantial functions than storing memory or it may be formed as part of a device with other functions, including but not limited to a memory of a computer, a chip set, and a sub assembly of an electronic device.
  • Fig. 1 shows an architectural overview of a system of an example embodiment of the invention
  • Fig. 2 shows an apparatus according to an example embodiment, which apparatus is suitable for use as a first apparatus of Fig. 1 ;
  • Fig. 3 shows an apparatus according to an example embodiment, which apparatus is suitable for use as a second apparatus of Fig. 1 ;
  • Fig. 4 shows a flow chart illustrating a process in the first apparatus of Fig. 1 , according to an example embodiment
  • Fig. 5 shows a flow chart illustrating a process in the second apparatus of Fig. 1 , according to an example embodiment
  • Fig. 6 shows a flow chart illustrating another process of an example embodiment e.g. in the first apparatus of Fig. 1 ;
  • Fig. 7 shows a flow chart illustrating another process of an example embodiment e.g. in the second apparatus of Fig. 1.
  • Fig. 1 shows an architectural overview of a system 100 of an example embodiment of the invention.
  • the system comprises a first apparatus 1 10, a plurality of second apparatuses 120 that comprise cameras, an optical metadata distributor 140 (or more than one in some example embodiments) configured to distribute optical indication 150 of the metadata via a scene 160.
  • the optical metadata distributor 140 is here drawn separately, as it can be built as a separate unit configured to communicate with the first apparatus over a wired or wireless connection (universal serial bus, Bluetooth, wireless local area network, infrared), although in another example embodiment, the optical metadata distributor 140 is comprised by the first apparatus 1 10.
  • Fig. 1 is simplified for easier illustration of at least one image object (the drawn house or details thereof) and a relatively uniform surface (for reflecting the optical indication 150) near the house such as a pavement.
  • system 100 also comprises a plurality of metadata distributors 140 each under control of the first apparatus 1 10.
  • the scene consists of one or more image objects that are being imaged by multiple cameras (comprised by the second apparatuses). Different cameras may have different views such that different cameras may be directed or focused to different parts of the scene.
  • the first apparatus 1 10 need not have different structure than the second apparatuses 120.
  • One apparatus is denoted as the first apparatus 1 10 for the purpose of simplifying this description.
  • the second apparatuses need not all be identical or similar by their structures.
  • the first apparatus comprises two or more cameras, wherein one of the cameras is dedicated to facilitating the optical metadata distribution and at least one other camera is dedicated for taking normal digital images (e.g. for a situation in which the first apparatus 1 10 operates as the second apparatus).
  • Fig. 2 shows an apparatus 200 according to an example embodiment, which apparatus 200 is suitable for use as the first apparatus 1 10 and comprises a processor 210, a memory 220 for storing metadata 222 that is configured to assist multiple-camera imaging of the scene 160; and an optical metadata distributor 140 configured to optically cast the optical indication 150 of the metadata 222 via the scene 160 under control of the processor 210.
  • the memory 220 further comprises computer program code or software 224 configured to control the operation of the apparatus 200 when executed by the processor 210.
  • Fig. 2 is further drawn to contain a camera 130 as in an example embodiment, the first apparatus is also capable of operating as the second apparatus.
  • the metadata 222 comprises at least one of: geographic position of the first apparatus 1 10 or of the scene 160; date; time; name of the scene; description of the scene; and camera settings (e.g. desired settings for use by second apparatuses).
  • the processor 210 is configured to automatically obtain the metadata 222.
  • the first apparatus 1 10 comprises a positioning circuitry configured to determine the position of the first apparatus 1 10. In this context, the positioning may refer to determining the position of the apparatus in three dimensions (e.g. Cartesian x-y-z dimensions) and in an example embodiment to additionally determining the orientation of the apparatus around one or more different dimensions (e.g. Cartesian x-y-z dimensions).
  • the processor 222 can be further configured to determine or estimate a position of the scene 160 based on the position of the first apparatus 1 10 and estimated direction and distance to the scene 160 (e.g. using a compass and focusing based distance estimation).
  • the processor is additionally or alternatively configured to determine or estimate position of the scene based on content of images captured by the camera, such as the time and position of the Sun and / or of the moon; the horizon; and orientation of walls or roofs of buildings.
  • the first apparatus 1 10 further comprises a user input for receiving the metadata 222 or a portion thereof from a user of the first apparatus 1 10.
  • the optical metadata distributor 140 comprises a projector configured to distribute the optical indication 150 in an automatically recognizable form.
  • the automatically recognizable form comprises, for example, any of: text; bar code; two-dimensional bar code; other symbols; and any combination thereof.
  • the optical metadata distributor 140 comprises a pulsating light source configured to cast on the scene 160 pulsating light that forms the optical indication 150.
  • the pulsating may be modulated by any of: amplitude modulation; pulse width modulation; pulse-position modulation; pulse-duration modulation; on-off keying; and any combination thereof.
  • the pulsating light may vary with a frequency less or equal to a frame rate of cameras used for imaging the scene 160.
  • the optical metadata distributor 140 is configured to distribute any of: visible light; infra-red light; and ultraviolet light.
  • the optical metadata distributor 140 is configured to the optical indication 150 in two or more parts.
  • One indication part comprises, for example: a time reference; scene description; contextual sensor data recorded by the first apparatus 1 10; geographic location; resource locator associated to the scene; imaging parameters for guiding cameras imaging the scene; timing information (such as interval) of subsequent indication parts; and any combination thereof.
  • the another indication part comprises: a timing indicator; a delta-coded indication of change in the location, imaging parameters or timing information; and any combination thereof.
  • the optical metadata distributor 140 is controlled by the processor to cast the optical indication 150 via a surface of the scene.
  • the apparatus further comprises a camera.
  • the processor is further configured to identify the surface using the camera.
  • the processor is further configured to detect when the camera is being used for imaging and to identify the surface so that the surface is not in focus of the camera, if the distributed optical indication 150 is visible to the camera.
  • the processor is further configured to geometrically correct the distribution of the optical indication 150 on the surface taking into account the geometry of the surface with relation to the optical metadata distributor 140.
  • the geometry comprises, for example, the shape and / or angle of the surface with regard to the optical metadata distributor 140.
  • the processor is further configured to detect focused portion of the scene and to identify the surface at or near the focused portion.
  • the imaging refers to capturing of video image or a series of sub-sequent still images with at least partial overlap.
  • Fig. 3 shows an apparatus 300 according to an example embodiment, which apparatus 300 is suitable for use as the second apparatus 120.
  • the apparatus 300 comprises a processor 310; a memory 320 configured to store camera captured image information 322 of the scene 160 (comprising the optical indication 150) and to store computer program code or software 324 for controlling the apparatus 300 when run by the processor 310; and a camera 130 configured to form the image information by imaging of a scene.
  • the apparatus 300 lacks the camera 130 and instead has a remote camera interface 330 (e.g. universal serial bus, Bluetooth, wireless local area network, and / or HDMI interface) configured to receive the image information from an external camera 130.
  • a remote camera interface 330 e.g. universal serial bus, Bluetooth, wireless local area network, and / or HDMI interface
  • the processor 310 is further configured to control the operation of the camera based on the metadata 222 that is obtained from the optical indication 150.
  • the processor 310 can be further configured to control, based on the metadata 222, at least one of: focusing of the camera; focal length of the camera; white balance of the camera; resolution of the imaging performed by the camera; field of view of the imaging performed by the camera; viewing direction of the camera; time base of the camera; and illumination of the scene by the camera and/or by other common or separate illumination equipment.
  • the camera 130 is configured to operate with visible light.
  • the camera 130 can be configured to operate with invisible light.
  • the apparatus 300 comprises or uses two or more cameras 130 for obtaining image information using visible light and invisible light.
  • the processor 310 can be configured to modify the image information so as to conceal the optical indication 150 if visible in images defined by the image information.
  • the processor 310 is configured to attach the metadata 222 to the image information as separate data. Alternatively or additionally, the processor 310 is configured to attach the metadata 222 to the image information by watermarking.
  • the optical indication 150 invisibly e.g. as a bar code, 2-dimensional bar code, quick response code, or other machine readable form.
  • the projection can be made using infrared, ultraviolet or other invisible wavelength that is detectable with camera sensor, and / or using brief pulses that short that the optical indication 150 appears invisible to human eye.
  • invisibility may generally refer to non-recognizable by human eye. That is, some people with very fast visual perception might notice minor flicker of light, but not recognize the optical indication 150, if the optical indication 150 is produced in isolated image frames lasting no more than e.g. 50 milliseconds.
  • the second apparatuses 120 image the scene 160 and obtain from their image information the projected optical indication 150 so that they can have similar metadata 222 stored on or with images taken at the same time of the same place or otherwise associated with the images e.g. in metadata parts of image files. After obtaining the information on scene 160, the second apparatuses 120 then recognize the metadata 222 from the optical indication 150 e.g. using OCR or bar code recognition and store the metadata 222 in image/video frame metadata.
  • the second apparatuses 120 are configured to use the metadata 222 obtained from the optical indication 150 for video synchronization. Making use of the metadata 222, all of the second apparatuses 120 can synchronize their time reference to one global clock based on which the optical indication 150 has been projected on the scene.
  • multi-camera clock synchronization is performed (e.g. for video capture) so that the first apparatus 1 10 first sends a base metadata data package as a first optical indication 150 (e.g. with complete clock data). Subsequently, say once a second, 5 seconds or 10 seconds, the first apparatus sends only one pulse.
  • the subsequent updating optical indications 150 are sent with predetermined times and durations according to a predetermined schedule so that each second apparatus 120 can interpolate its exact timing with respect to the first apparatus e.g. using Vernier interpolation.
  • the optical indication 150 is formed by transmitting modulated infrared radiation that is reflected from the target, recorded and decoded with suitable infrared detectors.
  • Such infrared detectors need not be cameras: in one robust and simple example embodiment, they are formed of an infrared detector such as an infrared transistor and a pipe or optical light converging element configured to provide the infrared detector field of view corresponding to that of its associated camera 130.
  • the field of view of the infrared detector is changeable correspondingly to that of the camera 130 e.g. by moving a detector deeper or less deep in the pipe or by zooming in / out by the optical light converging element.
  • the first apparatus 1 10 is configured e.g. using suitable software to select on the scene 160 an area that is suitable for the optical indication 150 projection, for example large continuous objects at or near desired imaging target, and to cause projecting the optical indication 150 on that area.
  • the first apparatus is configured e.g. using suitable software to selects on the scene 160 a plurality of areas that are suitable for the optical indication 150 projection, for example large continuous objects at or near desired imaging target, and to cause projecting the optical indication 150 on those areas for assisting second apparatuses 120 that obtain their image information from different parts of the scene. For example, some second apparatuses 120 may see one area too blurred because of its focal field and some may be simply directed to other imaging parts or be obscured from seeing one area because of any intermediate objects or the shape of the scene.
  • the first apparatus 1 10 is configured to (e.g. using suitable software) to identify one or more objects in focus and to cause the projection of the visual indication 150 so that the area in focus is avoided.
  • the first apparatus 1 10 is configured (e.g. using suitable software) to identify one or more objects in focus and to cause the projection of the visual indication in the vicinity of these objects if the visual indication 150 is related to the objects.
  • This example embodiment is usable with an example embodiment in which different image objects are associated with individual (possibly locally or globally unique) metadata for tagging these objects to the second apparatuses 120.
  • the projected visual indication 150 is usable for automatically steering the cameras 130 to particular (possibly different) focal distances, exposure times, and even to correct spread of positions (to enable better panoramas and 3d scene reconstruction).
  • the positioning can be implemented e.g. using automatic turning mechanisms, using adjustable cropping on a large-sensor camera, using image stabilization actuators to steer field of view of the camera in question, and / or by issuing (e.g. visual and / or audible instructions) to people operating the cameras 130.
  • the camera 130 is configured to be capable of detecting wavelengths beyond the visible light, i.e. in infrared or ultraviolet, for example. This can be achieved by adding a secondary camera for specific wavelengths, or using a specific amount of pixels in the main camera sensor for this function.
  • the camera 130 can be adapted capable of detecting infrared or ultraviolet wavelengths with a high enough resolution in the mobile device.
  • the cameras 130 and / or the second apparatuses can be equipped with suitable detectors that can be implemented using a single detector or one or more multiple-pixel sensors.
  • the optical metadata distributor 140 comprises e.g. a projector capable of projecting the optical indication 150 in the desired spectrum of light.
  • the optical metadata distributor 140 can be implemented simply with a suitable lamp, light emitting diode or other illuminator.
  • the first apparatus 1 10 comprises a camera 130 and the first apparatus is further configured to be capable of detecting optical indication 150 produced by another first apparatus 1 10.
  • the given first apparatus can detect the mutual locations of the other first apparatuses 1 10 based on the appearance of these visual indications 150, according to an example embodiment.
  • the given first apparatus can be further configured to select the area used for distributing its visual indication 150 to be non-overlapping with other visual indications 150 at least in one of: time and location.
  • the optical metadata distributor is configured to dynamically allocate a frequency band to be used for distributing the metadata and for detecting the metadata. The dynamic allocation may be based on one or more criteria.
  • the criteria may comprise any of: visibility of an optical metadata indication in prevailing conditions; predetermined scheme of using different ranges for given types of metadata; and availability of possible frequency bands.
  • the possible frequency bands comprise any one or more of: different visible light frequency bands (e.g.
  • nm to 420 nm, 430 nm to 450 nm and 460 nm to 480 nm and so on up to 700 nm i.e. relatively narrow bands with a guard range in between); low infrared band; medium infrared band; and high infrared band; and ultraviolet band.
  • Fig. 4 shows a flow chart illustrating a process in the first apparatus 1 10, according to an example embodiment, in which the relative pose of an imaging target or geometric properties such as distance, orientation and shape of surfaces are analyzed 410 within the field of view of the camera 130 from which the first apparatus 1 10 receives imaging information.
  • Term pose may herein refer to the location and orientation with respect to the camera 130.
  • the optical indicator is then adjusted 420 to avoid or reduce geometric distortions on projecting the optical indicator 150 onto a surface of the scene 160.
  • the optical indicator is projected 430 on its adjusted form for producing the optical indicator with reduced geometric distortion.
  • Fig. 5 shows a flow chart illustrating a process in the second apparatus 120, according to an example embodiment.
  • Captured video footage and / or one or more images are received 510 by the second apparatus 120 e.g. from the camera 130 of the second apparatus (that may be, as with other example embodiments, e.g. built-in or separate camera).
  • the optical indicator 150 is detected and the geometric properties or pose of its surfaces or at least some of its surfaces is estimated for the camera 130 with which the image(s) or video were captured.
  • the estimated pose is then associated 530 with the captured images or video footage e.g. by containing a numeric estimation of an angle between the camera and the optical indicator.
  • the image of the optical indicator 150 can simply be rotated.
  • the optical indicator 150 is projected on uneven surface and / or not perpendicularly so that the optical indicator 150 is geometrically distorted, and / or the camera 130 of the second apparatus 120 is not imaging the optical indicator quite perpendicularly.
  • the geometric distortions may involve e.g. of non-linear compression or expansion, trapezoid distortion or any other distortions.
  • Fig. 6 shows a flow chart illustrating another process of an example embodiment e.g. in the first apparatus 1 10 of Fig. 1.
  • the process comprises obtaining 610 metadata configured to assist multiple-camera imaging of a scene; and distributing 620 an optical indication of the metadata via the scene.
  • Fig. 7 shows a flow chart illustrating another process of an example embodiment e.g. in the second apparatus 120 of Fig. 1.
  • the process comprises: storing 710 camera captured image information of a scene; and obtaining 720 metadata from the image information.
  • a technical effect of one or more of the example embodiments disclosed herein is that multiple-camera imaging can be automatically controlled through metadata that is optically distributed without any preceding communications between different cameras that are imaging same scene.
  • Another technical effect of one or more of the example embodiments disclosed herein is that the metadata can be distributed to all the imaging cameras exactly simultaneously with reference to each camera. Even if one of the cameras were thousands of kilometers away, the image information obtained by such a camera would comprise metadata with matching timing with relation to events in the scene as another camera capturing image information 20 meters apart of the scene.
  • Another technical effect of one or more of the example embodiments disclosed herein is that no common control system is needed between the different cameras that participate to multiple-camera imaging.
  • crown imaging of one event can be automatically combined to a common multiple- camera presentation e.g. using Youtube® uploads of hundreds video clips.
  • the optical distribution of metadata using the visual indicators enables embedding metadata in events that copyright protected for automatic detection of illegal video uploads.
  • stitching of a panorama images can be facilitated by use of the optical indicator that helps initial correlating of different images.
  • Embodiments of the present invention may be implemented in software, hardware, application logic or a combination of software, hardware and application logic.
  • the software, application logic and/or hardware may reside on non-volatile memory of the first apparatus, of the second apparatus or any other apparatus in communication connection with one or more of the first and second apparatuses. If desired, part of the software, application logic and/or hardware may reside on the first apparatus, part of the software, application logic and/or hardware may reside on the second apparatus, and part of the software, application logic and/or hardware may reside on a server computer or other communicatively connected apparatus.
  • the application logic, software or an instruction set is maintained on any one of various conventional computer-readable media.
  • a "computer- readable medium” may be any non-transitory media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer, with one example of a computer described and depicted in Fig. 2 or 3.
  • a computer-readable medium may comprise a computer-readable storage medium that may be any media or means that can contain or store the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer.
  • the different functions discussed herein may be performed in a different order and/or concurrently with each other. Furthermore, if desired, one or more of the before- described functions may be optional or may be combined.

Abstract

A method, apparatus and computer program are disclosed for performing, at a first apparatus, obtaining metadata configured to assist multiple-camera imaging of a scene; and distributing an optical indication of the metadata via the scene; and at a second apparatus: storing camera captured image information of the scene; and obtaining the metadata from the image information.

Description

METHOD AND APPARATUS FOR MULTIPLE-CAMERA IMAGING TECHNICAL FIELD The present application generally relates to multiple-camera imaging. BACKGROUND
This section illustrates useful background information without admission of any technique described herein representative of the state of the art.
Photographing has developed with increasingly rapid pace following the advancements provided by digitized photographing and image processing. For instance, 3D and augmented reality footage are shot with two or more cameras. Moreover, different views can be enabled by providing users with multiple views of e.g. a sports event.
In real-time multiple-camera broadcast, numerous live video streams are transmitted. The user can be provided e.g. thumb-nail video images representing the available video streams so that the user can easily and intuitively select any of the views based on her own preferences. In off-line video, however, different video footages are recorded typically by different devices. A multiple-camera recording is then assembled from the different recordings. This process can be performed e.g. manually by grouping sequences of frames and mutually aligning them with a graphical user interface, e.g. by dragging with a computer mouse.
SUMMARY
Various aspects of examples of the invention are set out in the claims. According to a first example aspect of the present invention, there is provided an apparatus comprising: a processor configured to obtain metadata configured to assist multiple-camera imaging of a scene; and
an optical metadata distributor configured to distribute an optical indication of the metadata via the scene under control of the processor.
The scene may consist of one or more image objects that are being imaged by multiple cameras. Different cameras may have different views such that different cameras may be directed or focused to different parts of the scene. The optical metadata distributor may comprise a projector configured to distribute the metadata in an automatically recognizable form. The automatically recognizable form may comprise any of text; bar code; two-dimensional bar code; other symbols; and any combination thereof. The optical indicator needs not to be a static, but instead the optical indicator may be dynamic i.e. changing in course of time.
The optical metadata distributor may comprise a pulsating light source configured to cast on the scene pulsating light that is indicative of the metadata. The pulsating may be modulated by amplitude modulation; pulse width modulation; pulse-position modulation; pulse-duration modulation; and / or on-off keying. The pulsating light may vary with a frequency less or equal to a frame rate of cameras used for imaging the scene.
The optical metadata distributor may be configured to distribute any of: visible light; infra-red light; and ultraviolet light.
The optical metadata distributor may be configured to distribute metadata in two or more parts. One metadata part may comprise any of: a time reference; scene description; geographic location; resource locator associated to the scene; imaging parameters for guiding cameras imaging the scene; and timing information (such as interval) of subsequent metadata parts. Another metadata part may comprise any of: a timing indicator; a delta-coded indication of change in the location, imaging parameters or timing information; and any combination thereof.
The optical metadata distributor may be controlled by the processor to indicate the metadata via a surface of the scene.
The apparatus may further comprise a camera.
The processor may be further configured to identify the surface using the camera.
The processor may be further configured to detect when the camera is being used for imaging and to identify the surface so that the surface is not in focus of the camera, if the distributed metadata is visible to the camera. The processor may be further configured to geometrically correct the distribution of the metadata on the surface taking into account the geometry of the surface with relation to the optical metadata distributor. The geometry may comprise the shape and / or angle of the surface with regard to the optical metadata distributor. The processor may be further configured to detect focused portion of the scene and to identify the surface at or near the focused portion.
The imaging may refer to capturing of video image or a series of sub-sequent still images with at least partial overlap.
The metadata may comprise at least one of: geographic position of the apparatus or of the scene; resource locator associated to the scene; date; time; name of the scene; description of the scene; and camera settings. The processor may be configured to automatically obtain the metadata. The apparatus may comprise a positioning circuitry configured to determine the position of the apparatus. The positioning may refer to determining the position of the apparatus in three dimensions (e.g. Cartesian x-y-z dimensions) and to additionally determining the orientation of the apparatus around one or more different dimensions (e.g. Cartesian x-y-z dimensions). The processor may further configured to determine or estimate position of the scene based on the position of the apparatus and estimated direction and distance to the scene. Alternatively or additionally, the processor may further configured to determine or estimate position of the scene based on content of images captured by the camera, such as the time and position of the Sun and / or of the moon; the horizon; and orientation of walls or roofs of buildings. The apparatus may further comprise a user input for receiving the metadata or a portion thereof from a user of the apparatus.
According to a second example aspect of the present invention, there is provided an apparatus comprising:
a memory configured to store camera captured image information of a scene; and
a processor configured to obtain metadata from the image information.
The apparatus may further comprise a camera configured to form the image information by imaging of the scene. The processor may be further configured to control the operation of the camera based on the metadata. The processor may be further configured to control, based on the metadata, at least one of: focusing of the camera; focal length of the camera; white balance of the camera; resolution of the imaging performed by the camera; field of view of the imaging performed by the camera; time base of the camera; and illumination of the scene by the camera and/or by other common or separate illumination equipment.
The camera and the optical metadata distributor may be configured to dynamically allocate a frequency band to be used for distributing the metadata and for detecting the metadata. The dynamic allocation may be based on one or more criteria. The criteria may comprise any of: visibility of an optical metadata indication in prevailing conditions; predetermined scheme of using different ranges for given types of metadata; and availability of possible frequency bands. The camera may be configured to operate with visible light. Alternative and / or additionally, the camera may be configured to operate with invisible light. The processor may be configured to modify the image information so as to conceal the metadata if visible in images defined by the image information.
The processor may be configured to attach the metadata to the image information as separate data. Alternatively or additionally, the processor may be configured to attach the metadata to the image information by watermarking. According to a third example aspect of the present invention there is provided a method comprising:
obtaining metadata configured to assist multiple-camera imaging of a scene; and distributing an optical indication of the metadata via the scene. The distributing of the optical indication may comprise projecting the metadata in an automatically recognizable form. The automatically recognizable form may comprise any of text; bar code; two-dimensional bar code; other symbols; and any combination thereof. The optical indicator needs not to be a static, but instead the optical indicator may be dynamic i.e. changing in course of time.
The method may further comprise identifying using a camera a surface of the scene on which the optical indication is to be distributed.
The method may further comprise detecting when the camera is being used for imaging and identifying the surface so that the surface is not in focus of the camera, if the distributed metadata is visible to the camera. The method may further comprise geometrically correcting the distribution of the metadata on the surface taking into account the geometry of the surface with relation to the optical metadata distributor. The method may further comprise detecting focused portion of the scene and identifying the surface at or near the focused portion.
The method may further comprise automatically obtaining the metadata. The method may further comprise determining the position of the apparatus. The method may further comprise determining or estimating a position of the scene based on the position of the apparatus and estimated direction and distance to the scene. Alternatively or additionally, the method may further comprise determining or estimating a position of the scene based on content of images captured by the camera. The method may further comprise receiving the metadata or a portion thereof from a user of the apparatus.
According to a fourth example aspect of the present invention, there is provided a method comprising:
storing camera captured image information of a scene; and
obtaining metadata from the image information.
The method may further comprise forming the image information by imaging of the scene with a camera.
The method may further comprise controlling the operation of the camera based on the metadata. The controlling of the operation of the camera may comprise controlling at least one of: focusing of the camera; focal length of the camera; white balance of the camera; resolution of the imaging performed by the camera; field of view of the imaging performed by the camera; time base of the camera; and illumination of the scene by the camera and/or by other common or separate illumination equipment. The method may further comprise dynamically allocating a frequency band to be used for distributing the metadata and for detecting the metadata. The dynamic allocation may be based on one or more criteria. The criteria may comprise any of: visibility of an optical metadata indication in prevailing conditions; predetermined scheme of using different ranges for given types of metadata; and availability of possible frequency bands. The camera may be configured to operate with visible light. Alternative and / or additionally, the camera may be configured to operate with invisible light. The method may further comprise modifying the image information so as to conceal the metadata if visible in images defined by the image information.
The method may comprise attaching the metadata to the image information as separate data. Alternatively or additionally, the method may comprise attaching the metadata to the image information by watermarking. According to a fifth example aspect of the present invention there is provided a computer program comprising computer executable program code for:
obtaining metadata configured to assist multiple-camera imaging of a scene; and distributing an optical indication of the metadata via the scene;
when executed by an apparatus.
According to a sixth example aspect of the present invention there is provided a computer program comprising computer executable program code for:
storing camera captured image information of a scene; and
obtaining metadata from the image information.
when executed by an apparatus.
The computer program may be stored in a memory medium. The memory medium may be a non-transitory memory medium. The memory medium may comprise a digital data storage such as a data disc or diskette, optical storage, magnetic storage, holographic storage, opto-magnetic storage, phase-change memory, resistive random access memory, magnetic random access memory, solid-electrolyte memory, ferroelectric random access memory, organic memory or polymer memory. The memory medium may be formed into a device without other substantial functions than storing memory or it may be formed as part of a device with other functions, including but not limited to a memory of a computer, a chip set, and a sub assembly of an electronic device.
Different non-binding example aspects and embodiments of the present invention have been illustrated in the foregoing. The embodiments in the foregoing are used merely to explain selected aspects or steps that may be utilized in implementations of the present invention. Some embodiments may be presented only with reference to certain example aspects of the invention. It should be appreciated that corresponding embodiments may apply to other example aspects as well.
BRIEF DESCRIPTION OF THE DRAWINGS For a more complete understanding of example embodiments of the present invention, reference is now made to the following descriptions taken in connection with the accompanying drawings in which:
Fig. 1 shows an architectural overview of a system of an example embodiment of the invention;
Fig. 2 shows an apparatus according to an example embodiment, which apparatus is suitable for use as a first apparatus of Fig. 1 ;
Fig. 3 shows an apparatus according to an example embodiment, which apparatus is suitable for use as a second apparatus of Fig. 1 ;
Fig. 4 shows a flow chart illustrating a process in the first apparatus of Fig. 1 , according to an example embodiment;
Fig. 5 shows a flow chart illustrating a process in the second apparatus of Fig. 1 , according to an example embodiment;
Fig. 6 shows a flow chart illustrating another process of an example embodiment e.g. in the first apparatus of Fig. 1 ; and
Fig. 7 shows a flow chart illustrating another process of an example embodiment e.g. in the second apparatus of Fig. 1. DETAILED DESCRIPTION OF THE DRAWINGS
An example embodiment of the present invention and its potential advantages are understood by referring to Figs. 1 through 7 of the drawings. In this document, like reference signs denote like parts or steps.
Fig. 1 shows an architectural overview of a system 100 of an example embodiment of the invention. The system comprises a first apparatus 1 10, a plurality of second apparatuses 120 that comprise cameras, an optical metadata distributor 140 (or more than one in some example embodiments) configured to distribute optical indication 150 of the metadata via a scene 160. The optical metadata distributor 140 is here drawn separately, as it can be built as a separate unit configured to communicate with the first apparatus over a wired or wireless connection (universal serial bus, Bluetooth, wireless local area network, infrared), although in another example embodiment, the optical metadata distributor 140 is comprised by the first apparatus 1 10. Notice that Fig. 1 is simplified for easier illustration of at least one image object (the drawn house or details thereof) and a relatively uniform surface (for reflecting the optical indication 150) near the house such as a pavement.
In an example embodiment, the system 100 also comprises a plurality of metadata distributors 140 each under control of the first apparatus 1 10.
In an example embodiment, the scene consists of one or more image objects that are being imaged by multiple cameras (comprised by the second apparatuses). Different cameras may have different views such that different cameras may be directed or focused to different parts of the scene.
The first apparatus 1 10 need not have different structure than the second apparatuses 120. One apparatus is denoted as the first apparatus 1 10 for the purpose of simplifying this description. It should be also understood that the second apparatuses need not all be identical or similar by their structures. In an example embodiment, the first apparatus comprises two or more cameras, wherein one of the cameras is dedicated to facilitating the optical metadata distribution and at least one other camera is dedicated for taking normal digital images (e.g. for a situation in which the first apparatus 1 10 operates as the second apparatus).
Some example embodiments of the first apparatus and of the second apparatus will next be provided with reference to Figs. 2 and 3. Fig. 2 shows an apparatus 200 according to an example embodiment, which apparatus 200 is suitable for use as the first apparatus 1 10 and comprises a processor 210, a memory 220 for storing metadata 222 that is configured to assist multiple-camera imaging of the scene 160; and an optical metadata distributor 140 configured to optically cast the optical indication 150 of the metadata 222 via the scene 160 under control of the processor 210. The memory 220 further comprises computer program code or software 224 configured to control the operation of the apparatus 200 when executed by the processor 210. Fig. 2 is further drawn to contain a camera 130 as in an example embodiment, the first apparatus is also capable of operating as the second apparatus. In an example embodiment, the metadata 222 comprises at least one of: geographic position of the first apparatus 1 10 or of the scene 160; date; time; name of the scene; description of the scene; and camera settings (e.g. desired settings for use by second apparatuses). In an example embodiment, the processor 210 is configured to automatically obtain the metadata 222. In an example embodiment, the first apparatus 1 10 comprises a positioning circuitry configured to determine the position of the first apparatus 1 10. In this context, the positioning may refer to determining the position of the apparatus in three dimensions (e.g. Cartesian x-y-z dimensions) and in an example embodiment to additionally determining the orientation of the apparatus around one or more different dimensions (e.g. Cartesian x-y-z dimensions). The processor 222 can be further configured to determine or estimate a position of the scene 160 based on the position of the first apparatus 1 10 and estimated direction and distance to the scene 160 (e.g. using a compass and focusing based distance estimation). In an example embodiment, the processor is additionally or alternatively configured to determine or estimate position of the scene based on content of images captured by the camera, such as the time and position of the Sun and / or of the moon; the horizon; and orientation of walls or roofs of buildings. In an example embodiment, the first apparatus 1 10 further comprises a user input for receiving the metadata 222 or a portion thereof from a user of the first apparatus 1 10.
In an example embodiment, the optical metadata distributor 140 comprises a projector configured to distribute the optical indication 150 in an automatically recognizable form. The automatically recognizable form comprises, for example, any of: text; bar code; two-dimensional bar code; other symbols; and any combination thereof.
In an example embodiment, the optical metadata distributor 140 comprises a pulsating light source configured to cast on the scene 160 pulsating light that forms the optical indication 150. The pulsating may be modulated by any of: amplitude modulation; pulse width modulation; pulse-position modulation; pulse-duration modulation; on-off keying; and any combination thereof. The pulsating light may vary with a frequency less or equal to a frame rate of cameras used for imaging the scene 160.
In an example embodiment, the optical metadata distributor 140 is configured to distribute any of: visible light; infra-red light; and ultraviolet light.
In an example embodiment, the optical metadata distributor 140 is configured to the optical indication 150 in two or more parts. One indication part comprises, for example: a time reference; scene description; contextual sensor data recorded by the first apparatus 1 10; geographic location; resource locator associated to the scene; imaging parameters for guiding cameras imaging the scene; timing information (such as interval) of subsequent indication parts; and any combination thereof. In an example embodiment, the another indication part comprises: a timing indicator; a delta-coded indication of change in the location, imaging parameters or timing information; and any combination thereof.
In an example embodiment, the optical metadata distributor 140 is controlled by the processor to cast the optical indication 150 via a surface of the scene.
In an example embodiment, the apparatus further comprises a camera.
In an example embodiment, the processor is further configured to identify the surface using the camera.
In an example embodiment, the processor is further configured to detect when the camera is being used for imaging and to identify the surface so that the surface is not in focus of the camera, if the distributed optical indication 150 is visible to the camera.
In an example embodiment, the processor is further configured to geometrically correct the distribution of the optical indication 150 on the surface taking into account the geometry of the surface with relation to the optical metadata distributor 140. The geometry comprises, for example, the shape and / or angle of the surface with regard to the optical metadata distributor 140.
In an example embodiment, the processor is further configured to detect focused portion of the scene and to identify the surface at or near the focused portion. In an example embodiment, the imaging refers to capturing of video image or a series of sub-sequent still images with at least partial overlap.
Fig. 3 shows an apparatus 300 according to an example embodiment, which apparatus 300 is suitable for use as the second apparatus 120. The apparatus 300 comprises a processor 310; a memory 320 configured to store camera captured image information 322 of the scene 160 (comprising the optical indication 150) and to store computer program code or software 324 for controlling the apparatus 300 when run by the processor 310; and a camera 130 configured to form the image information by imaging of a scene. In another example embodiment, the apparatus 300 lacks the camera 130 and instead has a remote camera interface 330 (e.g. universal serial bus, Bluetooth, wireless local area network, and / or HDMI interface) configured to receive the image information from an external camera 130.
In an example embodiment, the processor 310 is further configured to control the operation of the camera based on the metadata 222 that is obtained from the optical indication 150. For example, the processor 310 can be further configured to control, based on the metadata 222, at least one of: focusing of the camera; focal length of the camera; white balance of the camera; resolution of the imaging performed by the camera; field of view of the imaging performed by the camera; viewing direction of the camera; time base of the camera; and illumination of the scene by the camera and/or by other common or separate illumination equipment.
In an example embodiment, the camera 130 is configured to operate with visible light. Alternative and / or additionally, the camera 130 can be configured to operate with invisible light. In an example embodiment, the apparatus 300 comprises or uses two or more cameras 130 for obtaining image information using visible light and invisible light. The processor 310 can be configured to modify the image information so as to conceal the optical indication 150 if visible in images defined by the image information.
In an example embodiment, the processor 310 is configured to attach the metadata 222 to the image information as separate data. Alternatively or additionally, the processor 310 is configured to attach the metadata 222 to the image information by watermarking.
In sake of simplicity, let us consider a use case in which a first apparatus 1 10 projects on the scene 160 the optical indication 150 invisibly e.g. as a bar code, 2-dimensional bar code, quick response code, or other machine readable form. The projection can be made using infrared, ultraviolet or other invisible wavelength that is detectable with camera sensor, and / or using brief pulses that short that the optical indication 150 appears invisible to human eye. Let us here clarify that invisibility may generally refer to non-recognizable by human eye. That is, some people with very fast visual perception might notice minor flicker of light, but not recognize the optical indication 150, if the optical indication 150 is produced in isolated image frames lasting no more than e.g. 50 milliseconds. The second apparatuses 120 image the scene 160 and obtain from their image information the projected optical indication 150 so that they can have similar metadata 222 stored on or with images taken at the same time of the same place or otherwise associated with the images e.g. in metadata parts of image files. After obtaining the information on scene 160, the second apparatuses 120 then recognize the metadata 222 from the optical indication 150 e.g. using OCR or bar code recognition and store the metadata 222 in image/video frame metadata.
In an example embodiment, the second apparatuses 120 are configured to use the metadata 222 obtained from the optical indication 150 for video synchronization. Making use of the metadata 222, all of the second apparatuses 120 can synchronize their time reference to one global clock based on which the optical indication 150 has been projected on the scene. In an example embodiment, multi-camera clock synchronization is performed (e.g. for video capture) so that the first apparatus 1 10 first sends a base metadata data package as a first optical indication 150 (e.g. with complete clock data). Subsequently, say once a second, 5 seconds or 10 seconds, the first apparatus sends only one pulse. In this manner, an initial synchronization can be performed at the frame rate of the cameras 130 of the second apparatuses 120 (with possibility of individual variation - some second apparatuses may have higher temporal resolution). In an example embodiment, the subsequent updating optical indications 150 are sent with predetermined times and durations according to a predetermined schedule so that each second apparatus 120 can interpolate its exact timing with respect to the first apparatus e.g. using Vernier interpolation. In an example embodiment, the optical indication 150 is formed by transmitting modulated infrared radiation that is reflected from the target, recorded and decoded with suitable infrared detectors. Such infrared detectors need not be cameras: in one robust and simple example embodiment, they are formed of an infrared detector such as an infrared transistor and a pipe or optical light converging element configured to provide the infrared detector field of view corresponding to that of its associated camera 130. In an example embodiment, the field of view of the infrared detector is changeable correspondingly to that of the camera 130 e.g. by moving a detector deeper or less deep in the pipe or by zooming in / out by the optical light converging element.
In an example embodiment, the first apparatus 1 10 is configured e.g. using suitable software to select on the scene 160 an area that is suitable for the optical indication 150 projection, for example large continuous objects at or near desired imaging target, and to cause projecting the optical indication 150 on that area. In another example embodiment, the first apparatus is configured e.g. using suitable software to selects on the scene 160 a plurality of areas that are suitable for the optical indication 150 projection, for example large continuous objects at or near desired imaging target, and to cause projecting the optical indication 150 on those areas for assisting second apparatuses 120 that obtain their image information from different parts of the scene. For example, some second apparatuses 120 may see one area too blurred because of its focal field and some may be simply directed to other imaging parts or be obscured from seeing one area because of any intermediate objects or the shape of the scene.
In an example embodiment, the first apparatus 1 10 is configured to (e.g. using suitable software) to identify one or more objects in focus and to cause the projection of the visual indication 150 so that the area in focus is avoided.
In an example embodiment, the first apparatus 1 10 is configured (e.g. using suitable software) to identify one or more objects in focus and to cause the projection of the visual indication in the vicinity of these objects if the visual indication 150 is related to the objects. This example embodiment is usable with an example embodiment in which different image objects are associated with individual (possibly locally or globally unique) metadata for tagging these objects to the second apparatuses 120.
In an example embodiment, the projected visual indication 150 is usable for automatically steering the cameras 130 to particular (possibly different) focal distances, exposure times, and even to correct spread of positions (to enable better panoramas and 3d scene reconstruction). The positioning can be implemented e.g. using automatic turning mechanisms, using adjustable cropping on a large-sensor camera, using image stabilization actuators to steer field of view of the camera in question, and / or by issuing (e.g. visual and / or audible instructions) to people operating the cameras 130.
In an example embodiment, the camera 130 is configured to be capable of detecting wavelengths beyond the visible light, i.e. in infrared or ultraviolet, for example. This can be achieved by adding a secondary camera for specific wavelengths, or using a specific amount of pixels in the main camera sensor for this function. For this purpose, the camera 130 can be adapted capable of detecting infrared or ultraviolet wavelengths with a high enough resolution in the mobile device. As mentioned in the foregoing, it is also possible to distribute the optical indicator 150 using pulsating light that may be visible or invisible to human eye. In case of infrared or ultraviolet pulsating light being used for this purpose, the cameras 130 and / or the second apparatuses can be equipped with suitable detectors that can be implemented using a single detector or one or more multiple-pixel sensors.
In an example embodiment, the optical metadata distributor 140 comprises e.g. a projector capable of projecting the optical indication 150 in the desired spectrum of light. In example embodiments in which pulsating light is being used for delivery of the optical indication 150, the optical metadata distributor 140 can be implemented simply with a suitable lamp, light emitting diode or other illuminator. In an example embodiment, the first apparatus 1 10 comprises a camera 130 and the first apparatus is further configured to be capable of detecting optical indication 150 produced by another first apparatus 1 10. In case that one or more other first apparatuses are projecting the visual indications 150 detectably by a given first apparatus, the given first apparatus can detect the mutual locations of the other first apparatuses 1 10 based on the appearance of these visual indications 150, according to an example embodiment. Further, the given first apparatus can be further configured to select the area used for distributing its visual indication 150 to be non-overlapping with other visual indications 150 at least in one of: time and location. Further or alternatively, in an example embodiment, the optical metadata distributor is configured to dynamically allocate a frequency band to be used for distributing the metadata and for detecting the metadata. The dynamic allocation may be based on one or more criteria. The criteria may comprise any of: visibility of an optical metadata indication in prevailing conditions; predetermined scheme of using different ranges for given types of metadata; and availability of possible frequency bands. In an example embodiment, the possible frequency bands comprise any one or more of: different visible light frequency bands (e.g. 400 nm to 420 nm, 430 nm to 450 nm and 460 nm to 480 nm and so on up to 700 nm i.e. relatively narrow bands with a guard range in between); low infrared band; medium infrared band; and high infrared band; and ultraviolet band.
Fig. 4 shows a flow chart illustrating a process in the first apparatus 1 10, according to an example embodiment, in which the relative pose of an imaging target or geometric properties such as distance, orientation and shape of surfaces are analyzed 410 within the field of view of the camera 130 from which the first apparatus 1 10 receives imaging information. Term pose may herein refer to the location and orientation with respect to the camera 130. The optical indicator is then adjusted 420 to avoid or reduce geometric distortions on projecting the optical indicator 150 onto a surface of the scene 160. The optical indicator is projected 430 on its adjusted form for producing the optical indicator with reduced geometric distortion.
Fig. 5 shows a flow chart illustrating a process in the second apparatus 120, according to an example embodiment. Captured video footage and / or one or more images are received 510 by the second apparatus 120 e.g. from the camera 130 of the second apparatus (that may be, as with other example embodiments, e.g. built-in or separate camera). The optical indicator 150 is detected and the geometric properties or pose of its surfaces or at least some of its surfaces is estimated for the camera 130 with which the image(s) or video were captured. The estimated pose is then associated 530 with the captured images or video footage e.g. by containing a numeric estimation of an angle between the camera and the optical indicator. For instance, in a very simplistic case in which the camera is imaging only perpendicular surfaces and the optical indicator 150 is oriented 20 degrees clock-wise with respect to the camera 130 that is imaging the optical indicator for the second apparatus 120, the image of the optical indicator 150 can simply be rotated. However, often the optical indicator 150 is projected on uneven surface and / or not perpendicularly so that the optical indicator 150 is geometrically distorted, and / or the camera 130 of the second apparatus 120 is not imaging the optical indicator quite perpendicularly. In such a case, the geometric distortions may involve e.g. of non-linear compression or expansion, trapezoid distortion or any other distortions. The compensative correcting of such geometric distortions is well-known from various projectors and the correction of geometric errors in captured images by computers (e.g. by post-processing) is also well-known e.g. in photogrammetry. Fig. 6 shows a flow chart illustrating another process of an example embodiment e.g. in the first apparatus 1 10 of Fig. 1. The process comprises obtaining 610 metadata configured to assist multiple-camera imaging of a scene; and distributing 620 an optical indication of the metadata via the scene. Fig. 7 shows a flow chart illustrating another process of an example embodiment e.g. in the second apparatus 120 of Fig. 1. The process comprises: storing 710 camera captured image information of a scene; and obtaining 720 metadata from the image information. Without in any way limiting the scope, interpretation, or application of the claims appearing below, a technical effect of one or more of the example embodiments disclosed herein is that multiple-camera imaging can be automatically controlled through metadata that is optically distributed without any preceding communications between different cameras that are imaging same scene. Another technical effect of one or more of the example embodiments disclosed herein is that the metadata can be distributed to all the imaging cameras exactly simultaneously with reference to each camera. Even if one of the cameras were thousands of kilometers away, the image information obtained by such a camera would comprise metadata with matching timing with relation to events in the scene as another camera capturing image information 20 meters apart of the scene. Another technical effect of one or more of the example embodiments disclosed herein is that no common control system is needed between the different cameras that participate to multiple-camera imaging. To the contrary, crown imaging of one event can be automatically combined to a common multiple- camera presentation e.g. using Youtube® uploads of hundreds video clips. Yet another technical effect of one or more of the example embodiments disclosed herein is that the optical distribution of metadata using the visual indicators enables embedding metadata in events that copyright protected for automatic detection of illegal video uploads. Another technical effect of one or more of the example embodiments disclosed herein is that stitching of a panorama images can be facilitated by use of the optical indicator that helps initial correlating of different images.
Embodiments of the present invention may be implemented in software, hardware, application logic or a combination of software, hardware and application logic. The software, application logic and/or hardware may reside on non-volatile memory of the first apparatus, of the second apparatus or any other apparatus in communication connection with one or more of the first and second apparatuses. If desired, part of the software, application logic and/or hardware may reside on the first apparatus, part of the software, application logic and/or hardware may reside on the second apparatus, and part of the software, application logic and/or hardware may reside on a server computer or other communicatively connected apparatus. In an example embodiment, the application logic, software or an instruction set is maintained on any one of various conventional computer-readable media. In the context of this document, a "computer- readable medium" may be any non-transitory media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer, with one example of a computer described and depicted in Fig. 2 or 3. A computer-readable medium may comprise a computer-readable storage medium that may be any media or means that can contain or store the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer.
If desired, the different functions discussed herein may be performed in a different order and/or concurrently with each other. Furthermore, if desired, one or more of the before- described functions may be optional or may be combined.
Although various aspects of the invention are set out in the independent claims, other aspects of the invention comprise other combinations of features from the described embodiments and/or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims.
It is also noted herein that while the foregoing describes example embodiments of the invention, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications which may be made without departing from the scope of the present invention as defined in the appended claims.

Claims

WHAT IS CLAIMED IS
1. An apparatus comprising:
a processor configured to obtain metadata configured to assist multiple-camera imaging of a scene; and
an optical metadata distributor configured to distribute an optical indication of the metadata via the scene under control of the processor.
2. The apparatus of claim 1 , wherein the scene consists of one or more image objects that are being imaged by multiple cameras.
3. The apparatus of claim 2, wherein different ones of the multiple cameras have different views of the scene.
4. The apparatus of any of preceding claims, wherein the optical metadata distributor comprises a projector configured to distribute the metadata in an automatically recognizable form.
5. The apparatus of claim 4, wherein the automatically recognizable form comprises any of: text; bar code; two-dimensional bar code; other symbols; and any combination thereof.
6. The apparatus of any of preceding claims, wherein the optical indication is dynamically changing.
7. The apparatus of any of preceding claims, wherein the optical metadata distributor comprises a pulsating light source configured to cast on the scene pulsating light that is indicative of the metadata.
8. The apparatus of any of preceding claims, wherein the optical metadata distributor is configured to distribute optical indicator using any of: visible light; infra-red light; and ultraviolet light.
9. The apparatus of any of preceding claims, wherein the optical metadata distributor is configured to distribute metadata in two or more parts.
10. The apparatus of any of preceding claims, wherein the optical metadata distributor is controlled by the processor to indicate the metadata via a surface of the scene.
1 1 . The apparatus of any of preceding claims, wherein the apparatus further comprises a camera.
12. The apparatus of claim 1 1 , wherein the processor is further configured to identify the surface using the camera.
13. The apparatus of claim 1 1 or 12, wherein the processor is further configured to detect when the camera is being used for imaging and to identify the surface so that the surface is not in focus of the camera.
14. The apparatus of any of claims 1 1 to 13, wherein the camera is dedicated to facilitating the optical metadata distribution.
15. The apparatus of any of preceding claims, wherein the processor is further configured to geometrically correct the distribution of the metadata on the surface taking into account the geometry of the surface with relation to the optical metadata distributor.
16. The apparatus of any of preceding claims, wherein the processor is further configured to detect focused portion of the scene and to identify the surface at or near the focused portion.
17. The apparatus of any of preceding claims, wherein the metadata comprises at least one of: geographic position of the apparatus or of the scene; resource locator associated to the scene; date; time; name of the scene; description of the scene; and camera settings.
18. The apparatus of any of preceding claims, wherein the processor is configured to automatically obtain the metadata.
19. An apparatus comprising:
a memory configured to store camera captured image information of a scene; a processor configured to obtain metadata from the image information.
20. The apparatus of claim 19, further comprising a camera configured to form the image information by imaging of a scene.
21 . The apparatus of claim 20, wherein the processor is further configured to control the operation of the camera based on the metadata.
22. The apparatus of claim 21 , wherein the processor is further configured to control, based on the metadata, at least one of: focusing of the camera; white balance of the camera; resolution of the imaging performed by the camera; field of view of the imaging performed by the camera; time base of the camera; and illumination of the scene by the camera and/or by other common or separate illumination equipment.
23. The apparatus of any of claims 20 to 22, wherein the camera is configured to operate with visible light.
24. The apparatus of any of claims 20 to 23, wherein the camera is configured to operate with invisible light.
25. A method comprising:
obtaining metadata configured to assist multiple-camera imaging of a scene; and distributing an optical indication of the metadata via the scene.
26. The method of claim 25, wherein the distributing of the optical indication comprises projecting the metadata in an automatically recognizable form.
27. The method of claim 25 or 26, further comprising identifying using a camera a surface of the scene on which the optical indication is to be distributed.
28. The method of claim 25, wherein the camera is used for imaging and identifying the surface so that the surface is not in focus of the camera.
29. The method of claim 27 or 28, further comprising geometrically correcting the distribution of the metadata on the surface taking into account the geometry of the surface with relation to the optical metadata distributor.
30. The method of any of claims 25 to 29, further comprising automatically obtaining the metadata.
31 . The method of claim 30, further comprising determining or estimating the position of the apparatus and using the determined or estimated position of the apparatus in the automatic obtaining of the metadata.
32. A method comprising:
storing camera captured image information of a scene; and
obtaining metadata from the image information.
33. The method of claim 32, further comprising forming the image information by imaging the scene with a camera.
34. The method of claim 33, further comprising controlling the operation of the camera based on the metadata.
35. The method of claim 34, wherein the controlling the operation of the camera comprises controlling at least one of: focusing of the camera; focal length of the camera; white balance of the camera; resolution of the imaging performed by the camera; field of view of the imaging performed by the camera; time base of the camera; and illumination of the scene by the camera and/or by other common or separate illumination equipment.
36. The method of any of claims 32 to 35, further comprising dynamically allocating a frequency band to be used for distributing the metadata and for detecting the metadata.
37. A computer program comprising computer executable program code for:
obtaining metadata configured to assist multiple-camera imaging of a scene; and
distributing an optical indication of the metadata via the scene;
when executed by an apparatus.
38. The computer program of claim 37, further comprising computer executable program code for causing the apparatus, when executing the computer program, to perform the method of any of claims 26 to 31 .
39. A computer program comprising computer executable program code for:
storing camera captured image information of a scene; and
obtaining metadata from the image information,
when executed by an apparatus.
40. The computer program of claim 39, further comprising computer executable program code for causing the apparatus, when executing the computer program, to perform the method of any of claims 33 to 36.
41 . A non-transitory memory medium comprising the computer program of any of claims 37 to 40.
PCT/FI2014/050033 2014-01-16 2014-01-16 Method and apparatus for multiple-camera imaging WO2015107257A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PCT/FI2014/050033 WO2015107257A1 (en) 2014-01-16 2014-01-16 Method and apparatus for multiple-camera imaging

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/FI2014/050033 WO2015107257A1 (en) 2014-01-16 2014-01-16 Method and apparatus for multiple-camera imaging

Publications (1)

Publication Number Publication Date
WO2015107257A1 true WO2015107257A1 (en) 2015-07-23

Family

ID=53542453

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/FI2014/050033 WO2015107257A1 (en) 2014-01-16 2014-01-16 Method and apparatus for multiple-camera imaging

Country Status (1)

Country Link
WO (1) WO2015107257A1 (en)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017034447A1 (en) * 2015-08-26 2017-03-02 Telefonaktiebolaget Lm Ericsson (Publ) Image capturing device and method thereof
EP3340611A1 (en) * 2016-12-20 2018-06-27 Nokia Technologies Oy An apparatus and associated methods for virtual reality scene capture
EP3465343A4 (en) * 2016-06-30 2020-07-29 Nokia Technologies Oy Method and apparatus for photographic image capture illumination

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102007041719A1 (en) * 2006-11-06 2008-05-15 Bauhaus-Universität Weimar Augmented reality producing method for use in e.g. TV studio, involves synchronizing camera with video projectors, and modulating images in luminance and/or chrominance in pixelwise, and integrating coded structure into projected images
US20090310859A1 (en) * 2008-06-11 2009-12-17 Vatics, Inc. Automatic color balance control method
US20130153651A1 (en) * 2011-12-20 2013-06-20 Elena A. Fedorovskaya Encoding information in illumination patterns

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102007041719A1 (en) * 2006-11-06 2008-05-15 Bauhaus-Universität Weimar Augmented reality producing method for use in e.g. TV studio, involves synchronizing camera with video projectors, and modulating images in luminance and/or chrominance in pixelwise, and integrating coded structure into projected images
US20090310859A1 (en) * 2008-06-11 2009-12-17 Vatics, Inc. Automatic color balance control method
US20130153651A1 (en) * 2011-12-20 2013-06-20 Elena A. Fedorovskaya Encoding information in illumination patterns

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
COTTING,D. ET AL.: "Embedding Imperceptible Patterns into Projected Images for Simultaneous Acquisition and Display''.", THIRD IEEE AND ACM INTERNATIONAL SYMPOSIUM ON MIXED AND AUGMENTED REALITY, ISMAR, November 2004 (2004-11-01), XP010769633 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2017034447A1 (en) * 2015-08-26 2017-03-02 Telefonaktiebolaget Lm Ericsson (Publ) Image capturing device and method thereof
US10171793B2 (en) 2015-08-26 2019-01-01 Telefonaktiebolaget Lm Ericsson (Publ) Image capturing device and method thereof
EP3465343A4 (en) * 2016-06-30 2020-07-29 Nokia Technologies Oy Method and apparatus for photographic image capture illumination
EP3340611A1 (en) * 2016-12-20 2018-06-27 Nokia Technologies Oy An apparatus and associated methods for virtual reality scene capture
WO2018115571A1 (en) * 2016-12-20 2018-06-28 Nokia Technologies Oy An apparatus and associated methods for virtual reality scene capture
US11055922B2 (en) 2016-12-20 2021-07-06 Nokia Technologies Oy Apparatus and associated methods for virtual reality scene capture

Similar Documents

Publication Publication Date Title
US11012622B2 (en) Digital 3D/360 degree camera system
US11176707B2 (en) Calibration apparatus and calibration method
TWI450111B (en) Method, computer useable medium, and device for guided photography based on image capturing device rendered user recommendations
US8208048B2 (en) Method for high dynamic range imaging
US10275898B1 (en) Wedge-based light-field video capture
CN105794202B (en) Depth for video and line holographic projections is bonded to
US9049369B2 (en) Apparatus, system and method for projecting images onto predefined portions of objects
US9756277B2 (en) System for filming a video movie
US11265620B1 (en) System and method to verify date and location for the creation of a video
US20150369593A1 (en) Orthographic image capture system
US10776971B2 (en) Metadata in multi image scenes
CN105191287A (en) Method of replacing objects in a video stream and computer program
GB2502065A (en) Video comparison apparatus for sports coverage
Côté et al. Live mobile panoramic high accuracy augmented reality for engineering and construction
US20120127203A1 (en) Mixed reality display
WO2018222932A1 (en) Video recording by tracking wearable devices
US20120127302A1 (en) Mixed reality display
KR20150127498A (en) Surveillance camera and surveillance system
WO2015107257A1 (en) Method and apparatus for multiple-camera imaging
JP2016021727A (en) Systems and methods for time-multiplexing temporal pixel-location data and regular image projection for interactive projection, and program
JP2004257872A (en) Position information acquiring system, position information acquiring device, position information acquiring method and program
JP2008005110A5 (en)
CN114788254A (en) Auxiliary focusing method, device and system
JP4210955B2 (en) Imaging apparatus, imaging control method, and imaging control program
CN108377327B (en) Panoramic camera and depth information acquisition method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14878602

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14878602

Country of ref document: EP

Kind code of ref document: A1