CN211018942U

CN211018942U - Imaging device with sensors of the same height

Info

Publication number: CN211018942U
Application number: CN201921573500.9U
Authority: CN
Inventors: 马龙·托马尔; 阿伦森·鲁思; 宾南·沙哈尔
Original assignee: HUMANEYES TECHNOLOGIES Ltd
Current assignee: HUMANEYES TECHNOLOGIES Ltd
Priority date: 2019-03-25
Filing date: 2019-09-20
Publication date: 2020-07-14
Anticipated expiration: 2029-09-20
Also published as: WO2020194190A1; CN211018941U; CN211019015U

Abstract

The utility model discloses an imaging device for acquire a plurality of simultaneous video image in order to produce 180 three-dimensional and 360 panoramic video streams, imaging device includes: a rigid body comprising two major surfaces and a bottom surface, there being a mechanical mount interface on the bottom surface; and three cameras having ultra-wide angle lenses, each camera defining a respective optical axis. The first camera is on the first major surface and its optical axis defines a first direction, the second camera is on the second major surface and its optical axis defines a second direction opposite to the first direction, and the third camera is on the second major surface and its optical axis defines a third direction parallel to the second direction. The apparatus includes a video display module including a display screen. The imaging device is upright, all three cameras being vertically aligned with each other at the same height, within a tolerance of one-half of any diameter of the respective ultra-wide angle lens.

Description

Imaging device with sensors of the same height

Technical Field

The present invention relates to an imaging device with multiple sensors for acquiring simultaneous real-time video images for combining and converting into 180 ° stereoscopic and/or 360 ° panoramic video images, and in particular to a single device that does not require mechanical manipulation for changing the image acquisition mode.

Background

Dual mode cameras have been introduced which allow the user to capture simultaneous images for conversion into 180 ° stereoscopic and 360 ° panoramic video images. Such dual mode cameras typically require a user to select between a stereoscopic mode and a panoramic mode by manipulating one or more mechanical devices, such as folding or unfolding a moveable camera platform, or changing orientation. Furthermore, such dual mode cameras typically allow for stereo or panoramic modes, but do not allow for both modes to exist simultaneously, as the built-in requirements for mechanical manipulation and/or orientation changes tend to make it impractical or even impossible to acquire images simultaneously for creating both stereo and panoramic images. Therefore, there is a need for a multi-mode, multi-sensor camera that will allow multiple simultaneous images to be captured for combination and conversion into either or both of 180 ° stereoscopic and 360 ° panoramic video images.

Disclosure of Invention

According to an embodiment of the present invention, an imaging apparatus for acquiring a plurality of simultaneous video images to generate 180 ° stereoscopic and 360 ° panoramic video streams, comprises: (a) a rigid body comprising first and second major surfaces; (b) first, second and third cameras fixedly mounted to the rigid body, each camera including a respective ultra-wide angle lens and defining a respective optical axis, wherein: (i) said first camera being positioned such that its lens extends from said first major surface and its optical axis defines a first direction; (ii) the second camera is positioned such that its lens extends from the second major surface and its optical axis defines a second direction that is opposite, or no more than 5 degrees opposite, the first direction; and (iii) the third camera is positioned such that its lens extends from the second major surface and its optical axis defines a third direction, the third direction being parallel to the second direction or within 10 degrees of parallel; (c) a video display module comprising a display screen mounted on the first major surface, the display screen operable to display and/or play 180 ° or 360 ° images in response to input received through the display screen and/or through a display control interface; and (d) a mechanical mount interface mounted on the first minor surface of the rigid body disposed closer to each of the first and second cameras than the third camera.

In some embodiments, a majority of a footprint of the mechanical mount interface on the first minor surface falls within a boundary of orthographic projections of the first and second cameras on the first minor surface.

In some embodiments, all footprints of the mechanical mount interface on the first minor surface fall within a boundary of orthographic projections of the first and second cameras on the first minor surface.

In some embodiments, a footprint of the mechanical mount interface on the first secondary surface is aligned on two orthogonal axes with a midline center of an orthographic projection of the first and second cameras on the first secondary surface.

In some embodiments, the imaging device further comprises a first electronic status indicator on the first major surface and a second electronic status indicator on the second major surface.

In some embodiments, the second camera 520₂Optical axis 401₂Defining a second direction opposite to the first direction or notOver 1 degree.

In some embodiments, the third camera 520₃Optical axis 401₃A third direction is defined that is parallel to the second direction, or within 5 degrees of parallelism.

In some embodiments, the third camera 520₃Optical axis 401₃A third direction is defined that is parallel to the second direction, or within 1 degree of parallelism.

In some embodiments, the imaging device is configured to acquire a plurality of simultaneous video images in a first mode, a second mode, or a third mode; in a first mode, the first and second cameras acquire simultaneous video images for stitching into a 360 ° panoramic video image; in a second mode, the second and third cameras acquire simultaneous video images for composing a 180 ° stereoscopic video image; in a third mode, the first, second and third cameras acquire simultaneous video images for producing 180 ° stereoscopic and 360 ° panoramic video images; the device is arranged to respond to input received via the display screen and/or via a display-control interface.

In some embodiments, the second minor surface may comprise a concave, textured finger grip region extending across a majority of the second minor surface in one of two perpendicular directions relative to the first minor surface.

According to an embodiment of the present invention, there is provided an imaging device for acquiring a plurality of simultaneous video images to produce a 180 ° stereoscopic and 360 ° panoramic video stream, the imaging device comprising (a) a rigid body comprising first and second major surfaces and a bottom surface on which a mechanical mount interface is present, (b) first, second and third cameras fixedly mounted to the rigid body, each camera comprising a respective super-wide angle lens and defining a respective optical axis, wherein (i) the first camera is positioned such that its lens extends from the first major surface and its optical axis defines a first direction, (ii) the second camera is positioned such that its lens extends from the second major surface and its optical axis defines a second direction, the second direction being opposite to the first direction or no more than 5 degrees, and (iii) the third camera is positioned such that its lens extends from the second major surface and its optical axis defines a third direction, the third direction being parallel to the second direction or parallel to the first direction or no more than 5 degrees, and (iii) the third camera is positioned such that its lens extends from the second major surface and its optical axis defines a third direction, the third direction is parallel to the second direction or parallel to the second major surface, and is operably mounted on the second plane, wherein the display screen is positioned such that it is positioned within a plane when the display screen is positioned such that it is positioned perpendicular to the display screen, the display device has a display screen is positioned such that it has a display screen that it is positioned such that it is positioned within.

In some embodiments, the imaging device is unattended standing on a flat surface of its bottom surface, and the first, second and third cameras are vertically aligned with each other at the same height, within a tolerance of one-fifth of any diameter of the respective ultra-wide angle lens.

In some embodiments, the imaging device is unattended standing on a flat surface of its bottom surface, and the first, second and third cameras are vertically aligned with each other at the same height, within a tolerance of 1 mm.

In some embodiments, the first and second cameras are aligned with each other in the x-axis dimension within a tolerance of 1 millimeter.

In some embodiments, the mechanical mount interface is aligned with the first and second cameras in the x-axis dimension to within a 5 mm tolerance.

In some embodiments, the optical axis of the second camera defines a second direction that is opposite the first direction, or is not more than 1 degree opposite.

In some embodiments, the optical axis of the third camera defines a third direction, the third direction being parallel to the second direction, or within 5 degrees of parallelism.

In some embodiments, the optical axis of the third camera defines a third direction, the third direction being parallel to the second direction, or within 1 degree of parallelism.

According to an embodiment of the present invention, there is provided an imaging apparatus for acquiring a plurality of simultaneous video images to generate 180 ° stereoscopic and 360 ° panoramic video streams, the imaging apparatus including: (a) a rigid body comprising first and second major surfaces; (b) first, second and third cameras fixedly mounted to the rigid body, each camera including a respective ultra-wide angle lens and a respective planar array of photodetectors defining a respective photodetector plane, each camera defining a respective optical axis perpendicularly intersecting the respective photodetector plane, wherein: (i) said first camera being positioned such that its lens extends from said first major surface and its optical axis defines a first direction; (ii) the second camera is positioned such that its lens extends from the second major surface and its optical axis defines a second direction that is opposite, or no more than 5 degrees opposite, the first direction; and (iii) the third camera is positioned such that its lens extends from the second major surface and its optical axis defines a third direction, the third direction being parallel to the second direction or within 10 degrees of parallel; wherein the respective photodetector planes of the first and second cameras are non-coplanar.

In some embodiments, the photodetector plane of the second camera is further disposed outwardly from the second major surface at least 3 millimeters further than the photodetector plane of the third camera.

In some embodiments, (i) the second camera has a view angle of at least 180 °, and (ii) the view angle does not encompass any portion of the lens of the third camera.

In some embodiments, (i) the second camera has a view angle of at least 180 °, and (ii) the 180 ° connection of the view angle does not contain any lens of the third camera.

In some embodiments, the imaging device further comprises a video display module including a display screen mounted on the first major surface.

In some embodiments, (i) the first camera has a viewing angle of at least 180 °, and (ii) the viewing angle does not encompass any portion of the display screen.

In some embodiments, the imaging device is configured to acquire a plurality of simultaneous video images in a first mode, a second mode, or a third mode; in a first mode, the first camera and the second camera acquire simultaneous video images for splicing into a panoramic video image; in a second mode, the second and third cameras acquire simultaneous video images for composing a 180 ° stereoscopic video image; in a third mode, the first, second and third cameras acquire simultaneous video images for producing 180 ° stereoscopic and 360 ° panoramic video images; the device is arranged to respond to input received via the display screen and/or via a display-control interface.

In some embodiments, the imaging device further comprises a first minor surface and a second minor surface, the second minor surface comprising, relative to the first minor surface, a concave, textured finger grip region extending across a majority of the second minor surface in one of two perpendicular directions.

Drawings

The invention will be further described with reference to the accompanying drawings, in which the dimensions of the components and features shown in the drawings are chosen for clarity of presentation and are not necessarily to scale. In some drawings, the relative sizes of objects and the relative distances between objects may be exaggerated or minimized for convenience and clarity. In these drawings:

fig. 1A, 1B, 1C, and 1D are front, rear, bottom, and top views, respectively, of an imaging device including three cameras according to a specific embodiment of the present invention.

Fig. 2 is a bottom view of an imaging device including three cameras according to a specific embodiment of the present invention, showing the boundaries of the orthographic projection of the two cameras on the bottom surface of the device.

FIG. 3 is a bottom view of an imaging device including three cameras according to a specific embodiment of the invention, schematically illustrating respective photodetector planes of the three cameras

Fig. 4 is a bottom view of an imaging device including three cameras according to a specific embodiment of the invention, schematically illustrating the respective perspectives of the two cameras with respect to physical features on two major surfaces of the imaging device.

Figures 5A and 5B illustrate block diagrams of systems for acquiring simultaneous images, processing the images to create stereoscopic and/or panoramic video streams, delivering one or both of the streaming videos to a client browsing device,

fig. 6 and 7 show flowcharts of a method for delivering simultaneous real-time video streaming of panoramic and three-dimensional stereoscopic images according to a specific embodiment of the present invention

Detailed Description

The invention will be described herein by way of example only and with reference to the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTSReferring now to the details of the drawings in particular, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of the preferred embodiments of the present invention only, and are presented in the cause of providing what is believed to be the most useful and readily understood description of the principles and conceptual aspects of the invention. In this regard, no attempt is made to show structural details of the invention in more detail than is necessary for a fundamental understanding of the invention, the description taken with the drawings making apparent to those skilled in the art how the several forms of the invention may be embodied in practice. Like reference characters generally refer to like elements throughout the several views.

Note that: throughout this specification, subscripted reference numerals (e.g., 10)₁Or 10_A) May be used to indicate multiple separate appearances of a single kind of element, for example: 10₁Is a single appearance (from among multiple appearances) of the element 10. When not referring to a particular one of the plurality of separate appearances, that is, for a generic class, the same element may instead be referred to as being without a subscript (e.g., 10, instead of 10)₁)。

For convenience, in the context of the description herein, various terms are presented herein. To the extent that a definition is provided herein or elsewhere in this application, either explicitly or implicitly, such definition is understood to be consistent with the use of the defined term by those skilled in the relevant art. Further, such definitions are to be construed in the broadest possible sense consistent with such usage.

The term "module" as used herein refers to any combination of electronic hardware, software, and firmware necessary to perform the functions associated with the module, e.g., "stitching module" includes hardware, software, and firmware for "stitching" an image. The myriad of techniques available in each of the three technical categories of hardware, software, and firmware are well known in the particular arts of their technical endeavors, and the present invention is intended to encompass the use of any such techniques, whether known and commonly used within the time of the present invention or developed and made available during the life of any patent after disclosure.

The terms "imaging device", "sensor" and "camera" are used interchangeably in this specification and the appended claims, and all have the same meaning: a device for capturing digital images comprises (at least) a CMOS or CCD device and a lens, and optionally any number of mechanical and electronic components and accessories. In contrast, the term "imaging device" as used herein is specific to a device comprising a plurality of sensors/cameras/imaging devices. In this statement, the imaging device, sensor or camera may be a component of the imaging device, but this is not the case.

According to some embodiments of the present invention, an imaging device including multiple imaging devices may acquire simultaneous video images of a scene. Each of these imaging devices may have a wide viewing angle, e.g., 180 ° or more. The scene may contain a field of view of 180 ° or more or 360 °. While the video images may be overlapping, this means that at least two of the video images are overlapping, or equivalently at least abutting or nearly abutting. In some embodiments, one of the video images may overlap each of the other two video images. In some embodiments, the angle of view captured by one or more respective imaging devices is exactly 180 °, or alternatively the sum of the angles of view of two said imaging devices amounts to 360 °, and in this case the concept of "overlapping" as used herein is extended to include adjacent or abutting images. In some embodiments, the perspectives captured by one or more respective imaging devices are less than 180 °, or the perspectives of at least some of the respective imaging devices are non-overlapping, and in such cases, the concept of "stitching" as used herein is extended to include interpolation, such that the images may be considered "stitched" even if there is no actual overlap between them.

These images are acquired by the imaging device simultaneously or with a minimum delay of no more than a few milliseconds or tens or hundreds of milliseconds. The stitching module may create a panoramic image from two or more video images that overlap (or abut or nearly abut). Meanwhile, the 3D stereoscopic synthesis module may create a 3D stereoscopic image from two or more overlapped video images, which may include at least partially the same image as a video image used when stitching the panoramic image.

Panoramic stitching and 3D stereoscopic compositing of multiple and overlapping images, including asymmetric cropping, intra-camera calibration, image acquisition and manipulation, may be performed using any of the systems and methods described in pending international application PCT/IB2018/059771, filed 2011, 11, 7, the contents of which are incorporated herein by reference in their entirety. The above-described methods and systems are also disclosed in the following U.S. patent application No. 62823409, filed on 3, 25, 2019, and U.S. patent application No. 62/596,112, filed on 12, 7, 2018, the entire contents of which are incorporated herein by reference in their entirety.

In some embodiments, either or both of the stitching module and the 3D stereoscopic compositing module are included in the imaging device, and in some embodiments either or both of the stitching module and the 3D stereoscopic compositing module reside remotely, such as at one or more remote servers, which may be part of a server farm that includes modules dedicated to the stitching module and/or the 3D stereoscopic compositing module (and/or other modules that may be necessary for processing the video images, storing the images, and/or delivering them as video streams to the client, where delivering them to the client may include selecting which video stream to deliver to any particular client). Alternatively, one or more servers may be "in the cloud," that is, servers available from commercial computing operations, and may include any number of networked servers (including but not limited to shared, dedicated, and virtual servers) available to any number of unrelated customers. For convenience, the remote server is described throughout as "in the cloud," but as noted above, this is not necessarily the case in some embodiments.

In some embodiments, the imaging device may include a communication circuit configured for uploading the overlaid video images to one or more servers; and in other embodiments, the communication device is configured to upload the stitched panoramic image and the synthesized 3D stereoscopic image to one or more servers. Any reference herein to a communication circuit may include an integrated or "built-in" communication circuit, or a module that is readily attachable to the imaging device, for example, where the imaging device is equipped with an appropriate connector.

Reference is now made to fig. 1A, 1B, 1C and 1D, which respectively illustrate front, rear, bottom and top views of an imaging apparatus 500 capable of acquiring a plurality of simultaneous video images from which 180 ° stereoscopic and 360 ° panoramic video streams (or files) can be created.

The imaging apparatus 500 according to the particular embodiment is a "one-piece" device that is preferably constructed with a rigid frame 501. By "unitary" is meant herein that the device has no moving parts for positioning/repositioning the individual cameras, nor is there a folding mechanism to place one or more individual cameras in more or fewer locations for imaging in a particular mode (stereo or panoramic). Of course, the rigid frame 501 may be constructed of multiple parts, and it may also have movable or removable openings for connection and control.

The imaging device 500 shown in FIGS. 1A-1D has two "major surfaces" -a front surface and a back surface 540₁、540₂And four minor surfaces, the shape of the imaging apparatus 500 in all figures consistently shows a single example of a product design for clarity purposes, but the product may have any rectangular/prismatic shape (including, for example, a rectangle with or without cuts or rounded corners, or a rectangle with rounded edges or edges). Nevertheless, it may be desirable to provide a bottom surface 545 that allows imaging apparatus 500 to be placed on an unattended flat surface without support, whether for storage or for operation.

The imaging device 500 includes three cameras 520. A dual mode device has only two cameras and would necessarily require mechanical operation of one or both cameras in order to orient them correctly for the selected mode. The device may have more than three cameras, but three is the practical minimum number of cameras for having dual mode capability in a rigid design. Each of these cameras 520 is a wide-angle camera, meaning that it has a respective wide-angle lens 530 mounted thereon. Wide angle lens 530 may comprise a fish-eye lens, which advantageously has a viewing angle of at least 180 °, i.e., a hemispherical or larger angle or viewing angle. All three cameras 520 are operable to acquire images at least when the cameras are in the orientation shown in fig. 1A to 1D.

First camera 520₁Is mounted on the first main surface 540₁And two other cameras 520₂And 520₃Mounted on the second major surface 540₂The above. The two cameras 520₁、520₂Each having a respective optical axis 401₁、401₂They are: directly opposite each other and therefore collinear; or in exactly opposite directions but not collinear, and thus parallel; or in nearly opposite directions, meaning within a rotation from the opposite ± 10 °, or within a rotation from the opposite ± 5 °, or within a rotation of ± 1 °. Terms in the reverse or parallel "+ -. x.degree" refer to a range that means anywhere in the range from-x to + x, away from or parallel, andand in each case means the case "from-x ° to + x ° does not include".

The display screen 510 may be in contact with the first major surface 540₁Is flush or slightly recessed or otherwise displaced so as not to be flush based on aesthetic or ergonomic considerations. It is preferably a touch screen to facilitate receiving user input. Regardless of whether display 510 is a touch screen or not, display control interface 515 may be, for example, a button in the example shown in FIG. 1B. The display screen 510 is operable to display and/or playback 180 ° or 360 ° images in response to input received via the display screen 510 (if the touch screen 510) or the display control interface 515. The video display module may include a display screen 510, a display control interface 515, and any required electronics or software for performing video display functions.

First major surface 540₁Also included is an electronic status indicator 516₁It can be operated to display an on/off state or to indicate that photographing or recording or transmission is in progress. Electronic status indicator 516₁Can include any kind of electronic display, for example, a small L CD screen, or a plurality L ED. of two major surfaces 540₁、540₂Either of which may be flat or rounded and/or may optionally include raised or recessed surfaces for aesthetic or ergonomic purposes. First major surface 540₁Identified in the figure as the "front" of the imaging device 500 "

(and second major surface 540₂As a "back" face), but this is for convenience only and it has no meaning to design or function, particularly in view of the fact that imaging apparatus 500 is operable to acquire images in any and all directions.

Two cameras 520₂、520₃Are all mounted on the second major surface 540₂The above. Second electronic status indicator 516₂Can also be mounted on the second major surface 540₂The above. Second and third cameras 520₂、520₃Facing in the same direction as each other, meaning their respective optical axes 401₂、401₃Are parallel to each other, or within + -5 deg. of parallelism, orWithin ± 2 ° of parallelism, or within ± 1 ° of parallelism. By a second camera 520₂And a third camera 520₃The simultaneously acquired images may be used together to synthesize a stereoscopic video image, for example, a 180 ° stereoscopic video image for viewing on a 3D browsing device.

Second and third cameras 520₂、520₃Preferably at the same height H as each other_LENSWherein the height H_LENSIs the height in the y-dimension to the centerline as shown in fig. 1A, as measured in the y-dimension, when imaging device 500 is placed on a flat surface, i.e., has a bottom surface 545 on the flat surface. First camera 520₁Is also preferably mounted at the same height H as shown in fig. 1B_LENSAnd particularly in the second camera 520 "back-to-back" therewith₂At the same height. The expression "same height" may mean that the cameras 520 are exactly at the same height, or have respective heights different from each other, three respective lenses 530₁、530₂、530₃(or the smallest diameter of the respective diameters 908 if the lenses 530 are not all of the same diameter) of no more than 50% of the diameter 908, or no more than 20% of the diameter 908, or no more than 2mm, or no more than 1 mm.

Fig. 1D shows an example of a top surface 546 of the imaging device 500, the finger grip region 547 preferably extending across most of the top surface 546 in each direction in order to provide as large an area as possible with typically small equipment. The finger grip area 547 may be at least partially textured, as shown, with an area that is less smooth to touch than the surrounding surface. The texture pattern shown in FIG. 1D is non-limiting and shows only one possible example. In some embodiments, not shown, the finger grip area may be flat or convex, or concave but not textured.

FIG. 1C is a bottom view of imaging device 500 showing mechanical mount interface 550 mounted on bottom surface 545, interface 550 may include, for example, for attaching a tripod with a mating threaded connection elementThe threaded socket of (1). In some embodiments, it may be desirable for the cradle interface 550 to be closer to the second and third cameras 520 than the first camera 5201₂、520₃ Interface 550 is preferably mounted along a longitudinal centerline C L x, which longitudinal centerline C L x represents the "middle" of imaging device 500 in the z-dimension the term "middle" may be a line drawn along the central longitudinal axis of floor 545 or may be a line indicating the center of balance considered considering the weights of various elements mounted on both major surfaces 540 of the vehicle, it may be desirable for interface 550 to be positioned along x-dimension centerline C L x and centered on first and second cameras 520₁、520₂Optical axis 401₁、401₂As shown in fig. 1C. In one example, this location of the interface 550 provides maximum angular stability for capturing images of a 360 ° panorama, i.e., ensures that the two cameras 520 responsible for the task of acquiring 360 ° images are responsible₁、520₂Can be stably maintained at an optimum angle for capturing an image.

In some embodiments, the interface 550 is not exactly located at the centerline C L x and the optical axis 401₁、401₂But is located at the intersection of two cameras 520₁、520₂Within the coverage area of (a). Fig. 2 shows a first camera 520₁And a second camera 520₂(including respective lenses 530)₁、530₂) Orthogonal projection 590 on bottom surface 545 of imaging device 500 orthogonal projection 590 is a projection of the combined footprint of two cameras 5201. as described in the previous paragraph, mechanical mount interface 550, or at least a majority of interface 550, is preferably located within orthogonal projection 590, and it may be desirable for interface 550 to be substantially (+ -3 mm, or + -2 mm, or + -1 mm) around x-dimensional centerline C L x and optical axis 401₁、401₂Is centered on the "intersection" of, the optical axis 401₁、401₂Represented in fig. 2 as optical centerline C L_OPT。

The inner workings of the digital camera may include a planar array of photodetectors. Such a planar array may define a respective photodetector plane for each camera. This is illustrated in fig. 3, where a first camera 520₁A second camera 520₂And a third camera 520₃With respective photodetector planes P₁、P₂、P₃And (4) associating. As discussed above with respect to FIG. 1C, the first camera 520₁And a second camera 520₂Optical axis 401₁、401₂Directly opposite (collinear) each other, or opposite each other within a range of rotation of + -10 deg., or opposite each other within a range of rotation of + -5 deg., or opposite each other within a range of rotation of + -1 deg.. The respective optical axis 401 of each camera is perpendicular to the photodetector plane P of a given camera and, therefore, with the first and second cameras 520₁、520₂Associated photodetector plane P₁、P₂Respectively, parallel to each other, or within a parallelism of + -10 deg., or within a parallelism of + -5 deg., or within a parallelism of + -1 deg.. As discussed above with respect to FIG. 1C, the second camera 520₂And a third camera 520₃Respectively optical axis 401₂、401₃Are parallel to each other, or within a parallelism of + -5 deg., or within a parallelism of + -2 deg., or within a parallelism of + -1 deg., so that the respective photodetector planes P₂、P₃Parallel to each other, or within a parallelism of + -5 deg., or within a parallelism of + -2 deg., or within a parallelism of + -1 deg.. In these embodiments, the second camera 520₂And a third camera 520₃Of the corresponding photodetector plane P₂、P₃Out of plane, in an embodiment, when the second camera 520₂Than the third camera 520₃Further from the rigid body 501, the second camera 520₂Further extending from the rigid body 501, a second camera 520₂Than the third camera 520₃From the second major surface 540 of the rigid body 501₂At least 2mm and not more than 10mm, or at least 3mm and not more than 9mm, or at least 4mm and not more than 8 mm.

As shown schematically in FIG. 4, each camera 520 has a viewing angle α limited by its optics (e.g., its respective wide angle lens 530), and the second camera 520₂Preferably having a viewing angle α of at least 180 deg.₂. And a second camera 520₂And a third camera 520₃Associated photodetector plane P₂、P₃Is not coplanar with the third camera 520₃In contrast, the second camera 520₂Further from the second main surface 540 of the rigid body 501₂Extended, third camera 520₃And its wide angle lens 530₃Is located at the third camera 520₃And its wide angle lens 530₃In addition, (at least) a second camera 520₂Angle of view α ₂180 deg.. In FIG. 4, the marker AOV can be seen₂(showing a second camera 520)₂Angle of view α₂Limited) in the third camera 520₃ Wide angle lens 530₃And pass through it. On the other major surface 5401, a first camera 520₁And also has a viewing angle α of at least 180 deg₁And corresponding AOV₁The lines are well away from the display screen 510 and the first major surface 540₁Through any other feature of (a). The final effect is: using first and second cameras 520₂、520₂The obtained panoramic image the acquired panoramic image may be generated without capturing any external portion of the imaging apparatus 500, as in each of the examples described in this paragraph.

In these embodiments, imaging device 500 may provide, among other things, the following image capture modes: (i) using only the first camera 520₁And a second camera 520₂To capture images for creating (stitching) a 360 ° panoramic image; (ii) using only the second and third cameras 520₂、520₃To capture images for creating 180 ° 3D stereoscopic images; and (iii) use of all three cameras 520₁、520₂、520₃For acquiring images for creating synchronized panoramic and stereoscopic images, wherein there is a first and a second camera 520₁、520₂The acquired images may be used to create a panoramic image, the second and third cameras 520₂、520₃The acquired images may be used simultaneously to create a stereoscopic image. The image capture mode can be user-selected, using the display screen 510 (if a touch screen) and ≧ based

Or display the control interface 515 for selection.

The following paragraphs will discuss a system and method suitable for delivering synchronized panoramic and stereoscopic video streams, employing all three cameras 520₁、520₂、520₃。

Referring now to fig. 5A, a block diagram illustrates an example of a system 100 for delivering synchronized real-time panoramic and stereoscopic video streams. The system 100 shown in FIG. 5A includes an imaging device 500 that includes three camera devices 520₁、520₂、520₃. The imaging device 500 is preferably operable to use a camera device 520₁、520₂、520₃To acquire three corresponding overlapping real-time video images 111₁、111₂、111₃. The system 100 also includes a communication circuit 157 configured to, among other things, overlay the video image 111₁、111₂、111₃Upload to remote server 170, employ communication channel 201, which, similar to all other communication channels disclosed herein, can incorporate any combination of wired and wireless communication techniques, and is preferably configured to be accessed by imaging device 520₁、520₂、520₃The resolution that can be used to transmit video images, and at the throughput speed necessary to maintain substantially live and real-time transmission. The communication circuit 157 may be integrated into the imaging device 500, or when the term "server" is used herein, it is synonymous with the terms "server(s)", "remote server(s)" and "remote server", and we note that the particular distribution of load and functionality among the servers or processors is not relevant to the present invention.

According to these embodiments, the system 100 may further comprise: multiple modules residing on one or more remote servers 170 in cloud 199-for overlaying real-time video images 111₁、111₂、111₃A stitching module 175 that stitches together, the images being transferred by the imaging device 500 to the server 170; and a stereo image compositing module 176 for compositing from overlapping live video images111₁、111₂、111₃And synthesizing the 3D stereoscopic image. In connection with server 170, storage media 140 (e.g., non-transitory computer readable storage media) can reside in cloud 199.

As shown in fig. 5A, the stitching module 175 can be operated to transfer (i.e., download) live video images 111 from an overlay₁、111₂、111₃The stitched panoramic video images are transmitted to the plurality of devices 195 of the client browsing device 190 as a panoramic video stream 180 using the communication channel 202. Plurality 195 can include any number N of client browsing devices 190₁……190_NDepending on the operational capabilities of the server 170 and the

communication channel

202 or 203. The stereoscopic image composition module 176 may be operable to transmit (i.e., download) the live video images 111 from the overlay₁、111₂、111₃To multiple devices 195 of the same client browsing device 190 as a live 3D video stream 181 using a communication channel 203 for transmission. It is important to note that the entire process implemented in the system shown in FIG. 5A includes, but is not limited to, acquiring multiple overlapping real-time video images 111₁、111₂、111₃And superimposing these live video images 111₁、111₂、111₃Uploading to one or more remote servers 170, employing respective stitching modules 175 and stereoscopic image composition modules 176 to generate respective panoramic and stereoscopic images therefrom, and delivering the images to multiple devices 195 of client browsing device 190 as real-time panoramic video stream 180 and real-time 3D stereoscopic video stream 181, without delay in processing time, so as to deliver real-time video streams 180, 181 in real-time, i.e., within an amount of delay time acceptable in the industry of "live" and "real-time" transmissions. It should be apparent to those skilled in the art that not all of the components shown in fig. 5A are necessary for proper operation of the system 100, and that any of the imaging devices 500 disclosed herein are also suitable for use in the system 100.

Referring now to fig. 5B, a second embodiment of a system 100 for delivering a synchronized real-time video stream is shown in block diagram form. The system 100 shown in fig. 5B has similarities to the system 100 previously discussed with respect to fig. 5A, with a number of differences:

in the example shown in fig. 5B, the stitching module 175 and the stereoscopic image composition module 176 reside within the imaging device 500 and not in the cloud at the one or more servers 170, as is the case in the example of fig. 5A;

the communication circuitry 157 of the imaging apparatus 500 is configured to upload the processed (pre-stitched/pre-composited) video images to the one or more servers 170 as live streams (i.e., the "stitched" real-time panoramic video stream 180 and the "composited" 3D stereoscopic video stream 181).

The delivery module 178 at the one or more servers 170 is configured to deliver the respective real-time video streams 180, 181 to a plurality of devices 195 of the client browsing device 190. Although not shown as a component of the system 100, in some embodiments, the delivery module 178 may also be a component of the system 100.

It should be apparent to those skilled in the art that not all of the components shown in fig. 5A and 5B are necessary for proper operation of the system 100, nor are every possible component and interface for such a system 100 shown in fig. 5A and 5B. It should also be apparent that certain aspects of the two figures may be combined — as in one non-limiting example, it may be one particular module of the respective stitching module 175 and stereoscopic image composition module 176, which may be included in the imaging apparatus 500, and the other of the two modules may reside in the cloud 199, i.e., at one or more remote servers 170.

As is known in the art, images acquired by imaging devices having ultra-wide angle lenses, such as fish-eye lenses, are typically dewaxed to remove distortion effects of the lenses prior to further processing into complex images, such as panoramic images or stereoscopic images (or even prior to being considered simplex images). The dewarping module may be included as part of the system 100 for delivering synchronized real-time panoramic video stream 180 and stereoscopic video stream 181, or may otherwise be available for use by components of the system. Dewarping module 177 may reside in any of a variety of different locations as described in accordance with a particular embodiment.

Referring now to fig. 6, a flow diagram of a method for delivering simultaneous real-time video streaming of panoramic and three-dimensional stereoscopic images to multiple clients is presented. The method comprises four steps as follows:

in step S01, the imaging device 500 including the three imaging devices 520 is used to acquire the corresponding three simultaneous real-time video images 111 of the scene. Any of the imaging devices 500 disclosed herein would be suitable for performing the method.

At step S02, the plurality of real-time video images 111 are transferred from the imaging device 500 to one or more remote servers 170 using the communication circuit 157 and communication channel 201 of the imaging device 500.

Step S03 creates real-time panoramic video images and real-time 3D stereoscopic video images from the plurality of real-time video images 111 using the respective stitching module 175 and stereoscopic image composition module 176 residing at the one or more servers 190. In an embodiment, creating the real-time panoramic video image includes using a camera comprised of the first and second cameras 520₁、520₂Simultaneous real-time video images are acquired and creating the real-time 3D stereoscopic video images includes using a camera controlled by a second and third camera 520₂、520₃And simultaneously acquiring real-time video images.

Step S04, simultaneously delivering from one or more remote servers 170 to a client browsing device 190 in communication therewith said real-time panoramic video image and said real-time 3D stereoscopic video image, delivered as live video streams 180, 181 respectively, said delivering being such that: (i) the real-time panoramic video stream 180 is the first plurality of devices 191 or 193 that is delivered to the client browsing device 190, while the real-time 3D stereoscopic video stream is the second plurality of devices 192 or 194 that is delivered to the client browsing device 190. In some embodiments, the two sets of multiple devices 191, 192 do not overlap, while in some embodiments, the two sets of multiple devices 193, 194 overlap such that a set of client browsing devices 190 in the overlap has two real-time video streams delivered to them. These real-time video streams are synchronized with each other to achieve smooth or seamless switching (transition) between them.

In some embodiments, not all steps of the method are performed.

Referring now to fig. 7, a flow diagram of another method for delivering simultaneous real-time video streaming of panoramic and 3D stereoscopic images to multiple clients is presented. The method comprises four steps as follows:

in step S11, the imaging device 500 including the three imaging devices 520 is used to acquire the corresponding three real-time video images 111 of the scene. Any of the imaging devices 500 disclosed herein would be suitable for performing the method.

Step S12, creates a real-time panoramic video image and a real-time 3D stereoscopic video image from the plurality of real-time video images 111, using the respective stitching module 175 and stereoscopic image composition module 176 residing at the imaging device 500. In an embodiment, creating the real-time panoramic video image includes using a camera comprised of the first and second cameras 520₁、520₂Simultaneous real-time video images are acquired and creating the real-time 3D stereoscopic video images includes using a camera controlled by a second and third camera 520₂、520₃And simultaneously acquiring real-time video images.

In step S13, the real-time panoramic video image and the real-time 3D stereoscopic video image are transferred from the imaging apparatus 500 to the one or more remote servers 170 using the communication circuit 157 and the communication channel 201 of the imaging apparatus 500.

Step S14, simultaneously delivering from one or more remote servers 170 to a client browsing device 190 in communication therewith said real-time panoramic video image and said real-time 3D stereoscopic video image, delivered as live video streams 180, 181 respectively, said delivering being such that: (i) the real-time panoramic video stream 180 is the first plurality of devices 191 or 193 that is delivered to the client browsing device 190, while the real-time 3D stereoscopic video stream is the second plurality of devices 192 or 194 that is delivered to the client browsing device 190. In some embodiments, the two sets of multiple devices 191, 192 do not overlap, while in some embodiments, the two sets of multiple devices 193, 194 overlap such that a set of client browsing devices 190 in the overlap has two real-time video streams delivered to them. These real-time video streams are synchronized with each other to achieve smooth or seamless switching (transition) between them.

In some embodiments, not all steps of the method are performed. Steps from multiple methods may be combined into a synthetic method.

In some embodiments, either method may further include a storing and retrieving step in which the uploaded video images 111 or the uploaded stitched and composited panoramic and stereoscopic images are stored separately for subsequent processing, e.g., retrieval by a customer on demand.

The present invention has been illustrated by a detailed description of specific embodiments, which are provided by way of example, and which should not be taken as limiting the scope of the invention. The described embodiments comprise different features, which are not required in all embodiments of the invention. Some embodiments of the invention utilize only some of these features or possible combinations. While various embodiments of the invention have been described, these embodiments of the invention include various combinations of features described in the described embodiments, as will occur to those skilled in the art to which the invention pertains.

In this specification and in the claims, each of the verbs "comprise", "comprise" and "have", and their conjugates, are used to indicate a subject or object of the verb, but they are not necessarily a complete list of the subjects, members, elements or parts of the verb, or the subject or object. The singular forms "a", "an" and "the" include plural references unless the context clearly dictates otherwise. For example, the term "marker" or "at least one marker" may include a plurality of markers.

Claims

1. An imaging apparatus (500) for acquiring a plurality of simultaneous video images to produce 180 ° stereoscopic and 360 ° panoramic video streams, the imaging apparatus comprising:

a. a rigid body (501) comprising a first main surface (540)₁) And a second major surface (540)₂) And a bottom surface (545) on which a mechanical mount interface (550) is present;

b. first camera (520)₁) A second camera (520)₂) And a third camera (520)₃) Fixedly mounted to the rigid body (501), each camera (520) comprising a respective ultra-wide angle lens (530) and defining a respective optical axis (401), wherein:

i. the first camera (520)₁) Is arranged such that its lens (530)₁) From the first main surface (540)₁) Extends out and its optical axis (401)₁) Defining a first direction;

the second camera (520)₂) Is arranged such that its lens (530)₂) From the second main surface (540)₂) Extends out and its optical axis (401)₂) Defining a second direction that is opposite to the first direction, or is not more than 5 degrees opposite; and

the third camera (520)₃) Is arranged such that its lens (530)₃) From the second main surface (540)₂) Extends out and its optical axis (401)₃) Defining a third direction, the third direction being parallel to the second direction, or within 10 degrees of parallelism;

c. a video display module including a first major surface (540) mounted thereon₁) A display screen (510) on said display screen (510), said display screen (510) operable to display and/or play 180 ° or 360 ° images in response to input received through said display screen (510) and/or through a display control interface (515);

wherein:

i. the floor (545) defines an x-z plane oriented such that a long centerline C L x of the floor (545) has only an x-axis dimension;

a y-axis intersects the x-z plane perpendicularly and defines a vertical direction; and

when the imaging device (500) is unattended standing on a flat surface on its bottom surface (545), the first camera (520)₁) A second camera (520)₂) And a third camera (520)₃) Are vertically aligned with each other at the same height in the respective super wide-angle lenses (530)₁、530₂、530₃) Within a tolerance of one-half of the arbitrary diameter (908).

2. The imaging apparatus (500) of claim 1, wherein: the imaging device (500) is unattended standing on a flat surface of its bottom surface (545), the first camera (520)₁) A second camera (520)₂) And a third camera (520)₃) Are vertically aligned with each other at the same height in the respective super wide-angle lenses (530)₁、530₂、530₃) Within a tolerance of one fifth of the arbitrary diameter (908).

3. The imaging apparatus (500) of claim 1 or 2, wherein: the imaging device (500) is unattended standing on a flat surface of its bottom surface (545), the first camera (520)₁) A second camera (520)₂) And a third camera (520)₃) Are vertically aligned with each other at the same height, within a tolerance of 1 mm.

4. The imaging apparatus (500) of claim 1 or 2, wherein: the first camera (520)₁) And a second camera (520)₂) Are aligned with each other in the x-axis dimension to within a tolerance of 1 mm.

5. The imaging apparatus (500) of claim 1 or 2, wherein: the mechanical support interface (550) is to the first camera (520)₁) And a second camera (520)₂) Are aligned with each other in the x-axis dimension to within a 5 mm tolerance。

6. The imaging apparatus (500) of claim 1 or 2, wherein: the second camera (520)₂) Optical axis (401)₂) A second direction is defined that is opposite to the first direction, or no more than 1 degree opposite.

7. The imaging apparatus (500) of claim 1 or 2, wherein: the third camera (520)₃) Optical axis (401)₃) A third direction is defined that is parallel to the second direction, or within 5 degrees of parallelism.

8. The imaging apparatus (500) of claim 7, wherein: the third camera (520)₃) Optical axis (401)₃) A third direction is defined that is parallel to the second direction, or within 1 degree of parallelism.

9. The imaging apparatus (500) of claim 1 or 2, wherein: further comprising a first main surface (540)₁) First electronic status indicator (516) of (a)₁) And on the second main surface (540)₂) Second electronic status indicator (516)₂)。

10. The imaging apparatus (500) of claim 1 or 2, wherein: configured to acquire a plurality of simultaneous video images in a first mode, a second mode, or a third mode; in a first mode, a first camera (520)₁) And a second camera (520)₂) Acquiring simultaneous video images for splicing into a 360-degree panoramic video image; in the second mode, the second camera (520)₂) And a third camera (520)₃) Acquiring simultaneous video images for synthesizing a 180-degree stereoscopic video image; in a third mode, the first camera (520)₁) A second camera (520)₂) And a third camera (520)₃) Acquiring simultaneous video images for productionGenerating 180-degree stereo and 360-degree panoramic video images; the apparatus is arranged to respond to input received via the display screen (510) and/or via a display-control interface (515).