CN108460824B - Method, device and system for determining stereoscopic multimedia information - Google Patents

Method, device and system for determining stereoscopic multimedia information Download PDF

Info

Publication number
CN108460824B
CN108460824B CN201710091162.4A CN201710091162A CN108460824B CN 108460824 B CN108460824 B CN 108460824B CN 201710091162 A CN201710091162 A CN 201710091162A CN 108460824 B CN108460824 B CN 108460824B
Authority
CN
China
Prior art keywords
multimedia information
light source
active light
parallax
scanning
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201710091162.4A
Other languages
Chinese (zh)
Other versions
CN108460824A (en
Inventor
张文波
雷娟
李海洋
熊君君
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Samsung Electronics Co Ltd
Original Assignee
Samsung Electronics Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Samsung Electronics Co Ltd filed Critical Samsung Electronics Co Ltd
Priority to CN201710091162.4A priority Critical patent/CN108460824B/en
Publication of CN108460824A publication Critical patent/CN108460824A/en
Application granted granted Critical
Publication of CN108460824B publication Critical patent/CN108460824B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G06T15/10—Geometric effects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2219/00—Indexing scheme for manipulating 3D models or images for computer graphics
    • G06T2219/20—Indexing scheme for editing of 3D models
    • G06T2219/2008—Assembling, disassembling

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Computer Graphics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Geometry (AREA)
  • Architecture (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

The invention provides a method, a device and a system for determining three-dimensional multimedia information, wherein the method comprises the following steps: acquiring multimedia information respectively acquired by at least two multimedia information acquisition devices; and matching the acquired at least two pieces of multimedia information according to the at least one active light source to determine corresponding three-dimensional multimedia information. In the invention, the multimedia information collected by at least two multimedia information collecting devices comprises active light source marks, and each multimedia information can be matched according to the position difference of the same mark in different multimedia information to generate three-dimensional multimedia information; therefore, the overlapping ratio requirement on the acquisition angle between the multimedia information acquisition devices is greatly reduced, the number of the multimedia information acquisition devices can be greatly reduced, the weight, the volume and the cost of a three-dimensional multimedia information determining system are reduced, and the portability is greatly enhanced.

Description

Method, device and system for determining stereoscopic multimedia information
Technical Field
The invention relates to the technical field of visual processing, in particular to a method, a device and a system for determining stereoscopic multimedia information.
Background
VR (Virtual Reality) technology is a technology that provides an immersive sensation in a computationally generated, interactive, stereoscopic (three-dimensional) environment by comprehensively utilizing a computer graphics system and various interface devices such as display and control. With the development of computer software and hardware, VR has been developed in recent years, and a large number of consumer products emerge.
However, most of the existing VR products belong to the field of display technology, that is, play stereoscopic contents such as stereoscopic video for consumers to watch and enjoy. The development of production equipment for stereoscopic content is relatively limited. The generation of stereoscopic content is currently mainly performed by means of computer graphics, such as computer games, simulations, and the like. That is, the current stereoscopic content mainly includes virtual scenes, and the content from real scenes is still small. The inventor of the invention discovers that the reason for fewer real scenes in the stereoscopic content is that stereoscopic video acquisition of the real scenes is complex, technical difficulty is high, and a large amount of manual intervention is needed to generate a final panoramic product.
The generation or determination of stereoscopic video including real scenes is one of the sources of stereoscopic industry, and compared with other content generation methods, the photographed content is mainly real scenes, so that stereoscopic experience with excellent reality can be brought to users. The existing live-action stereoscopic content determination process involves related hardware and software. Hardware is mainly a plurality of groups of binocular shooting equipment at present, and software mainly comprises image processing technologies such as brightness, color and image quality adjustment; image stitching techniques, such as stitching of two-dimensional panoramas; depth restoration techniques such as binocular vision, multi-view, and the like.
However, the inventors of the present invention have noted that in order to achieve the display effect of virtual reality, the existing stereoscopic video determination method requires a plurality of image capturing apparatuses (e.g., 10 or 20 cameras), the number of image capturing apparatuses is large, resulting in the existing stereoscopic video determination apparatus being bulky, heavy (e.g., a weight of about 2 kg, a radius of at least about 20 cm), inconvenient to carry, and costly. Moreover, the number of image capturing apparatuses is large, the video data to be processed is also large, a large amount of computing resources are required to be consumed, and processing based on a server is likely to be required, further limiting portability and increasing cost.
Disclosure of Invention
The invention provides a method, a device and a system for determining three-dimensional multimedia information, aiming at the defects of the prior art, which are used for solving the problems of more imaging equipment (belonging to multimedia information acquisition equipment), larger volume, larger weight and/or higher cost in the prior art, so as to reduce the number of the multimedia information acquisition equipment, reduce the volume or weight of a system containing the multimedia information acquisition equipment or reduce the cost.
According to a first aspect, an embodiment of the present invention provides a method for determining stereoscopic multimedia information, including:
Acquiring multimedia information respectively acquired by at least two multimedia information acquisition devices;
and matching the acquired at least two pieces of multimedia information according to the at least one active light source to determine corresponding three-dimensional multimedia information.
According to a second aspect, an embodiment of the present invention further provides a device for determining stereoscopic multimedia information, including:
the multimedia information acquisition module is used for acquiring the multimedia information acquired by at least two multimedia information acquisition devices respectively;
the stereoscopic multimedia information determining module is used for matching the acquired at least two pieces of multimedia information according to the at least one active light source to determine corresponding stereoscopic multimedia information.
According to a third aspect, an embodiment of the present invention further provides a system for determining stereoscopic multimedia information, including: the device comprises at least two multimedia information acquisition devices, an active light source and the device for determining the three-dimensional multimedia information provided by the embodiment of the invention.
In the embodiment of the invention, at least two multimedia information acquisition devices acquire multimedia information respectively; setting an active light source in a determination system of the three-dimensional multimedia information, irradiating a shot object by the active light source, and irradiating a mark on the shot object; when the collected multimedia information contains a mark, the mark is equivalent to a matching point which is actively created; even if the repeated areas, feature points or image contents which can be mutually matched among the original multimedia information are less, the multimedia information can be matched according to the position difference of the same mark in different multimedia information, and the three-dimensional multimedia information can be generated; therefore, the overlapping ratio requirement on the acquisition angle between the multimedia information acquisition devices is greatly reduced, the number of the multimedia information acquisition devices can be greatly reduced, the weight, the volume and the cost of a three-dimensional multimedia information determining system are reduced, and the portability is greatly enhanced.
Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention.
Drawings
The foregoing and/or additional aspects and advantages of the invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:
fig. 1 is a schematic diagram of an example in which the positional relationship of two subjects in two images is different in the related art;
fig. 2 is a flow chart of a method for determining stereoscopic multimedia information according to the present invention;
fig. 3 is a flowchart illustrating an example of a method for determining stereoscopic multimedia information according to the first embodiment of the present invention;
fig. 4 is a schematic diagram of an example of adjusting the scanning density of an active light source according to the object distance corresponding to the photographed object according to the second embodiment of the present invention;
fig. 5 is a schematic diagram illustrating an example of adjusting the scanning density and/or scanning intensity of an active light source according to the surface curvature, surface hardness, and surface reflectivity of a subject according to the second embodiment of the present invention;
FIG. 6 is a diagram illustrating an example of adjusting the scanning density of an active light source according to the movement parameters of a photographed object according to the second embodiment of the present invention;
Fig. 7 is a schematic diagram of an example of adjusting the scanning intensity of an active light source according to the object distance or the acquisition environment corresponding to the photographed object according to the second embodiment of the present invention;
FIG. 8 is a schematic diagram illustrating an example of adjusting the light frequency of an active light source according to the collection environment according to the second embodiment of the present invention;
fig. 9 is a schematic diagram of a principle framework for adjusting scanning density, scanning intensity and/or optical frequency of an active light source according to a subject and/or an acquisition environment;
FIG. 10 is a diagram showing an example of a density distribution after adjusting the scanning density of the active light source according to the second embodiment of the present invention;
FIG. 11 is a schematic diagram showing an example of a density distribution after adjusting the scanning density of the active light source according to the movement parameters of the photographed object according to the second embodiment of the present invention;
fig. 12 is a schematic diagram showing an example of adjusting the scanning density/intensity of an active light source according to the size/surface curvature of a subject according to the second embodiment of the present invention;
fig. 13a is a schematic diagram of an example of an acquisition environment with/without ambient light interference according to the second embodiment of the present invention;
fig. 13b is a schematic diagram illustrating an example of adjusting the scanning intensity of the active light source according to the object distance of the photographed object according to the second embodiment of the present invention;
FIG. 14 is a diagram showing an example of adjusting the light frequency of an active light source according to the collection environment according to the second embodiment of the present invention;
fig. 15 is a schematic diagram of a specific example of the existing stereoscopic video capturing and viewing principle introduced in the third embodiment of the present invention;
fig. 16 is a schematic diagram showing an example of a first method for regularizing parallax of multimedia information according to a third embodiment of the present invention;
fig. 17 is a schematic diagram of an example of a second method for regularizing parallax of multimedia information according to a third embodiment of the present invention;
fig. 18 is a schematic diagram of the internal structure of a stereoscopic multimedia information determining device according to the fourth embodiment of the present invention.
Detailed Description
Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative only and are not to be construed as limiting the invention.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless expressly stated otherwise, as understood by those skilled in the art. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. The term "and/or" as used herein includes all or any element and all combination of one or more of the associated listed items.
It will be understood by those skilled in the art that all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs unless defined otherwise. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.
The inventor of the present invention has found through research that, by directly reducing the number of image pickup apparatuses, problems are likely to occur in determining a stereoscopic video using the existing method, and it is easy to cause that a stereoscopic video cannot be obtained.
Specifically, after the number of the image capturing apparatuses is directly reduced, each group of two image capturing apparatuses is still responsible for capturing stereoscopic video of one angle. The first situation is that after the number is reduced, the difference of the acquisition angles between the two image capturing devices in the group is too large, so that the problems that a repeated area does not exist between the two acquired video frame images, the repeated area is very small, or the deformation of the repeated area is large and the like are easily caused, and the two acquired video frame images cannot be matched, so that a stereoscopic video cannot be generated. For example, although there is a repetitive region between binocular video frame images, since parallax is large, after the acquired video frame images are subjected to 3D to 2D projection (for example, spherical projection, cylindrical projection, mercator projection, or gaussian-kriging projection, etc.) transformation, there is a large deformation of the repetitive region in both left-eye and right-eye frame images, resulting in difficulty in matching the left-eye and right-eye frame images.
The second condition is that after the number is reduced, the difference of the acquisition angles between the groups is too large, so that the problems that repeated areas between video frame images acquired by the same-purpose camera equipment of different groups are very small or the repeated areas are large in deformation and the like are easily caused, the video frame images acquired by the two groups cannot be matched, the splicing cannot be performed, and a stereoscopic video cannot be generated.
Even more serious, the number of image pickup apparatuses is directly reduced, and the above-described several cases may all have consequences. Therefore, the existing method needs to adopt a large number of image pickup devices to ensure that a large overlap ratio exists between the shooting angles of different image pickup devices, so that enough overlap areas, feature points or image contents exist between video frame images shot by different image pickup devices for matching, and the success rate of generating the stereoscopic video is ensured.
The inventors of the present invention have further studied and found that even without reducing the number of image pickup apparatuses, problems of image mismatch still easily occur with the existing methods.
In the third case, the images to be stitched originate from different image capturing apparatuses, and the positions of the respective image capturing apparatuses are fixed and different, resulting in different angles at which the same subject is observed by the different image capturing apparatuses. Thus, the same subject has a positional difference in the fields of view of different photographing apparatuses; and the closer the object distance between the photographed object and the photographing apparatus is to the distance between the collecting apparatuses, the larger the position difference is. The presence of such a positional difference is likely to cause a positional relationship of two subjects in each image to be stitched to be different, and is likely to cause image stitching failure.
Fig. 1 is a schematic diagram of an example in which the positional relationship of two subjects in two images is different. In fig. 1, two objects in a real scene are elliptical and triangular, cameras a/B respectively represent image capturing apparatuses a/B, and cameras a/B photographs respectively represent images captured by the image capturing apparatuses a/B. In fig. 1, the ellipse and triangle in the real scene are on a straight line, and the ellipse is behind the front triangle, and the ellipse can be blocked by the triangle in the front view. However, since the observation (photographing) angles of the image pickup apparatuses a and B are different, the ellipse in the image picked up by the image pickup apparatus a is located on the right side of the triangle, and the ellipse in the image picked up by the image pickup apparatus B is located on the left side of the triangle. The two images cannot be matched due to the fact that the left-right position relationship of the ellipse and the triangle in the two images is different.
Fourth, there is no texture in both the left and right eye images (e.g., white walls) resulting in a mismatch between the two eye images. Alternatively, the left and right eye images contain large areas of repeated textures or similar textures (e.g., tiles, grass, or curtain walls of architectural surfaces, etc.), which are prone to mismatch or failure of match.
Based on the above findings, the present invention provides a determination system for stereoscopic multimedia information, comprising: at least two multimedia information acquisition devices and an active light source.
The multimedia information acquisition device in the invention can be specifically an image pickup device, a photographing device and/or the like.
The system for determining stereoscopic multimedia information of the present invention further comprises: a determination device for the stereoscopic multimedia information.
The invention provides a method for determining three-dimensional multimedia information, a flow diagram of the method is shown in fig. 2, and the method comprises the following steps: s201, acquiring multimedia information respectively acquired by at least two multimedia information acquisition devices; s202, matching the acquired at least two pieces of multimedia information according to at least one active light source, and determining corresponding three-dimensional multimedia information.
In the present invention, the multimedia information includes: pictures, and/or video. The pictures may be individual pictures or may be consecutive pictures. The video includes a frame image.
The active light source refers to radiation capable of emitting light attributes, including but not limited to electromagnetic waves in the visible light band and the non-visible light band, and may encode the light radiation so that it has global uniqueness or local uniqueness in spatial distribution.
It can be understood that in the present invention, at least two multimedia information collecting devices collect multimedia information respectively; setting an active light source in a determination system of the three-dimensional multimedia information, irradiating a shot object by the active light source, and irradiating a mark on the shot object; when the collected multimedia information contains a mark, the mark is equivalent to a matching point which is actively created; even if the repeated areas, feature points or image contents which can be mutually matched among the original multimedia information are less, the multimedia information can be matched according to the position difference of the same mark in different multimedia information, and the three-dimensional multimedia information can be generated; therefore, the overlapping ratio requirement on the acquisition angle between the multimedia information acquisition devices is greatly reduced, the number of the multimedia information acquisition devices can be greatly reduced, the weight, the volume and the cost of a three-dimensional multimedia information determining system are reduced, and the portability is greatly enhanced.
Several embodiments of the technical solution of the present invention are specifically described below.
Example 1
The inventor of the present invention has studied and found that an existing method for determining a stereoscopic video includes: shooting and obtaining video frame images of a plurality of angles through a plurality of groups of camera equipment respectively; wherein each group of image capturing apparatuses includes left-eye and right-eye image capturing apparatuses; the video frame images for each angle include left eye and right eye frame images. And converting frame images shot and acquired by the same-eye image pickup equipment in the image pickup equipment of each group of different angles into 2D images, and then splicing the 2D wide-angle or panoramic images of the purpose, thereby obtaining left-eye and right-eye 2D wide-angle or panoramic images. When the left-view and right-view 2D wide-angle or panoramic images are simultaneously played in the stereoscopic playing device, a user can feel that the real scene in the 2D wide-angle or panoramic images is as if the user were around the body, so that the user generates an immersive sensation of being in the body, and the display effect of certain virtual reality is achieved.
Another existing method for determining stereoscopic video is mainly different from the above method for determining stereoscopic video in that the step of stitching images is different. Specifically, the step of stitching the images mainly includes: and searching sparse matching points in different images, matching with a post-processing technology, so as to obtain dense matching, and finally, realizing image stitching according to the dense matching. Such methods are computationally long, requiring powerful servers, or server farm support.
The first embodiment of the invention expands and introduces a system and a method for determining stereoscopic multimedia information.
The number of the multimedia information acquisition devices is at least two. The number and arrangement of the multimedia information collecting devices of the present invention can be determined by those skilled in the art according to the required view angle range of the stereoscopic multimedia information and the view angle range of the adopted multimedia information collecting device.
Preferably, the view angle range required by the stereoscopic multimedia information can reach 360 degrees. When a fisheye lens, an ultra-wide angle lens or a common zoom lens is respectively arranged in the multimedia information acquisition equipment, the view angle range is the maximum view angle range of the lens. The maximum view angle range of the fish-eye lens can reach 180 degrees and even can reach about 220 degrees; the wide-angle end view angle range of the ultra-wide-angle lens can reach about 108-110 degrees; the wide-angle end field angle range of a common zoom lens can reach 70-80 degrees.
The number of the multimedia information collecting devices may be 2 to 10 and may be an odd number; when an odd number of multimedia information collecting devices is used, at least one of the multimedia information collecting devices can be multiplexed as a constituent part of a different group of multimedia information collecting devices. For example, when 7 multimedia information collecting devices are used, the 6 th multimedia information collecting device can be multiplexed, the 5 th and 6 th multimedia information collecting devices form the 3 rd group of multimedia information collecting devices, and the 6 th and 7 th multimedia information collecting devices form the 4 th group of multimedia information collecting devices.
For example, when the required viewing angle range of the stereoscopic multimedia information is 180 degrees and a fisheye lens having a 180-degree viewing field is used in the multimedia information collecting device, 2 multimedia information collecting devices may be used, the optical axes are arranged in parallel, and the optical axes are all right opposite to the 90 th direction in the required viewing angle range.
For example, when the required viewing angle range of the stereoscopic multimedia information is 360 degrees, and the multimedia information collecting device is provided with an ultra-wide angle lens, for example, a viewing angle greater than 180 degrees, 4 multimedia information collecting devices may be used, and the optical axes of the 4 multimedia information collecting devices are respectively opposite to the 0 th, 90 th, 180 th and 270 th directions in the required viewing angle range, so that each angle is covered by two multimedia information collecting devices.
The required view angle range of the stereoscopic multimedia information is typically a horizontal view angle range. Further, when the required viewing angle range of the stereoscopic multimedia information is generally a stereoscopic spherical range, lenses in the multimedia information collecting device may be all set as fisheye lenses; or, in the case of maintaining an ultra-wide angle lens or a normal zoom lens, the multimedia information collecting devices are added in upward and downward directions (directions toward zenith and geocenter when the plane is parallel to the horizontal plane) perpendicular to the plane in which the respective multimedia information collecting devices are located.
The active light source in the present invention may include at least one active light emitting cell array.
When the number of the active light emitting unit arrays is large, the adjacent active light emitting unit arrays are connected, which is equivalent to that the active light emitting unit arrays are connected into a whole, and the whole irradiation angle can cover 360 degrees.
When the number of the included active light emitting unit arrays is small, the active light emitting unit arrays are distributed in a scattered manner within a coplanar 360-degree range, and each active light emitting unit array is responsible for irradiating a visual field angle range. The positions of the active luminous unit arrays and the multimedia information acquisition devices are reasonably configured, so that the light spots emitted by each active luminous unit array can fall in the visual field range of at least two multimedia information acquisition devices.
When only one active light emitting unit array is included, the active light source can be integrally rotated, and the rotating surface is parallel to the plane where the plurality of multimedia information acquisition devices are located; so that the light emitted from the active light emitting cell array in the active light source can cover 360 degrees.
The active light emitting unit array comprises a plurality of active light emitting units. The light frequency, switching (matching the scanning density and/or scanning interval time of the active light emitting cell array, or active light source), light intensity of each active light emitting cell can be controlled individually.
Preferably, the plurality of active light emitting cells in each active light emitting cell array may be divided into a plurality of groups of active light emitting cells. The light frequency, the switch and the light intensity of the same group of active light emitting units are uniformly controlled.
For example, one active light emitting unit array includes six groups of active light emitting units, and the corresponding groups of active light emitting units are turned on according to the required gear of the scanning density. Specifically, when the scanning density of the active light emitting unit array in the active light source is required to be zero, all the active light emitting units in the active light emitting unit array are turned off; when the required scanning density is the lowest grade scanning density, a group of active light-emitting units are started to emit light; when the required scanning density is the highest scanning density, the six groups of active light emitting units are started to emit light.
Further, in order to make the array emit light more uniformly, one active light emitting unit can be selected from six groups of active light emitting units respectively to be adjacent to each other, and the active light emitting units are spliced into one active light emitting unit combination. When each active light emitting unit is hexagonal, the active light emitting units are spliced into a honeycomb-shaped structure, and the whole array is also in the honeycomb-shaped structure.
The light frequency of the active light source is matched with the detectable frequency of the multimedia information acquisition equipment, namely, the light frequency of the active light source is the detectable frequency of the multimedia information acquisition equipment. For example, the optical frequency of the active light source may be electromagnetic waves in the infrared band, the visible band, or the ultraviolet band, or the like.
Further, the active light emitting units in the active light source can be point light sources, linear light sources or coded light spots and other distribution forms.
The aggregation degree of the emitted light of the active light source needs to reach the set requirement. For example, a laser may be used. The active light emitting unit in the active light source may be a laser. Further, the active light emitting unit may be an optical frequency tunable laser, such as a DBR (Distributed Bragg Reflector, distributed bragg reflection) laser. Further, a plurality of lasers of different optical frequencies may be configured for each active light emitting unit.
The method of determining stereoscopic multimedia information is described in detail below.
Acquiring multimedia information respectively acquired by at least two multimedia information acquisition devices; matching the acquired at least two pieces of multimedia information according to the at least one active light source; and determining corresponding stereoscopic multimedia information according to the matched at least two pieces of multimedia information.
Specifically, at least two multimedia information acquisition devices acquire multimedia information independently; at least one active light source independently irradiates while rotating; the multimedia information respectively collected by the at least two multimedia information collecting devices may include marks irradiated on the photographed object by the active light source.
The subject may include at least one of: subject, subject environment.
The mark irradiated on the photographed object by the active light source comprises at least one of the following: light spots, grid light, coded light spots; the light spot comprises at least one of the following: single light spot, light spot set and light spot cluster
Preferably, when the active light source includes a single active light emitting unit array, at least two multimedia information collecting devices respectively and independently collect multimedia information. At least one active light source independently illuminates while rotating.
Preferably, the positions of at least two multimedia information collecting devices are known when the positions are fixed. According to the principle that the active light irradiates to assist the multimedia information acquisition equipment to acquire the multimedia information, when at least two multimedia information acquisition equipment acquire the multimedia information respectively, an active light emitting unit array with the irradiation angle in the active light source matched with the acquisition direction of the multimedia information acquisition equipment is called to irradiate; and matching the acquired multimedia information according to the mark irradiated on the shot object.
Preferably, when the active light source includes a plurality of active light emitting unit arrays, for each pair of the multimedia information collecting devices, the active light emitting unit arrays whose irradiation angles are matched with the collection directions of the pair of the multimedia information collecting devices are called for irradiation. The acquisition direction may be any direction in the acquisition view angle of the multimedia information acquisition device, for example, the direction in which the optical axis is located.
Specifically, for the case that a plurality of active light emitting unit arrays in the active light source are mutually connected to form an integral array covering 360 degrees, for each pair of multimedia information acquisition equipment, the active light emitting unit arrays with the irradiation angles matched with the acquisition directions of the pair of multimedia information acquisition equipment can be selected to irradiate through switching on and switching off the active light emitting units in the plurality of active light emitting unit arrays, so that the acquisition view angles (including the acquisition directions) of the multimedia information acquisition equipment are basically consistent with the irradiation angles of the active light source. The multimedia information acquisition equipment with basically consistent angles comprises marks of most active light sources irradiating a shot object.
For the case that a plurality of active light emitting unit arrays in the active light source are not connected with each other, but discrete to cover respective view angles, for each pair of multimedia information acquisition devices, the active light emitting unit array with the largest overlapping area of the irradiation angle and the acquisition view angle of the pair of multimedia information acquisition devices is selected for irradiation. The pair of multimedia information acquisition devices can acquire multimedia information which contains marks irradiated on a shot object by the active light source as much as possible.
Matching the acquired at least two pieces of multimedia information according to the at least one active light source, including: and matching the different multimedia information according to the position difference of the marks irradiated on the shot object by the active light source in the different multimedia information. Wherein the indicia comprises at least one of: light spots, grid light, encoded light spots.
Preferably, the corresponding stereoscopic multimedia information is determined according to the position difference of the marks irradiated on the photographed object by the active light source in different multimedia information.
Specifically, for each group of two multimedia information acquisition devices, the multimedia information respectively acquired by the two multimedia information acquisition devices contains marks irradiated on the photographed object by the active light source. Either a single spot or a plurality of spots, e.g. an array of spots, or other encoded spot distribution.
Taking a light spot array as an example, when a plurality of designated active light emitting units in the active light source irradiate the designated light spot array, all or a part of the light spot array is contained in the collected multimedia information. By designing the distribution form of each light spot in the light spot array, the relative position of each light spot can be unique; that is, each spot in the array of spots is unique.
For each group of two multimedia information acquisition devices, each mark in the multimedia information acquired by one multimedia information acquisition device is determined, and the mark is determined from the multimedia information acquired by the other multimedia information acquisition device; determining the position difference of the mark in the two pieces of multimedia information; and matching the two pieces of multimedia information according to the respective position differences of the marks, determining the parallax between the two pieces of multimedia information, and further determining the corresponding three-dimensional multimedia information as the corresponding three-dimensional multimedia information or a part thereof.
Preferably, the multimedia information collected by the multimedia information collecting devices in different collecting directions is processed, for example spliced, according to the position difference in the multimedia information collected by the multimedia information collecting devices in different collecting directions of the mark irradiated on the photographed object by the active light source.
Specifically, for two adjacent groups of multimedia information acquisition devices with different acquisition view angles (including acquisition directions), the active light source irradiates the same mark on the photographed object, and the mark falls within the acquisition view angle range of one-mesh (for example, left-mesh) multimedia information acquisition device of one group and the acquisition view angle range of the corresponding-mesh (for example, left-mesh) multimedia information acquisition device of the other group. Therefore, the multimedia information collected by the same-purpose multimedia information collecting equipment of two adjacent groups contains the same mark. For example, the mark may be a single spot or an array of spots.
For two adjacent groups of multimedia information acquisition equipment with the same purpose, each mark in the multimedia information acquired by one multimedia information acquisition equipment is determined, and the mark is determined from the multimedia information acquired by the other multimedia information acquisition equipment; determining the position difference of the mark in the two pieces of multimedia information; and matching the two pieces of multimedia information according to the respective position differences of the marks, determining the parallax between the two pieces of multimedia information, splicing the two pieces of multimedia information according to the parallax, and the like, and further determining the corresponding three-dimensional multimedia information or a part of the three-dimensional multimedia information.
Preferably, the method for determining stereoscopic multimedia information according to the first embodiment of the present invention further includes: according to the shot object and/or the acquisition environment, at least one of the scanning density, the scanning intensity, the optical frequency and the scanning interval time of the active light source are adjusted; the details will be described in detail in the following second embodiment, which is not repeated here.
Preferably, the method for determining stereoscopic multimedia information according to the first embodiment of the present invention further includes:
the parallax of the collected multimedia information is regularized, and the specific content will be described in detail in the following third embodiment, which is not repeated here.
A flowchart of an example of a method for determining stereoscopic multimedia information according to the first embodiment of the present invention is described below, as shown in fig. 3.
The following steps in the flow are presented in sequence in fig. 3: the parameters loaded into the camera represent parameters loaded into the multimedia information acquisition equipment, and the camera represents the multimedia information acquisition equipment; the active light source is turned on. Acquiring image representing multimedia information by the multimedia information acquisition equipment; the non-distortion and face-lifting means that distortion and face-lifting correction are carried out on the acquired multimedia information; registering information of the active light source; regularization of parallax; visual sequence detection, such as parallax detection, is performed to determine matching relations and splicing sequences among multimedia information acquired by the multimedia information acquisition equipment with different acquisition view angles; seam searching, for example, marks at the outline of the same shot object in two pieces of multimedia information to be spliced are relatively dense, curves to be spliced (for example, folding lines with multiple marks as nodes) are drawn according to the marks, and the two pieces of multimedia information are spliced according to the curves to be spliced; color mixing; stereoscopic vision inspection; stereoscopic rendering of the panoramic image; and outputting the stereoscopic multimedia information.
In the first embodiment of the invention, even if the repeated area is smaller, the deformation of the repeated area is larger, the position relationship is different, or a large number of repeated textures exist among the collected multimedia information of different multimedia information collecting devices, the position difference of the mark of the active light source irradiated on the shot object in the different multimedia information can be determined, the different multimedia information is matched according to the position difference, and the parallax among the different multimedia information is determined; further, according to the parallax between the multimedia information acquired by the same group of multimedia information acquisition equipment, determining the stereoscopic multimedia information corresponding to the group of view angles; and splicing the multimedia information according to the parallax among the multimedia information of different groups of corresponding purposes, and determining the virtual reality (surround three-dimensional) multimedia information corresponding to a plurality of groups of view angles. The matching success rate of the multimedia information is greatly improved, and the efficiency of determining the stereoscopic multimedia information can be improved.
Example two
In the second embodiment of the present invention, a method for adjusting at least one of a scanning density, a scanning intensity, an optical frequency, and a scanning interval time of an active light source according to a subject and/or an acquisition environment is specifically described.
Specifically, the scanning density and/or scanning intensity of the active light source are adjusted according to the spatial attribute of the photographed object and/or the content attribute of the photographed object. The spatial properties of the subject include at least one of: object distance and movement parameters corresponding to the shot object. The content attribute of the subject includes at least one of: the size and surface feature properties of the subject; the surface-feature attributes include at least one of: surface curvature, surface hardness, surface reflectivity.
Preferably, the scanning density of the active light source is adaptively adjusted according to the object distance corresponding to the photographed object.
There are two methods for determining the object distance corresponding to the object:
the first method is that the active light source periodically performs omnibearing (360 degrees) scanning irradiation on surrounding real scenes, and the multimedia acquisition equipment performs omnibearing acquisition on the surrounding real scenes; and determining the object distance of each surrounding shot object according to the acquired multimedia information.
The second method is to keep the active light source in the off state, and directly collect all-round objects in the surrounding real scene by the multimedia information collecting equipment; and determining rough object distances of all surrounding shot objects according to the acquired multimedia information.
According to the object distance and the rough object distance which are respectively determined by the first method and the second method, the scanning density of the active light source aiming at the shot objects with different object distances is adaptively adjusted.
Preferably, the greater the object distance, the greater the corresponding scan density. Specifically, the scanning density of the active light source is adjusted to irradiate a subject having a larger subject distance, and the scanning density is adjusted to irradiate a subject having a smaller subject distance.
Fig. 4 is a schematic diagram of an example of adjusting the scanning density of the active light source according to the object distance corresponding to the subject. The steps in the flow in fig. 4 sequentially include: the full scanning means that the full-directional multimedia information acquisition is carried out on the surrounding reality scene or the full-directional multimedia information acquisition is carried out on the surrounding reality scene under the cooperation of the full-directional scanning irradiation of the active light source; the depth map represents a map containing depth information obtained based on comprehensive acquisition; the active light source density adjustment means that the scanning density of the active light source is adjusted; the spatial distribution represents the spatial distribution of the light emitted by the active light source after adjustment of the scanning density of the active light source.
Therefore, the scanning density of the active light source is adjusted according to the object distances of different shot objects (such as objects) in the same reality scene, so that the active light source is prevented from always working in the state of the maximum scanning density, the energy consumption of the active light source is saved, the working time of the multimedia information acquisition equipment can be prolonged, the capacity, the volume and the weight of functional equipment (such as a battery) can be reduced, and the portability and the mobility of a determination system of the three-dimensional multimedia information are further enhanced.
Preferably, the scanning density and/or scanning intensity of the active light source are/is adjusted according to the content attribute of the photographed object. The content attribute of the subject includes at least one of: the size and surface feature properties of the subject; the surface-feature attributes include at least one of: surface curvature, surface hardness, surface reflectivity.
Specifically, the real-world object (e.g., an object) generally has characteristics of a size, a degree of surface curvature, a flexible surface, and a reflective surface, and according to these characteristics, the active light source is adaptively adjusted according to the scheme, so that energy consumption is saved while the same effect is maintained.
Size: the multimedia information of the shot object is acquired through the multimedia information acquisition equipment, the multimedia information containing the shot object is detected, identified and segmented, the size of the shot object in the view angle of the multimedia information acquisition equipment is judged, and the scanning density of the active light source in space is adjusted according to the size.
The shot object with larger size is usually closer to the multimedia information acquisition equipment (namely, the object distance is smaller), or has larger projection area than the shot object with the same object distance and smaller size; even if the scanning density of the active light source for the shot object with larger size is reduced, the area where the shot object is located in the multimedia information acquired by the multimedia information acquisition equipment can also contain more marks. On the contrary, the scanning density of the active light source for the shot object with smaller size needs to be increased, so that the shot object in the multimedia information acquired by the multimedia information acquisition device can contain enough marks in the area.
And the surface curvature is obtained by collecting the multimedia information of the shot object through the multimedia information collecting equipment, detecting and identifying the multimedia information containing the shot object, judging the surface curvature of the shot object according to the identification result, and adjusting the space scanning density of the active light source based on the surface curvature. In addition, when the active light source is called to perform full-angle full-density scanning on the surrounding real scene, the multimedia information acquisition equipment is synchronously called to acquire the multimedia information in full-angle on the surrounding real scene, and the surface curvature of a shot object in the multimedia information is determined according to the multimedia information acquired under the cooperation of the active light source and the irradiation; according to the determined surface curvature of the shot object, the scanning density of the active light source for the space region where the shot object is located is adjusted; the follow-up active light source scans and irradiates the space region of the shot object according to the adjusted scanning density, and meanwhile, the multimedia information synchronously collects the multimedia information of the shot object.
When the emitted light of the active light source irradiates on objects with different surface curvatures, the mathematical description relationship of the plane can be determined by three points on the surface with small curvature, such as the plane, and the space scanning density of the active light source can be minimized; for surfaces with large curvatures, more points are needed to build mathematical models of the surfaces, so that the spatial scanning density of the active light source is required to be larger and to increase with increasing curvature, so that enough marking information can be provided to effectively achieve matching and post-processing.
More preferably, when the surface curvature of the photographed object is a preset curvature gear, the scanning density of the active light source for the photographed object is adjusted to be corresponding to the preset scanning density gear.
For example, when the surface curvature of the object is a preset large curvature gear and a preset small curvature gear, the scanning density of the active light source for the object is respectively adjusted to be a corresponding dense scanning density gear and a corresponding sparse scanning density gear.
Flexible surface: the multimedia information of the shot object is acquired through the multimedia information acquisition equipment, the multimedia information containing the shot object is detected and identified, whether the surface of the shot object is flexible and easy to deform is judged according to the identification result and knowledge stored in advance, and the spatial scanning density of the active light source is adjusted according to the detection result and the knowledge.
When the deformation rate of the flexible surface is high, the time interval of the active light source scanning needs to be adjusted to reflect the shape change of the real object. Therefore, the scanning interval time of the active light source is adjusted according to the deformation rate of the photographed object. For example, a balloon may have different deformation rates from no gas to full gas, and when the inflation rate is slow, the deformation is slow under the view angle of multimedia information acquisition, so the scanning time interval for scanning the balloon may be longer; correspondingly, when the inflation rate is high, the deformation is high, so that the shape of the balloon is correctly reflected, and the scanning time interval is short.
The flexible surface is easily deformed, and the deformed surface is similar to the surface with larger curvature. It is therefore desirable that the active light source illuminate the subject with a large surface curvature at a large scanning density so that the multimedia information acquisition device simultaneously acquires a sufficient number of substantially non-deforming marks on the subject to facilitate subsequent matching of the multimedia information based on the marks.
More preferably, when the surface hardness of the shot object is flexible and rigid, the scanning density of the active light source for the shot object is respectively adjusted to be a dense scanning density gear and a sparse scanning density gear.
Reflective surface: the mirror surface appearance of the shot object can reflect the light rays emitted by the active light source to other places, the mark is difficult to be reserved on the surface of the shot object, the mark on the surface of the shot object is easy to be acquired by the multimedia information acquisition equipment, and the difficulty is easy to be caused for matching between subsequent multimedia information based on the mark. Therefore, from the multimedia information acquired by the multimedia information acquisition apparatus, the specular surface of the subject having reflectivity is detected and identified. The active light source is adjusted to avoid emitting light to the surface of the mirror portion of the subject. Or, based on a reflection model acquired in advance, parallax adjustment is performed on the subject in the mirror surface, so that the parallax of the subject seen through the mirror surface is reflected as truly as possible.
More preferably, when the surface reflectivity of the object is specular reflection, the scanning intensity of the active light source for the area within the edge of the object is adjusted to a low scanning intensity gear, and/or the scanning intensity for the area within the edge of the object is adjusted to a high scanning intensity gear. Preferably, the low scan intensity shift may specifically include zero scan intensity, i.e. no scan.
The principle of adjusting the scanning intensity for the region within the edge of the object to the high scanning intensity range is as follows: when the surface reflectivity of the photographed object is diffuse reflection, for the multimedia information collecting device, the object distance is the distance of the photographed object to the multimedia information collecting device when the multimedia information of the photographed object is collected.
When the surface reflectivity of the photographed object is specular reflection, for the multimedia information collecting device, when collecting the multimedia information of the photographed object, the equivalent object distance includes the distance from the mirror to the multimedia information collecting device and the distance from the mirror to the photographed object (imaging surface) according to the principle of specular reflection. That is, at this time the equivalent object distance at which the multimedia information collecting device collects the multimedia information of the object including the mark increases; thus, an active light source (in the form of a scan) is required to emit light of a higher intensity than diffuse reflection in order to keep the multimedia information collecting device able to collect a sufficiently large number of marks of a sufficiently good quality to keep the matching of subsequent mark-based multimedia information smooth.
Fig. 5 is a schematic diagram of an example of adjusting the scanning density and/or scanning intensity of an active light source according to the surface curvature, surface hardness, surface reflectivity of a subject. The steps in the flow in fig. 5 sequentially include: full scanning means that the surrounding real scene is subjected to full-scale multimedia information acquisition or is subjected to full-scale multimedia information acquisition under the cooperation of full-scale scanning and irradiation of an active light source; the color camera detecting/recognizing an object representing that the object as a subject is detected/recognized from the multimedia information acquired by the color camera as the multimedia information acquisition device; the object surface characteristic represents the surface characteristic attribute of the determined shot object; the active light source density adjustment means that the scanning density of the active light source is adjusted; the spatial distribution represents the spatial distribution of the light emitted by the active light source after adjustment of the scanning density of the active light source.
Preferably, the scanning density of the active light source is adjusted according to the movement parameters of the photographed object.
Specifically, a real-world subject includes two movement attributes, namely, relative stationary and relative movement. For a moving subject, the invention adopts a detection and prediction mode to control the switching of the active light source. For a multimedia information acquisition device, the movement of the subject may also be divided into tangential movement and/or radial movement with respect to the multimedia information acquisition device.
Firstly, a multimedia information acquisition device acquires multimedia information of a shot object, detects and tracks the moving shot object in the multimedia information, predicts moving parameters (such as a moving track, a moving speed and/or a moving direction) of the shot object, and based on the moving parameters, turns on and off different active light emitting units of an active light source and/or adaptively pre-adjusts scanning density of the active light source; so that when the subject moves to the next position, the active light source can synchronously track the irradiation to the next position or increase the scanning (irradiation) density of the next position according to the predicted movement parameter, and stop the irradiation to other positions or reduce the scanning (irradiation) density of the other positions.
Secondly, the multimedia information acquisition equipment acquires the multimedia information of the mobile shot object under the cooperation of the active light source. The other steps are the same as those of the first method, and are not repeated.
Fig. 6 is a schematic diagram of an example of adjusting the scanning density of an active light source according to a movement parameter of a subject. The steps in the flow in fig. 6 sequentially include: color camera motion vector estimation, which means estimating a motion vector (movement parameter) of a subject in multimedia information acquired by a color camera as a multimedia acquisition device; object position prediction, which means predicting a moving position of an object as a subject; active light source adjustment, which means adjustment of scanning density and/or scanning intensity of an active detector as an active light source; the active light source is aligned to the new position, which means that the detection of the movement parameter of the next period and the adjustment of the active light source are performed at the new movement position.
Preferably, the scanning intensity of the active light source is adjusted according to the object distance or the acquisition environment corresponding to the shot object.
In particular, in addition to adaptively adjusting the scanning density of the active light source, the intensity of the light emitted by the active light source in a scanning manner may also be adaptively adjusted.
Firstly, multimedia information under the cooperation of an active light source is collected by multimedia information collection equipment, the parallax of a mark irradiated on a shot object is identified, and further the depth information of the mark is determined and used as the object distance of the shot object where the mark is positioned; according to the object distance, the energy intensity of the emitted light of the active light source is adjusted. The greater the object distance, the greater the intensity adjustment of the light emitted by the active light source.
And secondly, turning off the active light source, independently collecting multimedia information by the multimedia information collecting equipment, identifying parallax on the shot object from the multimedia information, further determining the depth information of the shot object, and adjusting the energy intensity emitted by the active light source according to the depth information of the shot object serving as a surrounding scene.
Moreover, this manner of adjusting the intensity of the light emitted in a scanning fashion is related to the environment of the site, such as indoors and outdoors, whether daytime or evening, whether there is an interfering light source in the site, etc. When the background light intensity of the surrounding environment is weak, for example indoors and/or at night, the intensity of the light emitted by the active light source is reduced. When the background light intensity of the surrounding environment is strong, for example, when an interfering light source exists, the intensity of the light emitted by the active light source is increased.
Preferably, the scanning intensity of the active light source is limited below a preset safe scanning intensity threshold in view of the safety of the human eye. The active light source's distance of action is also suitably limited, for example to 5-10 meters.
Fig. 7 is a schematic diagram of an example of adjusting the scanning intensity of an active light source according to the object distance or the acquisition environment corresponding to the subject. The steps of the flow in fig. 7 sequentially include: an active light source, which means that the active light source is called for irradiation; a depth map representing depth maps of surrounding subjects; the color camera is used for representing and calling the color camera belonging to the multimedia information acquisition equipment to acquire the multimedia information; environmental analysis, namely analyzing the acquisition environment of the site and identifying the illumination condition of the site environment; adjusting the emission intensity based on the distance means adjusting the energy intensity of the light emitted by the active light source based on the object distance and/or the acquisition environment.
Preferably, the light frequency of the active light source is adjusted according to the collection environment.
The inventors of the present invention have noted that ambient light sources may interfere with the active light source, rendering the active light source inoperable under certain conditions, and thus, adaptation to the ambient light source is necessary. In the invention, the following mechanism is adopted, and the main frequency of the environment light source is avoided by changing the frequency of the light emitted by the active light source, so that the self-adaption is achieved.
Specifically, the light frequency of the active light source is adjusted according to the light frequency of the interfering light source in the acquisition environment.
Preferably, the light frequency of the active light source is adjusted to be out of the range of the light frequency of the interference light source in the acquisition environment. Or, invoking the active light source to emit light with various frequencies in an attempted manner; and determining the selected frequency according to the test result of the multimedia information collected under the irradiation of the light with each frequency, and further adjusting the light frequency of the active light source to the selected frequency.
Fig. 8 is a schematic diagram of one example of adjusting the light frequency of an active light source according to the acquisition environment. FIG. 8 shows pulses of frequency A/B, indicating light pulses of frequency A/B being emitted; verification by image inspection means that verification is performed by image (belonging to multimedia information) inspection, and the image quality under the irradiation of the light of frequencies A and B is determined; the color camera scene analysis is used for analyzing the real scene contained in the multimedia information acquired by the multimedia information acquisition equipment; environment recognition, namely, recognizing the illumination condition of the field environment; selecting the frequency, namely selecting the light frequency corresponding to the best image quality as the frequency of the light emitted by the active light source, which is equivalent to adjusting the light frequency of the active light source; the right-most spherical chart shows that the multimedia information is acquired under the irradiation of an active light source for adjusting the light frequency, and the stereoscopic multimedia information is determined.
In fact, the scanning density, scanning intensity and/or light frequency of the active light source may be adjusted in combination according to the subject and/or the acquisition environment.
Fig. 9 is a schematic diagram of a principle framework for adjusting the scanning density, scanning intensity and/or optical frequency of an active light source according to the photographed object and/or the acquisition environment. In fig. 9, from left to right, full scanning is shown to perform full-scale multimedia information acquisition on a surrounding real scene, or perform full-scale multimedia information acquisition on the surrounding real scene under the cooperation of full-scale scanning irradiation of an active light source; the depth index is used for determining depth information from the multimedia information acquired in all directions; the color camera scene analysis is used for representing scene analysis of the multimedia information acquired by the multimedia information acquisition equipment; distance adaptation, intensity adaptation, frequency adaptation, movement adaptation, structure and proportion adaptation represent distance (object distance) adaptation, light energy (intensity) adaptation, frequency adaptation, adaptation to movement of a subject, structure and proportion adaptation, respectively; light intensity adjustment, frequency selection, active light source correction respectively represent adjustment of the intensity of light emitted by an active light source in a scanning form, adjustment of the intensity of light of the active light source, selection (adjustment) of the light frequency of the active light source, correction (adjustment) of the emitted light of the active light source based on movement of a subject, correction of the emitted light of the active light source based on structure and proportion; these constitute the operation of the active light source.
FIG. 10 is a schematic diagram of an example density distribution after adjusting the scanning density of the active light source. In fig. 10, a circle indicates a VR acquisition system, i.e., a stereoscopic multimedia information determination system in the present invention, in which each multimedia information acquisition device and active light source are disposed. The distance between the object and the system is far longer than the distance between the multimedia information acquisition equipment and the active light source in the system, so that the distance between the multimedia information acquisition equipment and the active light source can be ignored; the distance of the object from the system can be considered to be equal to the distance between the object and the active light source, i.e. the drawing in the figure can be considered to represent the active light source. Similarly, the distance between the object and the VR acquisition system in the subsequent illustration is equal to the distance between the object and the active light source, and will not be described again.
In fig. 10, the distance of the object from the active light source is inversely proportional to the spatial scanning density and directly proportional to the scanning intensity; the line thickness represents the scan intensity and how much the line represents the scan density. The active light source emits light with higher density to the angle of the object A at a short distance, emits light with medium density to the angle of the object B at a medium distance, and emits light with lower density to the angle of the object C at a long distance; less dense light or no light is emitted to other angles.
Fig. 11 is a schematic diagram showing an example of density distribution after adjusting the scanning density of the active light source according to the movement parameters of the subject. In fig. 11, the solid line corresponds to the original position of the object, the dotted line corresponds to the next time position of the object, and the scanning density varies with distance. The solid line between the active light source and the tangential/radial moving object represents the light emitted by the active light source for the moving object at the current moment, the dotted line represents the light emitted by the active light source for the position of the moving object at the next moment, and the change from the solid line to the dotted line represents that the active light source adjusts the spatial scanning density along the moving track of the photographed object.
An application scenario of adjusting the scanning density, scanning intensity and/or optical frequency of the active light source in the second embodiment of the present invention is specifically described below.
Application scenario one: the multimedia information acquisition device is stationary, and surrounding subjects are stationary. The method comprises the steps of firstly calling an active light source to scan the periphery at the maximum scanning density, simultaneously calling a multimedia information acquisition device to acquire the surrounding multimedia information in an omnibearing manner, determining the distance information of surrounding shot objects according to the multimedia information, and then distributing different scanning densities according to the distance between the shot objects. This operation may be performed at a certain period.
And (2) an application scene II: the multimedia information acquisition device is stationary, and surrounding subjects are stationary. The method comprises the steps of calling a multimedia information acquisition device to acquire surrounding multimedia information in an omnibearing manner, roughly determining distance information of surrounding shot objects according to the multimedia information by a binocular/multi-view visual method, and guiding an active light source to distribute different scanning densities according to the distance between the shot objects, wherein the operation can be performed in a certain period.
And (3) an application scene III: the multimedia information acquisition device is stationary, and surrounding subjects are stationary. Invoking a multimedia information acquisition device to acquire surrounding multimedia information in an omnibearing manner, roughly determining the distance information of surrounding shot objects according to the multimedia information by a binocular/multi-view vision method, and guiding an active light source to distribute different scanning densities according to the distance between each shot object; on the basis, the multimedia information acquisition equipment is further called to acquire the multimedia information of surrounding shot objects under the cooperation of the active light source, the distance information of the roughly determined shot objects is accurately measured according to the further acquired multimedia information, the scanning density of the active light source is further adjusted, and the operation can be carried out in a one-to-one period.
And application scene IV: there is relative motion between the multimedia information acquisition device and the part of the surrounding part of the subject. And calling an active light source to scan the surrounding in an omnibearing manner with the maximum scanning density, calling a multimedia information acquisition device to acquire surrounding multimedia information in an omnibearing manner, determining the distance information of surrounding shot objects according to the multimedia information, and distributing the scanning density to different shot objects according to the obtained distance information. Meanwhile, according to the time sequence information, the movement parameters such as the movement direction of the shot object are predicted, so that tracking scanning of the same relative movement shot object is realized.
Application scenario five: there is relative motion between the multimedia information acquisition device and the part of the surrounding part of the subject. The method comprises the steps of tracking and collecting multimedia information of a mobile shot object by calling multimedia information collecting equipment, predicting the rough distance and the movement direction of the mobile shot object, guiding an active light source to distribute specific scanning density to the shot object, and predicting the movement direction of the shot object according to time sequence information, so that tracking scanning of the same relative mobile shot object is realized.
Application scene six: there is relative motion between the multimedia information acquisition device and the part of the surrounding part of the subject. The multimedia information acquisition equipment predicts the rough distance and the movement direction of a mobile shot object, guides an active light source to distribute specific scanning density to the shot object, and predicts the movement direction of the shot object according to time sequence information; after the primary scanning is completed by the active light source, calling the multimedia information acquisition equipment to track and acquire the multimedia information of the moving shot object or acquire the multimedia information in all directions again under the cooperation of the active light source; according to the multimedia information which is tracked and collected or collected in an omnibearing way again, determining an accurate measurement result of the movement parameters of the shot object, and updating the position information of the shot object, thereby realizing tracking and scanning of the same relative movement shot object.
Application scene seven: the system for determining the stereoscopic multimedia information to which the multimedia information acquisition device belongs has integral relative motion relative to surrounding objects, such as the system for determining the stereoscopic multimedia information is moving. At the moment, an active light source is called to conduct omnibearing scanning at the maximum scanning density, and meanwhile, a multimedia information acquisition device is called to acquire omnibearing multimedia information under the cooperation of the active light source; the current distance (object distance) of each surrounding subject is determined based on the multimedia information. After different scanning densities are distributed for shot objects with different distances according to the current distance, an active light source is called to carry out optical scanning according to the adjusted scanning densities, meanwhile, a multimedia information acquisition device matched with the optical scanning angle is called to acquire multimedia information, the relative motion trail of the multimedia information acquisition device and the shot objects is judged according to the multimedia information, the positions of the multimedia information acquisition device are predicted, and self-adaptive scanning density distribution is carried out on all the shot objects again.
Application scenario eight: the system for determining the stereoscopic multimedia information to which the multimedia information acquisition device belongs has integral relative motion relative to surrounding objects, such as the system for determining the stereoscopic multimedia information is moving. At this time, the multimedia information acquisition equipment acquires all-round multimedia information; the method comprises the steps of determining the current distance (object distance) of each surrounding shot object according to multimedia information, roughly judging the movement of the multimedia information acquisition equipment according to the current distance of each surrounding shot object, and distributing different scanning densities to shot objects with different distances according to the current distance. And then, calling the active light source to perform optical scanning with the adjusted scanning density, and calling the multimedia information acquisition equipment matched with the optical scanning angle to acquire multimedia information, judging the relative motion trail of the multimedia information acquisition equipment and the shot object according to the multimedia information (namely the scanning result after the active light source is adaptive), and predicting the position of the multimedia information acquisition equipment to perform adaptive scanning density distribution again on all the shot objects. Fig. 12 shows a schematic diagram of one example of adjusting the scanning density/intensity of the active light source according to the size/surface curvature of the subject. FIG. 12 shows that during the adjustment of the spatial scan density and scan intensity of the active light source, a dense scan density is assigned to the angle at which the object (i.e., subject) with high surface curvature/small size is located; for the angle of a flat/large-size object, sparse scanning density is distributed; for angles occupied by the non-presence (significance or target) object, the scanning density is not distributed, namely the active light emitting units in the active light source, the optical axis directions of which are located in the angle range occupied by the non-presence (significance or target) object, are turned off.
Application scene nine: the surface curvature of the shot object is smaller, such as a planar wall and a table, after the active light source scans at the maximum density, the scanning density of the active light source related to the shot object is adjusted according to the surface curvature of the shot object, and the surface of the shot object with small curvature is scanned only by using sparse density, so that the energy consumption is saved.
Application scene ten: the surface curvature of the shot object is large, such as a cup, a baseball or a part of the surface of the shot object is large, and after the active light source scans with the maximum density, the active light source scans with the denser scanning density according to the complexity of the curvature, so that a better effect is achieved.
An application scene eleven, calling a multimedia information acquisition device to acquire multimedia information of a shot object, analyzing textures of the shot object in the multimedia information, and predicting surface curvature according to the textures; for example, the table may have a flat portion with less texture and a smooth curvature, while the edge portion may have a strong edge and a larger curvature. Based on this information, the subject, or different parts of the subject, are instructed to detect using different scan densities.
Calling multimedia information acquisition equipment to acquire multimedia information of a shot object, detecting and identifying the shot object in the multimedia information, and determining the surface curvature of the shot object according to the identification result and priori knowledge; if the table is identified, the curvature of the plane part is small and the curvature of the edge part is large. Based on this information, the subject, or different parts of the subject, are instructed to detect using different scan densities.
And calling multimedia information acquisition equipment to acquire multimedia information of the shot object, performing three-dimensional reconstruction on the shot object in the multimedia information, determining the surface curvature of the shot object according to a reconstruction result, guiding to detect the shot object or different parts of the shot object based on the information, and using different scanning densities.
Fourteen application scenarios: the multimedia information acquisition equipment is called to acquire the multimedia information of the shot object, the shot object in the multimedia information is detected, identified and the reflection model of the surface of the shot object is analyzed, so that the density and the energy (namely the intensity) of the active light source scanning can be distributed according to the surface characteristics of the shot object according to the prior knowledge of the shot object, which is related to the reflection model obtained by analysis and the identification result. If the mirror surface is reflected, the distance measurement is completed by adopting a mode of projecting no active light source energy and only projecting the energy to the edge of the mirror surface or projecting active light source rays with enough intensity and scanning density determined according to the content in the mirror; for a surface that mixes specular and diffuse, active rays of sufficient intensity and density are transmitted to ensure that sufficient effective reflection can be formed at the surface to complete distance measurement of the subject or mark projected thereon.
Fifteen application scenarios: and calling the multimedia information acquisition equipment to acquire the multimedia information of the periodic scene, analyzing the scene in the multimedia information, and guiding the active light source to work with smaller energy when analyzing the scene without the interference of other light sources, such as indoor or outdoor at night, so as to reduce the power consumption and the safety risk. Fig. 13a shows an example of an acquisition environment with/without ambient light interference. In fig. 13a, the object distance is the same, but the light intensity emitted by the active light source will be adjusted accordingly when there is ambient light interference. The left part of the specific map 13a shows that the active light source emits light with higher intensity in a scene with ambient light interference; the right part shows that the active light source emits light of lower intensity in a scene without ambient light interference.
Sixteen application scenarios: and (3) calling the active light source to perform trial light irradiation under the current scene, such as adopting different emission intensities. When the emission intensity is lower, if the distance estimation of the shot object is only performed according to the multimedia information independently collected by the multimedia information collecting device, but the reliable distance result cannot be estimated according to the multimedia information collected by the multimedia information collecting device under the irradiation of the active light source, the emission intensity needs to be improved. On the other hand, under the condition of higher distance confidence, the emission intensity of the active light source is reduced, and the distance confidence is guided to reach a preset threshold value.
Seventeen application scenarios: when the multimedia information acquisition equipment judges the rough distance of the shot object according to binocular/multiview, smaller active light source emission intensity is adopted for the shot object at the near position, and smaller active light source emission intensity is adopted for the shot object at the far position. Fig. 13b shows an example of adjusting the scanning intensity of the active light source according to the object distance of the subject. In fig. 13b, the light intensity is shown to be large by the thick line when the object distance is long, and the light intensity is shown to be small by the thin line when the object distance is short. The active light source in fig. 13b emits light with a lower emitted light intensity for close range objects and with a higher emitted light intensity for longer range objects.
Eighteen application scenarios: and scanning surrounding shot objects by using the active light source, adopting smaller active light source emission intensity for the shot objects at the near position and adopting closer active light source emission intensity for the shot objects at the far position according to the obtained distance information.
Application scenario nineteen: analyzing the scene according to the multimedia information collected by the multimedia information collecting equipment aiming at the surrounding scene, and identifying common interference light sources such as sun; and on-site environments such as sunny days, cloudy days; to guide the selection of the frequency by the active light source and avoid the spectrum range of the interference light source. Fig. 14 is a schematic diagram of one example of adjusting the light frequency of an active light source according to the acquisition environment. In fig. 14, the solid line and the broken line represent different light wavelengths, and when the broken line wavelengths are similar to the ambient light, the system is irradiated with only solid line light different from the ambient light wavelengths. For example, the solid line represents light at frequency A; the dotted line represents the light with the frequency of B, firstly, the active light source is called to emit the light with the frequencies of A and B, when the environment interference light is determined to be in the wave band with the frequency of B, the active light source is controlled to stop emitting the light with the frequency of B, and the light with the frequency of A is selected to eliminate the environment interference.
Application scene twenty: and calling the active light source to try by using different light frequencies, and finally selecting the light frequency with the best effect under the current scene to scan and emit.
Application scenario twenty-one: and adjusting the scanning interval time of the active light source according to the moving speed of the shot object. Specifically, for a moving object in the environment, the moving speed of the object is determined according to the multimedia information of the protected object acquired by the multimedia information acquisition device, so as to determine the interval time of active light source scanning. For stationary subjects, the time interval may be longer, and for fast moving subjects, the maximum scan frequency may be used for scanning.
Application scene twenty-two: in a complex environment, the above comprehensive adaptation in terms of scan density, scan frequency, scan intensity and scan spectrum occur simultaneously, overlapping one another.
Application scenario twenty-three: when the flexible object is deformed, the deformation rate of the flexible object determines the shape change rate under the visual angle of the multimedia acquisition equipment, so that when the deformation rate is large, the scanning time interval of an active light source which irradiates the flexible object is small, so that the real-time shape of the object can be accurately reflected; conversely, when the deformation rate is small, the scanning time interval for the flexible object may be long. Furthermore, as the surface curvature of the object is also changed during the deformation process, the spatial scanning density of the active light source can also be changed according to the related scene.
In the second embodiment of the present invention, at least one of the scanning density, the scanning intensity, and the scanning interval time of the active light source may be adjusted according to the subject and/or the acquisition environment. On the basis of guaranteeing the matching of multimedia information based on the mark irradiated by the active light source on the shot object, the power consumption of the active light source is reduced, the duration of the whole system where the active light source is located can be prolonged, and/or the capacity of a configuration power supply in the system can be reduced, the volume and the weight of the system are further reduced, and the portability and the mobility are improved. And the light frequency of the active light source is adjusted according to the acquisition environment, so that the anti-interference performance can be improved, the quality of the multimedia information matched based on the mark is improved, and the quality of the three-dimensional multimedia information is integrally improved.
Example III
The third embodiment of the invention specifically introduces a method for regularizing the parallax of the acquired multimedia information. The regularization of the parallax refers to a process of modifying parallax distortion caused by a large difference in orientation of cameras according to an existing matching relationship, and sorting the parallax distortion into the same parallax state as that when the eyes observe the parallax distortion.
The inventors of the present invention found that, ideally, a user views which azimuth angle, a pair of binocular imaging apparatuses needs to be provided at which direction angle to simulate viewing (capturing an image) by both eyes of the user. Therefore, the existing product is generally provided with more pairs of camera equipment to simplify the bionic simulation, but the viewing angles of users are infinite, and the existing equipment cannot simulate all the viewing angles, so that the parallax of the existing stereoscopic video in a large angle range is unstable.
The reason why the parallax of the existing stereoscopic video is unstable is specifically analyzed as follows.
Fig. 15 is a schematic diagram of a specific example of the existing stereoscopic video capturing and viewing principle. For ease of understanding, any one set (including left-eye and right-eye) of image capturing apparatuses is selected from the existing 10-20 image capturing apparatuses to explain the principle of capturing and viewing stereoscopic video. The optical axes of a group of binocular imaging apparatuses are on the same plane, and for ease of understanding, it is assumed that the optical axes of the binocular imaging apparatuses are on the same horizontal plane. At this time, fig. 15 can be understood as a horizontal cross-sectional view or a top perspective view of the stereoscopic view of the binocular imaging apparatus.
Left and right eyes represent left and right eye image pickup apparatuses, respectively; d between the left and right eyes represents a base line length between the two image pickup apparatuses; the center of a circle is the midpoint of arrangement of the plurality of image pickup apparatuses, and when it is only a binocular image pickup apparatus, the center of arrangement is degenerated to be the midpoint of the base line of the binocular image pickup apparatus. R in fig. 15 represents the distance from the subject to the center of the circle. A and B are two shot objects with the same object distance but different angles respectively.
In the process of acquiring the stereoscopic video, the positions of the optical axes of the left eye and the right eye are fixed; for example, as shown in fig. 15, assuming that the center is denoted as O, the optical axes of the left eye of cam_l and the right eye of cam_r are parallel to the straight line where AO is located. When the left eye and the right eye acquire binocular frame images of the object located at the point a, parallax between the binocular frame images corresponding to the point a is lossless. However, when the left eye and the right eye acquire binocular frame images of the object located at the point B, the equivalent baseline distance between the left eye and the right eye is reduced, resulting in a loss of parallax between the binocular frame images corresponding to the point a.
Further analysis, the center of light O will be represented by the points on the circle and left-hand object 1 The included angle between the line segment which is the two end points and the negative direction (leftwards) of the transverse axis is marked as alpha; will be centered on the circle at the point and right of interest O 2 The included angle between the line segment with two endpoints and the negative direction (left) of the transverse axis is recorded as beta; the angle between the two line segments is noted as θ. For example, point A corresponds to alpha 1 、β 1 And O 1 AO 2 Included angle theta of (2) A The method comprises the steps of carrying out a first treatment on the surface of the Point B corresponds to alpha 2 、β 2 And O 1 BO 2 Included angle theta of (2) B If the left image is held so that the point B coincides with the point A, θ B Shown as O' 1 BO’ 2 I.e. the dotted line portion
From knowledge of the triangle, 180- α+θ+β=180, θ=α - β. The angle θ corresponds to the parallax, and the larger the angle θ is, the larger the parallax is, and the smaller the angle θ is, the smaller the parallax is.
From the point A on the left part to the point B on the right part in FIG. 15 (representing any point on the circle offset by the point A), O 1 (points on a circle) O 2 The three-point relationship may be equivalent to line segment O in the left portion of the graph 1 O 2 At the level of over O 1 And O 2 Rotated to the dashed line position. As shown, for point B (representing any point on the circle offset from point A), which does not coincide with point A, angle θ A >θ B 。
Based on graphic knowledge, only θ at point A has a maximum value, α 1 -β 1 Alpha is greater than the value of 2 -β 2 Values. The parallax corresponding to the point a is the largest, and the more the point on the circle deviates from the point a, the smaller the corresponding parallax.
Therefore, when a user views, the user visually feels that the shot object at the point A is nearest, other shot objects deviating from the point A are farther, and the shot object with a larger angle deviating from the point A is farther, so that parallax is unstable when the user views, and the user experience is easily reduced.
Even the multi-group binocular imaging apparatus adopted in the prior art cannot solve the problem of parallax instability. Specifically, only the parallax of the subject to which each group of binocular imaging apparatuses is facing (the position resembling the point a) is lossless (maximum), the parallaxes of the subjects at other angles are all lossy (smaller), and the larger the angle of deviation is, the smaller the corresponding parallax is, and the parallax of the subject at the boundary of the adjacent two groups of imaging apparatuses is minimum.
In the third embodiment of the invention, the defect of unstable parallax between the multimedia information acquired at different angles is eliminated by regularizing the parallax of the multimedia information acquired by at least two multimedia information acquisition devices.
Specifically, a parallax line and a lossless parallax line are described first. The parallax lines include a vertical parallax line and a horizontal parallax line. The lossless parallax lines include lossless vertical parallax lines and lossless horizontal parallax lines.
The vertical parallax line is a line intersecting with a plane perpendicular to the optical axes of the at least two multimedia information collecting devices and passing through the arrangement midpoints of the respective multimedia information collecting devices and the outer surfaces of the collecting vision fields of the at least two multimedia information collecting devices. When the optical axes of at least two multimedia information acquisition devices are coplanar, the center of a polygon with the optical center of each multimedia information acquisition device as each vertex or the point with the minimum sum of the lengths of the vertices in the polygon is taken as the arrangement center.
Preferably, when the overall acquisition view of at least two multimedia information acquisition devices is spherical, a part of spherical or spheroid, the arrangement midpoint of each multimedia information acquisition device coincides with the sphere center of the sphere, the part of spherical or spheroid. The spheroid may include an ellipsoid or the like. At this time, the parallax line is embodied as a meridian of a sphere (or spheroid) acquisition view.
The lossless vertical parallax line is a parallax line with equal distance from each pixel point on the parallax line to each multimedia information acquisition device in the pair of multimedia information acquisition devices. Preferably, the plane of the lossless vertical parallax line is generally parallel to the optical axis of the pair of multimedia information collecting devices. The pair of multimedia information acquisition devices observe that any point on the lossless parallax line is free of parallax loss (i.e., parallax is the largest), and can be considered to be for the lossless parallax line.
A first method of regularizing the disparity of multimedia information is described below.
And determining the depth information of each pixel point in the multimedia information according to the depth information of the mark irradiated on the shot object by the active light source in the multimedia information acquired by any pair of multimedia information acquisition equipment.
Specifically, depth information of a mark irradiated on a subject by an active light source in multimedia information acquired by any pair of multimedia information acquisition devices is determined.
And for the pixel points in the multimedia information, which fall in the range of the marking area or in the vicinity of the marking, the depth information marked in the multimedia information collected by the pair of multimedia information collecting equipment is used as the depth information of the pixel points in the multimedia information collected by the pair of multimedia information collecting equipment.
And for the pixel points outside the range of the marking area or outside the marking neighborhood, taking the depth information of the pixel points in the multimedia information collected by the pair of multimedia information collecting equipment as the depth information of the pixel points in the multimedia information collected by the pair of multimedia information collecting equipment according to the nearest marks around the pixel points.
Preferably, the depth information of the pixel point in the multimedia information collected by the pair of multimedia information collecting devices can be determined through interpolation according to the depth information of the nearest mark around the pixel point in the multimedia information collected by the pair of multimedia information collecting devices.
Further, for sparse labeling methods, dense, pixel or sub-pixel level depth information obtained by image matching of unlabeled portions may be combined based on labeled depth information.
And determining the position of the virtual binocular multimedia information acquisition equipment corresponding to any pair of the multimedia information acquisition equipment according to the yaw angle of the mark irradiated on the shot object by the active light source.
The yaw angle of the mark is an included angle between a plane of a parallax line to which the mark belongs and a plane of a lossless vertical parallax line. Preferably, the baseline distances between the pairs of virtual binocular multimedia information gathering devices are equal, e.g., all equal to the standard baseline distance. Further, the standard baseline distance may be equal to a baseline distance between a pair of multimedia information gathering devices facing the lossless vertical disparity line.
Preferably, when the overall acquisition view of at least two multimedia information acquisition devices is a spherical (or spheroid) view, the yaw angle of the mark irradiated on the object by the active light source is the included angle between the plane of the meridian to which the mark belongs and the plane of the lossless vertical parallax line (the meridian at this time).
And projecting the pixel points to an imaging surface of the virtual binocular multimedia information acquisition equipment according to the depth information of each pixel point in the multimedia information acquired by any pair of multimedia information acquisition equipment and the position of the corresponding virtual binocular multimedia information acquisition equipment to form corresponding virtual binocular multimedia information.
Fig. 16 is a schematic diagram of an example of a first method of regularizing parallaxes of multimedia information. In fig. 16, cam_ L, cam _r represents a left-eye multimedia information collecting device and a right-eye multimedia information collecting device of any pair of multimedia information collecting devices, respectively; d. r, A, B, alpha 1 And beta 1 The meaning of each representation is the same as that in fig. 15 described above, and is not described in detail. Virtual cam_l and virtual cam_r represent virtual binocular multimedia information collecting devices; the angle between the line segment with the points on the circle and the optical center of the virtual cam_l as two end points and the negative direction (left) of the transverse axis is denoted as alpha 2 The method comprises the steps of carrying out a first treatment on the surface of the The angle between the line segment with the points on the circle and the optical center of the virtual cam_R as two end points and the negative direction (left) of the transverse axis is recorded as beta 2 . Yaw_a represents the Yaw angle of point a, which is 0 degrees because the parallax line to which point a belongs is a lossless parallax line. Yaw_b represents the Yaw angle of the point B, namely the included angle between the plane of the parallax line of the point B and the plane of the lossless parallax line of the point a is equal to the degree of the angle AOB in the figure, and O is a dot.
Projecting each pixel point in the multimedia information acquired by any pair of multimedia information acquisition equipment to form corresponding virtual binocular multimedia information; the method is equivalent to projecting pixel points in two original multimedia information into virtual binocular multimedia information. Because the baseline distance between the virtual binocular multimedia information acquisition devices is regularized (equal to the standard baseline distance), the parallax between the virtual binocular multimedia information corresponding to the virtual binocular multimedia information acquisition devices is regularized and stable, so that the visual perception that the parallax is stable when a user views the virtual binocular multimedia information can be ensured, the object distance of a photographed object is lifelike, the immersive sensation of the user can be brought to the scene easily, and the user experience can be improved.
A second method of regularizing the disparity of multimedia information is described below.
The horizontal parallax line is a line intersecting a plane parallel to the optical axes of the at least two multimedia information collecting devices and an outer surface of the collecting field of view of the at least two multimedia information collecting devices.
The horizontal lossless parallax line is an intersection line with the plane where the optical axes of the at least two multimedia information acquisition devices are located and the outer surface of the acquisition visual field of the at least two multimedia information acquisition devices.
And determining the parallax difference value of each pixel point according to the yaw angle and/or pitch angle of each pixel point in the acquired multimedia information and the corresponding relation among the yaw angles, pitch angles and parallax difference values of a plurality of pixel points, which are established in advance.
Preferably, the overall acquisition view of at least two multimedia information acquisition devices is in a three-dimensional shape (e.g. spherical), and the panoramic multimedia information acquired by the at least two multimedia information acquisition devices is three-dimensional and has a three-dimensional surface (e.g. spherical). And converting the panoramic multimedia information from three dimensions to two dimensions, such as cylindrical projection, to obtain two-dimensional panoramic multimedia information. Further, in the case of cylindrical projection, the vertical parallax line in the three-dimensional multimedia information is projected as a vertical line segment in the two-dimensional multimedia information. The horizontal parallax line in the three-dimensional multimedia information is projected to be represented as a horizontal line segment in the two-dimensional multimedia information.
According to the parallax difference value of the pixel point, carrying out position adjustment on the corresponding line of the vertical parallax line and/or the horizontal parallax line of the pixel point in the projected two-dimensional multimedia information; thereby obtaining the projected two-dimensional multimedia information after parallax regularization.
Preferably, the parallax difference of the pixel points is determined by the following method: taking the parallax between the virtual binocular multimedia information related to the pixel points as the parallax after the pixel points are corrected; and then determining the parallax difference between the corrected parallax of the pixel point and the original parallax.
More preferably, for the pixel point of each three-dimensional position in the whole acquisition view field of at least two multimedia acquisition devices, depth information of the pixel point in the corresponding pair of multimedia information acquisition devices is determined, and then an actual parallax value of each pixel point is determined.
For each pixel point in the multimedia information, the projection of the pixel point in the virtual binocular multimedia information of the virtual binocular multimedia information acquisition equipment corresponding to the pair of multimedia information acquisition equipment (determined by the first method) is utilized to determine the corrected (i.e. ideal) parallax value of the pixel point.
And in the process of converting the multimedia information from three dimensions to two dimensions, determining the yaw angle and/or pitch angle of each pixel point in the multimedia information.
And for the pixel points of each three-dimensional position in the whole acquisition view field of at least two multimedia information acquisition devices, establishing the corresponding relation among the actual parallax value, the corrected parallax value, the yaw angle and the pitch angle of the pixel points. The calculation amount of the step for establishing the corresponding relation is large, and the step can be completed off-line.
And then, for any pixel point in the multimedia information acquired by the multimedia information acquisition equipment, determining the actual parallax value, yaw angle and pitch angle of the pixel point, and directly searching the corresponding corrected parallax value according to the corresponding relation. The step similar to the table look-up is utilized to replace the complex step of determining the corrected parallax value, so that the workload of determining the corrected parallax value is greatly saved. Therefore, the step of determining the corrected parallax value of the subsequent pixel point can be completed on line according to the actual parallax value of the pixel point, the corrected parallax value, and the correspondence between the yaw angle and the pitch angle.
And further determining a parallax difference between the corrected parallax of the pixel point and the actual parallax (namely, the original parallax).
According to the parallax difference value of the pixel point, carrying out position adjustment on the corresponding line of the vertical parallax line and/or the horizontal parallax line of the pixel point in the projected two-dimensional multimedia information; thereby obtaining the projected two-dimensional multimedia information after parallax regularization.
Further, a dense matching is established firstly, and the technology is the same as the first method, and is based on sparse mark points and matched with an image matching technology to obtain a dense matching result. And then calibrating objects with different distances at different angles in an off-line manner according to the yaw angle and the pitch angle of any point on the spherical surface, so as to obtain the parallax value of the object with any depth and the parallax deviation of the object with the same distance as the parallax lossless line point on any angle. When the system works, according to the pitch angle, the yaw angle and the current parallax, the delta required by the current point to be converted into the correct parallax value is reversely pushed, and parallax adjustment is carried out according to the delta.
Fig. 17 is a schematic diagram of an example of a second method of regularizing parallaxes of multimedia information. In fig. 17, the four sub-images are divided in the order of the arrows.
The sphere of the first sub-graph represents that the acquisition view field of at least two multimedia information acquisition devices is a spherical view field, the meridian represents a parallax line, and the sphere represents panoramic multimedia information acquired by at least two multimedia information acquisition devices.
The arrow between the first sub-graph and the second sub-graph represents a three-dimensional to two-dimensional transformation, such as a cylindrical projection, of the spherical panoramic multimedia information.
The second sub-graph represents the projected two-dimensional panoramic multimedia information. The disparity line, which is the warp line in the first sub-graph, appears as a vertical line segment in the second sub-graph. The weft of the first sub-graph appears as a corresponding horizontal line segment in the second sub-graph. In fact, through cylindrical projection, pixels on latitudes of 0 degrees (similar to the equator) are theoretically undistorted, while pixels on other latitudes are distorted to some extent. Thus, the horizontal parallax line in the second method is embodied as a latitudinal parallax line; the lossless horizontal parallax line is embodied as a lossless latitudinal parallax line, i.e., a latitudinal parallax line having a latitude of 0.
In the course of cylindrical projection, the yaw angle of each parallax line can be obtainedAnd pitch) θ. In the second sub-graph, < > >Respectively representing the yaw angles of parallax lines to which A, B points belong; θ A 、θ B The pitch angle of the parallax line to which the A, B point belongs is shown. />The difference value is expressed as the distance between the vertical line segments of the points A and B; θ A 、θ B The difference between the points is expressed as the distance between the horizontal line segments to which the points A and B belong.
D' in the third sub-graph represents the disparity after pixel correction, i.e., the disparity between the virtual binocular multimedia information related to the pixel. According to the previous analysis, the parallax after the pixel correction is obtained by projection of the pixel (i.e., positional shift of the pixel). And determining a parallax difference value between the corrected parallax of the pixel point and the original parallax according to the corrected parallax of the pixel point and the yaw angle and pitch angle of the pixel point. For example, the parallax difference between the points A and B is Δd A 、Δd B 。
The vertical line segments of the points A and B in the fourth sub-graph are respectively according to delta d A 、Δd B Move to obtain parallax regularized (projected) twoAnd (5) maintaining the multimedia information.
In the third embodiment of the invention, the parallax of the multimedia information is regularized, so that the parallax stability of the multimedia information is greatly improved, the stable parallax and the vivid object distance of the photographed object are visually perceived by a user when the user views the virtual binocular multimedia information, the immersive sense of the user is easily brought to the user, and the user experience is improved.
Example IV
In a fourth embodiment of the present invention, there is described a determination apparatus for stereoscopic multimedia information, wherein a frame diagram of an internal structure of the determination apparatus is shown in fig. 18, and the determination apparatus includes: a multimedia information acquisition module 1801 and a stereoscopic multimedia information determination module 1802.
The multimedia information acquisition module 1801 is configured to acquire multimedia information acquired by at least two multimedia information acquisition devices respectively.
The stereo multimedia information determining module 1802 is configured to match the acquired at least two pieces of multimedia information according to at least one active light source, and determine corresponding stereo multimedia information.
Preferably, the stereo multimedia information determining module 1802 is specifically configured to invoke an active light emitting unit array in which an irradiation angle in the active light source is matched with an acquisition direction of the multimedia information acquisition device to irradiate when at least two multimedia information acquisition devices acquire multimedia information respectively; and matching the acquired multimedia information according to the mark irradiated on the shot object.
Preferably, the stereo multimedia information determining module 1802 is specifically configured to, when the active light source includes a plurality of active light emitting unit arrays, invoke, for each pair of multimedia information collecting devices, the active light emitting unit arrays having an illumination angle matching the collection direction of the pair of multimedia information collecting devices to illuminate.
Preferably, the stereo multimedia information determining module 1802 is specifically configured to match different multimedia information according to a position difference of a mark irradiated on a subject by an active light source in the different multimedia information; wherein the indicia comprises at least one of: light spots, grid light, encoded light spots.
Preferably, the stereo multimedia information determining module 1802 is further configured to process the multimedia information collected by the multimedia information collecting devices in different collecting directions according to the position difference in the multimedia information collected by the multimedia information collecting devices in different collecting directions of the mark irradiated on the photographed object by the active light source.
More preferably, as shown in fig. 18, the apparatus for determining stereoscopic multimedia information according to the fourth embodiment of the present invention further includes: an active light source adjustment module 1803.
The active light source adjustment module 1803 is configured to adjust at least one of a scanning density, a scanning intensity, an optical frequency, and a scanning interval time of the active light source according to a subject and/or an acquisition environment.
Preferably, the active light source adjustment module 1803 is specifically configured to adjust a scanning density and/or a scanning intensity of the active light source according to a spatial attribute of the subject and/or a content attribute of the subject; the spatial properties of the subject include at least one of: object distance and movement parameters corresponding to the shot object; the content attribute of the subject includes at least one of: the size and surface feature properties of the subject; the surface-feature attributes include at least one of: surface curvature, surface hardness, surface reflectivity.
Preferably, the active light source adjustment module 1803 is specifically configured to adjust the scanning density of the active light source for the photographed object to a corresponding preset scanning density gear when the surface curvature of the photographed object is a preset curvature gear; when the surface hardness of the shot object is flexible and rigid respectively, the scanning density of the active light source for the shot object is adjusted to be a dense scanning density gear and a sparse scanning density gear respectively; when the surface reflectivity of the shot object is specular reflection, the scanning intensity of the active light source for the area within the edge of the shot object is adjusted to be in a low scanning intensity gear, and/or the scanning intensity for the area within the edge of the shot object is adjusted to be in a high scanning intensity gear.
Preferably, the active light source adjustment module 1803 is specifically configured to adjust the light frequency of the active light source according to the light frequency of the interference light source in the collection environment.
Preferably, the active light source adjustment module 1803 is specifically configured to adjust a scanning interval time of the active light source according to a moving speed of the photographed object; and adjusting the scanning interval time of the active light source according to the deformation rate of the shot object.
More preferably, as shown in fig. 18, the apparatus for determining stereoscopic multimedia information according to the fourth embodiment of the present invention further includes: the disparity regularization module 1804.
The disparity regularization module 1804 is configured to regularize disparities of the acquired multimedia information.
Preferably, the parallax regularization module 1804 is specifically configured to determine depth information of each pixel point in the multimedia information according to the depth information marked in the multimedia information collected by any pair of the multimedia information collecting devices; determining the position of a virtual binocular multimedia information acquisition device corresponding to any pair of multimedia information acquisition devices according to the yaw angle of each pixel point in the multimedia information; the yaw angle of each pixel point is an included angle between the plane of the vertical parallax line to which the pixel point belongs and the plane of the lossless vertical parallax line; projecting each pixel point to an imaging surface of the virtual binocular multimedia information acquisition equipment according to the depth information of the pixel point and the position of the virtual binocular multimedia information acquisition equipment to form corresponding virtual binocular multimedia information; the vertical parallax line is a plane which is vertical to the optical axes of at least two multimedia information acquisition devices and passes through the arrangement center of each multimedia information acquisition device and an intersection line of the outer surfaces of the acquisition visual fields of the at least two multimedia information acquisition devices; the lossless vertical parallax line is a parallax line on which the distance from each pixel point to each of the pair of multimedia information collecting devices is equal.
Preferably, the parallax regularization module 1804 is further configured to determine a parallax difference value of each pixel point according to a yaw angle and/or a pitch angle of the pixel point in the collected multimedia information and a corresponding relationship between the yaw angle, the pitch angle and the parallax difference value of the pixel points, which are pre-established; according to the parallax difference value of the pixel point, carrying out position adjustment on a corresponding line of a vertical parallax line and/or a horizontal parallax line of the pixel point in the projected two-dimensional multimedia information; the horizontal parallax line is a line intersecting a plane parallel to the optical axes of the at least two multimedia information collecting devices and an outer surface of the collecting field of view of the at least two multimedia information collecting devices.
Preferably, the parallax regularization module 1804 is specifically configured to determine the parallax difference of the pixel points by the following method: taking the parallax between the virtual binocular multimedia information related to the pixel points as the parallax after the pixel points are corrected; and then determining the parallax difference between the corrected parallax of the pixel point and the original parallax.
The above-mentioned multimedia information obtaining module 1801, the stereoscopic multimedia information determining module 1802, the active light source adjusting module 1803, and the parallax regularization module 1804 may refer to the foregoing summary of the first embodiment and the specific contents of the first to third embodiments, which are not described herein.
Those skilled in the art will appreciate that the present invention includes apparatuses related to performing one or more of the operations described herein. These devices may be specially designed and constructed for the required purposes, or may comprise known devices in general purpose computers. These devices have computer programs stored therein that are selectively activated or reconfigured. Such a computer program may be stored in a device (e.g., a computer) readable medium or any type of medium suitable for storing electronic instructions and respectively coupled to a bus, including, but not limited to, any type of disk (including floppy disks, hard disks, optical disks, CD-ROMs, and magneto-optical disks), ROMs (Read-Only memories), RAMs (Random Access Memory, random access memories), EPROMs (Erasable Programmable Read-Only memories), EEPROMs (Electrically Erasable Programmable Read-Only memories), flash memories, magnetic cards, or optical cards. That is, a readable medium includes any medium that stores or transmits information in a form readable by a device (e.g., a computer).
It will be understood by those within the art that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by computer program instructions. Those skilled in the art will appreciate that the computer program instructions can be implemented in a processor of a general purpose computer, special purpose computer, or other programmable data processing method, such that the blocks of the block diagrams and/or flowchart illustration are implemented by the processor of the computer or other programmable data processing method.
Those of skill in the art will appreciate that the various operations, methods, steps in the flow, acts, schemes, and alternatives discussed in the present invention may be alternated, altered, combined, or eliminated. Further, other steps, means, or steps in a process having various operations, methods, or procedures discussed herein may be alternated, altered, rearranged, disassembled, combined, or eliminated. Further, steps, measures, schemes in the prior art with various operations, methods, flows disclosed in the present invention may also be alternated, altered, rearranged, decomposed, combined, or deleted.
The foregoing is only a partial embodiment of the present invention, and it should be noted that it will be apparent to those skilled in the art that modifications and adaptations can be made without departing from the principles of the present invention, and such modifications and adaptations are intended to be comprehended within the scope of the present invention.

Claims (17)

1. A method for determining stereoscopic multimedia information, comprising:
acquiring multimedia information respectively acquired by at least two multimedia information acquisition devices;
and matching the acquired at least two pieces of multimedia information according to the marks of at least one active light source on the shot object, which are included in the acquired at least two pieces of multimedia information, and determining corresponding three-dimensional multimedia information according to the matched at least two pieces of multimedia information.
2. The method of claim 1, wherein the step of determining the position of the substrate comprises,
when the at least two multimedia information acquisition devices acquire multimedia information respectively, the active light-emitting unit array with the irradiation angle in the active light source matched with the acquisition direction of the multimedia information acquisition devices is called to irradiate.
3. The method according to claim 2, wherein the invoking the active light emitting unit array with the irradiation angle matched with the collection direction of the multimedia information collection device to irradiate comprises:
When the active light source comprises a plurality of active light emitting unit arrays, for each pair of multimedia information acquisition equipment, the active light emitting unit arrays with the irradiation angles matched with the acquisition directions of the pair of multimedia information acquisition equipment are called for irradiation.
4. The method of claim 1, wherein said matching the acquired at least two pieces of multimedia information comprises:
matching different multimedia information according to the position difference of the marks irradiated on the shot object by the active light source in the different multimedia information; wherein the indicia comprises at least one of: light spots, grid light, coded light spots; the light spot comprises at least one of the following: a single spot, a collection of spots, a cluster of spots.
5. The method of claim 4, wherein the matching the different multimedia information based on the position difference in the different multimedia information of the mark irradiated on the subject by the active light source, further comprises:
and processing the multimedia information acquired by the multimedia information acquisition equipment in different acquisition directions according to the position difference among the multimedia information acquired by the multimedia information acquisition equipment in different acquisition directions according to the marks irradiated on the shot object by the active light source.
6. The method of any one of claims 1-5, further comprising:
and adjusting at least one of scanning density, scanning intensity, light frequency and scanning interval time of the active light source according to the shot object and/or the acquisition environment.
7. The method of claim 6, wherein adjusting the scan density and/or scan intensity of the active light source according to the subject comprises:
according to the space attribute of the shot object and/or the content attribute of the shot object, the scanning density and/or the scanning intensity of the active light source are/is adjusted;
the spatial attributes of the subject include at least one of: object distance and movement parameters corresponding to the shot object;
the content attribute of the subject includes at least one of: the size and surface feature properties of the subject; the surface-feature attributes include at least one of: surface curvature, surface hardness, surface reflectivity.
8. The method of claim 7, wherein adjusting the scanning density and/or scanning intensity of the active light source according to the surface feature property of the subject comprises:
when the surface curvature of the shot object is a preset curvature gear, the scanning density of the active light source aiming at the shot object is adjusted to be corresponding to the preset scanning density gear;
When the surface hardness of the shot object is flexible and rigid respectively, the scanning density of the active light source for the shot object is adjusted to be a dense scanning density gear and a sparse scanning density gear respectively;
when the surface reflectivity of the shot object is specular reflection, the scanning intensity of the active light source for the area inside the edge of the shot object is adjusted to be a low scanning intensity gear, and/or the scanning intensity for the area inside the edge of the shot object is adjusted to be a high scanning intensity gear.
9. The method of claim 6, wherein adjusting the light frequency of the active light source according to the acquisition environment comprises:
and adjusting the light frequency of the active light source according to the light frequency of the interference light source in the acquisition environment.
10. The method of claim 6, wherein adjusting the scan interval time of the active light source according to the subject comprises:
according to the moving speed of the shot object, adjusting the scanning interval time of the active light source;
and adjusting the scanning interval time of the active light source according to the deformation rate of the shot object.
11. The method of any one of claims 4-5, further comprising:
And regularizing the parallax of the acquired multimedia information.
12. The method of claim 11, wherein regularizing the disparity of the acquired multimedia information, comprising:
determining the depth information of each pixel point in the multimedia information according to the depth information of the mark in the multimedia information acquired by any pair of multimedia information acquisition equipment;
determining the position of a virtual binocular multimedia information acquisition device corresponding to any pair of multimedia information acquisition devices according to the yaw angle of each pixel point in the multimedia information; the yaw angle of each pixel point is an included angle between the plane of the vertical parallax line to which the pixel point belongs and the plane of the lossless vertical parallax line;
according to the depth information of each pixel point and the position of the virtual binocular multimedia information acquisition equipment, projecting the pixel point to an imaging surface of the virtual binocular multimedia information acquisition equipment to form corresponding virtual binocular multimedia information;
the vertical parallax line is a plane which is vertical to the optical axes of the at least two multimedia information acquisition devices and passes through the arrangement center of each multimedia information acquisition device, and is an intersection line with the outer surface of the acquisition visual field of the at least two multimedia information acquisition devices; the lossless vertical parallax line is a parallax line with equal distance from each pixel point on the lossless vertical parallax line to each multimedia information acquisition device in the pair of multimedia information acquisition devices.
13. The method of claim 12, wherein the regularizing the disparity of the acquired multimedia information further comprises:
determining the parallax difference value of each pixel point according to the yaw angle and/or pitch angle of each pixel point in the acquired multimedia information and the corresponding relation among the yaw angle, pitch angle and parallax difference values of a plurality of pixel points, which are established in advance;
according to the parallax difference value of the pixel point, carrying out position adjustment on a corresponding line of a vertical parallax line and/or a horizontal parallax line of the pixel point in the projected two-dimensional multimedia information;
the horizontal parallax line is a line intersecting a plane parallel to the optical axes of the at least two multimedia information collecting devices and an outer surface of a collecting field of view of the at least two multimedia information collecting devices.
14. The method of claim 13, wherein the disparity difference for the pixel is determined by:
taking the parallax between the virtual binocular multimedia information related to the pixel points as the parallax after the pixel points are corrected; and then determining the parallax difference between the corrected parallax of the pixel point and the original parallax.
15. A system for determining stereoscopic multimedia information, comprising: at least two multimedia information collecting devices, an active light source, and an electronic device configured to perform the method of any of claims 1-14.
16. An electronic device, the electronic device comprising:
one or more processors;
a memory;
wherein one or more application programs are stored in the memory, which when executed by the one or more processors, perform the method of any of claims 1-14.
17. A computer readable storage medium for storing a computer program which, when run on a processor, causes the processor to perform the method of any one of claims 1 to 14.
CN201710091162.4A 2017-02-20 2017-02-20 Method, device and system for determining stereoscopic multimedia information Active CN108460824B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710091162.4A CN108460824B (en) 2017-02-20 2017-02-20 Method, device and system for determining stereoscopic multimedia information

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710091162.4A CN108460824B (en) 2017-02-20 2017-02-20 Method, device and system for determining stereoscopic multimedia information

Publications (2)

Publication Number Publication Date
CN108460824A CN108460824A (en) 2018-08-28
CN108460824B true CN108460824B (en) 2024-04-02

Family

ID=63222045

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710091162.4A Active CN108460824B (en) 2017-02-20 2017-02-20 Method, device and system for determining stereoscopic multimedia information

Country Status (1)

Country Link
CN (1) CN108460824B (en)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109493354B (en) * 2018-10-10 2021-08-06 中国科学院上海技术物理研究所 Target two-dimensional geometric shape reconstruction method based on multi-view images
CN111213368B (en) * 2019-05-23 2021-07-13 深圳市瑞立视多媒体科技有限公司 Rigid body identification method, device and system and terminal equipment
CN110636266B (en) * 2019-10-10 2020-10-16 珠海格力电器股份有限公司 Security monitoring method and device based on electric appliance and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101729918A (en) * 2009-10-30 2010-06-09 无锡景象数字技术有限公司 Method for realizing binocular stereo image correction and display optimization
CN102510512A (en) * 2011-11-17 2012-06-20 杭州电子科技大学 Binocular-vision-node based self-positioning method for dynamic three-dimensional multimedia sensor network
CN103268604A (en) * 2013-05-10 2013-08-28 清华大学 Binocular video depth map calculating method

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014200589A2 (en) * 2013-03-15 2014-12-18 Leap Motion, Inc. Determining positional information for an object in space

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101729918A (en) * 2009-10-30 2010-06-09 无锡景象数字技术有限公司 Method for realizing binocular stereo image correction and display optimization
CN102510512A (en) * 2011-11-17 2012-06-20 杭州电子科技大学 Binocular-vision-node based self-positioning method for dynamic three-dimensional multimedia sensor network
CN103268604A (en) * 2013-05-10 2013-08-28 清华大学 Binocular video depth map calculating method

Also Published As

Publication number Publication date
CN108460824A (en) 2018-08-28

Similar Documents

Publication Publication Date Title
US10560687B2 (en) LED-based integral imaging display system as well as its control method and device
CN108475327A (en) three-dimensional acquisition and rendering
CN106327454B (en) The method and apparatus of composograph
CN101422035A (en) Image high-resolution upgrading device, image high-resolution upgrading method, image high-resolution upgrading program and image high-resolution upgrading system
CN106570938A (en) OPENGL based panoramic monitoring method and system
CN108460824B (en) Method, device and system for determining stereoscopic multimedia information
CN112233165B (en) Baseline expansion implementation method based on multi-plane image learning visual angle synthesis
CN103281507B (en) Video-phone system based on true Three-dimensional Display and method
AU2022271399B2 (en) Techniques for multi-view neural object modeling
CN101729920A (en) Method for displaying stereoscopic video with free visual angles
US20150373320A1 (en) Visual cognition system
CN115731336B (en) Image rendering method, image rendering model generation method and related devices
KR101399274B1 (en) multi 3-DIMENSION CAMERA USING MULTI PATTERN BEAM AND METHOD OF THE SAME
Waizenegger et al. Real-time patch sweeping for high-quality depth estimation in 3D video conferencing applications
US20150145861A1 (en) Method and arrangement for model generation
CN115115747A (en) Illumination rendering method and device, electronic equipment and storage medium
EP4150560B1 (en) Single image 3d photography with soft-layering and depth-aware inpainting
Yang et al. Real-time light-field generation based on the visual hull for the 3D light-field display with free-viewpoint texture mapping
CN109309827B (en) Multi-user real-time tracking device and method for 360-degree suspended light field three-dimensional display system
Yang View-dependent Pixel Coloring: A Physically-based Approach for 2D View Synthesis
Babahajiani Geometric computer vision: Omnidirectional visual and remotely sensed data analysis
CN116939186B (en) Processing method and device for automatic associative covering parallax naked eye space calculation
Thatte Cinematic virtual reality with head-motion parallax
CN117857769B (en) Self-adaptive multi-camera capturing and real-time free view video rendering method and system
CN109407329B (en) Space light field display method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant