CN117974742A - Binocular image generation method, device, equipment, storage medium and program product - Google Patents

Abstract

The present disclosure relates to a binocular image generation method, device, equipment, storage medium and program product. The method comprises: obtaining multiple images corresponding to a target scene, wherein the multiple images are acquired from multiple angles of the target scene; using the multiple images to train a neural radiation field model to obtain a trained neural radiation field model; using the trained neural radiation field model to generate a left eye image based on a preset left eye light; and using the trained neural radiation field model to generate a right eye image based on a preset right eye light.

Description

Binocular image generation method, device, equipment, storage medium and program product

Technical Field

The present disclosure relates to the field of computer vision technology, and in particular to a binocular image generation method, device, equipment, storage medium and program product.

Background technique

By providing the viewer with a left-eye image and a right-eye image with appropriate parallax, a stereoscopic visual effect can be provided to the viewer. The left-eye image represents an image provided to the left eye for viewing, and the right-eye image represents an image provided to the right eye for viewing. The viewing device may be VR (Virtual Reality) glasses, AR (Augmented Reality) glasses, etc.

In the related art, specially designed shooting equipment is needed to obtain left-eye images and right-eye images. Some even need to rely on 3D (3 Dimensions) sensors or require precisely calibrated camera arrays. The hardware cost is high and the implementation is difficult.

Summary of the invention

The present disclosure provides a technical solution for generating binocular images.

According to one aspect of the present disclosure, a method for generating a binocular image is provided, comprising:

Acquire multiple images corresponding to a target scene, wherein the multiple images are acquired from multiple angles of the target scene;

Using the multiple images to train a neural radiation field model to obtain a trained neural radiation field model;

The neural radiation field model completed through the training generates a left eye image based on a preset left eye light;

The neural radiation field model completed through the training generates a right eye image based on the preset right eye light.

By acquiring multiple images acquired from multiple angles of the target scene, using the multiple images to train a neural radiation field model to obtain a trained neural radiation field model, generating a left-eye image based on a preset left-eye light through the trained neural radiation field model, and generating a right-eye image based on a preset right-eye light through the trained neural radiation field model, thereby generating a binocular image corresponding to the target scene by utilizing the multiple images acquired from multiple angles of the target scene, which can reduce hardware costs, reduce the difficulty of obtaining binocular images, and generate accurate and high-quality binocular images.

In a possible implementation manner, the multiple images include panoramic information of the target scene, the left-eye image is a panoramic image corresponding to the left eye, and the right-eye image is a panoramic image corresponding to the right eye.

In this implementation, by using multiple images containing panoramic information of the target scene, a left-eye panoramic image and a right-eye panoramic image can be generated, thereby providing a binocular panoramic image, that is, a panoramic image with a stereoscopic effect can be provided, thereby providing a stereoscopic panoramic effect to a viewer using a viewing device.

In a possible implementation manner, any image information in any image among the multiple images is at least contained in another image among the multiple images.

In this implementation, by repeatedly collecting the same visual information of the target scene from different angles, the accuracy of three-dimensional reconstruction of the target scene can be improved, thereby improving the quality of the ultimately generated left-eye panoramic image and right-eye panoramic image.

In a possible implementation manner, the multiple images are acquired by moving a single camera.

In this implementation, a single camera is used to acquire multiple images corresponding to the target scene, and a left-eye image and a right-eye image are generated based on the multiple images acquired, thereby reducing the equipment cost for generating binocular images. Moreover, in the method of using a single camera, there is no need to adjust the internal parameters of different cameras, which can further reduce the complexity of operation. In addition, in this implementation, the user only needs to take multiple images of the target scene from multiple angles without having to operate complex equipment.

In a possible implementation, the radius of the minimum circumscribed circle of the multiple acquisition points corresponding to the multiple images is within a preset radius range; wherein the acquisition point corresponding to any one of the multiple images represents the optical center position of the lens when the image is acquired; and the left boundary of the preset radius range is greater than 0.

In this implementation, by controlling the radius of the minimum circumscribed circle of the multiple acquisition points corresponding to the multiple images to be within a preset radius range, it is possible to ensure that there is a suitable parallax between the images of adjacent perspectives in the multiple images, thereby facilitating accurate and efficient three-dimensional reconstruction of the target scene.

In a possible implementation, the using the multiple images to train the neural radiation field model to obtain the trained neural radiation field model includes:

Based on the multiple images, determining a plurality of camera extrinsic parameters corresponding one-to-one to the multiple images;

Based on the multiple images and the multiple camera extrinsic parameters, a neural radiation field model is trained to obtain a trained neural radiation field model.

By training the neural radiation field model through this implementation method, the neural radiation field model can learn the three-dimensional visual information of the target scene.

In one possible implementation,

The preset left-eye light is a ray, the endpoint of the preset left-eye light is on a preset circle, the preset left-eye light is on a tangent of the preset circle, and the direction of the preset left-eye light is the clockwise direction of the preset circle;

The preset right-eye light is a ray, the endpoint of the preset right-eye light is on the preset circle, the preset right-eye light is on the tangent of the preset circle, and the direction of the preset right-eye light is counterclockwise of the preset circle;

Wherein, the diameter of the preset circle is the preset pupil distance.

By adopting this implementation method, it is possible to generate left-eye images and right-eye images with appropriate parallax and high quality.

In one possible implementation, the method is applied to any one of a virtual reality device, an augmented reality device, a mixed reality device, an artificial intelligence device, and a digital twin system.

According to one aspect of the present disclosure, a device for generating a binocular image is provided, comprising:

An acquisition module, used to acquire multiple images corresponding to a target scene, wherein the multiple images are acquired from multiple angles of the target scene;

A training module, used to train a neural radiation field model using the multiple images to obtain a trained neural radiation field model;

A first generating module, configured to generate a left eye image based on a preset left eye light by using the trained neural radiation field model;

The second generating module is used to generate a right eye image based on a preset right eye light through the trained neural radiation field model.

In a possible implementation manner, the multiple images include panoramic information of the target scene, the left-eye image is a panoramic image corresponding to the left eye, and the right-eye image is a panoramic image corresponding to the right eye.

In a possible implementation manner, any image information in any image among the multiple images is at least contained in another image among the multiple images.

In a possible implementation manner, the multiple images are acquired by moving a single camera.

In a possible implementation, the radius of the minimum circumscribed circle of the multiple acquisition points corresponding to the multiple images is within a preset radius range; wherein the acquisition point corresponding to any one of the multiple images represents the optical center position of the lens when the image is acquired; and the left boundary of the preset radius range is greater than 0.

In a possible implementation, the training module is used to:

Based on the multiple images, determining a plurality of camera extrinsic parameters corresponding one-to-one to the multiple images;

Based on the multiple images and the multiple camera extrinsic parameters, a neural radiation field model is trained to obtain a trained neural radiation field model.

In one possible implementation,

The preset left-eye light is a ray, the endpoint of the preset left-eye light is on a preset circle, the preset left-eye light is on a tangent of the preset circle, and the direction of the preset left-eye light is the clockwise direction of the preset circle;

The preset right-eye light is a ray, the endpoint of the preset right-eye light is on the preset circle, the preset right-eye light is on the tangent of the preset circle, and the direction of the preset right-eye light is counterclockwise of the preset circle;

Wherein, the diameter of the preset circle is the preset pupil distance.

In one possible implementation, the device is applied to any one of a virtual reality device, an augmented reality device, a mixed reality device, an artificial intelligence device, and a digital twin system.

According to one aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor.

According to one aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.

In an embodiment of the present disclosure, a plurality of images acquired from a plurality of angles of the target scene are acquired, and a neural radiation field model is trained using the plurality of images to obtain a trained neural radiation field model. A left-eye image is generated based on a preset left-eye light by the trained neural radiation field model, and a right-eye image is generated based on a preset right-eye light by the trained neural radiation field model. Thus, by utilizing the plurality of images acquired from a plurality of angles of the target scene to generate a binocular image corresponding to the target scene, hardware costs can be reduced, the difficulty of obtaining binocular images can be reduced, and accurate and high-quality binocular images can be generated.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

FIG1 shows a flow chart of a binocular image generation method provided by an embodiment of the present disclosure.

FIG. 2 is a schematic diagram showing a method for collecting multiple images corresponding to a target scene in a binocular image generation method provided in an embodiment of the present disclosure.

FIG. 3 a shows a schematic diagram of a left-eye panoramic image generated by the binocular image generation method provided by an embodiment of the present disclosure.

FIG3 b is a schematic diagram showing a right-eye panoramic image generated by the binocular image generation method provided by an embodiment of the present disclosure.

FIG. 4 a is a schematic diagram showing a preset left-eye light in a binocular image generation method provided in an embodiment of the present disclosure.

FIG. 4 b is a schematic diagram showing a preset right-eye light in the binocular image generation method provided in an embodiment of the present disclosure.

FIG5 shows a block diagram of a binocular image generating device provided by an embodiment of the present disclosure.

FIG. 6 shows a block diagram of an electronic device 1900 provided by an embodiment of the present disclosure.

Detailed ways

Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

The term "and/or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and/or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

The embodiments of the present disclosure provide a binocular image generation method, a binocular image generation device, an electronic device, a storage medium and a program product. A plurality of images acquired from a target scene from multiple angles are acquired, and a neural radiation field model is trained using the plurality of images to obtain a trained neural radiation field model. A left-eye image is generated based on a preset left-eye light by the trained neural radiation field model, and a right-eye image is generated based on a preset right-eye light by the trained neural radiation field model. Thus, a binocular image corresponding to the target scene is generated by using the plurality of images acquired from a target scene from multiple angles, thereby reducing hardware costs, reducing the difficulty of obtaining binocular images, and generating accurate and high-quality binocular images.

Exemplarily, the binocular image generation method, binocular image generation device, electronic device, storage medium and program product of the embodiments of the present disclosure can be applied to application fields such as virtual reality technology (VR), augmented reality technology (AR), mixed reality technology (MR), artificial intelligence technology (e.g., artificial intelligence painting) and digital twins. For example, it can be applied to any one of virtual reality devices, artificial intelligence devices, augmented reality devices, mixed reality devices and digital twin systems. It should be noted that the present disclosure does not limit specific application fields or scenarios.

The binocular image generation method provided by the embodiment of the present disclosure is described in detail below with reference to the accompanying drawings.

FIG1 shows a flow chart of a binocular image generation method provided by an embodiment of the present disclosure. In a possible implementation, the execution subject of the binocular image generation method may be a binocular image generation device, for example, the binocular image generation method may be executed by a terminal device or a server or other electronic device. Among them, the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device or a wearable device, etc. In some possible implementations, the binocular image generation method may be implemented by a processor calling a computer-readable instruction stored in a memory. As shown in FIG1 , the binocular image generation method includes steps S11 to S14.

In step S11, a plurality of images corresponding to a target scene are acquired, wherein the plurality of images are acquired from capturing the target scene from multiple angles.

In step S12, the plurality of images are used to train a neural radiation field model to obtain a trained neural radiation field model.

In step S13, the neural radiation field model completed through the training generates a left eye image based on the preset left eye light.

In step S14, the neural radiation field model completed through the training generates a right eye image based on the preset right eye light.

In an embodiment of the present disclosure, a binocular image may include a left-eye image and a right-eye image, wherein the left-eye image may represent an image provided to the left eye for viewing, and the right-eye image may represent an image provided to the right eye for viewing. There is parallax between the left-eye image and the right-eye image, and both the left-eye image and the right-eye image may be two-dimensional images. Among them, the parallax may represent the directional difference produced by observing the same target from two points at a certain distance. The angle between the two points viewed from the target may be referred to as the parallax angle between the two points. In some application scenarios, the binocular image may also be referred to as a 3D (3 Dimensions, three-dimensional) image, a stereo image, a binocular stereo image, a binocular vision image, etc., which is not limited here.

The left-eye image and the right-eye image generated by the embodiment of the present disclosure may be provided to a viewing device for display, so as to provide a stereoscopic visual effect to a viewer using the viewing device. The viewing device may be any device capable of providing a stereoscopic visual effect by displaying the left-eye image and the right-eye image. For example, the viewing device may be VR (Virtual Reality) glasses, AR (Augmented Reality) glasses, etc., which are not limited here.

In the disclosed embodiments, the target scene may represent any scene for which a binocular image is to be generated. The multiple images corresponding to the target scene may be images obtained by performing image acquisition and/or video acquisition of the target scene from multiple angles. For example, images of the target scene may be taken from multiple angles to obtain multiple images. For another example, a video of the target scene may be taken from multiple angles, and multiple images may be captured from the video.

In the disclosed embodiment, when the multiple images are collected from multiple angles, the camera can be manually controlled by the user or mechanically controlled, which is not limited here. The method of manually controlling the camera movement by the user not only does not require the user to operate complex equipment, but also reduces hardware costs, thereby reducing the difficulty of implementing multi-angle image collection, and can be applied to a wider range of scenarios.

In a possible implementation, the multiple images are acquired by moving a single camera. In this implementation, multiple images corresponding to the target scene are acquired by using a single camera, and left-eye images and right-eye images are generated based on the multiple images acquired, thereby reducing the cost of equipment for generating binocular images. Moreover, in the method of using a single camera, there is no need to adjust the internal parameters of different cameras, which can further reduce the complexity of operation. In addition, in this implementation, the user only needs to take multiple images of the target scene from multiple angles without having to operate complex equipment.

In another possible implementation, the multiple images may be acquired by moving at least two cameras. In this implementation, the internal parameters of the at least two cameras may be adjusted so that the internal parameters of the at least two cameras are consistent or similar. After adjusting the internal parameters of the at least two cameras, the at least two cameras may be moved to obtain multiple images of the target scene from multiple angles.

In a possible implementation, the radius of the minimum circumscribed circle of the multiple acquisition points corresponding to the multiple images is within a preset radius range; wherein the acquisition point corresponding to any one of the multiple images represents the optical center position of the lens when the image is acquired; and the left boundary of the preset radius range is greater than 0.

There is a special point on the main axis of the lens. The propagation direction of any light passing through this point remains unchanged. This point is called the optical center. The center of the convex lens can be roughly regarded as the optical center. Usually, the lens of a camera is equivalent to a convex lens.

In this implementation, the minimum circumscribed circle of the multiple projection points corresponding to the multiple acquisition points can be used as the minimum circumscribed circle of the multiple acquisition points. The multiple acquisition points can be projected onto the same horizontal plane to obtain multiple projection points corresponding to the multiple acquisition points.

For example, the preset radius interval may be (50 cm, 100 cm]. Of course, those skilled in the art may flexibly set the preset radius interval according to the actual application scenario requirements, and no limitation is made here.

In this implementation, the range of camera movement can be determined according to a preset viewing point and a preset radius interval, and the camera movement can be controlled within the range.

In this implementation, by controlling the radius of the minimum circumscribed circle of the multiple acquisition points corresponding to the multiple images to be within a preset radius range, it is possible to ensure that there is a suitable parallax between the images of adjacent perspectives in the multiple images, thereby facilitating accurate and efficient three-dimensional reconstruction of the target scene.

In one possible implementation, the multiple images include panoramic information of the target scene, the left-eye image is a panoramic image corresponding to the left eye, and the right-eye image is a panoramic image corresponding to the right eye. In this implementation, the multiple images may include 360-degree image information of the target scene. In this implementation, the binocular image may be called a binocular panoramic image, the left-eye image may be called a left-eye panoramic image, and the right-eye image may be called a right-eye panoramic image. In some application scenarios, the binocular panoramic image may also be called a 3D panoramic image, a stereoscopic panoramic image, a panoramic stereo image, a binocular stereoscopic panoramic image, etc., which is not limited here.

In this implementation, by using multiple images containing panoramic information of the target scene, a left-eye panoramic image and a right-eye panoramic image can be generated, thereby providing a binocular panoramic image, that is, a panoramic image with a stereoscopic effect can be provided, thereby providing a stereoscopic panoramic effect to a viewer using a viewing device.

As an example of this implementation, any image information in any one of the multiple images is at least contained in another one of the multiple images. In this example, the same visual information of the target scene can be repeatedly collected from different angles so that the different collected images contain the same visual information of the target scene. For example, when the camera is moved, each object in the target scene can be photographed more than twice.

In this example, by repeatedly collecting the same visual information of the target scene from different angles, the accuracy of three-dimensional reconstruction of the target scene can be improved, thereby improving the quality of the ultimately generated left-eye panoramic image and right-eye panoramic image.

As another example of this implementation, image information in any one of the multiple images may not be included in another image of the multiple images.

FIG2 is a schematic diagram showing a method for collecting multiple images corresponding to a target scene in a binocular image generation method provided by an embodiment of the present disclosure. As shown in FIG2 , a camera 22 can be moved near a viewing point 21, and the camera 22 can be controlled to shoot the target scene outward. Among them, the radius of the minimum circumscribed circle 23 of each acquisition point of the camera 22 can be between 50 cm and 1 m. By controlling the camera 22 to capture images at multiple different positions, the images captured can cover image information at various angles outside the minimum circumscribed circle 23. By capturing more images of the target scene from more angles, more accurate three-dimensional reconstruction is facilitated, thereby facilitating the generation of left-eye and right-eye images with higher resolutions.

Fig. 3a is a schematic diagram of a left-eye panoramic image generated by the binocular image generation method provided by an embodiment of the present disclosure. Fig. 3b is a schematic diagram of a right-eye panoramic image generated by the binocular image generation method provided by an embodiment of the present disclosure. In Figs. 3a and 3b, the left-eye panoramic image and the right-eye panoramic image are panoramic images of spherical projection.

In the embodiment of the present disclosure, the multiple images can be used to train the Neural Radiance Fields (NeRF) model. During the training process of the neural radiation field model, the parameters of the neural radiation field model can be updated by minimizing the pixel difference between the known image (i.e., the multiple images) and the image obtained by rendering. Among them, the neural radiation field model introduces a fully connected neural network into the three-dimensional representation of the scene. By using the multiple images as supervision, the neural radiation field model can implicitly reconstruct the target scene in three dimensions, and then generate a two-dimensional image of a new angle by rendering under a new perspective. The neural radiation field model in the embodiment of the present disclosure can adopt a neural radiation field algorithm or an improved algorithm thereof, which is not limited here.

In a possible implementation, the multiple images are used to train the neural radiation field model to obtain a trained neural radiation field model, including: based on the multiple images, determining a number of camera extrinsic parameters corresponding one-to-one to the multiple images; based on the multiple images and the multiple camera extrinsic parameters, training the neural radiation field model to obtain a trained neural radiation field model.

In this implementation, a method such as Structure From Motion (SFM) can be used to recover multiple camera extrinsic parameters corresponding to the multiple images one by one based on the multiple images. The camera extrinsic parameters corresponding to any one of the multiple images may include rotation information and translation information. As an example of this implementation, the camera extrinsic parameters corresponding to any one of the images may be represented by an extrinsic matrix, the rotation information may be represented by a rotation matrix, and the translation information may be represented by a translation vector.

For any image among the multiple images, the three-dimensional coordinates X=(x, y, z) of the space point corresponding to the image and the two-dimensional viewing direction d=(θ, Φ) can be determined based on the camera extrinsics corresponding to the image. In the training stage of the neural radiation field model, the input of the neural radiation field model may include the three-dimensional coordinates of the space point corresponding to the image and the two-dimensional viewing direction. By training the neural radiation field model in this implementation method, the neural radiation field model can learn the three-dimensional visual information of the target scene.

In the embodiment of the present disclosure, the left eye image and the right eye image are rendered by the trained neural radiation field model. In the embodiment of the present disclosure, the execution order of step S13 and step S14 is not limited. For example, step S13 and step S14 can be executed at the same time. For another example, step S13 can be executed first, and then step S14 can be executed. For another example, step S14 can be executed first, and then step S13 can be executed.

In a possible implementation, the preset left eye light is a ray, the endpoint of the preset left eye light is on a preset circle, the preset left eye light is on a tangent of the preset circle, and the direction of the preset left eye light is in a clockwise direction of the preset circle; the preset right eye light is a ray, the endpoint of the preset right eye light is on the preset circle, the preset right eye light is on a tangent of the preset circle, and the direction of the preset right eye light is in a counterclockwise direction of the preset circle; wherein the diameter of the preset circle is a preset pupil distance.

In this implementation, the value range of the preset pupil distance may be 5 cm to 12 cm. As an example of this implementation, the pupil distance of the target user may be used as the preset pupil distance.

In this implementation, the diameter of the preset circle is equal to the preset pupil distance. In this implementation, the preset left eye light and the preset right eye light can be sampled according to the light sampling direction of ODS (Omni-Directional Stereo). Figure 4a shows a schematic diagram of the preset left eye light in the binocular image generation method provided by the embodiment of the present disclosure. In Figure 4a, the ray with an arrow represents the preset left eye light. As shown in Figure 4a, the preset left eye light is a ray, the endpoint of the preset left eye light is on the preset circle, the preset left eye light is on the tangent of the preset circle, and the direction of the preset left eye light is the clockwise direction of the preset circle. Figure 4b shows a schematic diagram of the preset right eye light in the binocular image generation method provided by the embodiment of the present disclosure. In Figure 4b, the ray with an arrow represents the preset right eye light. As shown in Figure 4b, the preset right eye light is a ray, the endpoint of the preset right eye light is on the preset circle, the preset right eye light is on the tangent of the preset circle, and the direction of the preset right eye light is the counterclockwise direction of the preset circle.

For example, a panoramic image using spherical projection includes n columns of pixels, and each column of pixels represents 360/n degrees of light. Then, after determining the preset left eye light, n columns of pixels can be rendered column by column to obtain a left eye panoramic image. After determining the preset right eye light, n columns of pixels can be rendered column by column to obtain a right eye panoramic image.

By adopting this implementation method, it is possible to generate left-eye images and right-eye images with appropriate parallax and high quality.

Of course, in other implementations, those skilled in the art can flexibly set the preset left eye light direction and the preset right eye light direction according to actual application scenario requirements and/or personal preferences, which is not limited here.

The binocular image generation method provided in the embodiments of the present disclosure can be applied to virtual reality, augmented reality, three-dimensional reconstruction, computer vision, deep learning and other fields, and is not limited here.

The following is an explanation of the binocular image generation method provided by the embodiment of the present disclosure through a specific application scenario. In this application scenario, the image acquisition method shown in FIG. 2 can be used to acquire images of the target scene from different angles by moving a single camera to obtain multiple images. Among them, the multiple images may include 360-degree image information of the target scene, and any image information in any of the multiple images may be included in at least another image of the multiple images. After acquiring the multiple images, a number of camera extrinsics corresponding to the multiple images can be determined based on the multiple images, and a neural radiation field model can be trained based on the multiple images and the multiple camera extrinsics. After the training of the neural radiation field model is completed, a left-eye panoramic image can be generated based on the preset left-eye light as shown in FIG. 4a through the trained neural radiation field model, and a right-eye panoramic image can be generated based on the preset right-eye light as shown in FIG. 4b through the trained neural radiation field model.

It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

In addition, the present disclosure also provides a binocular image generation device, an electronic device, a computer-readable storage medium, and a computer program product, all of which can be used to implement any binocular image generation method provided by the present disclosure. The corresponding technical solutions and technical effects can be found in the corresponding records of the method part and will not be repeated here.

FIG5 is a block diagram of a binocular image generation device provided by an embodiment of the present disclosure. As shown in FIG5 , the binocular image generation device includes:

An acquisition module 51 is used to acquire multiple images corresponding to a target scene, wherein the multiple images are acquired from multiple angles of the target scene;

A training module 52 is used to train a neural radiation field model using the multiple images to obtain a trained neural radiation field model;

A first generating module 53, configured to generate a left eye image based on a preset left eye light by using the trained neural radiation field model;

The second generating module 54 is used to generate a right eye image based on the preset right eye light through the trained neural radiation field model.

In a possible implementation manner, the multiple images include panoramic information of the target scene, the left-eye image is a panoramic image corresponding to the left eye, and the right-eye image is a panoramic image corresponding to the right eye.

In a possible implementation manner, any image information in any image among the multiple images is at least contained in another image among the multiple images.

In a possible implementation manner, the multiple images are acquired by moving a single camera.

In a possible implementation, the radius of the minimum circumscribed circle of the multiple acquisition points corresponding to the multiple images is within a preset radius range; wherein the acquisition point corresponding to any one of the multiple images represents the optical center position of the lens when the image is acquired; and the left boundary of the preset radius range is greater than 0.

In a possible implementation, the training module 52 is used to:

Based on the multiple images, determining a plurality of camera extrinsic parameters corresponding one-to-one to the multiple images;

Based on the multiple images and the multiple camera extrinsic parameters, a neural radiation field model is trained to obtain a trained neural radiation field model.

In one possible implementation,

The preset left-eye light is a ray, the endpoint of the preset left-eye light is on a preset circle, the preset left-eye light is on a tangent of the preset circle, and the direction of the preset left-eye light is the clockwise direction of the preset circle;

The preset right-eye light is a ray, the endpoint of the preset right-eye light is on the preset circle, the preset right-eye light is on the tangent of the preset circle, and the direction of the preset right-eye light is counterclockwise of the preset circle;

Wherein, the diameter of the preset circle is the preset pupil distance.

In one possible implementation, the device is applied to any one of a virtual reality device, an augmented reality device, a mixed reality device, an artificial intelligence device, and a digital twin system.

In an embodiment of the present disclosure, a plurality of images acquired from a plurality of angles of the target scene are acquired, and a neural radiation field model is trained using the plurality of images to obtain a trained neural radiation field model. A left-eye image is generated based on a preset left-eye light by the trained neural radiation field model, and a right-eye image is generated based on a preset right-eye light by the trained neural radiation field model. Thus, by utilizing the plurality of images acquired from a plurality of angles of the target scene to generate a binocular image corresponding to the target scene, hardware costs can be reduced, the difficulty of obtaining binocular images can be reduced, and accurate and high-quality binocular images can be generated.

In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. Its specific implementation and technical effects can refer to the description of the above method embodiments. For the sake of brevity, they will not be repeated here.

The present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

The embodiment of the present disclosure further provides a computer program, including a computer-readable code. When the computer-readable code is executed in an electronic device, a processor in the electronic device executes the above method.

The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in an electronic device, a processor in the electronic device executes the above method.

An embodiment of the present disclosure also provides an electronic device, comprising: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory to execute the above method.

The electronic device may be provided as a terminal, a server, or a device in other forms.

FIG6 shows a block diagram of an electronic device 1900 provided in an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Referring to FIG6 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each of which corresponds to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

The electronic device 1900 may further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input/output (I/O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Microsoft's server operating system (Windows Server ^™ ), Apple's graphical user interface-based operating system (Mac OSX ^™ ), a multi-user multi-process computer operating system (Unix ^™ ), a free and open source Unix-like operating system (Linux ^™ ), an open source Unix-like operating system (FreeBSD ^™ ), or the like.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

The present disclosure may be a system, a method and/or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing/processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and/or edge servers. The network adapter card or network interface in each computing/processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing/processing device.

The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

Various aspects of the present disclosure are described herein with reference to the flowcharts and/or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and/or block diagram and the combination of each box in the flowchart and/or block diagram can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions/actions specified in one or more boxes in the flowchart and/or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and/or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions/actions specified in one or more boxes in the flowchart and/or block diagram.

The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and/or flow chart, and the combination of the square boxes in the block diagram and/or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

If the technical solution of the embodiments of the present disclosure involves personal information, the product using the technical solution of the embodiments of the present disclosure has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of the embodiments of the present disclosure involves sensitive personal information, the product using the technical solution of the embodiments of the present disclosure has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as the camera, a clear and obvious sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he agrees to collect his personal information; or on the device for processing personal information, when the personal information processing rules are notified by obvious signs/information, the individual's authorization is obtained through pop-up information or by asking the individual to upload his personal information by himself; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (12)

1. A method of generating a binocular image, comprising:

acquiring a plurality of images corresponding to a target scene, wherein the images are acquired from a plurality of angles for the target scene;

training a nerve radiation field model by adopting the images to obtain a trained nerve radiation field model;

Generating a left eye image based on a preset left eye light line through the trained nerve radiation field model;

and generating a right eye image based on the preset right eye light rays through the trained nerve radiation field model.

2. The method of claim 1, wherein the plurality of images includes panoramic information of the target scene, the left-eye image is a panoramic image corresponding to a left eye, and the right-eye image is a panoramic image corresponding to a right eye.

3. The method of claim 2, wherein any image information in any one of the plurality of images is contained at least in another one of the plurality of images.

4. The method of claim 1, wherein the plurality of images are acquired by moving a single camera.

5. The method of claim 1, wherein the radius of the smallest circumscribed circle of the plurality of acquisition points corresponding to the plurality of images is within a preset radius interval; the image acquisition device comprises a plurality of images, a plurality of imaging units and a lens, wherein the acquisition point position corresponding to any one of the plurality of images represents the optical center position of the lens when the image is acquired; the left boundary of the preset radius interval is greater than 0.

6. The method of any one of claims 1 to 5, wherein training a neural radiation field model using the plurality of images to obtain a trained neural radiation field model comprises:

Determining a plurality of camera external parameters corresponding to the plurality of images one by one based on the plurality of images;

And training a nerve radiation field model based on the images and the camera external parameters to obtain a trained nerve radiation field model.

7. The method according to any one of claims 1 to 5, wherein,

The preset left eye light is a ray, the end point of the preset left eye light is on a preset circle, the preset left eye light is on a tangent line of the preset circle, and the direction of the preset left eye light is clockwise of the preset circle;

the preset right eye ray is a ray, the end point of the preset right eye ray is on the preset circle, the preset right eye ray is on the tangent line of the preset circle, and the direction of the preset right eye ray is the anticlockwise direction of the preset circle;

The diameter of the preset circle is a preset interpupillary distance.

8. The method according to any one of claims 1 to 5, wherein the method is applied to any one of a virtual reality device, an augmented reality device, a mixed reality device, an artificial intelligence device, and a digital twin system.

9. A binocular image generating apparatus, comprising:

The acquisition module is used for acquiring a plurality of images corresponding to a target scene, wherein the images are acquired from a plurality of angles for the target scene;

The training module is used for training the nerve radiation field model by adopting the images to obtain a trained nerve radiation field model;

the first generation module is used for generating a left eye image based on a preset left eye light line through the trained nerve radiation field model;

and the second generation module is used for generating a right eye image based on the preset right eye light rays through the trained nerve radiation field model.

10. An electronic device, comprising:

one or more processors;

a memory for storing executable instructions;

Wherein the one or more processors are configured to invoke the memory-stored executable instructions to perform the method of any of claims 1 to 8.

11. A computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method of any of claims 1 to 8.

12. A computer program product comprising computer readable code, or a non-transitory computer readable storage medium carrying computer readable code, which when run in an electronic device, causes a processor in the electronic device to perform the method of any one of claims 1 to 8.