CN117455754A - Image conversion method and related equipment - Google Patents
Image conversion method and related equipment Download PDFInfo
- Publication number
- CN117455754A CN117455754A CN202311415758.7A CN202311415758A CN117455754A CN 117455754 A CN117455754 A CN 117455754A CN 202311415758 A CN202311415758 A CN 202311415758A CN 117455754 A CN117455754 A CN 117455754A
- Authority
- CN
- China
- Prior art keywords
- dimensional image
- viewpoint
- image
- processed
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 65
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 25
- 238000009877 rendering Methods 0.000 claims abstract description 46
- 238000012545 processing Methods 0.000 claims description 31
- 230000008439 repair process Effects 0.000 claims description 17
- 230000004927 fusion Effects 0.000 claims description 14
- 238000003860 storage Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 8
- 230000008569 process Effects 0.000 description 15
- 238000004422 calculation algorithm Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 11
- 230000018109 developmental process Effects 0.000 description 10
- 238000003062 neural network model Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 6
- 230000005012 migration Effects 0.000 description 6
- 238000013508 migration Methods 0.000 description 6
- 238000013528 artificial neural network Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 4
- 238000000605 extraction Methods 0.000 description 4
- 238000012549 training Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000013475 authorization Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000009471 action Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000001427 coherent effect Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000003708 edge detection Methods 0.000 description 1
- 230000003631 expected effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000003709 image segmentation Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003472 neutralizing effect Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 238000003786 synthesis reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20228—Disparity calculation for image-based rendering
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Graphics (AREA)
- Image Processing (AREA)
Abstract
The application provides an image conversion method and related equipment, which are used for acquiring a two-dimensional image to be processed, generating a plurality of viewpoint images corresponding to the two-dimensional image to be processed based on the two-dimensional image to be processed and preset viewpoint conditions, and performing viewpoint rendering based on the two-dimensional image to be processed and the viewpoint images, so that the obtained target three-dimensional image is smoother and more natural, the problem of artifacts can be avoided, and the quality of the converted high-dimensional image can be ensured.
Description
Technical Field
The present disclosure relates to the field of image display technologies, and in particular, to an image conversion method and related devices.
Background
With the development of user viewing demands, in order to be able to more meet the user viewing demands, it is necessary to convert a low-dimensional image (for example, a two-dimensional image) into a high-dimensional image.
Based on the above situation, in the prior art, the image conversion manner may cause serious artifact problem in the converted high-dimensional image (for example, three-dimensional image), and the quality of the converted high-dimensional image cannot be ensured.
Disclosure of Invention
In view of the foregoing, an object of the present application is to provide a method and related apparatus for converting an image display format, which are used for solving the above-mentioned problems.
Based on the above object, a first aspect of the present application provides an image conversion method, including:
acquiring a two-dimensional image to be processed;
generating a plurality of viewpoint images corresponding to the two-dimensional image to be processed based on the two-dimensional image to be processed and preset viewpoint conditions;
and performing viewpoint rendering based on the two-dimensional image to be processed and the plurality of viewpoint images to obtain a target three-dimensional image.
Optionally, the acquiring the two-dimensional image to be processed includes:
acquiring an original two-dimensional image to be processed;
and resetting the original two-dimensional image to be processed according to a preset size to obtain the two-dimensional image to be processed.
Optionally, the preset viewpoint conditions include a viewpoint number and a viewpoint angle, the viewpoint number and the viewpoint angle being determined based on display parameters of a display device for displaying the target three-dimensional image;
the generating a plurality of viewpoint images corresponding to the two-dimensional image to be processed based on the two-dimensional image to be processed and a preset viewpoint condition includes:
acquiring an original two-dimensional image to be processed, and acquiring a depth image from the original two-dimensional image to be processed;
extracting a foreground image and a background image from the depth image, and determining the relative position relationship between the foreground image and the background image;
determining a plurality of target viewpoints corresponding to the foreground images according to the viewpoint number and the viewpoint angle;
determining a plurality of first viewpoint images corresponding to the foreground images according to the plurality of target viewpoints, and determining a plurality of second viewpoint images corresponding to the background images based on the relative position relationship;
and fusing the first viewpoint images and the second viewpoint images to generate viewpoint images corresponding to the two-dimensional image to be processed.
Optionally, the acquiring a depth image from the original two-dimensional image to be processed includes:
performing parallax prediction on the original two-dimensional image to be processed to obtain a parallax prediction result;
acquiring an image acquisition unit focal length and an image acquisition unit interval corresponding to the original two-dimensional image to be processed;
performing product processing on the focal length of the image acquisition unit and the distance between the image acquisition units to obtain a product processing result;
and carrying out ratio determination processing on the parallax prediction result by using the product processing result to obtain depth information, and constructing the depth image based on the depth information.
Optionally, the performing viewpoint rendering based on the two-dimensional image to be processed and the multiple viewpoint images to obtain a target three-dimensional image includes:
repairing the two-dimensional image to be processed and the multiple viewpoint images to obtain multiple repaired viewpoint images;
and splicing the plurality of repair viewpoint images to obtain a target three-dimensional image.
Optionally, the repairing the two-dimensional image to be processed and the multiple viewpoint images to obtain multiple repaired viewpoint images includes:
extracting features of the two-dimensional image to be processed to obtain first features, and extracting features of the viewpoint images to obtain second features;
performing feature fusion on the first feature and the plurality of second features to obtain a plurality of feature fusion results;
performing feature offset processing on the feature fusion results to obtain a plurality of feature offset results;
and carrying out cavity restoration on the characteristic offset results to obtain a plurality of restoration viewpoint images.
Optionally, the stitching the multiple repair viewpoint images to obtain a target three-dimensional image includes:
splicing the plurality of repair viewpoint images in a preset dimension direction of the three-dimensional space to obtain a spliced image;
and performing viewpoint interweaving treatment on the spliced images to obtain target dimension images.
Based on the same inventive concept, a second aspect of the present application provides an image conversion apparatus, comprising:
the acquisition module is configured to acquire a two-dimensional image to be processed;
the viewpoint image generation module is configured to generate a plurality of viewpoint images corresponding to the two-dimensional image to be processed based on the two-dimensional image to be processed and preset viewpoint conditions;
and the viewpoint rendering module is configured to perform viewpoint rendering based on the two-dimensional image to be processed and the plurality of viewpoint images to obtain a target three-dimensional image.
Based on the same inventive concept, a third aspect of the present application provides an electronic device comprising a memory, a processor and a computer program stored on the memory and executable by the processor, the processor implementing the method as described in the first aspect above when executing the computer program.
Based on the same inventive concept, a fourth aspect of the present application provides a non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of the first aspect above.
Based on the same inventive concept, a fifth aspect of the present application provides a computer program product comprising computer program instructions which, when run on a computer, cause the computer to perform the method according to the first aspect.
From the above, it can be seen that the image display form conversion method and related device provided by the present application acquire a two-dimensional image to be processed, generate multiple viewpoint images corresponding to the two-dimensional image to be processed based on the two-dimensional image to be processed and preset viewpoint conditions, and perform viewpoint rendering based on the two-dimensional image to be processed and the multiple viewpoint images, so that the obtained target three-dimensional image is smoother and more natural, further the problem of artifacts can be avoided, and the quality of the converted high-dimensional image can be ensured.
Drawings
In order to more clearly illustrate the technical solutions of the present application or related art, the drawings that are required to be used in the description of the embodiments or related art will be briefly described below, and it is apparent that the drawings in the following description are only embodiments of the present application, and other drawings may be obtained according to these drawings without inventive effort to those of ordinary skill in the art.
FIG. 1 is a flowchart of an image conversion method according to an embodiment of the present application;
fig. 2A is a first schematic diagram of an image conversion process according to an embodiment of the present application;
fig. 2B is a second schematic diagram of an image conversion process according to an embodiment of the present application;
fig. 3 is a block diagram of the image conversion apparatus according to the embodiment of the present application;
fig. 4 is a schematic diagram of an electronic device according to an embodiment of the present application.
Detailed Description
For the purposes of making the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail below with reference to the accompanying drawings.
It should be noted that unless otherwise defined, technical or scientific terms used in the embodiments of the present application should be given the ordinary meaning as understood by one of ordinary skill in the art to which the present application belongs. The terms "first," "second," and the like, as used in embodiments of the present application, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The word "comprising" or "comprises", and the like, means that elements or items preceding the word are included in the element or item listed after the word and equivalents thereof, but does not exclude other elements or items. The terms "connected" or "connected," and the like, are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", etc. are used merely to indicate relative positional relationships, which may also be changed when the absolute position of the object to be described is changed.
It can be appreciated that before using the technical solutions of the embodiments in the present application, the user is informed about the type, the use range, the use scenario, etc. of the related personal information in an appropriate manner, and the authorization of the user is obtained.
For example, in response to receiving an active request from a user, a prompt is sent to the user to explicitly prompt the user that the operation it is requesting to perform will require personal information to be obtained and used with the user. Therefore, the user can select whether to provide personal information to the software or hardware such as the electronic equipment, the application program, the server or the storage medium for executing the operation of the technical scheme according to the prompt information.
As an alternative but non-limiting implementation, in response to receiving an active request from a user, the manner in which the prompt information is sent to the user may be, for example, a popup, in which the prompt information may be presented in a text manner. In addition, a selection control for the user to select to provide personal information to the electronic device in a 'consent' or 'disagreement' manner can be carried in the popup window.
It will be appreciated that the above-described notification and user authorization acquisition process is merely illustrative, and not limiting of the implementation of the present application, and that other ways of satisfying relevant legal regulations may be applied to the implementation of the present application.
With the development of the viewing requirements of users in the related art, in order to be able to more meet the viewing requirements of users, it is required to convert a low-dimensional image (for example, a two-dimensional image) into a high-dimensional image. Based on the above situation, in the prior art, the image conversion manner may cause serious artifact problem in the converted high-dimensional image (for example, three-dimensional image), and the quality of the converted high-dimensional image cannot be ensured.
In addition, in the related art, high-dimensional image transformation is performed on the existing low-dimensional image, and most of the low-dimensional image is manufactured in the later stage by a large amount of manpower by utilizing professional software, but the cost of manufacturing in the later stage of manpower is high, time and labor are wasted, and the method is not suitable for the general demands of users.
The embodiment of the application provides an image conversion method, which is used for performing viewpoint rendering based on a two-dimensional image to be processed and a plurality of viewpoint images, so that the obtained target three-dimensional image is smoother and more natural, the problem of artifacts can be avoided, and the quality of the converted high-dimensional image can be ensured.
As shown in fig. 1, the method of the present embodiment includes:
step 101, acquiring a two-dimensional image to be processed.
In this step, the two-dimensional image to be processed is acquired by an image acquisition device (e.g., camera, video camera) in the terminal device; alternatively, the two-dimensional image to be processed may be acquired from an album of the terminal device.
Step 102, generating a plurality of viewpoint images corresponding to the two-dimensional image to be processed based on the two-dimensional image to be processed and preset viewpoint conditions.
In this step, the viewpoint conditions are determined based on display parameters of a display device for displaying the target three-dimensional image, the viewpoint conditions including the number of viewpoints and the viewpoint angle, and the display device may be a desktop computer, a mobile phone, a mobile computer, a tablet computer, a media player, a smart wearable device.
The viewpoint image represents a visual difference image at a different viewing angle position.
And generating a plurality of viewpoint images corresponding to the two-dimensional images to be processed based on the two-dimensional images to be processed and preset viewpoint conditions by using a software development kit (Software Development Kit, SDK), wherein the software development kit can be in the form of Application (APP) or web page.
And step 103, performing viewpoint rendering based on the two-dimensional image to be processed and the plurality of viewpoint images to obtain a target three-dimensional image.
In the step, the viewpoint rendering is performed on the dimension image to be processed and the viewpoint image by using a software development kit (Software Development Kit, SDK), wherein the software development kit can be in the form of Application (APP) or web page.
And performing viewpoint rendering training in advance by using the neural network to obtain a viewpoint rendering neural network model capable of performing viewpoint rendering according to the two-dimensional image to be processed and the plurality of viewpoint images. And then, the two-dimensional image to be processed and the plurality of viewpoint images are subjected to viewpoint rendering by utilizing the viewpoint rendering neural network model, so that the images are restored in the viewpoint rendering process, the obtained target three-dimensional image is smoother and more natural, the problem of artifacts can be avoided by avoiding the existence of holes without pixel values, the quality of the converted target three-dimensional image can be further ensured, the method is suitable for different versions and different types of display equipment, the conversion cost can be saved, and the method is suitable for the general demands of users.
Through the scheme, the two-dimensional image to be processed is obtained, a plurality of viewpoint images corresponding to the two-dimensional image to be processed are generated based on the two-dimensional image to be processed and the preset viewpoint conditions, viewpoint rendering is carried out based on the two-dimensional image to be processed and the viewpoint images, the obtained target three-dimensional image is smoother and more natural, the problem of artifacts can be avoided, and the quality of the converted high-dimensional image can be guaranteed.
In some embodiments, in step 101, the acquiring a two-dimensional image to be processed includes:
and A1, acquiring an original two-dimensional image to be processed.
And step A2, resetting the original two-dimensional image to be processed according to a preset size to obtain the two-dimensional image to be processed.
In the above-described scheme, the original two-dimensional image to be processed is acquired by an image acquisition device (e.g., camera, video camera).
The size of the original two-dimensional image to be processed is set to be the corresponding size of the preset size in an enlarging or reducing mode, and the size is preferably reduced to be one half of the width of the original two-dimensional image to be processed, so that the original two-dimensional image to be processed can be spliced into the target three-dimensional image conveniently.
In some embodiments, the preset viewpoint conditions include a number of viewpoints and a viewpoint angle, which are determined based on display parameters of a display device for displaying the target three-dimensional image.
In step 102, the generating, based on the two-dimensional image to be processed and the preset viewpoint conditions, a plurality of viewpoint images corresponding to the two-dimensional image to be processed includes:
and step B1, acquiring an original two-dimensional image to be processed, and acquiring a depth image from the original two-dimensional image to be processed.
And B2, extracting a foreground image and a background image from the depth image, and determining the relative position relationship between the foreground image and the background image.
And B3, determining a plurality of target viewpoints corresponding to the foreground images according to the viewpoint number and the viewpoint angle.
Step B4, determining a plurality of first viewpoint images corresponding to the foreground images according to the plurality of target viewpoints, and determining a plurality of second viewpoint images corresponding to the background images based on the relative position relation;
and fusing the first viewpoint images and the second viewpoint images to generate viewpoint images corresponding to the two-dimensional image to be processed.
In the above-described scheme, a depth image (depth image) is also called a range image (range image), and refers to an image in which a distance (depth) from an image capturing device to each point in a scene is taken as a pixel value.
The method comprises the steps of presetting the number of viewpoints and the angles of the viewpoints according to the requirements of the display equipment on the number of the viewpoints and the angles of the viewpoints, and determining a plurality of target viewpoints corresponding to the foreground images according to the preset number of the viewpoints and the preset angles of the viewpoints.
The plurality of first viewpoint images whose view angle positions with respect to the respective target viewpoints are determined with respect to the foreground image, and since the relative positional relationship between the foreground image and the background image is unchanged, the plurality of second viewpoint images corresponding to the background image can be determined based on the relative positional relationship.
Finally, the following processes are executed for each first view image and each corresponding second view image, so that a plurality of view images are obtained:
the first view image and the second view image are fused into the same view image.
The viewpoint image determined by the depth image can calculate the coordinates of all points in the whole viewpoint image aiming at the target three-dimensional image through the distance and the coordinates of a certain point, thereby being convenient for obtaining the target three-dimensional image subsequently.
In some embodiments, in step B1, the acquiring a depth image from the original two-dimensional image to be processed includes:
and step B11, performing parallax prediction on the original two-dimensional image to be processed to obtain a parallax prediction result.
And step B12, acquiring the focal length of the image acquisition unit corresponding to the original two-dimensional image to be processed and the distance between the image acquisition units.
And step B13, performing product processing on the focal length of the image acquisition unit and the distance between the image acquisition units to obtain a product processing result.
And step B14, carrying out ratio determination processing on the parallax prediction result by using the product processing result to obtain depth information, and constructing the depth image based on the depth information.
In the above scheme, the depth estimation processing is performed on the original two-dimensional image to be processed by using a monocular depth estimation algorithm (for example, monoscopic 2), so as to obtain a depth image, and the procedure is as follows:
and performing parallax prediction training in advance by using the neural network to obtain a parallax prediction neural network model capable of performing parallax prediction according to the original two-dimensional image to be processed. And performing parallax prediction on the original two-dimensional image to be processed by using the parallax prediction neural network model to obtain a parallax prediction result, wherein the parallax prediction result is a parallax image, each position of the parallax image is a pixel difference value which is stored by taking a pixel as a unit, and the pixel difference value is a pixel difference of a same-name point pair in a column coordinate of a left view minus a column coordinate on a right view.
And reconstructing a depth image by utilizing the parallax image, when the image acquisition unit is a binocular camera, acquiring a focal length when the binocular camera shoots an original two-dimensional image to be processed and a distance between the binocular camera, multiplying the focal length and the distance to obtain a multiplication result, determining a ratio of a parallax prediction result by utilizing the multiplication result to obtain depth information, and constructing the depth image based on the depth information, wherein the method can be represented as follows:
d=bf/D, wherein D represents depth information, b represents an image acquisition unit pitch, f represents an image acquisition unit focal length, and D represents a parallax prediction result.
And the monocular depth estimation algorithm is utilized to carry out depth estimation processing on the original two-dimensional image to be processed, so that the processing cost can be saved in a mode of obtaining the depth image.
In some embodiments, step 103, performing viewpoint rendering based on the two-dimensional image to be processed and the multiple viewpoint images to obtain a target three-dimensional image, including:
and C1, repairing the two-dimensional image to be processed and the plurality of viewpoint images to obtain a plurality of repaired viewpoint images.
And C2, splicing the plurality of repair viewpoint images to obtain a target three-dimensional image.
In the scheme, the viewpoint rendering training is performed in advance by utilizing the neural network to obtain the viewpoint rendering neural network model capable of performing viewpoint rendering according to the two-dimensional image to be processed and the plurality of viewpoint images. And performing viewpoint rendering on the two-dimensional image to be processed and the plurality of viewpoint images by using the viewpoint rendering neural network model to obtain a target three-dimensional image.
And repairing the two-dimensional image to be processed and the multiple viewpoint images in the viewpoint rendering process to obtain multiple repaired viewpoint images, so that the multiple repaired viewpoint images are smoother and more natural, and the existence of holes without pixel values is avoided. Then, a plurality of repair viewpoint images are spliced, so that the problem that artifacts appear in the obtained target three-dimensional image can be avoided, and the quality of the converted high-dimensional image can be ensured.
In some embodiments, step C1 comprises:
and C11, performing feature extraction on the two-dimensional image to be processed to obtain first features, and performing feature extraction on the viewpoint images to obtain second features.
And step C12, carrying out feature fusion on the first feature and the plurality of second features to obtain a plurality of feature fusion results.
And step C13, performing feature offset processing on the feature fusion results to obtain a plurality of feature offset results.
And step C14, carrying out cavity restoration on the characteristic offset results to obtain a plurality of restoration viewpoint images.
In the scheme, the viewpoint rendering training is performed in advance by utilizing the neural network to obtain the viewpoint rendering neural network model capable of performing viewpoint rendering according to the two-dimensional image to be processed and the plurality of viewpoint images. And then, performing viewpoint rendering on the two-dimensional image to be processed and the plurality of viewpoint images by using the viewpoint rendering neural network model, so that the images are repaired in the viewpoint rendering process, the obtained target three-dimensional image is smoother and more natural, and the existence of a hole without a pixel value is avoided.
The process of performing viewpoint rendering on the two-dimensional image to be processed and the plurality of viewpoint images by the viewpoint rendering neural network model is as follows:
and respectively carrying out preliminary extraction on the two-dimensional image to be processed and the plurality of viewpoint images to obtain a first feature corresponding to the two-dimensional image to be processed and a plurality of second features corresponding to the plurality of viewpoint images. And then fusing the first feature and the plurality of second features together to obtain a plurality of fused feature results, and neutralizing and utilizing the plurality of image features to realize the advantage complementation of the plurality of features by fusing the first feature and the plurality of second features, so that the fused feature results can neutralize and utilize the plurality of image features to realize the advantage complementation of the plurality of features, and the robustness and the accuracy of the obtained fused feature results are improved.
And performing feature migration processing on the multiple feature fusion results to obtain multiple feature migration results, wherein the feature migration results represent images obtained through the feature migration processing, and in order to avoid the problem that the feature migration results have holes without pixel values, the quality of the target three-dimensional image is affected, and the holes are required to be repaired on the multiple feature migration results.
The process of repairing the cavity of the characteristic offset result is as follows:
the hole area in the characteristic offset result needs to be detected, and the hole area detection can be performed in different manners, such as threshold segmentation, edge detection or a method based on texture characteristics to determine the hole area needing to be repaired.
After the hole area to be repaired is determined, the value of each pixel in the hole area needs to be estimated, the estimation of the pixel value can be performed based on the context information of the characteristic offset result, and the value of the missing pixel can be estimated by using methods such as average value, median filtering or nearest neighbor pixel value of the adjacent pixels.
After the estimated value of each pixel in the hole area is determined, the estimated values of the pixels are applied to the characteristic offset result to fill the holes, so that the hole repairing process is completed, and the repairing viewpoint image is obtained.
In some embodiments, step C2 comprises:
and step C21, splicing the plurality of repair viewpoint images in the preset dimension direction of the three-dimensional space to obtain a spliced image.
And step C22, performing viewpoint interweaving treatment on the spliced images to obtain target dimension images.
In the above scheme, in the process of stitching the multiple repair viewpoint images, in order to ensure consistency and consistency of stitched images, pixel unbalance is avoided.
The correspondence between multiple repair viewpoint images can be obtained by stitching based on the feature points, by finding the same feature points in different repair viewpoint images, and by aligning the feature points. Based on the corresponding relations, parameters such as brightness, color, contrast and the like among different restoration viewpoint images are uniformly adjusted by utilizing algorithms such as a histogram equalization algorithm according to brightness calculation and color restoration calculation, so that the finally generated spliced image is more coordinated and coherent.
In addition, in the process of aligning the feature points, in order to ensure the accuracy of the feature points, some matching algorithms, such as Scale-invariant feature transform (Scale-invariant feature transform, SIFT) algorithm, accelerated robust feature algorithm (Speeded Up Robust Features, SURF), rapid feature point extraction (Oriented FAST and Rotated BRIEF, ORB) algorithm, and the like, may be used.
The overall effect of the stitching, such as smoothness and definition, can be maximized under the condition that the corresponding relation of the feature points is ensured. When the expected effect cannot be achieved by the local feature point matching or linear splicing method, global optimization is performed by the method. The way here based on global optimization may be an image segmentation algorithm.
The plurality of repair viewpoint images are stitched in a preset dimension direction of the three-dimensional space, where the preset dimension direction may be set according to a specific case, and here, a wide dimension, i.e., a y direction is preferable.
In order to ensure that the spliced image can present the effect of the target three-dimensional image, viewpoint interweaving processing is required to be carried out on the spliced image, so that the effect of the target three-dimensional image can be presented on the display device.
Based on the same inventive concept, an application scenario corresponding to the image conversion method of the above embodiment is specifically described, as shown in fig. 2A, specifically as follows:
firstly, an original image (namely an original two-dimensional image to be processed) is selected on a terminal device, an incoming software development kit (Software Development Kit, SDK) is used for processing the incoming original image, the incoming original image is scaled to a certain size, such as half of the width of the original image, the two-dimensional image to be processed is obtained, a depth map (namely a depth image) is generated according to the original image, a viewpoint map (namely a viewpoint image) is generated according to the depth map, and finally the viewpoint map is scaled to the same size as the two-dimensional image to be processed.
And finally, combining the two-dimensional image to be processed and the viewpoint image to a 3D picture (namely, a target three-dimensional image), and then displaying the 3D picture on a terminal device supporting 3D display.
As shown in fig. 2B, the process of generating a depth map from an original image is: a depth image is generated based on the original 2D image (i.e. the original two-dimensional image to be processed) using a depth estimation algorithm.
Based on the depth images, depth images (i.e., viewpoint images) corresponding to a plurality of viewpoints, for example, viewpoint 1 depth image, viewpoint 2 depth image, and viewpoint 3 depth image, are calculated.
And then performing viewpoint rendering by using a viewpoint rendering algorithm according to the original 2D image and the viewpoint images (viewpoint 1 depth image, viewpoint 2 depth image and viewpoint 3 depth image) to generate rendered viewpoint images (namely repair viewpoint image) (rendering viewpoint 1, rendering viewpoint 2 and rendering viewpoint 3).
Finally, splicing the rendering viewpoint images, and presenting a target dimension image on the 3D display device, wherein the splicing process is as follows:
taking two viewpoints as an example, firstly performing viewpoint synthesis, splicing a rendering viewpoint 1 (left viewpoint) and a rendering viewpoint 2 (right viewpoint) in a wide dimension to obtain a synthesized 3D two-viewpoint image (i.e. spliced image), and performing viewpoint interleaving on the synthesized 3D two-viewpoint image, so that a target three-dimensional image is presented on a 3D display device.
It should be noted that, the method of the embodiments of the present application may be performed by a single device, for example, a computer or a server. The method of the embodiment can also be applied to a distributed scene, and is completed by mutually matching a plurality of devices. In the case of such a distributed scenario, one of the devices may perform only one or more steps of the methods of embodiments of the present application, and the devices may interact with each other to complete the methods.
It should be noted that some embodiments of the present application are described above. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims may be performed in a different order than in the embodiments described above and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
Based on the same inventive concept, the application also provides an image conversion device corresponding to the method of any embodiment.
Referring to fig. 3, the image conversion apparatus includes:
an acquisition module 301 configured to acquire a two-dimensional image to be processed;
a viewpoint image generating module 302 configured to generate a plurality of viewpoint images corresponding to the two-dimensional image to be processed based on the two-dimensional image to be processed and a preset viewpoint condition;
the viewpoint rendering module 303 is configured to perform viewpoint rendering based on the two-dimensional image to be processed and the plurality of viewpoint images, so as to obtain a target three-dimensional image.
In some embodiments, the acquisition module 301 is specifically configured to:
acquiring an original two-dimensional image to be processed;
and resetting the original two-dimensional image to be processed according to a preset size to obtain the two-dimensional image to be processed.
In some embodiments, the preset viewpoint conditions include a number of viewpoints and a viewpoint angle, the number of viewpoints and the viewpoint angle being determined based on display parameters of a display device for displaying the target three-dimensional image;
the viewpoint image generation module 302 includes:
a depth image acquisition unit configured to acquire an original two-dimensional image to be processed, and acquire a depth image from the original two-dimensional image to be processed;
a relative position determining unit configured to extract a foreground image and a background image from the depth image, and determine a relative positional relationship between the foreground image and the background image;
a viewpoint determining unit configured to determine a plurality of target viewpoints corresponding to the foreground image in accordance with the number of viewpoints and the viewpoint angle;
a viewpoint image determining unit configured to determine a plurality of first viewpoint images corresponding to the foreground image from the plurality of target viewpoints, and a plurality of second viewpoint images corresponding to the background image based on the relative positional relationship;
and a fusion unit configured to fuse the plurality of first viewpoint images and the plurality of second viewpoint images, and generate a plurality of viewpoint images corresponding to the two-dimensional image to be processed.
In some embodiments, the depth image acquisition unit is specifically configured to:
performing parallax prediction on the original two-dimensional image to be processed to obtain a parallax prediction result;
acquiring an image acquisition unit focal length and an image acquisition unit interval corresponding to the original two-dimensional image to be processed;
performing product processing on the focal length of the image acquisition unit and the distance between the image acquisition units to obtain a product processing result;
and carrying out ratio determination processing on the parallax prediction result by using the product processing result to obtain depth information, and constructing the depth image based on the depth information.
In some embodiments, the viewpoint rendering module 303 includes:
the restoration unit is configured to perform restoration processing on the two-dimensional image to be processed and the plurality of viewpoint images to obtain a plurality of restoration viewpoint images;
and the splicing unit is configured to splice the plurality of repair viewpoint images to obtain a target three-dimensional image.
In some embodiments, the repair unit is specifically configured to:
extracting features of the two-dimensional image to be processed to obtain first features, and extracting features of the viewpoint images to obtain second features;
performing feature fusion on the first feature and the plurality of second features to obtain a plurality of feature fusion results;
performing feature offset processing on the feature fusion results to obtain a plurality of feature offset results;
and carrying out cavity restoration on the characteristic offset results to obtain a plurality of restoration viewpoint images.
In some embodiments, the stitching unit is specifically configured to:
splicing the plurality of repair viewpoint images in a preset dimension direction of the three-dimensional space to obtain a spliced image;
and performing viewpoint interweaving treatment on the spliced images to obtain target dimension images.
For convenience of description, the above devices are described as being functionally divided into various modules, respectively. Of course, the functions of each module may be implemented in the same piece or pieces of software and/or hardware when implementing the present application.
The device of the foregoing embodiment is configured to implement the corresponding image conversion method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which is not described herein.
Based on the same inventive concept, the application also provides an electronic device corresponding to the method of any embodiment, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the processor implements the image conversion method of any embodiment when executing the program.
Fig. 4 shows a more specific hardware architecture of an electronic device according to this embodiment, where the device may include: a processor 401, a memory 402, an input/output interface 403, a communication interface 404, and a bus 405. Wherein the processor 401, the memory 402, the input/output interface 403 and the communication interface 404 are in communication connection with each other inside the device via a bus 405.
The processor 401 may be implemented by a general purpose CPU (Central Processing Unit ), a microprocessor, an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, etc. for executing relevant programs to implement the technical solutions provided in the embodiments of the present disclosure.
The Memory 402 may be implemented in the form of ROM (Read Only Memory), RAM (RandomAccess Memory ), static storage device, dynamic storage device, or the like. Memory 402 may store an operating system and other application programs, and when implementing the solutions provided by the embodiments of the present specification by software or firmware, the relevant program code is stored in memory 402 and invoked for execution by processor 401.
The input/output interface 403 is used to connect with an input/output module to realize information input and output. The input/output module may be configured as a component in a device (not shown) or may be external to the device to provide corresponding functionality. Wherein the input devices may include a keyboard, mouse, touch screen, microphone, various types of sensors, etc., and the output devices may include a display, speaker, vibrator, indicator lights, etc.
The communication interface 404 is used to connect a communication module (not shown in the figure) to enable communication interaction between the present device and other devices. The communication module may implement communication through a wired manner (such as USB, network cable, etc.), or may implement communication through a wireless manner (such as mobile network, WIFI, bluetooth, etc.).
Bus 405 includes a path to transfer information between components of the device (e.g., processor 401, memory 402, input/output interface 403, and communication interface 404).
It should be noted that, although the above device only shows the processor 401, the memory 402, the input/output interface 403, the communication interface 404, and the bus 405, in the implementation, the device may further include other components necessary for realizing normal operation. Furthermore, it will be understood by those skilled in the art that the above-described apparatus may include only the components necessary to implement the embodiments of the present description, and not all the components shown in the drawings.
The electronic device of the foregoing embodiment is configured to implement the corresponding image conversion method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiment, which is not described herein.
Based on the same inventive concept, corresponding to any of the above embodiments of the method, the present application further provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the image conversion method according to any of the above embodiments.
The computer readable media of the present embodiments, including both permanent and non-permanent, removable and non-removable media, may be used to implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device.
The storage medium of the above embodiment stores computer instructions for causing the computer to perform the image conversion method according to any one of the above embodiments, and has the advantages of the corresponding method embodiments, which are not described herein.
Those of ordinary skill in the art will appreciate that: the discussion of any of the embodiments above is merely exemplary and is not intended to suggest that the scope of the application (including the claims) is limited to these examples; the technical features of the above embodiments or in the different embodiments may also be combined within the idea of the present application, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of brevity.
Additionally, well-known power/ground connections to Integrated Circuit (IC) chips and other components may or may not be shown within the provided figures, in order to simplify the illustration and discussion, and so as not to obscure the embodiments of the present application. Furthermore, the devices may be shown in block diagram form in order to avoid obscuring the embodiments of the present application, and this also takes into account the fact that specifics with respect to implementation of such block diagram devices are highly dependent upon the platform on which the embodiments of the present application are to be implemented (i.e., such specifics should be well within purview of one skilled in the art). Where specific details (e.g., circuits) are set forth in order to describe example embodiments of the application, it should be apparent to one skilled in the art that embodiments of the application can be practiced without, or with variation of, these specific details. Accordingly, the description is to be regarded as illustrative in nature and not as restrictive.
While the present application has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of those embodiments will be apparent to those skilled in the art in light of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.
The present embodiments are intended to embrace all such alternatives, modifications and variances which fall within the broad scope of the appended claims. Accordingly, any omissions, modifications, equivalents, improvements and/or the like which are within the spirit and principles of the embodiments are intended to be included within the scope of the present application.
Claims (11)
1. An image conversion method, comprising:
acquiring a two-dimensional image to be processed;
generating a plurality of viewpoint images corresponding to the two-dimensional image to be processed based on the two-dimensional image to be processed and preset viewpoint conditions;
and performing viewpoint rendering based on the two-dimensional image to be processed and the plurality of viewpoint images to obtain a target three-dimensional image.
2. The method of claim 1, wherein the acquiring a two-dimensional image to be processed comprises:
acquiring an original two-dimensional image to be processed;
and resetting the original two-dimensional image to be processed according to a preset size to obtain the two-dimensional image to be processed.
3. The method according to claim 1, wherein the preset viewpoint conditions include a number of viewpoints and a viewpoint angle, the number of viewpoints and the viewpoint angle being determined based on display parameters of a display device for displaying the target three-dimensional image;
the generating a plurality of viewpoint images corresponding to the two-dimensional image to be processed based on the two-dimensional image to be processed and a preset viewpoint condition includes:
acquiring an original two-dimensional image to be processed, and acquiring a depth image from the original two-dimensional image to be processed;
extracting a foreground image and a background image from the depth image, and determining the relative position relationship between the foreground image and the background image;
determining a plurality of target viewpoints corresponding to the foreground images according to the viewpoint number and the viewpoint angle;
determining a plurality of first viewpoint images corresponding to the foreground images according to the plurality of target viewpoints, and determining a plurality of second viewpoint images corresponding to the background images based on the relative position relationship;
and fusing the first viewpoint images and the second viewpoint images to generate viewpoint images corresponding to the two-dimensional image to be processed.
4. A method according to claim 3, wherein said obtaining a depth image from said original two-dimensional image to be processed comprises:
performing parallax prediction on the original two-dimensional image to be processed to obtain a parallax prediction result;
acquiring an image acquisition unit focal length and an image acquisition unit interval corresponding to the original two-dimensional image to be processed;
performing product processing on the focal length of the image acquisition unit and the distance between the image acquisition units to obtain a product processing result;
and carrying out ratio determination processing on the parallax prediction result by using the product processing result to obtain depth information, and constructing the depth image based on the depth information.
5. The method according to claim 1, wherein performing viewpoint rendering based on the two-dimensional image to be processed and the plurality of viewpoint images to obtain a target three-dimensional image includes:
repairing the two-dimensional image to be processed and the multiple viewpoint images to obtain multiple repaired viewpoint images;
and splicing the plurality of repair viewpoint images to obtain a target three-dimensional image.
6. The method according to claim 5, wherein performing restoration processing on the two-dimensional image to be processed and the plurality of viewpoint images to obtain a plurality of restored viewpoint images includes:
extracting features of the two-dimensional image to be processed to obtain first features, and extracting features of the viewpoint images to obtain second features;
performing feature fusion on the first feature and the plurality of second features to obtain a plurality of feature fusion results;
performing feature offset processing on the feature fusion results to obtain a plurality of feature offset results;
and carrying out cavity restoration on the characteristic offset results to obtain a plurality of restoration viewpoint images.
7. The method of claim 1, wherein stitching the plurality of repair viewpoint images to obtain a target three-dimensional image comprises:
splicing the plurality of repair viewpoint images in a preset dimension direction of the three-dimensional space to obtain a spliced image;
and performing viewpoint interweaving treatment on the spliced images to obtain target dimension images.
8. An image conversion apparatus, comprising:
the acquisition module is configured to acquire a two-dimensional image to be processed;
the viewpoint image generation module is configured to generate a plurality of viewpoint images corresponding to the two-dimensional image to be processed based on the two-dimensional image to be processed and preset viewpoint conditions;
and the viewpoint rendering module is configured to perform viewpoint rendering based on the two-dimensional image to be processed and the plurality of viewpoint images to obtain a target three-dimensional image.
9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the method of any one of claims 1 to 7 when the program is executed by the processor.
10. A non-transitory computer readable storage medium storing computer instructions for causing a computer to perform the method of any one of claims 1 to 7.
11. A computer program product comprising computer program instructions which, when run on a computer, cause the computer to perform the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311415758.7A CN117455754A (en) | 2023-10-27 | 2023-10-27 | Image conversion method and related equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311415758.7A CN117455754A (en) | 2023-10-27 | 2023-10-27 | Image conversion method and related equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117455754A true CN117455754A (en) | 2024-01-26 |
Family
ID=89579434
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311415758.7A Pending CN117455754A (en) | 2023-10-27 | 2023-10-27 | Image conversion method and related equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117455754A (en) |
-
2023
- 2023-10-27 CN CN202311415758.7A patent/CN117455754A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110264509B (en) | Method, apparatus, and storage medium for determining pose of image capturing device | |
| US9699380B2 (en) | Fusion of panoramic background images using color and depth data | |
| US20180014003A1 (en) | Measuring Accuracy of Image Based Depth Sensing Systems | |
| US10547822B2 (en) | Image processing apparatus and method to generate high-definition viewpoint interpolation image | |
| CN107025660B (en) | Method and device for determining image parallax of binocular dynamic vision sensor | |
| US10580143B2 (en) | High-fidelity 3D reconstruction using facial features lookup and skeletal poses in voxel models | |
| US20130258064A1 (en) | Apparatus and method for reconstructing high density three-dimensional image | |
| US10957021B2 (en) | Method for rendering a final image from initial images acquired by a camera array, corresponding device, computer program product and computer-readable carrier medium | |
| WO2012135220A2 (en) | Real-time depth extraction using stereo correspondence | |
| CN105578023A (en) | Image quick photographing method and device | |
| US20170019615A1 (en) | Image processing method, non-transitory computer-readable storage medium and electrical device thereof | |
| Yang et al. | Dynamic 3D scene depth reconstruction via optical flow field rectification | |
| US10650488B2 (en) | Apparatus, method, and computer program code for producing composite image | |
| US10298914B2 (en) | Light field perception enhancement for integral display applications | |
| CN115861515A (en) | Three-dimensional face reconstruction method, computer program product and electronic device | |
| CN115131507B (en) | Image processing method, image processing device and meta space three-dimensional reconstruction method | |
| CN112634439B (en) | 3D information display method and device | |
| CN117455754A (en) | Image conversion method and related equipment | |
| CN112652056B (en) | 3D information display method and device | |
| US9077963B2 (en) | Systems and methods for generating a depth map and converting two-dimensional data to stereoscopic data | |
| US10325378B2 (en) | Image processing apparatus, image processing method, and non-transitory storage medium | |
| CN107633498B (en) | Image dark state enhancement method and device and electronic equipment | |
| CN112615993A (en) | Depth information acquisition method, binocular camera module, storage medium and electronic equipment | |
| Yu et al. | Racking focus and tracking focus on live video streams: a stereo solution | |
| Lee et al. | Real-time virtual-viewpoint image synthesis algorithm using Kinect camera |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |