CN111787303B - Three-dimensional image generation method and device, storage medium and computer equipment - Google Patents
Three-dimensional image generation method and device, storage medium and computer equipment Download PDFInfo
- Publication number
- CN111787303B CN111787303B CN202010479011.8A CN202010479011A CN111787303B CN 111787303 B CN111787303 B CN 111787303B CN 202010479011 A CN202010479011 A CN 202010479011A CN 111787303 B CN111787303 B CN 111787303B
- Authority
- CN
- China
- Prior art keywords
- target
- imaging
- camera
- tof camera
- dimensional image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/271—Image signal generators wherein the generated image signals comprise depth maps or disparity maps
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
Abstract
The application discloses a method, a device, a storage medium and computer equipment for generating a three-dimensional image, wherein the method comprises the following steps: acquiring overlapped imaging of the TOF camera and the visible light camera in a shooting range; performing target identification in overlay imaging; the method comprises the steps that distance measurement is conducted on each identified target through a TOF camera to obtain depth information of each target, wherein the depth information is the distance information from the target to the TOF camera; and generating the three-dimensional image of the overlapped imaging according to the depth information, wherein the three-dimensional image comprises all the targets in the overlapped imaging, and because the overlapped imaging area in which the TOF camera and the visible light camera are overlapped in the imaging range is obtained, the visible light camera is used for imaging and the TOF camera is used for obtaining the depth information, the three-dimensional image with the contour color consistent with that of the original object of the target is constructed, the image is vivid and the calculated amount is less.
Description
Technical Field
The present application relates to the field of camera technologies, and in particular, to a method and an apparatus for generating a three-dimensional image, a storage medium, and a computer device.
Background
In the prior art, a camera can only shoot an image of a two-dimensional plane generally, but with the development of science and technology, people no longer meet the shooting mode, and pursue shooting of a more vivid three-dimensional image is pursued.
Disclosure of Invention
The application mainly aims to provide a three-dimensional image generation method, a storage medium and computer equipment, and aims to solve the technical problem that in the prior art, three-dimensional images generated by a TOF camera are inconvenient to watch.
Based on the above purpose of the invention, an embodiment of the present application provides a method for generating a three-dimensional image, where the method is applied to an intelligent device, the intelligent device includes a TOF camera and a visible light camera, and the method for generating a three-dimensional image includes:
acquiring overlapped imaging of the TOF camera and the visible light camera in a camera shooting range;
performing target identification in the overlay imaging;
the TOF camera is used for ranging each identified target to obtain depth information of each target, wherein the depth information is the distance information from the target to the TOF camera;
generating a three-dimensional image of the overlay imaging from the depth information, the three-dimensional image including all of the targets in the overlay imaging.
Further, the step of acquiring overlapped imaging in which the TOF camera and the visible light camera overlap in a shooting range further includes:
acquiring visible imaging in the camera shooting range of the visible light camera;
and acquiring the overlapped imaging in the visible imaging according to the maximum object distance detected by the TOF camera and the distance between the TOF camera and the optical center of the lens of the visible light camera.
Further, the step of acquiring the overlay imaging in the visible imaging according to the maximum object distance detectable by the TOF camera and the distance between the TOF camera and the optical center of the lens of the visible light camera includes:
obtaining a TOF camera shooting range of the TOF camera according to the focal length of the TOF camera and the maximum object distance detectable by the TOF camera;
obtaining a shooting area of the visible light camera in the TOF shooting range according to the distance between the TOF camera and the optical center of the lens of the visible light camera and the focal length of the visible light camera;
and calculating the area occupied by the camera shooting area when the visible light camera is used for imaging to obtain the overlapped imaging.
Further, the step of performing object recognition in the overlay imaging includes:
and identifying the image corresponding to the overlapped imaging through a preset identification model to obtain each target in the overlapped imaging.
Further, the step of obtaining depth information of each target by ranging each identified target through the TOF camera includes:
labeling each target in the overlapped imaging to obtain contour information of each target, wherein the contour information comprises pixel information of the contour of the target;
and ranging the target according to the pixel information to obtain the distance information from the physical point corresponding to each pixel point in the contour of the target to the TOF camera.
Further, the step of generating a three-dimensional image corresponding to each of the targets according to the depth information includes:
calculating target data of each target according to the depth information, wherein the target data comprises the actual distance between the targets and the outline information of each target;
and constructing a three-dimensional electronic map corresponding to each target in a preset actual coordinate system according to the distance of each target and the outline of each target to obtain the three-dimensional image.
Further, the step of generating the overlay-imaged three-dimensional image from the depth information may be followed by:
acquiring three-dimensional information of each target in the three-dimensional image, wherein the three-dimensional information comprises distance information between the targets;
and displaying the three-dimensional image and the three-dimensional information through a display screen.
The embodiment of the present application further provides a device for generating a three-dimensional image, including:
the acquisition imaging unit is used for acquiring overlapped imaging of the TOF camera and the visible light camera in an imaging range;
an identifying target unit for performing target identification in the overlay imaging;
the target ranging unit is used for ranging each identified target through the TOF camera to obtain depth information of each target, wherein the depth information is the distance information from the target to the TOF camera;
an image generation unit for generating a three-dimensional image of the overlay imaging in dependence on the depth information, the three-dimensional image comprising all the objects in the overlay imaging.
The embodiment of the present application also provides a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed, the method for generating a three-dimensional image as described above is implemented.
The embodiment of the present application further provides a computer device, which includes a processor, a memory, and a computer program stored on the memory and capable of running on the processor, and when executed, the computer program implements the method for generating a three-dimensional image as described above.
The beneficial effect of this application:
the application provides a three-dimensional image generation method, a storage medium and computer equipment, wherein in the three-dimensional image generation method, an overlapped imaging area formed by overlapping a TOF camera and a visible light camera in a camera shooting range is obtained, then a target in the area is identified, and distance measurement is carried out on the target to obtain depth information so as to construct a corresponding three-dimensional image.
Drawings
Fig. 1 is a schematic flow chart of a three-dimensional image generation method according to an embodiment of the present application;
FIG. 2 is a planar image of a coordinate system established with the optical center of the lens of the TOF camera as the origin according to an embodiment of the present disclosure;
fig. 3 is a block diagram schematically illustrating a structure of a three-dimensional image generation apparatus according to an embodiment of the present application;
FIG. 4 is a block diagram illustrating the structure of one embodiment of a storage medium of the present application;
FIG. 5 is a block diagram illustrating the structure of one embodiment of a computer device of the present application.
The implementation, functional features and advantages of the objectives of the present application will be further explained with reference to the accompanying drawings.
Detailed Description
The technical solutions in the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In addition, descriptions in this application as to "first", "second", etc. are for descriptive purposes only and are not to be construed as indicating or implying relative importance or implicit to the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present application.
Referring to fig. 1, a flow diagram of a three-dimensional image generation method provided in the present application may be executed by a three-dimensional image generation apparatus, where the three-dimensional image generation apparatus may be specifically implemented in a form of software or hardware, and an embodiment of the present application provides a three-dimensional image generation method applied in an intelligent device, where the intelligent device includes a TOF camera and a visible light camera, and specifically, the method includes:
step S1: acquiring overlapped imaging of the TOF camera and the visible light camera in a camera shooting range;
step S2: performing target identification in the overlay imaging;
step S3: the TOF camera is used for ranging each identified target to obtain depth information of each target, wherein the depth information is the distance information from the target to the TOF camera;
step S4: generating a three-dimensional image of the overlay imaging from the depth information, the three-dimensional image including all of the targets in the overlay imaging.
In this embodiment, the TOF camera is manufactured based on the existing TOF technology, which is an abbreviation of Time of Flight (Time of Flight) technology, that is, the sensor emits modulated near-infrared light and reflects the modulated near-infrared light after encountering an object, and the sensor converts the distance between the camera and the shot object by calculating the Time difference or the phase difference between the emitted light and the reflected light, so as to obtain depth information.
As described in step S1, the above-mentioned smart device includes multiple modes, such as a mode of taking images by a TOF camera or a visible light camera alone, in which a three-dimensional contour image and a normal two-dimensional image of an object are obtained, respectively, and a mode of generating a three-dimensional image of a scene by the TOF camera or the visible light camera, when the smart device enters the mode, the 3D modeling mode is turned on, first, overlapping images of the TOF camera and the visible light camera overlapping in a camera range are obtained, it is required that the cameras take images of the object, each of the cameras has a corresponding imaging plane, the scene in the imaging plane is determined according to an optical center and a focal length of the lens and a size of the imaging plane, the TOF camera has a farthest distance measurement, which is denoted as L for convenience of description, the camera range of the TOF camera is within the farthest distance measurement, and the distance of the object is not limited by the visible light camera, therefore, the overlapped imaging is the imaging of the overlapped area and is a part of the visible imaging of the visible light camera, so that the appearance of the shot object can be displayed in the overlapped imaging, and the depth information of the object in the process can be acquired by the TOF camera.
As described in the above step S2, the object recognition is performed on the overlapped image, for example, the scene is recognized by the model or the scene in the overlapped image is determined by comparing the pictures in the preset database, wherein the scene is the recognized object. In one embodiment, the step S2 includes the steps of identifying the target in the overlay imaging according to a preset identification model:
step S21: and identifying the image corresponding to the overlapped imaging through a preset identification model to obtain each target in the overlapped imaging.
In this embodiment, images corresponding to overlapping imaging are identified through a preset identification model, for example, an AI intelligent identification algorithm is used to perform object identification processing on pixels of the overlapping imaging, and identify multiple targets in an overlapping imaging area, for example, targets such as a table, a chair, a box, and the like, where the preset identification model may be implemented through an SSD algorithm or a DSST algorithm, and for example, the identification model is an SSD 16 network structure, the network structure includes a convolutional layer, a fully-connected layer, and a pooling layer, and first, target features are extracted through a CNN network, and then, classifications of the targets are obtained through calculation through a VGG16 network, and the classifications include categories of devices commonly used in families such as a table, a sofa, a chair, a refrigerator, and the like. When the identification model is trained, firstly, samples are collected to obtain a data set, the samples comprise samples of multiple categories such as the table, the sofa, the chair and the refrigerator, then, a preset initial model is input for training, then, the performance of the SSD algorithm is improved through methods such as loss compensation and augmentation, for example, loss values are calculated through a loss function, parameter gradients are calculated through network back propagation, and then, parameters of the model are updated until the model converges, so that the identification model is obtained.
As described in step S3, the TOF camera measures the distance of each identified target to obtain corresponding depth information, that is, the TOF technique measures the distance of each identified target, and if the plane of each target facing the TOF camera is a relative plane when the target actually faces the TOF camera, the distance from each point on the relative plane to the TOF camera is the depth information, in an embodiment, the step S3 includes:
step S31: labeling each target in the overlapped imaging to obtain contour information of each target, wherein the contour information comprises pixel information of the contour of the target;
step S32: and ranging the target according to the pixel information to obtain the distance information from the physical point corresponding to each pixel point in the contour of the target to the TOF camera.
In this embodiment, to obtain more detailed depth information, first labeling each target in the overlay imaging, while distinguishing each target, obtaining profile information corresponding to each target according to the labeling, where the profile information includes pixel information of a target profile, and pixels of the target profile are used to represent a whole profile of the target for image display, for example, a camera pixel of a visible light camera is a fixed value, when labeling the target, the target may be framed by a rectangular frame, then the profile of each target is partitioned according to edge pixels of each target, and the target is measured according to a pixel size corresponding to each target, to obtain distance information of each pixel point in each target profile, that is, the depth information, where the distance information is distance information from each object point in the profile of the target to a TOF camera, and each point corresponds to the object pixel point one by one, each surface of the target to be known consists of points, and the real object points are all points of the surface of the target facing the TOF camera in practice; the pixels needing to be known are basic units of image display, the depth information of each target contour used for displaying is obtained by obtaining the depth information of the pixels corresponding to the targets, and a basis is further provided for subsequently establishing a three-dimensional graph of each target.
As described in step S4, after the depth information of each target is obtained, a three-dimensional image corresponding to each target is constructed through 3D modeling according to the depth information, because the depth information includes information about the distance between an actual object point corresponding to each pixel point and a TOF camera during imaging of the target, a three-dimensional shape of each target can be obtained through the information, and then a three-dimensional image of the target is constructed, and the target is a target in visible imaging of a visible light camera, the three-dimensional profile image of the target is constructed, and simultaneously, RGB primary colors actually captured by each target, that is, colors displayed by superimposing three color channels of red (R), green (G), and blue (B) in reality are obtained through imaging of the visible light camera, so that the constructed three-dimensional image is consistent with an actual target original object.
Specifically, step S4 includes:
step S41: calculating target data of each target according to the depth information, wherein the target data comprises the actual distance between the targets and the outline information of each target;
step S42: and constructing a three-dimensional electronic map corresponding to each target in a preset actual coordinate system according to the distance of each target and the outline of each target to obtain the three-dimensional image.
In this embodiment, the target data of each target is obtained through the depth information, and the target data is contour information of the target, such as length, width, height, geometric center, centroid, depth, and the like of the target, and the actual distance between the targets calculated through the contour information, wherein, the center point of each target is first calculated, if the target is in a regular shape, the geometric center is calculated as the center point through the contour of the target, if the target is in an irregular shape, the centroid of the target is used as the center point, then a preset actual coordinate system is established, in the actual coordinate system, the TOF camera is used as the coordinate origin or the ground is used as the reference surface of the XY surface, three-dimensional coordinates of a plurality of targets are established, and then the calculation is performed by using the side length of a plane triangle, or the calculation formula of the spatial coordinate distance between two points in the three-dimensional coordinates, for example, a camera may first observe a ground plane, use the ground plane as an XY reference plane of the coordinate system, and use a perpendicular line perpendicular to a junction of the XY plane as a Z axis, so that a distance between center points of two targets in a space may be obtained by a coordinate distance formula from a midpoint to a point in the space, and thus, a distance between the targets may be obtained, thereby constructing a three-dimensional electronic map of the targets, thereby obtaining the three-dimensional image.
In one embodiment, the step S1 includes:
step S11: acquiring visible imaging in the camera shooting range of the visible light camera;
step S12: and acquiring the overlapped imaging in the visible imaging according to the maximum object distance detected by the TOF camera and the distance between the TOF camera and the optical center of the lens of the visible light camera.
In the embodiment, firstly, a visible image within a camera shooting range of a visible light camera is obtained, and then an imaging area overlapped by the TOF camera and the visible light camera within the camera shooting range is found out in the visible image, specifically, the TOF camera can detect the maximum object distance and the distance between the TOF camera and the lens optical center of the visible light camera, where the TOF camera detects the maximum object distance, that is, the maximum distance measurement of the TOF camera, and as the TOF camera has the maximum distance measurement, the shooting range of the TOF camera and the corresponding TOF image can be obtained by using a small-hole imaging principle, similarly, the shooting range and the imaging surface of the visible light camera can also be obtained, in an intelligent device, the two lenses are in the same plane, so that the closer the optical centers of the two lenses are, the more the overlapped part is, the farther the optical center distance of the lens is, and the less the overlapped part is, the distance between the two lenses is not limited in the embodiment, and the distance can be set according to actual needs.
Preferably, the step S12 includes:
step S121: obtaining a TOF camera shooting range of the TOF camera according to the focal length of the TOF camera and the maximum object distance detected by the TOF camera;
step S122: obtaining a shooting area of the visible light camera in the TOF shooting range according to the distance between the TOF camera and the optical center of the lens of the visible light camera and the focal length of the visible light camera;
step S123: and calculating the area occupied by the camera shooting area when the visible light camera is used for imaging to obtain the overlapped imaging.
In this embodiment, the TOF camera range of the TOF camera is obtained by first detecting the maximum object distance through the focal length of the TOF camera and the TOF camera, because the focal length and the maximum object distance of the TOF camera are both determined, the size of the corresponding imaging plane can also be determined, the TOF camera range of the TOF camera is further determined through the pinhole imaging principle, the visible camera range of the visible light camera can also be correspondingly determined through the focal length of the visible light camera, and the optical center distance between the two lenses is fixed, so that the camera overlapping region of the two cameras, that is, the camera area of the visible light camera in the TOF camera range is obtained, and then the region occupied by the camera area in the visible light camera during imaging is calculated to obtain the above overlapping imaging, specifically, the overlapping imaging can be calculated through a preset formula, for example, the formula set through the similar triangle principle, the coordinates of the overlay imaging can also be calculated by establishing a coordinate system.
For example, referring to fig. 2, a coordinate system is established by taking the optical center of the lens of the TOF camera as an origin O, taking a straight line where a connecting line of the TOF camera and the optical center of the lens of the visible light camera is located as a Y axis, and taking a straight line which passes through the origin and is perpendicular to the connecting line as an X axis; because the TOF camera has effective ranging, the effective ranging L of the TOF camera is set, and the distance from the optical center of the lens of the visible light camera to the optical center of the lens of the TOF camera is K. According to the principle of similar triangles, the coordinate values of the overlapped imaging boundaries can be obtained, and then the pixels corresponding to the coordinates are obtained according to the size of the imaging pixels.
Take one of the planes passing through the two optical centers as an example: effective distance measurement of the TOF camera is L, and the focal length of the TOF camera is F1Focal length of the visible light camera is F2The optical center distance between the TOF camera and the visible light camera is K, the overlapping shooting range of the TOF camera and the visible light camera obtained by the imaging principle is BG, the imaging area of the overlapping shooting range in the imaging plane of the visible light camera is DE, and the upper edge coordinate of the imaging plane of the TOF camera is A (X)1，Y1) Wherein X is1Is a focal length F1，Y1The longitudinal dimension of an imaging plane known to the TOF camera is obtained according to the similar triangle principle to obtain the ordinate Y of the lower edge B of the maximum object distance of the TOF camera2Its horizontal axis coordinate X2is-L, then Y2＝Y1*L/F1(ii) a The optical center coordinates of the visible light camera are C (0, K), and the corresponding upper edge point D (X) of the overlapped imaging4，Y4) Ordinate Y thereof4Comprises the following steps: y is4＝|Y2-K|*F2L + K, abscissa X thereof4Is a focal length F2The coordinate of the lower edge point E of the imaging plane of the visible light camera is (X)5，Y5) Wherein X is5Is the focal length F of a visible light camera2，Y5The longitudinal dimension of the imaging plane is known for visible light cameras.
Thus, the upper and lower edge positions of the overlapped imaging area in one plane can be obtained, and the upper and lower edge positions of the overlapped imaging area in all planes can be obtained according to the method, namely all edge coordinates of the overlapped imaging area in the three-dimensional space can be obtained.
In one embodiment, after the step S4, the method further includes:
step S5: acquiring three-dimensional information of each target in the three-dimensional image, wherein the three-dimensional information comprises distance information between the targets;
step S6: and displaying the three-dimensional image and the three-dimensional information through a display screen.
In this embodiment, after the three-dimensional image is constructed, the three-dimensional image may be displayed through a display screen, and at the same time, specific information of each object may be displayed, so that a user may know the situation of each object more.
The application also provides a three-dimensional image generation device, which is used for executing the three-dimensional image generation method, and the three-dimensional image generation device can be specifically realized in a form of software or hardware. Referring to fig. 3, the three-dimensional image generating apparatus includes:
an acquisition imaging unit 100, configured to acquire overlapped imaging in which the TOF camera and the visible light camera overlap in a shooting range;
an object recognition unit 200 for performing object recognition in the overlay imaging;
a target ranging unit 300, configured to perform ranging on each identified target through the TOF camera to obtain depth information of each target, where the depth information is distance information from the target to the TOF camera;
an image generation unit 400 configured to generate a three-dimensional image of the overlay imaging according to the depth information, wherein the three-dimensional image includes all the targets in the overlay imaging.
In this embodiment, the TOF camera is manufactured based on the existing TOF technology, which is an abbreviation of Time of Flight (Time of Flight) technology, that is, the sensor emits modulated near-infrared light and reflects the modulated near-infrared light after encountering an object, and the sensor converts the distance between the camera and the shot object by calculating the Time difference or the phase difference between the emitted light and the reflected light, so as to obtain depth information.
As described in the above-mentioned obtaining and imaging unit 100, the above-mentioned smart device includes multiple modes, such as a mode of taking images by a TOF camera or a visible light camera alone, at which a three-dimensional profile image and a normal two-dimensional image of an object are obtained, and a mode of generating a three-dimensional image of a scene by the TOF camera or the visible light camera, respectively, when the smart device enters the mode, the 3D modeling mode is turned on, first, overlapping imaging in which the TOF camera and the visible light camera overlap in a camera range is obtained, it is required that the cameras take images of the object, each of the cameras has a corresponding imaging plane, the scene in the imaging plane is determined according to an optical center, a focal length, and a size of the imaging plane of the lens, the TOF camera has a farthest distance measurement, which is described as L for convenience of describing the farthest distance measurement, and the camera has no limitation on the distance of the object within the farthest distance measurement, therefore, the overlapped imaging is the imaging of the overlapped area and is a part of the visible imaging of the visible light camera, so that the appearance of the shot object can be displayed in the overlapped imaging, and the depth information of the object in the process can be acquired by the TOF camera.
As described in the above-mentioned object recognition unit 200, the object recognition is performed on the overlapped imaging, for example, a scene is recognized through a model or a scene in the overlapped imaging is determined by comparing pictures in a preset database, where the scene is the above-mentioned recognized object. In one embodiment, the identifying the target unit 200, when the target in the overlay imaging is identified by a preset identification model, includes:
and the model identification subunit is used for identifying the image corresponding to the overlapped imaging through a preset identification model to obtain each target in the overlapped imaging.
In this embodiment, images corresponding to overlapping imaging are identified through a preset identification model, for example, an AI intelligent identification algorithm is used to perform object identification processing on pixels of the overlapping imaging, and identify multiple targets in an overlapping imaging area, for example, targets such as a table, a chair, a box, and the like, where the preset identification model may be implemented through an SSD algorithm or a DSST algorithm, and for example, the identification model is an SSD 16 network structure, the network structure includes a convolutional layer, a fully-connected layer, and a pooling layer, and first, target features are extracted through a CNN network, and then, classifications of the targets are obtained through calculation through a VGG16 network, and the classifications include categories of devices commonly used in families such as a table, a sofa, a chair, a refrigerator, and the like. When the identification model is trained, firstly, samples are collected to obtain a data set, the samples comprise samples of multiple categories such as the table, the sofa, the chair and the refrigerator, then, a preset initial model is input for training, then, the performance of the SSD algorithm is improved through methods such as loss compensation and augmentation, for example, loss values are calculated through a loss function, parameter gradients are calculated through network back propagation, and then, parameters of the model are updated until the model converges, so that the identification model is obtained.
As described in the above object ranging unit 300, the TOF camera is used to perform ranging on each identified object to obtain corresponding depth information, that is, the TOF technology is used to perform ranging on each identified object, if in practice each object faces the TOF camera, a plane of each object facing the TOF camera is a relative plane, and a distance from each point on the relative plane to the TOF camera is the depth information, in an embodiment, the above object ranging unit 300 includes:
a labeling target subunit, configured to label each target in the overlay imaging to obtain contour information of each target, where the contour information includes pixel information of a contour of the target;
and the target ranging subunit is used for ranging the target according to the pixel information to obtain distance information from the physical point corresponding to each pixel point in the contour of the target to the TOF camera.
In this embodiment, to obtain more detailed depth information, first labeling each target in the overlay imaging, while distinguishing each target, obtaining profile information corresponding to each target according to the labeling, where the profile information includes pixel information of a target profile, and pixels of the target profile are used to represent a whole profile of the target for image display, for example, a camera pixel of a visible light camera is a fixed value, when labeling the target, the target may be framed by a rectangular frame, then the profile of each target is partitioned according to edge pixels of each target, and the target is measured according to a pixel size corresponding to each target, to obtain distance information of each pixel point in each target profile, that is, the depth information, where the distance information is distance information from each object point in the profile of the target to a TOF camera, and each point corresponds to the object pixel point one by one, each surface of the target to be known consists of points, and the real object points are all points of the surface of the target facing the TOF camera in practice; the pixels needing to be known are basic units of image display, the depth information of each target contour used for displaying is obtained by obtaining the depth information of the pixels corresponding to the targets, and a basis is further provided for subsequently establishing a three-dimensional graph of each target.
As described in the image generating unit 400, after the depth information of each target is obtained, a three-dimensional image corresponding to each target is constructed through 3D modeling according to the depth information, since the depth information includes information about the distance from an actual object point corresponding to each pixel point to a TOF camera during imaging of the target, a three-dimensional shape of each target can be obtained through the information, and then a three-dimensional image of the target is constructed, and the target is a target in visible imaging of a visible light camera, the three-dimensional profile image of the target is constructed, and simultaneously, RGB primary colors actually shot by each target, that is, colors displayed by superimposing three color channels of red (R), green (G), and blue (B) in reality are obtained through imaging of the visible light camera, so that the constructed three-dimensional image is consistent with an actual target original object.
Specifically, the image generation unit 400 includes:
a data calculating subunit, configured to calculate target data of each target according to the depth information, where the target data includes an actual distance between the targets and contour information of each target;
and the image constructing subunit is used for constructing a three-dimensional electronic map corresponding to each target in a preset actual coordinate system according to the distance of each target and the outline of each target so as to obtain the three-dimensional image.
In this embodiment, the target data of each target is obtained through the depth information, and the target data is contour information of the target, such as length, width, height, geometric center, centroid, depth, and the like of the target, and the actual distance between the targets calculated through the contour information, wherein, the center point of each target is first calculated, if the target is in a regular shape, the geometric center is calculated as the center point through the contour of the target, if the target is in an irregular shape, the centroid of the target is used as the center point, then a preset actual coordinate system is established, in the actual coordinate system, the TOF camera is used as the coordinate origin or the ground is used as the reference surface of the XY surface, three-dimensional coordinates of a plurality of targets are established, and then the calculation is performed by using the side length of a plane triangle, or the calculation formula of the spatial coordinate distance between two points in the three-dimensional coordinates, for example, a camera may first observe a ground plane, use the ground plane as an XY reference plane of the coordinate system, and use a perpendicular line perpendicular to a junction of the XY plane as a Z axis, so that a distance between center points of two targets in a space may be obtained by a coordinate distance formula from a midpoint to a point in the space, and thus, a distance between the targets may be obtained, thereby constructing a three-dimensional electronic map of the targets, thereby obtaining the three-dimensional image.
In one embodiment, the above-mentioned acquisition imaging unit 100 includes:
the acquisition imaging subunit is used for acquiring visible imaging in the camera shooting range of the visible light camera;
and the acquisition overlapping subunit is used for acquiring the overlapping imaging in the visible imaging according to the maximum object distance detected by the TOF camera and the distance between the TOF camera and the optical center of the lens of the visible light camera.
In the embodiment, firstly, a visible image within a camera shooting range of a visible light camera is obtained, and then an imaging area overlapped by the TOF camera and the visible light camera within the camera shooting range is found out in the visible image, specifically, the TOF camera can detect the maximum object distance and the distance between the TOF camera and the lens optical center of the visible light camera, where the TOF camera detects the maximum object distance, that is, the maximum distance measurement of the TOF camera, and as the TOF camera has the maximum distance measurement, the shooting range of the TOF camera and the corresponding TOF image can be obtained by using a small-hole imaging principle, similarly, the shooting range and the imaging surface of the visible light camera can also be obtained, in an intelligent device, the two lenses are in the same plane, so that the closer the optical centers of the two lenses are, the more the overlapped part is, the farther the optical center distance of the lens is, and the less the overlapped part is, the distance between the two lenses is not limited in the embodiment, and the distance can be set according to actual needs.
Preferably, the acquiring overlapping sub-unit includes:
the range obtaining module is used for obtaining a TOF camera shooting range of the TOF camera according to the focal length of the TOF camera and the maximum object distance detected by the TOF camera;
the obtaining region module is used for obtaining a shooting region of the visible light camera in the TOF shooting range according to the distance between the TOF camera and the optical center of the lens of the visible light camera and the focal length of the visible light camera;
and the calculation region module is used for calculating the region occupied by the camera shooting region in the visible imaging when the visible camera is used for imaging so as to obtain the overlapped imaging.
In this embodiment, the TOF camera range of the TOF camera is obtained by first detecting the maximum object distance through the focal length of the TOF camera and the TOF camera, because the focal length and the maximum object distance of the TOF camera are both determined, the size of the corresponding imaging plane can also be determined, the TOF camera range of the TOF camera is further determined through the pinhole imaging principle, the visible camera range of the visible light camera can also be correspondingly determined through the focal length of the visible light camera, and the optical center distance between the two lenses is fixed, so that the camera overlapping region of the two cameras, that is, the camera area of the visible light camera in the TOF camera range is obtained, and then the region occupied by the camera area in the visible light camera during imaging is calculated to obtain the above overlapping imaging, specifically, the overlapping imaging can be calculated through a preset formula, for example, the formula set through the similar triangle principle, the coordinates of the overlay imaging can also be calculated by establishing a coordinate system.
For example, referring to fig. 2, a coordinate system is established by taking the optical center of the lens of the TOF camera as an origin O, taking a straight line where a connecting line of the TOF camera and the optical center of the lens of the visible light camera is located as a Y axis, and taking a straight line which passes through the origin and is perpendicular to the connecting line as an X axis; because the TOF camera has effective ranging, the effective ranging L of the TOF camera is set, and the distance from the optical center of the lens of the visible light camera to the optical center of the lens of the TOF camera is K. According to the principle of similar triangles, the coordinate values of the overlapped imaging boundaries can be obtained, and then the pixels corresponding to the coordinates are obtained according to the size of the imaging pixels.
Take one of the planes passing through the two optical centers as an example: effective distance measurement of the TOF camera is L, and the focal length of the TOF camera is F1Focal length of the visible light camera is F2The optical center distance between the TOF camera and the visible light camera is K, the overlapping shooting range of the TOF camera and the visible light camera obtained by the imaging principle is BG, the imaging area of the overlapping shooting range in the imaging plane of the visible light camera is DE, and the upper edge coordinate of the imaging plane of the TOF camera is A (X)1，Y1) Wherein X is1Is a focal length F1，Y1The longitudinal dimension of an imaging plane known to the TOF camera is obtained according to the similar triangle principle to obtain the ordinate Y of the lower edge B of the maximum object distance of the TOF camera2Its horizontal axis coordinate X2is-L, then Y2＝Y1*L/F1(ii) a The optical center coordinates of the visible light camera are C (0, K), and the corresponding upper edge point D (X) of the overlapped imaging4，Y4) Ordinate Y thereof4Comprises the following steps: y is4＝|Y2-K|*F2L + K, abscissa X thereof4Is a focal length F2The coordinate of the lower edge point E of the imaging plane of the visible light camera is (X)5，Y5) Wherein X is5Is the focal length F of a visible light camera2，Y5The longitudinal dimension of the imaging plane is known for visible light cameras.
Thus, the upper and lower edge positions of the overlapped imaging area in one plane can be obtained, and the upper and lower edge positions of the overlapped imaging area in all planes can be obtained according to the method, namely all edge coordinates of the overlapped imaging area in the three-dimensional space can be obtained.
In one embodiment, the apparatus for generating a three-dimensional image further includes:
the distance acquiring unit is used for acquiring three-dimensional information of each target in the three-dimensional image, and the three-dimensional information comprises distance information between each target;
and the display image unit is used for displaying the three-dimensional image and the three-dimensional information through a display screen.
In this embodiment, after the three-dimensional image is constructed, the three-dimensional image may be displayed through a display screen, and at the same time, specific information of each object may be displayed, so that a user may know the situation of each object more.
Referring to fig. 4, the present application also provides a computer-readable storage medium 21, in which a computer program 22 is stored in the storage medium 21, and when the computer program runs on a computer, the computer is caused to execute the method for generating a three-dimensional image described in the above embodiment.
Referring to fig. 5, the present application also provides a computer device 34 containing instructions, the computer device includes a memory 31 and a processor 33, the memory 31 stores a computer program 22, and the processor 33 implements the three-dimensional image generation method described in the above embodiment when executing the computer program 22.
In the above embodiments, the implementation may be wholly or partially realized by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product.
The computer program product includes one or more computer instructions. When loaded and executed on a computer, cause the processes or functions described in accordance with the embodiments of the application to occur, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a network of computers, or other programmable device. The computer instructions may be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another, for example, from one website site, computer, server, or data center to another website site, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can store or a data storage device, such as a server, a data center, etc., that is integrated with one or more available media. The usable medium may be a magnetic medium (e.g., floppy Disk, hard Disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., Solid State Disk (SSD)), among others.
The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application, or which are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.
Claims (7)
1. A three-dimensional image generation method is applied to intelligent equipment and is characterized in that the intelligent equipment comprises a TOF camera and a visible light camera, and the three-dimensional image generation method comprises the following steps:
acquiring overlapped imaging of the TOF camera and the visible light camera in a camera shooting range;
performing target identification in the overlay imaging;
the TOF camera is used for ranging each identified target to obtain depth information of each target, wherein the depth information is the distance information from the target to the TOF camera; the method specifically comprises the following steps: labeling each target in the overlapped imaging to obtain contour information of each target, wherein the contour information comprises pixel information of the contour of the target; ranging the targets according to the pixel information to obtain depth information of each target, wherein the depth information is the distance information from an actual point corresponding to each pixel point in the contour of each target to the TOF camera;
generating a three-dimensional image of the overlay imaging from the depth information, the three-dimensional image including all of the targets in the overlay imaging;
the step of obtaining overlapping imaging of the TOF camera and the visible light camera overlapping in a camera shooting range further includes:
acquiring visible imaging in the camera shooting range of the visible light camera;
acquiring the overlapped imaging in the visible imaging according to the maximum object distance detectable by the TOF camera and the distance between the TOF camera and the optical center of a lens of the visible light camera;
the step of obtaining the overlay imaging in the visible imaging according to the maximum object distance detected by the TOF camera and the distance between the TOF camera and the optical center of the lens of the visible light camera includes:
obtaining a TOF camera shooting range of the TOF camera according to the focal length of the TOF camera and the maximum object distance detectable by the TOF camera;
obtaining a shooting area of the visible light camera in the TOF shooting range according to the distance between the TOF camera and the optical center of the lens of the visible light camera and the focal length of the visible light camera;
and calculating the area occupied by the camera shooting area when the visible light camera is used for imaging to obtain the overlapped imaging.
2. The method of generating a three-dimensional image according to claim 1, wherein the step of performing object recognition in the overlay imaging includes:
and identifying the image corresponding to the overlapped imaging through a preset identification model to obtain each target in the overlapped imaging.
3. The method for generating a three-dimensional image according to claim 1, wherein the step of generating a three-dimensional image corresponding to each of the objects based on the depth information includes:
calculating target data of each target according to the depth information, wherein the target data comprises the actual distance between the targets and the outline information of each target;
and constructing a three-dimensional electronic map corresponding to each target in a preset actual coordinate system according to the distance of each target and the outline of each target to obtain the three-dimensional image.
4. A method of generating a three-dimensional image according to claim 1, wherein the step of generating the overlay-imaged three-dimensional image from the depth information is followed by:
acquiring three-dimensional information of each target in the three-dimensional image, wherein the three-dimensional information comprises distance information between the targets;
and displaying the three-dimensional image and the three-dimensional information through a display screen.
5. A three-dimensional image generation device is applied to intelligent equipment and is characterized in that the intelligent equipment comprises a TOF camera and a visible light camera, and the three-dimensional image generation device comprises:
the acquisition imaging unit is used for acquiring overlapped imaging of the TOF camera and the visible light camera in an imaging range;
an identifying target unit for performing target identification in the overlay imaging;
the target ranging unit is used for ranging each identified target through the TOF camera to obtain depth information of each target, wherein the depth information is the distance information from the target to the TOF camera; the method specifically comprises the following steps: labeling each target in the overlapped imaging to obtain contour information of each target, wherein the contour information comprises pixel information of the contour of the target; ranging the targets according to the pixel information to obtain depth information of each target, wherein the depth information is the distance information from an actual point corresponding to each pixel point in the contour of each target to the TOF camera;
a generating image unit for generating a three-dimensional image of the overlay imaging according to the depth information, the three-dimensional image including all the targets in the overlay imaging;
an acquisition imaging unit comprising:
the acquisition imaging subunit is used for acquiring visible imaging in the camera shooting range of the visible light camera;
the acquisition overlapping subunit is used for acquiring the overlapping imaging in the visible imaging according to the maximum object distance detected by the TOF camera and the distance between the TOF camera and the optical center of the lens of the visible light camera;
the acquisition overlap sub-unit comprises:
the range obtaining module is used for obtaining a TOF camera shooting range of the TOF camera according to the focal length of the TOF camera and the maximum object distance detected by the TOF camera;
the obtaining region module is used for obtaining a shooting region of the visible light camera in the TOF shooting range according to the distance between the TOF camera and the optical center of the lens of the visible light camera and the focal length of the visible light camera;
and the calculation region module is used for calculating the region occupied by the camera shooting region in the visible imaging when the visible camera is used for imaging so as to obtain the overlapped imaging.
6. A storage medium, which is a computer-readable storage medium having a computer program stored thereon, the computer program, when executed, implementing a method of generating a three-dimensional image according to any one of claims 1 to 4.
7. A computer device comprising a processor, a memory, and a computer program stored on the memory and executable on the processor, the computer program when executed implementing a method of generating a three-dimensional image as claimed in any one of claims 1 to 4.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010479011.8A CN111787303B (en) | 2020-05-29 | 2020-05-29 | Three-dimensional image generation method and device, storage medium and computer equipment |
| PCT/CN2020/127202 WO2021238070A1 (en) | 2020-05-29 | 2020-11-06 | Three-dimensional image generation method and apparatus, and computer device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010479011.8A CN111787303B (en) | 2020-05-29 | 2020-05-29 | Three-dimensional image generation method and device, storage medium and computer equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111787303A CN111787303A (en) | 2020-10-16 |
| CN111787303B true CN111787303B (en) | 2022-04-15 |
Family
ID=72754493
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010479011.8A Active CN111787303B (en) | 2020-05-29 | 2020-05-29 | Three-dimensional image generation method and device, storage medium and computer equipment |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111787303B (en) |
| WO (1) | WO2021238070A1 (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111787303B (en) * | 2020-05-29 | 2022-04-15 | 深圳市沃特沃德软件技术有限公司 | Three-dimensional image generation method and device, storage medium and computer equipment |
| CN112672076A (en) * | 2020-12-11 | 2021-04-16 | 展讯半导体(成都)有限公司 | Image display method and electronic equipment |
| CN117253195B (en) * | 2023-11-13 | 2024-02-27 | 广东申立信息工程股份有限公司 | IPC safety monitoring method, monitoring system, computer equipment and readable storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104599314A (en) * | 2014-06-12 | 2015-05-06 | 深圳奥比中光科技有限公司 | Three-dimensional model reconstruction method and system |
| CN108681726A (en) * | 2018-06-26 | 2018-10-19 | 深圳阜时科技有限公司 | 3D chip modules, identity recognition device and electronic equipment |
| CN109814127A (en) * | 2017-11-22 | 2019-05-28 | 浙江舜宇智能光学技术有限公司 | High-resolution TOF imaging system and method |
| CN110827408A (en) * | 2019-10-31 | 2020-02-21 | 上海师范大学 | Real-time three-dimensional reconstruction method based on depth sensor |
| CN111033571A (en) * | 2017-08-22 | 2020-04-17 | 索尼公司 | Image processing apparatus and image processing method |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10021284B2 (en) * | 2015-08-27 | 2018-07-10 | Samsung Electronics Co., Ltd. | Epipolar plane single-pulse indirect TOF imaging for automotives |
| US10306254B2 (en) * | 2017-01-17 | 2019-05-28 | Seiko Epson Corporation | Encoding free view point data in movie data container |
| CN111787303B (en) * | 2020-05-29 | 2022-04-15 | 深圳市沃特沃德软件技术有限公司 | Three-dimensional image generation method and device, storage medium and computer equipment |
-
2020
- 2020-05-29 CN CN202010479011.8A patent/CN111787303B/en active Active
- 2020-11-06 WO PCT/CN2020/127202 patent/WO2021238070A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104599314A (en) * | 2014-06-12 | 2015-05-06 | 深圳奥比中光科技有限公司 | Three-dimensional model reconstruction method and system |
| CN111033571A (en) * | 2017-08-22 | 2020-04-17 | 索尼公司 | Image processing apparatus and image processing method |
| CN109814127A (en) * | 2017-11-22 | 2019-05-28 | 浙江舜宇智能光学技术有限公司 | High-resolution TOF imaging system and method |
| CN108681726A (en) * | 2018-06-26 | 2018-10-19 | 深圳阜时科技有限公司 | 3D chip modules, identity recognition device and electronic equipment |
| CN110827408A (en) * | 2019-10-31 | 2020-02-21 | 上海师范大学 | Real-time three-dimensional reconstruction method based on depth sensor |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111787303A (en) | 2020-10-16 |
| WO2021238070A1 (en) | 2021-12-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110568447B (en) | Visual positioning method, device and computer readable medium | |
| CN111787303B (en) | Three-dimensional image generation method and device, storage medium and computer equipment | |
| US10977818B2 (en) | Machine learning based model localization system | |
| US9129435B2 (en) | Method for creating 3-D models by stitching multiple partial 3-D models | |
| KR101758058B1 (en) | Apparatus and method for estimating camera motion using depth information, augmented reality system | |
| US11847796B2 (en) | Calibrating cameras using human skeleton | |
| CN104715479A (en) | Scene reproduction detection method based on augmented virtuality | |
| CN110798677B (en) | Three-dimensional scene modeling method and device, electronic device, readable storage medium and computer equipment | |
| WO2013056188A1 (en) | Generating free viewpoint video using stereo imaging | |
| CN109215085B (en) | Article statistical method using computer vision and image recognition | |
| WO2020063987A1 (en) | Three-dimensional scanning method and apparatus and storage medium and processor | |
| CN109934873B (en) | Method, device and equipment for acquiring marked image | |
| da Silveira et al. | Dense 3D scene reconstruction from multiple spherical images for 3-DoF+ VR applications | |
| TWI599987B (en) | System and method for combining point clouds | |
| EP2946360B1 (en) | Foot tracking | |
| CN107504917A (en) | A kind of three-dimensional dimension measuring method and device | |
| CN110310325B (en) | Virtual measurement method, electronic device and computer readable storage medium | |
| CN112184793B (en) | Depth data processing method and device and readable storage medium | |
| JP7341736B2 (en) | Information processing device, information processing method and program | |
| Weinmann et al. | Fusing passive and active sensed images to gain infrared-textured 3D models | |
| JP2014112057A (en) | Real-time display method of expected coupling precision, and shape measurement system | |
| CN112270693A (en) | Method and device for detecting motion artifact of time-of-flight depth camera | |
| Conen et al. | Rectification and robust matching using oriented image triplets for minimally invasive surgery | |
| JP2002135807A (en) | Method and device for calibration for three-dimensional entry | |
| Liu et al. | Fusion of depth maps based on confidence |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20220216 Address after: 518000 floor 2, building 3, Dexin Chang wisdom Park, No. 23 Heping Road, Qinghua community, Longhua street, Longhua District, Shenzhen, Guangdong Province Applicant after: Shenzhen waterward Software Technology Co.,Ltd. Address before: 518000 B, 503, 602, digital city building, garden city, 1079 Shekou Road, Shekou, Nanshan District, Shenzhen, Guangdong. Applicant before: SHENZHEN WATER WORLD Co.,Ltd. |
|
| TA01 | Transfer of patent application right | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |