CN114782692A - House model repairing method and device, electronic equipment and readable storage medium - Google Patents
House model repairing method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN114782692A CN114782692A CN202210422989.XA CN202210422989A CN114782692A CN 114782692 A CN114782692 A CN 114782692A CN 202210422989 A CN202210422989 A CN 202210422989A CN 114782692 A CN114782692 A CN 114782692A
- Authority
- CN
- China
- Prior art keywords
- information
- panoramic image
- repaired
- target image
- repairing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 74
- 230000011218 segmentation Effects 0.000 claims description 35
- 230000002159 abnormal effect Effects 0.000 claims description 14
- 238000006243 chemical reaction Methods 0.000 claims description 9
- 238000001914 filtration Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 230000001360 synchronised effect Effects 0.000 description 5
- 230000000694 effects Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 238000013507 mapping Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- UDHXJZHVNHGCEC-UHFFFAOYSA-N Chlorophacinone Chemical compound C1=CC(Cl)=CC=C1C(C=1C=CC=CC=1)C(=O)C1C(=O)C2=CC=CC=C2C1=O UDHXJZHVNHGCEC-UHFFFAOYSA-N 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Images
Classifications
-
- G06T5/77—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30168—Image quality inspection
Abstract
The application discloses a house model repairing method and device, electronic equipment and a readable storage medium, and belongs to the technical field of house model modeling. The house model repairing method comprises the following steps: acquiring a panoramic image in the house model; identifying target image features in the panoramic image; determining a region to be repaired in the panoramic image according to the characteristics of the target image; acquiring point cloud information of target image features; and repairing the area to be repaired according to the point cloud information.
Description
Technical Field
The application belongs to the technical field of house model modeling, and particularly relates to a house model repairing method, a house model repairing device, electronic equipment and a readable storage medium.
Background
In recent years, with the increasing precision of depth cameras, house modeling methods based on depth cameras are becoming more popular. In the prior art, because the camera shooting process is influenced by highlight factors, the highlight area cannot acquire effective depth, and a certain degree of cavities can exist in the house model.
Disclosure of Invention
The embodiment of the application aims to provide a house model repairing method, a house model repairing device, an electronic device and a readable storage medium, which can accurately find a to-be-repaired area with a hole in a panoramic image, and accurately repair the detected to-be-repaired area according to point cloud information of the panoramic image, so that the repairing controllability is improved, and the accuracy of a repairing position is improved.
In a first aspect, an embodiment of the present application provides a method for repairing a house model, including: acquiring a panoramic image in the house model; identifying target image features in the panoramic image; determining a region to be repaired in the panoramic image according to the characteristics of the target image; acquiring point cloud information of target image features; and repairing the area to be repaired according to the point cloud information.
In a second aspect, an embodiment of the present application provides a house model repairing apparatus, which is applied to a virtual reality device, and includes: the acquisition module is used for acquiring a panoramic image in the house model; the identification module is used for identifying the target image characteristics in the panoramic image; the determining module is used for determining a region to be repaired in the panoramic image according to the characteristics of the target image; the acquisition module is also used for acquiring point cloud information of the target image characteristics; and the repairing module is used for repairing the area to be repaired according to the point cloud information.
In a third aspect, embodiments of the present application provide an electronic device, which includes a processor, a memory, and a program or instructions stored on the memory and executable on the processor, where the program or instructions, when executed by the processor, implement the steps of the method according to the first aspect.
In a fourth aspect, the present application provides a readable storage medium, on which a program or instructions are stored, which when executed by a processor implement the steps of the house model repairing method according to the first aspect.
In a fifth aspect, the present application provides a chip, where the chip includes a processor and a communication interface, where the communication interface is coupled to the processor, and the processor is configured to execute a program or instructions to implement the steps of the house model repairing method according to the first aspect.
In a sixth aspect, the present application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the house model repairing method according to the first aspect.
In the embodiment of the application, the panoramic image corresponding to the house model is obtained, the panoramic image is subjected to semantic segmentation, and each image feature in the panoramic image can be obtained, so that the target image feature in the panoramic image is identified and obtained. The target image features are image features which are easy to generate holes in the house model. And detecting the area to be repaired in the target feature, and acquiring point cloud information in the target image feature. And performing hole repairing on the area to be repaired according to the point cloud information.
According to the method and the device, the target image characteristics in the panoramic image are identified, the area to be repaired with the cavity in the panoramic image can be accurately found, the detected area to be repaired is accurately repaired according to the point cloud information of the panoramic image, and the repairing controllability and the repairing position accuracy are improved.
Drawings
Fig. 1 shows one of the flow diagrams of a house model repairing method provided by the embodiment of the present application;
fig. 2 illustrates a second flow chart of the house model repairing method provided in the embodiment of the present application;
fig. 3 shows a third schematic flowchart of a house model repairing method provided by the embodiment of the present application;
fig. 4 shows a fourth flowchart of a house model repairing method provided by the embodiment of the present application;
fig. 5 shows a fifth flowchart of a house model repairing method provided by the embodiment of the present application;
FIG. 6 is a schematic diagram illustrating a house model repairing method provided by an embodiment of the application;
fig. 7 shows a structural block diagram of a house model repairing device provided in an embodiment of the present application;
fig. 8 shows a block diagram of an electronic device provided in an embodiment of the present application;
fig. 9 shows a hardware structure diagram of an electronic device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below clearly with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are some, but not all, embodiments of the present application. All other embodiments that can be derived by one of ordinary skill in the art from the embodiments given herein are intended to be within the scope of the present disclosure.
The terms first, second and the like in the description and in the claims of the present application are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the application may be practiced in sequences other than those illustrated or described herein, and that the terms "first," "second," and the like are generally used herein in a generic sense and do not limit the number of terms, e.g., the first term can be one or more than one. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.
The house model repairing method, the house model repairing apparatus, the electronic device and the readable storage medium provided in the embodiments of the present application are described in detail below with reference to fig. 1 to 9 through specific embodiments and application scenarios thereof.
An embodiment of the present application provides a house model repairing method, fig. 1 shows one of flow diagrams of the house model repairing method provided by the embodiment of the present application, and as shown in fig. 1, the house model repairing method includes:
102, acquiring a panoramic image in a house model;
and step 110, repairing the area to be repaired according to the point cloud information.
The embodiment provides a scheme for performing semantic segmentation on a panoramic image through a semantic segmentation technology and repairing a hole caused by strong light in the target image characteristics obtained by segmentation.
In the embodiment of the application, the panoramic image corresponding to the house model is obtained, the panoramic image is subjected to semantic segmentation, and each image feature in the panoramic image can be obtained, so that the target image feature in the panoramic image is identified and obtained. The target image features are image features which are easy to generate holes in the house model. And detecting the area to be repaired in the target feature, and acquiring point cloud information in the target image feature. And performing cavity repair on the area to be repaired according to the point cloud information.
Specifically, after the panoramic image corresponding to the house model is obtained, the target image characteristics of the cavity which is easily affected by strong light in the panoramic image are identified, so that the area to be repaired in the panoramic image is accurately searched, and the area to be repaired in the panoramic image is repaired.
In the related art, the method for repairing the hole area in the image in a self-adaptive mode by means of a deep learning framework depends on a large data training set, and the problem that the repairing result is uncontrollable exists. The color identification is carried out on the whole image to determine the position to be repaired where the hole exists, and the hole area of the depth map caused by strong light is difficult to detect, so that the repairing accuracy is influenced.
It is worth explaining that the scheme for repairing the cavity problem caused by strong light in the house model provided by the application has the advantages of simplicity, easiness in use and good expansibility because the target image characteristics in the panoramic image are identified and the area to be repaired in the panoramic image is positioned according to the target image characteristics.
According to the method and the device, the target image characteristics in the panoramic image are identified, the area to be repaired with the cavity in the panoramic image can be accurately found, the detected area to be repaired is accurately repaired according to the point cloud information of the panoramic image, and the repairing controllability and the repairing position accuracy are improved.
In some embodiments of the present application, identifying a target image feature in a panoramic image comprises: inputting the panoramic image into a semantic segmentation network to obtain target image characteristics; wherein the target image features include one or a combination of: wall surface characteristics, ground surface characteristics, table top characteristics and bed surface characteristics.
This embodiment presents a scheme of recognizing a target image in a panoramic image by a semantic segmentation technique.
In the embodiment of the application, after the panoramic image in the house model is obtained, the panoramic image is input to a semantic segmentation network for semantic segmentation, so that the target image characteristics in the panoramic image are obtained.
Specifically, one or a combination of wall surface features, ground surface features, desktop features and bed surface features in the panoramic image is identified through a semantic segmentation technology.
It can be understood that in the process of modeling the house model, in the collected panoramic image of the house, the wall surface, the ground surface, the desktop surface and the bed surface of the house are easily affected by strong light to generate holes. According to the method and the device, the semantic segmentation network of the wall surface features, the ground surface features, the desktop features and the bed surface features can be identified through training, the target image features are identified through the semantic segmentation network, and the to-be-repaired area in the panoramic image can be conveniently determined according to the target image features.
It should be noted that not only the target image features can be identified, but also mask information of corresponding points can be acquired through the semantic segmentation network.
According to the method and the device, the semantic segmentation information of the wall surface, the ground surface, the desktop and the bed surface is acquired by means of the depth learning framework, the target image characteristics in the panoramic image are determined, and the to-be-repaired area in the panoramic image can be conveniently determined subsequently according to the target image characteristics.
In some embodiments of the present application, fig. 2 shows a second flowchart of a house model repairing method provided in the embodiments of the present application, and as shown in fig. 2, determining an area to be repaired in a panoramic image according to a target image feature includes:
step 202, determining the position information of the target image characteristics in the panoramic image according to the mask information of the target image characteristics;
and step 204, determining a region to be repaired in the panoramic image according to the brightness information and the position information of the target image characteristics.
In the embodiment, a scheme for searching the area to be repaired in the panoramic image according to the characteristics of the target image is provided.
In the embodiment of the application, not only can the target image characteristics in the panoramic image be identified through the semantic segmentation network, but also mask information corresponding to the target image characteristics can be acquired. And according to the mask information, the corresponding position information of the target image characteristics in the panoramic image can be obtained. And acquiring brightness information of the panoramic image, searching the brightness information of the target image characteristic according to the position information, and screening to obtain a cavity area in the target image characteristic, namely the area to be repaired in the panoramic image according to the brightness information.
Specifically, the target image features include one or a combination of wall features, ground features, desktop features, and bed surface features, and each feature corresponds to corresponding mask information. And according to the mask information, a corresponding area of the corresponding feature in the panoramic image can be determined, so that the position information of the target image feature in the panoramic image can be determined. The brightness information of the target image characteristics can be determined by identifying the brightness information of the panoramic image and according to the position information, and the highlight area in the panoramic image, namely the area to be repaired, is screened according to the highlight area.
In the embodiment of the application, the target image characteristics and the mask information corresponding to the target image characteristics are identified through a semantic segmentation technology, and accordingly, the area to be repaired in the panoramic image is accurately positioned, so that the accuracy of repairing the area to be repaired in the panoramic image subsequently is improved.
In some embodiments of the present application, fig. 3 shows a third flowchart of a house model repairing method provided in an embodiment of the present application, and as shown in fig. 3, determining an area to be repaired in a panoramic image according to luminance information and position information of a target image feature includes:
the histogram corresponding to the histogram information is a luminance information and pixel number histogram;
and step 304, determining the area to be repaired according to the histogram information and the position information.
The embodiment provides a scheme for accurately positioning the area to be repaired in the panoramic image according to the histogram information of the panoramic image and the position information of the target image characteristic.
In the embodiment of the application, the brightness information of each pixel in the panoramic image is extracted, and the histogram information of the panoramic image can be acquired according to the brightness information of each pixel. According to the position information of the target image characteristic, a highlight area in the panoramic image, namely the area to be repaired, can be determined together with the histogram information.
According to the embodiment of the application, the area to be repaired can be accurately positioned through the histogram information of the panoramic image and the position information of the target image characteristic in the panoramic image, and the problem that the hole repairing of the panoramic image is inaccurate due to the fact that the area to be repaired is difficult to detect is solved.
In some embodiments of the present application, fig. 4 shows a fourth flowchart of the house model repairing method provided in the embodiments of the present application, and as shown in fig. 4, acquiring histogram information of a panoramic image includes:
This embodiment presents a scheme of converting a panoramic image from an RGB (red, green, blue) space to an HSV (hue, saturation, brightness) space and determining histogram information from luminance information therein.
In the embodiment of the application, before the histogram information of the panoramic image is acquired, the panoramic image is mapped to the HSV space from the RGB space, and the histogram information of the panoramic image is established according to the brightness information in the HSV space.
According to the embodiment of the application, the brightness information of the panoramic image can be more accurately determined by mapping the RGB space to the HSV space, so that the obtained histogram information is more accurate, and the accuracy of repairing the holes in the panoramic image is further improved.
In some embodiments of the present application, fig. 5 shows a fifth schematic flow chart of the house model repairing method provided in the embodiments of the present application, and as shown in fig. 5, repairing the region to be repaired according to the point cloud information includes:
step 504, determining a three-dimensional point corresponding to a pixel in the area to be repaired according to the camera parameters and the plane information of the panoramic image;
and step 506, repairing the region to be repaired according to the depth information of the three-dimensional point.
The embodiment provides a scheme for repairing the area to be repaired in the panoramic image according to the point cloud information.
It should be noted that after the panoramic image is subjected to semantic segmentation, not only the target image information can be determined, but also the point cloud information corresponding to the target image feature after the semantic segmentation can be obtained.
In the embodiment of the application, according to the point cloud information corresponding to the target image feature, the plane information corresponding to the target image feature can be determined through the least square technology. The area to be repaired in the panoramic image comprises a plurality of pixels, and the three-dimensional point of each pixel in the area to be repaired can be determined according to the camera parameters and the plane information in the panoramic image. The three-dimensional points correspond to pixels in the area to be repaired one by one, and the depth information of the three-dimensional points is configured on the corresponding pixels in the area to be repaired, so that the area to be repaired is repaired.
Specifically, the target image features include one or a combination of wall surface features, ground surface features, desktop features and bed surface features, and the plane information corresponding to the target image features is a plane model determined through point cloud information. The method comprises the steps of connecting a ray to each pixel in a region to be repaired by using a camera model of a panoramic image and taking an optical center of the camera model as an original point, intersecting the ray with a plane model, obtaining an intersection point which is a three-dimensional point corresponding to the pixel of the region to be repaired, filling depth information of the three-dimensional point to the corresponding pixel of the region to be repaired, and completing repairing of the region to be repaired.
In the embodiment of the application, the three-dimensional point corresponding to each pixel in the area to be repaired can be determined through the camera parameters of the camera model and the plane information determined according to the point cloud information. And then filling depth of corresponding pixels in the area to be repaired according to the depth information of the three-dimensional points, so that the problem of cavities caused by strong light is repaired, the cavities caused by the strong light do not exist in the repaired panoramic image, and the depth of the filled area to be repaired is matched with the depth of the surrounding scene.
In some embodiments of the present application, determining plane information of the target image feature from the point cloud information includes: filtering abnormal information in the point cloud information; and determining plane information corresponding to the point cloud information with the abnormal information filtered out.
The embodiment provides a scheme for determining plane information through point cloud information.
In the embodiment of the application, the point cloud information of the target image characteristics can be determined by performing semantic segmentation on the panoramic image. In order to improve the accuracy of the point cloud information, abnormal information in the point cloud information needs to be filtered, and then the point cloud information with the abnormal information removed is filtered to determine plane information through least square.
Specifically, point cloud information corresponding to the target image features can be extracted according to the semantic segmentation information. Due to inaccuracy of semantic precision, firstly, ranac (Random Sample Consensus) is carried out on the acquired point cloud to remove some outliers (abnormal information), and then least square is used for estimating plane information.
According to the embodiment of the application, abnormal information is filtered out from the point cloud information obtained by semantic segmentation, so that the accuracy of plane information determined by least square can be improved, the accuracy of repairing the region to be repaired is further improved, and the problem of model cavities under strong light is effectively solved.
In some embodiments of the present application, fig. 6 shows a schematic flowchart of a house model repairing method provided in an embodiment of the present application, and as shown in fig. 6, the house model repairing method includes: inputting a panoramic image, performing semantic segmentation, detecting an area to be repaired, extracting point cloud information, and repairing a depth map.
Wherein, semantic segmentation: and acquiring semantic segmentation information of a wall surface, a ground surface, a desktop and a bed surface in the house by means of a deep learning frame, namely target image characteristics. And acquires mask information corresponding thereto.
Detecting a region to be repaired: according to mask information obtained by semantic segmentation, a corresponding region of a certain class of target image features in the panoramic image is obtained, then an RGB space of the panoramic image is mapped into an HSV space, corresponding histogram information is calculated for a V channel (a brightness channel), and finally a region to be repaired is determined according to the histogram information.
Extracting point cloud information: according to the semantic segmentation information, point cloud information corresponding to the target image features can be extracted. Because the semantic has the problem of low precision, Ranac is carried out on the acquired point cloud information to remove some outliers, and then least square is utilized to estimate plane information.
Repairing a depth map: and connecting a ray with pixels in the area to be repaired by using the camera model and taking the optical center as an origin, and intersecting the ray with plane information. The intersection points are three-dimensional points corresponding to pixels of the area to be repaired, and then the depth of the intersection points is filled to be the depth value of the area to be repaired.
According to the house model repairing method provided by the embodiment of the application, the execution main body can be a house model repairing device. In the embodiment of the present application, a house model repairing method executed by a house model repairing apparatus is taken as an example, and the house model repairing apparatus provided in the embodiment of the present application is described.
In some embodiments of the present application, a house model repairing apparatus is provided, and fig. 7 shows a block diagram of a house model repairing apparatus provided in an embodiment of the present application, as shown in fig. 7, a house model repairing apparatus 700, and the house model repairing apparatus 700 includes:
an obtaining module 702, configured to obtain a panoramic image in a house model;
an identifying module 704 for identifying a target image feature in the panoramic image;
a determining module 706, configured to determine, according to the target image feature, a region to be repaired in the panoramic image;
the obtaining module 702 is further configured to obtain point cloud information of the target image feature;
and the repairing module 708 is used for repairing the area to be repaired according to the point cloud information.
According to the method and the device, the target image characteristics in the panoramic image are identified, the area to be repaired with the cavity in the panoramic image can be accurately found, the detected area to be repaired is accurately repaired according to the point cloud information of the panoramic image, and the repairing controllability and the repairing position accuracy are improved.
In some embodiments of the present application, the house model restoration device 700 further includes:
the input module is used for inputting the panoramic image into a semantic segmentation network so as to obtain target image characteristics; wherein the target image features include one or a combination of: wall surface characteristics, ground surface characteristics, table top characteristics, and bed surface characteristics.
According to the method and the device, the semantic segmentation information of the wall surface, the ground surface, the desktop and the bed surface is acquired by means of the depth learning framework, the target image characteristics in the panoramic image are determined, and the to-be-repaired area in the panoramic image can be conveniently determined subsequently according to the target image characteristics.
In some embodiments of the present application, the determining module 706 is further configured to determine, according to mask information of the target image feature, position information of the target image feature in the panoramic image;
the determining module 706 is further configured to determine an area to be repaired in the panoramic image according to the brightness information and the position information of the target image feature.
In the embodiment of the application, the target image characteristics and the mask information corresponding to the target image characteristics are identified through a semantic segmentation technology, and accordingly the area to be repaired in the panoramic image is accurately positioned, so that the accuracy of repairing the area to be repaired in the panoramic image subsequently is improved.
In some embodiments of the present application, the obtaining module 702 is further configured to obtain histogram information of the panoramic image;
the histogram corresponding to the histogram information is a luminance information and pixel number histogram;
the determining module 706 is further configured to determine the region to be repaired according to the histogram information and the location information.
According to the embodiment of the application, the area to be repaired can be accurately positioned through the histogram information of the panoramic image and the position information of the target image characteristic in the panoramic image, and the problem that the hole repairing of the panoramic image is inaccurate due to the fact that the area to be repaired is difficult to detect is solved.
In some embodiments of the present application, the house model restoration apparatus 700 further includes:
the conversion module is used for carrying out color space conversion on the panoramic image;
the determining module 706 is further configured to determine the histogram information according to the brightness information of the panoramic image after color space conversion.
According to the embodiment of the application, the brightness information of the panoramic image can be more accurately determined by mapping the RGB space to the HSV space, so that the obtained histogram information is more accurate, and the accuracy of repairing the holes in the panoramic image is further improved.
In some embodiments of the present application, the determining module 706 is further configured to determine plane information of the target image feature according to the point cloud information;
the determining module 706 is further configured to determine a three-dimensional point corresponding to a pixel in the area to be repaired according to the camera parameter and the plane information of the panoramic image;
the repairing module 708 is further configured to repair the region to be repaired according to the depth information of the three-dimensional point.
In the embodiment of the application, the three-dimensional point corresponding to each pixel in the area to be repaired can be determined through the camera parameters of the camera model and the plane information determined according to the point cloud information. And then, filling depth is carried out on corresponding pixels in the area to be repaired according to the depth information of the three-dimensional points, so that the hole problem caused by the strong light is repaired, the hole generated by the strong light does not exist in the repaired panoramic image, and the depth of the filled area to be repaired is matched with the depth of the surrounding scene.
In some embodiments of the present application, the house model restoration device 700 further includes:
the filtering module is used for filtering abnormal information in the point cloud information;
the determining module 706 is further configured to determine plane information corresponding to the point cloud information with the abnormal information filtered out.
According to the method and the device, abnormal information is filtered out from the point cloud information obtained by semantic segmentation, so that the accuracy of plane information determined by least square can be improved, the accuracy of repairing the region to be repaired is further improved, and the problem of mold cavities under strong light is effectively solved.
The house model repairing device in the embodiment of the present application may be an electronic device, or may be a component in an electronic device, such as an integrated circuit or a chip. The electronic device may be a terminal, or may be a device other than a terminal. The electronic Device may be, for example, a Mobile phone, a tablet computer, a notebook computer, a palm computer, a vehicle-mounted electronic Device, a Mobile Internet Device (MID), an Augmented Reality (AR)/Virtual Reality (VR) Device, a robot, a wearable Device, an ultra-Mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like, and may also be a server, a Network Attached Storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), an assistant, or a self-service machine, and the embodiments of the present application are not limited in particular.
The house model restoration device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which is not specifically limited in the embodiment of the present application.
The house model repairing device provided by the embodiment of the application can realize each process realized by the method embodiment, and is not described again for avoiding repetition.
Optionally, an electronic device is further provided in an embodiment of the present application, fig. 8 shows a block diagram of a structure of the electronic device according to the embodiment of the present application, as shown in fig. 8, an electronic device 800 includes a processor 802 and a memory 804, a program or an instruction that can be executed on the processor 802 is stored in the memory 804, and when the program or the instruction is executed by the processor 802, the steps of the embodiment of the method are implemented, and the same technical effect can be achieved, and details are not repeated here to avoid repetition.
It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic device and the non-mobile electronic device described above.
Fig. 9 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present application.
The electronic device 900 includes, but is not limited to: a radio frequency unit 901, a network module 902, an audio output unit 903, an input unit 904, a sensor 905, a display unit 906, a user input unit 907, an interface unit 908, a memory 909, and a processor 910.
Those skilled in the art will appreciate that the electronic device 900 may further include a power source (e.g., a battery) for supplying power to various components, and the power source may be logically connected to the processor 910 through a power management system, so as to manage charging, discharging, and power consumption management functions through the power management system. The electronic device structure shown in fig. 9 does not constitute a limitation to the electronic device, and the electronic device may include more or less components than those shown, or combine some components, or arrange different components, and thus, the description is omitted here.
The processor 910 is configured to obtain a panoramic image in the house model;
a processor 910 configured to identify a target image feature in the panoramic image;
a processor 910, configured to determine, according to a target image feature, an area to be repaired in a panoramic image;
the processor 910 is further configured to obtain point cloud information of the target image feature;
and the processor 910 is configured to repair the area to be repaired according to the point cloud information.
According to the method and the device, the target image characteristics in the panoramic image are identified, the area to be repaired with the cavity in the panoramic image can be accurately found, the detected area to be repaired is accurately repaired according to the point cloud information of the panoramic image, and the repairing controllability and the repairing position accuracy are improved.
Further, the processor 910 is configured to input the panoramic image into a semantic segmentation network to obtain a target image feature; wherein the target image features include one or a combination of: wall surface characteristics, ground surface characteristics, table top characteristics and bed surface characteristics.
According to the method and the device, the semantic segmentation information of the wall surface, the ground surface, the desktop and the bed surface is acquired by means of the depth learning framework, the target image characteristics in the panoramic image are determined, and the to-be-repaired area in the panoramic image can be conveniently determined subsequently according to the target image characteristics.
Further, the processor 910 is further configured to determine, according to the mask information of the target image feature, location information of the target image feature in the panoramic image;
the processor 910 is further configured to determine an area to be repaired in the panoramic image according to the brightness information and the position information of the target image feature.
In the embodiment of the application, the target image characteristics and the mask information corresponding to the target image characteristics are identified through a semantic segmentation technology, and accordingly, the area to be repaired in the panoramic image is accurately positioned, so that the accuracy of repairing the area to be repaired in the panoramic image subsequently is improved.
Further, the processor 910 is further configured to obtain histogram information of the panoramic image;
the histogram corresponding to the histogram information is a luminance information and pixel number histogram;
the processor 910 is further configured to determine an area to be repaired according to the histogram information and the location information.
According to the embodiment of the application, the area to be repaired can be accurately positioned through the histogram information of the panoramic image and the position information of the target image characteristic in the panoramic image, and the problem that the hole repairing of the panoramic image is inaccurate due to the fact that the area to be repaired is difficult to detect is solved.
Further, a processor 910 configured to perform color space conversion on the panoramic image;
the processor 910 is further configured to determine the histogram information according to the brightness information of the panoramic image after color space conversion.
According to the embodiment of the application, the brightness information of the panoramic image can be more accurately determined by mapping the RGB space to the HSV space, so that the obtained histogram information is more accurate, and the accuracy of repairing the holes in the panoramic image is further improved.
Further, the processor 910 is further configured to determine plane information of the target image feature according to the point cloud information;
the processor 910 is further configured to determine a three-dimensional point corresponding to a pixel in the area to be repaired according to the camera parameter and the plane information of the panoramic image;
the processor 910 is further configured to repair the area to be repaired according to the depth information of the three-dimensional point.
In the embodiment of the application, the three-dimensional point corresponding to each pixel in the area to be repaired can be determined through the camera parameters of the camera model and the plane information determined according to the point cloud information. And then, filling depth is carried out on corresponding pixels in the area to be repaired according to the depth information of the three-dimensional points, so that the hole problem caused by the strong light is repaired, the hole generated by the strong light does not exist in the repaired panoramic image, and the depth of the filled area to be repaired is matched with the depth of the surrounding scene.
Further, the processor 910 is configured to filter out abnormal information in the point cloud information;
the processor 910 is further configured to determine plane information corresponding to the point cloud information with abnormal information filtered out.
According to the method and the device, abnormal information is filtered out from the point cloud information obtained by semantic segmentation, so that the accuracy of plane information determined by least square can be improved, the accuracy of repairing the region to be repaired is further improved, and the problem of mold cavities under strong light is effectively solved.
It should be understood that, in the embodiment of the present application, the input Unit 904 may include a Graphics Processing Unit (GPU) 9041 and a microphone 9042, and the Graphics Processing Unit 9041 processes image data of a still picture or a video obtained by an image capturing device (such as a camera) in a video capturing mode or an image capturing mode. The display unit 906 may include a display panel 9061, and the display panel 9061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 907 includes at least one of a touch panel 9071 and other input devices 9072. A touch panel 9071 also referred to as a touch screen. The touch panel 9071 may include two parts, a touch detection device and a touch controller. Other input devices 9072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail herein.
The memory 909 may be used to store software programs as well as various data. The memory 909 may mainly include a first storage area storing a program or an instruction and a second storage area storing data, wherein the first storage area may store an operating system, an application program or an instruction (such as a sound playing function, an image playing function, and the like) required for at least one function, and the like. Further, the memory 909 may include volatile memory or nonvolatile memory, or the memory 909 may include both volatile and nonvolatile memory. The non-volatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable PROM (EEPROM), or a flash Memory. The volatile Memory may be a Random Access Memory (RAM), a Static Random Access Memory (Static RAM, SRAM), a Dynamic Random Access Memory (Dynamic RAM, DRAM), a Synchronous Dynamic Random Access Memory (Synchronous DRAM, SDRAM), a Double Data Rate Synchronous Dynamic Random Access Memory (Double Data Rate SDRAM, ddr SDRAM), an Enhanced Synchronous SDRAM (ESDRAM), a Synchronous Link DRAM (SLDRAM), and a Direct Memory bus RAM (DRRAM). The memory 909 in the embodiments of the subject application includes, but is not limited to, these and any other suitable types of memory.
The embodiment of the present application further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the embodiment of the house model repairing method, and can achieve the same technical effect, and in order to avoid repetition, details are not described here again.
The processor is the processor in the electronic device in the above embodiment. Readable storage media, including computer readable storage media such as computer read only memory ROM, random access memory RAM, magnetic or optical disks, and the like.
The embodiment of the present application further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction to implement each process of the embodiment of the house model repairing method, and the same technical effect can be achieved.
It should be understood that the chips mentioned in the embodiments of the present application may also be referred to as system-on-chip, system-on-chip or system-on-chip, etc.
The embodiment of the present application provides a computer program product, where the program product is stored in a storage medium, and the program product is executed by at least one processor to implement the processes of the embodiment of the house model repairing method as described above, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Further, it should be noted that the scope of the methods and apparatuses in the embodiments of the present application is not limited to performing the functions in the order illustrated or discussed, but may include performing the functions in a substantially simultaneous manner or in a reverse order based on the functions recited, e.g., the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.
Through the description of the foregoing embodiments, it is clear to those skilled in the art that the method of the foregoing embodiments may be implemented by software plus a necessary general hardware platform, and certainly may also be implemented by hardware, but in many cases, the former is a better implementation. Based on such understanding, the technical solutions of the present application may be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (such as a mobile phone, a computer, a server, or a network device) to execute the method of the embodiments of the present application.
While the present embodiments have been described with reference to the accompanying drawings, it is to be understood that the invention is not limited to the precise embodiments described above, which are meant to be illustrative and not restrictive, and that various changes may be made therein by those skilled in the art without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A house model restoration method, comprising:
acquiring a panoramic image in the house model;
identifying a target image feature in the panoramic image;
determining a region to be repaired in the panoramic image according to the target image characteristics;
acquiring point cloud information of the target image features;
and repairing the area to be repaired according to the point cloud information.
2. The method for repairing a house model according to claim 1, wherein the identifying a target image feature in the panoramic image comprises:
inputting the panoramic image into a semantic segmentation network to obtain target image characteristics;
wherein the target image features include one or a combination of: wall surface characteristics, ground surface characteristics, table top characteristics and bed surface characteristics.
3. The house model repairing method according to claim 1, wherein the determining the area to be repaired in the panoramic image according to the target image feature comprises:
determining the position information of the target image characteristic in the panoramic image according to the mask information of the target image characteristic;
and determining the area to be repaired in the panoramic image according to the brightness information and the position information of the target image characteristic.
4. The house model repairing method according to claim 3, wherein the determining the area to be repaired in the panoramic image according to the brightness information and the position information of the target image feature comprises:
acquiring histogram information of the panoramic image, wherein a histogram corresponding to the histogram information is a luminance information and pixel number histogram;
and determining the area to be repaired according to the histogram information and the position information.
5. The house model repairing method according to claim 4, wherein the obtaining histogram information of the panoramic image comprises:
performing color space conversion on the panoramic image;
and determining the histogram information according to the brightness information of the panoramic image after color space conversion.
6. The house model repairing method according to any one of claims 1 to 5, wherein the repairing the area to be repaired according to the point cloud information comprises:
determining plane information of the target image features through the point cloud information;
determining a three-dimensional point corresponding to a pixel in the area to be repaired according to the camera parameter of the panoramic image and the plane information;
and repairing the area to be repaired according to the depth information of the three-dimensional point.
7. The method for repairing house model according to claim 6, wherein the determining the plane information of the target image feature through the point cloud information comprises:
filtering abnormal information in the point cloud information;
and determining the plane information corresponding to the point cloud information with the abnormal information filtered out.
8. The utility model provides a prosthetic devices of house model which characterized in that is applied to virtual reality equipment, includes:
the acquisition module is used for acquiring a panoramic image in the house model;
the identification module is used for identifying the target image characteristics in the panoramic image;
the determining module is used for determining a region to be repaired in the panoramic image according to the target image characteristics;
the acquisition module is also used for acquiring point cloud information of the target image characteristics;
and the repairing module is used for repairing the area to be repaired according to the point cloud information.
9. An electronic device, comprising:
a memory having a program or instructions stored thereon;
a processor for implementing the steps of the method of repairing a house model according to any one of claims 1 to 7 when said program or instructions are executed.
10. A readable storage medium on which a program or instructions are stored, characterized in that said program or instructions, when executed by a processor, carry out the steps of the method of repairing a house model according to any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210422989.XA CN114782692A (en) | 2022-04-21 | 2022-04-21 | House model repairing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210422989.XA CN114782692A (en) | 2022-04-21 | 2022-04-21 | House model repairing method and device, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114782692A true CN114782692A (en) | 2022-07-22 |
Family
ID=82430971
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210422989.XA Pending CN114782692A (en) | 2022-04-21 | 2022-04-21 | House model repairing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114782692A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115908163A (en) * | 2022-11-01 | 2023-04-04 | 北京城市网邻信息技术有限公司 | Hole repairing method, device and storage medium |
-
2022
- 2022-04-21 CN CN202210422989.XA patent/CN114782692A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115908163A (en) * | 2022-11-01 | 2023-04-04 | 北京城市网邻信息技术有限公司 | Hole repairing method, device and storage medium |
| CN115908163B (en) * | 2022-11-01 | 2023-09-08 | 北京城市网邻信息技术有限公司 | Hole repairing method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110430443B (en) | Method and device for cutting video shot, computer equipment and storage medium | |
| US9756261B2 (en) | Method for synthesizing images and electronic device thereof | |
| US10824910B2 (en) | Image processing method, non-transitory computer readable storage medium and image processing system | |
| CN111124888B (en) | Method and device for generating recording script and electronic device | |
| US20130258198A1 (en) | Video search system and method | |
| US10347000B2 (en) | Entity visualization method | |
| CN109934873B (en) | Method, device and equipment for acquiring marked image | |
| CN107909569B (en) | Screen-patterned detection method, screen-patterned detection device and electronic equipment | |
| CN109064525A (en) | A kind of picture format conversion method, device, equipment and storage medium | |
| CN114782692A (en) | House model repairing method and device, electronic equipment and readable storage medium | |
| CN108764248B (en) | Image feature point extraction method and device | |
| CN111475677A (en) | Image processing method, image processing device, storage medium and electronic equipment | |
| CN111062924A (en) | Image processing method, device, terminal and storage medium | |
| CN110751668A (en) | Image processing method, device, terminal, electronic equipment and readable storage medium | |
| CN110717060A (en) | Image mask filtering method and device and storage medium | |
| CN105516735A (en) | Representation frame acquisition method and representation frame acquisition apparatus | |
| CN114299271A (en) | Three-dimensional modeling method, three-dimensional modeling apparatus, electronic device, and readable storage medium | |
| CN114758054A (en) | Light spot adding method, device, equipment and storage medium | |
| US9798932B2 (en) | Video extraction method and device | |
| CN114329675A (en) | Model generation method, model generation device, electronic device, and readable storage medium | |
| CN114119695A (en) | Image annotation method and device and electronic equipment | |
| CN114565777A (en) | Data processing method and device | |
| CN114245193A (en) | Display control method and device and electronic equipment | |
| CN110196638B (en) | Mobile terminal augmented reality method and system based on target detection and space projection | |
| CN113747076A (en) | Shooting method and device and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |