CN111583417A - Method and device for constructing indoor VR scene with combined constraint of image semantics and scene geometry, electronic equipment and medium - Google Patents

Abstract

The embodiment of the disclosure discloses a method, a device, electronic equipment and a medium for constructing an indoor VR scene constrained by image semantics and scene geometry. One embodiment of the method comprises: obtaining an image and the height of a camera from the ground; preprocessing the image; inputting the preprocessed image into a depth residual error network trained in advance, and outputting the characteristic information of the image; detecting information of straight lines in the characteristic information; inputting the information of the straight line into a convolutional neural network, and outputting the information of the wall connection part in the image; determining indoor layout information; inputting the characteristic information into a pre-trained target detection network, and outputting the indoor object information in the image; obtaining the position information of the indoor object; and finishing the construction of an indoor and virtual reality interactive scene. According to the embodiment, the real image scene is accurately and quickly restored through the construction of the indoor VR interactive scene.

Description

A method for constructing indoor VR scenes with joint constraints of image semantics and scene geometry Laws, devices, electronic equipment and media

technical field

Embodiments of the present disclosure relate to the field of computer technology, and in particular, to a method, apparatus, electronic device, and medium for constructing an indoor VR scene with joint constraints of image semantics and scene geometry.

Background technique

With the rapid development of virtual reality technology (VR, Virtual Reality), the realization of interactivity has become easier. VR technology is increasingly used in our daily life. For example, in interior decoration, VR technology should be incorporated into interior decoration to interact with customers to design, thus solving the problem of the diversity of customers' requirements for interior decoration. Due to the high requirements for details in virtual reality scenes, manual production of scenes not only consumes a lot of manpower and material resources, but also has a high cost, which is one of the reasons that restrict the development of virtual reality technology. The existing solutions include template-based scene generation, using various devices to collect real-world 3D information and convert it into 3D models. These generation methods rely on a set of pre-designed scene templates, and then generate 3D scenes in batches by setting parameters. However, these methods have great limitations and the accuracy is not high.

SUMMARY OF THE INVENTION

This summary of the disclosure serves to introduce concepts in a simplified form that are described in detail in the detailed description that follows. The content section of this disclosure is not intended to identify key features or essential features of the claimed technical solution, nor is it intended to be used to limit the scope of the claimed technical solution.

Some embodiments of the present disclosure propose a method, apparatus, electronic device, and medium for indoor VR scene construction with joint constraints of image semantics and scene geometry, to solve the technical problems mentioned in the background section above.

In a first aspect, some embodiments of the present disclosure provide a method for constructing an indoor VR scene with joint constraints of image semantics and scene geometry. The method includes: obtaining an indoor image and the height of the camera from the ground when shooting the above-mentioned indoor image; The image is preprocessed; the preprocessed indoor image is input into the pre-trained deep residual network, and the feature information of the indoor image is output; the information of the straight line in the feature information is detected; the information of the straight line is input to the convolution In the neural network, the information of the wall connecting part in the above-mentioned indoor image is output, wherein the above-mentioned wall connecting part is a straight line connecting between the walls in the above-mentioned indoor image; based on the height of the above-mentioned camera from the ground and the above-mentioned wall connecting part to obtain indoor layout information; input the above-mentioned feature information into a pre-trained target detection network, and output the indoor object information in the above-mentioned indoor image; based on the above-mentioned indoor object information and the information of the above-mentioned wall connection part, obtain the position of the indoor object information; according to the above-mentioned indoor layout information and the above-mentioned position information of the indoor objects, the construction of the indoor and virtual reality interaction scene is completed.

In a second aspect, some embodiments of the present disclosure provide an apparatus for constructing an indoor VR scene with joint constraints of image semantics and scene geometry. The apparatus includes: an acquisition unit configured to acquire an indoor image and to capture the indoor image when the camera is far from the ground The processing unit is configured to preprocess the indoor image; the first input and output unit is configured to input the preprocessed indoor image into the pre-trained deep residual network, and output the characteristics of the indoor image The detection unit is configured to detect the information of the straight line in the above-mentioned characteristic information; the second input and output unit is configured to input the information of the above-mentioned straight line into the convolutional neural network, and output the information of the connecting part of the wall in the above-mentioned indoor image , wherein the wall connecting part is a straight line connecting the walls in the indoor image; the first determining unit is configured to obtain indoor layout information based on the height of the camera from the ground and the information of the wall connecting part The third input and output unit is configured to input the above-mentioned feature information into the pre-trained target detection network, and outputs the indoor object information in the above-mentioned indoor image; The second determination unit is configured to be based on the above-mentioned indoor object information and the above-mentioned wall. Connect the information of the part to obtain the position information of the indoor objects; the third determination unit is configured to complete the construction of the indoor and virtual reality interaction scene according to the indoor layout information and the position information of the indoor objects.

In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, when one or more programs are stored by one or more The processor executes such that the one or more processors implement the method according to any one of the first and second aspects.

In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium on which a computer program is stored, wherein the program implements the method according to any one of the first and second aspects when executed by a processor.

One of the above-mentioned various embodiments of the present disclosure has the following beneficial effects: First, an indoor image is obtained and the height of the camera from the ground when the above-mentioned indoor image is captured. Furthermore, the indoor images are preprocessed, and the processed data makes it easier for the deep learning network to learn, and the accuracy of the network is improved. Furthermore, the pre-processed indoor image is input into a pre-trained deep residual network to extract the feature information of the indoor image, which lays a foundation for the recognition of the connecting part of the wall and object recognition. Then, the information of the straight lines in the above feature information is detected and stored, so as to detect all the straight lines that may be the connecting parts of the wall. This makes it possible to use a convolutional neural network to score all lines that might be connected parts of a wall. The indoor layout information is basically determined according to the height of the camera from the ground and the information of the connecting part of the wall. The above feature information is input into the pre-trained target detection network, which is used to determine the position, size and other information of indoor objects in the above indoor image. Then, based on the above-mentioned indoor object information and the above-mentioned information of the wall connecting part, the position information of the indoor object is obtained. Finally, the construction of the indoor and virtual reality interaction scene is completed according to the above-mentioned indoor layout information and the above-mentioned position information of the indoor objects.

Description of drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent when taken in conjunction with the accompanying drawings and with reference to the following detailed description. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that the originals and elements are not necessarily drawn to scale.

1 is a flowchart of some embodiments of a method for indoor VR scene construction according to the present disclosure.

FIG. 2 is a schematic structural diagram of some embodiments of an apparatus for indoor VR scene construction according to the present disclosure.

3 is a schematic structural diagram of an electronic device suitable for implementing some embodiments of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the protection scope of the present disclosure.

In addition, it should be noted that, for the convenience of description, only the parts related to the related invention are shown in the drawings. The embodiments of this disclosure and features of the embodiments may be combined with each other without conflict.

It should be noted that concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "a" and "a plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, they should be understood as "one or a plurality of". multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of these messages or information.

The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

Referring to FIG. 1 , a flow 100 of some embodiments of a method for indoor VR scene construction according to the present disclosure is shown. This method can be performed by the server. The method for generating classification information includes the following steps:

Step 101: Obtain an indoor image and the height of the camera from the ground when the indoor image is captured.

In some embodiments, the execution subject (which may be a server) of the method for indoor VR scene construction can obtain the indoor image and the height of the camera from the ground when the indoor image is captured in various ways. For example, the above-mentioned executive body may acquire the indoor image and the height of the camera from the ground when the indoor image is captured through a wired connection or a wireless connection. It should be pointed out that the above wireless connection methods may include but are not limited to 3G/4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or developed in the future .

Step 102, preprocessing the indoor image.

In some embodiments, the above-mentioned execution body performs preprocessing on the indoor image acquired in step 101, and the above-mentioned preprocessing refers to the processing performed before feature extraction, segmentation and matching are performed on the input image. The main purpose of image preprocessing is to eliminate irrelevant information in images, restore useful real information, enhance the detectability of relevant information and simplify data to the greatest extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition. The preprocessing process generally includes steps such as digitization, geometric transformation, normalization, smoothing, restoration and enhancement.

In some optional implementations of some embodiments, the above-mentioned preprocessing of the indoor image includes: adjusting the above-mentioned indoor image to a predetermined resolution.

Step 103: Input the preprocessed indoor image into a pre-trained deep residual network, and output the feature information of the indoor image.

In some embodiments, the indoor image preprocessed by the main body is input to a pre-trained deep residual network (Deep Residual Network, ResNet) to extract feature information of the indoor image. Among them, the ResNet-50 neural network is used as the feature extraction neural network. The input of the neural network is 224*224*3 (length*width*number of color channels), and the output is an image feature tensor of 1024*1*256. ResNet uses a network structure called Residual Block, through three layers of 1*1*64 convolution, 3*3*64 convolution, and 1*1*256 convolution, and ReLU is used between layers The activation function effectively reduces the number of parameters in the neural network.

Rectified Linear Unit (ReLU), also known as rectified linear unit, is a commonly used excitation function in neural networks. ReLU uses the following formula to process the output of each layer of the neural network:

f(x)=max{0,x}

Step 104: Detect the information of the straight line in the feature information.

In some embodiments, the above-mentioned execution subject may use the characteristic information of the above-mentioned indoor image obtained in the above-mentioned step 103. The straight line detection algorithm is used to detect the straight line in the input panoramic image, and the detected straight line position is stored as the feature of wall position discrimination. Wherein, the detection of the above-mentioned straight line detection algorithm may include but not limited to at least one of the following: CannyLines algorithm, Hough transform.

Step 105: Input the information of the straight line into the convolutional neural network, and output the information of the connecting part of the wall in the indoor image.

In some embodiments, the above-mentioned executive body may input the information of the above-mentioned straight line obtained in step 104 into a convolutional neural network, and output the information of the wall connection part in the above-mentioned indoor image. Among them, in the wall position discrimination stage, a convolutional neural network (CNN) is used as the discriminator to score the straight line positions extracted in the feature extraction, and the straight lines are determined by the score to be the connecting part between the walls. The neural network iterates using the following loss function:

是像素样本i的真实判定结果，P_i为像素样本i预测属于墙角的概率，n为图像中的像素点总数。该神经网络训练实为对该损失函数计算值最小化的过程。Among them, f _loss is the loss function, which is formed by adding two parts: the first part is the loss of boundary pixels belonging to the room layout part, and the second part is the loss of pixels belonging to the connection part between walls. α and β are self-defined parameters, which respectively represent the weight of the two parts of the pixel calculation results in the loss value, which need to be determined according to the actual model training parameters;

is the true judgment result of pixel sample i, P _i is the predicted probability of pixel sample i belonging to the corner of the wall, and n is the total number of pixels in the image. The neural network training is actually the process of minimizing the calculated value of the loss function.

In order to optimize the generated wall information to improve the reconstruction accuracy, the following scoring formula is used to evaluate the final generated wall information:

Among them, Score(L) is the score, C is the set of detected angle coordinates (x, y) between walls; l _c ∈ C indicates that l _c is an angle pixel coordinate point to be detected in the set C; l _f is the set of all detected floor pixel coordinate points (x, y), _{lf is a floor pixel coordinate point to be detected in the set L f ;} P _corner , P _floor respectively represent for each detected wall or The floor part, the probability that this pixel belongs to this target. ω _corner and ω _floor are weight parameters, which need to be determined through experiments for different data sets, and the two values can also be determined through Grid Search.

Step 106 , obtaining indoor layout information based on the height of the camera from the ground and the information on the connecting portion of the wall.

In some embodiments, firstly, according to the height of the camera from the ground, the xy coordinates of the upper and lower corners in the image to the xyz coordinates in the three-dimensional environment are calculated through projection transformation. Then, the xyz coordinates of the floor and the roof are calculated once for each wall corner, and the average of all the above values is taken as the reconstructed roof and floor positions. Then, all the identified possible wall positions are traversed and screened, and only a series of walls that are perpendicular to each other between the walls are retained as candidates. Then, vote on the selected candidate walls, and delete the walls whose length is less than 0.16m and the angle formed by the two ends of the wall and the camera position is less than 5°. For the distortion phenomenon in the panoramic image, the image stretching process is performed to improve the reconstruction accuracy. Finally, the indoor layout information is obtained.

Step 107: Input the above feature information into a pre-trained target detection network, and output the indoor object information in the above indoor image.

In the indoor object recognition stage, the R-FCN neural network is used for object recognition and localization. The input of the R-FCN neural network is the feature tensor output in step 103. In addition to the feature layer, the network also includes three parts: the region candidate network RPN (Region Proposal Network), the position-sensitive prediction layer, and an RoI pooling layer.

卷积层最终对每个物体分类输出k²个概率。随后对概率谱进行池化，该池化操作计算公式如下：Among them, the original image is cut in a grid format, and several regions of interest (RoI) that may contain the target to be identified are split. In order to detect the targets in each area and solve the translation invariance problem of neural network features, so that the neural network can maintain the original accuracy when migrating to a new task, R-FCN introduces a position-sensitive pooling operation. The specific method is: for each region candidate frame generated, it is divided into k*k position-sensitive regions, and the size of each region is about

The convolutional layer finally outputs k ² probabilities for each object classification. Then the probability spectrum is pooled, and the calculation formula of the pooling operation is as follows:

为该RoI区域(i，j)坐标处对于c个类别的评分图谱，(x0，y0)为该RoI的中心坐标。

表示ROI中(x,y)相对坐标转换为绝对坐标后，属于第c个类别的评分值Among them, rc (i, j, Θ) is the pooling calculation result for category _c at the coordinates of the RoI region (i, j), where Θ is the learning parameter of the network. (x, y) is the pixel value in the (i, j)th grid point.

is the score map for c categories at the coordinates of the RoI region (i, j), and (x0, y0) is the center coordinate of the RoI.

Indicates the score value belonging to the c-th category after the (x, y) relative coordinates in the ROI are converted to absolute coordinates

When training the R-FCN network, the following loss function is used to evaluate the performance of the neural network:

表示待识别目标s属于分类c^*的概率；L_regression(t，t^*)表示计算该边界框对于正确的标注(ground truth)的回归损失，t^*为对应的ground truth box；

为分类的交叉熵损失；L_regression为检测到目标的边界区域回归损失，λ为不同分类间的平衡权重。若要使得训练时各待检测目标具有相同权重，可设置λ＝1。Among them, t _{x, y, w, h} is the current bounding box (bounding box), x, y, w, h represent the center coordinates, length and width of the box, respectively; s is the box t _{x, y, w, h} The target to be identified in the inner; c ^* is the real classification of the RoI area of the training set, c ^* =0 means the classification is background; c ^* >0 means that the expression in the square brackets is true, the value is 1, otherwise it is 0;

Represents the probability that the target s to be recognized belongs to the classification c ^* ; L _regression (t, t ^* ) represents the regression loss of the bounding box for the correct annotation (ground truth), and t ^* is the corresponding ground truth box;

is the cross-entropy loss of classification; L _regression is the regression loss of the boundary region of the detected target, and λ is the balance weight between different classifications. To make each target to be detected have the same weight during training, λ=1 can be set.

After using R-FCN to detect the target in the image, output the center coordinate P _obj of each recognized target and the length and width information of the object position.

Step 108 , based on the above-mentioned indoor object information and the above-mentioned information of the connecting part of the wall, obtain the position information of the indoor object.

In some embodiments, the two-dimensional panoramic image is first cropped according to the position coordinates of each wall, and the position of the wall is identified by using the above-mentioned depth residual network, and the position of each wall connection in the two-dimensional panoramic image is detected. The coordinates P _wall , and the detected center coordinates P _obj of each target area frame are extracted, and the relative coordinates of P _obj to each P _wall are calculated to obtain the position of the object in the reconstructed three-dimensional scene.

Step 109: Complete the construction of the indoor and virtual reality interaction scene according to the indoor layout information and the position information of the indoor objects.

In some embodiments, the construction of the virtual reality interaction scene is completed by using the indoor layout information obtained in step 106 and the position information of indoor objects obtained in step 108 .

Continuing to refer to FIG. 2 , as an implementation of the above methods in the above figures, the present disclosure provides some embodiments of an apparatus for constructing an indoor VR scene with joint constraints of image semantics and scene geometry. These apparatus embodiments are similar to those described above in FIG. 1 . Corresponding to the method embodiments, the apparatus can be specifically applied to various electronic devices.

As shown in FIG. 2, the apparatus 200 for generating classification information in some embodiments includes:

Acquisition unit 201 , processing unit 202 , first input and output unit 203 , detection unit 204 , second input and output unit 205 , first determination unit 206 , third input and output unit 207 , second determination unit 208 , third determination unit 209 . The acquisition unit is configured to obtain indoor images and the height of the camera from the ground when the indoor images are captured; the processing unit is configured to preprocess the indoor images; the first input and output unit is configured to The above-mentioned indoor image is input into the pre-trained deep residual network, and the characteristic information of the above-mentioned indoor image is output; the detection unit is configured to detect the information of the straight line in the above-mentioned characteristic information; the second input and output unit is configured to The information is input into the convolutional neural network, and the information of the wall connection part in the indoor image is output, wherein the wall connection part is a straight line connected between the walls in the indoor image; the first determination unit is configured The indoor layout information is obtained based on the height of the camera from the ground and the information of the connecting part of the wall; the third input and output unit is configured to input the above feature information into the pre-trained target detection network, and output the above indoor image. Indoor object information; a second determining unit, configured to obtain position information of indoor objects based on the above-mentioned indoor object information and information of the above-mentioned wall connecting parts; a third determining unit, configured to be based on the above-mentioned indoor layout information and the above-mentioned indoor objects location information to complete the construction of indoor and virtual reality interaction scenes.

It can be understood that the units recorded in the apparatus 200 correspond to the respective steps in the method described with reference to FIG. 1 . Therefore, the operations, features, and beneficial effects described above with respect to the method are also applicable to the apparatus 200 and the units included therein, and details are not described herein again.

Referring next to FIG. 3 , a schematic structural diagram of an electronic device server 300 suitable for implementing some embodiments of the present disclosure is shown. The server shown in FIG. 3 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 3, an electronic device 300 may include a processing device (eg, a central processing unit, a graphics processor, etc.) 301 that may be loaded into random access according to a program stored in a read only memory (ROM) 302 or from a storage device 308 Various appropriate actions and processes are executed by the programs in the memory (RAM) 303 . In the RAM 303, various programs and data necessary for the operation of the electronic device 300 are also stored. The processing device 301 , the ROM 302 , and the RAM 303 are connected to each other through a bus 304 . An input/output (I/O) interface 305 is also connected to bus 304 .

Typically, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 307 of a computer, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. Communication means 309 may allow electronic device 300 to communicate wirelessly or by wire with other devices to exchange data. While FIG. 3 shows electronic device 300 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in FIG. 3 can represent one device, and can also represent multiple devices as needed.

In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method illustrated in the flowchart. In some such embodiments, the computer program may be downloaded and installed from the network via the communication device 309 , or from the storage device 308 , or from the ROM 302 . When the computer program is executed by the processing device 301, the above-mentioned functions defined in the methods of some embodiments of the present disclosure are performed.

It should be noted that, in some embodiments of the present disclosure, the computer-readable medium described above may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the foregoing two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In some embodiments of the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. Rather, in some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code therein. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the client and server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can communicate with digital data in any form or medium (eg, a communications network) interconnected. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks), as well as any currently known or future development network of.

The above-mentioned computer-readable medium may be included in the above-mentioned apparatus; or may exist alone without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the indoor image and the height of the camera from the ground when shooting the above-mentioned indoor image; Perform preprocessing; input the preprocessed indoor image into a pre-trained deep residual network, and output the feature information of the indoor image; detect the information of the straight line in the feature information; input the information of the straight line into the convolutional neural network In the network, the information of the wall connection part in the above-mentioned indoor image is output, wherein the above-mentioned wall connection part is a straight line connected between the walls in the above-mentioned indoor image; based on the height of the camera from the ground and the wall connection part. information to obtain indoor layout information; input the above feature information into a pre-trained target detection network, and output the indoor object information in the above indoor image; based on the above indoor object information and the above information of the wall connection part, obtain the position information of the indoor object ; Complete the construction of the indoor and virtual reality interaction scene according to the above-mentioned indoor layout information and the above-mentioned position information of the indoor objects.

Computer program code for carrying out operations of some embodiments of the present disclosure may be written in one or more programming languages, including object-oriented programming languages—such as Java, Smalltalk, C++, or a combination thereof, Also included are conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider to via Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The units described in some embodiments of the present disclosure may be implemented by means of software, and may also be implemented by means of hardware. The described unit may also be set in the processor, for example, it may be described as: an acquisition unit, a processing unit, a first input and output unit, a detection unit, a second input and output unit, a first determination unit, a third input and output unit, The second determination unit and the third determination unit. The names of these units do not limit the unit itself in some cases. For example, the acquisition unit can also be described as "a unit that obtains indoor images and the height of the camera from the ground when capturing the above indoor images."

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chips (SOCs), Complex Programmable Logical Devices (CPLDs) and more.

The above descriptions are merely some preferred embodiments of the present disclosure and illustrations of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solution formed by the specific combination of the above-mentioned technical features, and should also cover, without departing from the above-mentioned inventive concept, the above-mentioned Other technical solutions formed by any combination of technical features or their equivalent features. For example, a technical solution is formed by replacing the above-mentioned features with the technical features disclosed in the embodiments of the present disclosure (but not limited to) with similar functions.

Claims (9)

1. A method for constructing an indoor VR scene with joint constraints of image semantics and scene geometry, comprising: obtaining an indoor image and the height of the camera from the ground when taking the indoor image; preprocessing the indoor image; Inputting the preprocessed indoor image into a pre-trained deep residual network, and outputting feature information of the indoor image; detecting the information of the straight line in the feature information; Inputting the information of the straight line into the convolutional neural network, and outputting the information of the wall connecting part in the indoor image, wherein the wall connecting part is the straight line connecting the walls in the indoor image; Obtain indoor layout information based on the height of the camera from the ground and the information of the connecting part of the wall; Inputting the feature information into a pre-trained target detection network, and outputting indoor object information in the indoor image; Based on the information of the indoor object and the information of the connecting part of the wall, obtain the position information of the indoor object; The construction of the indoor and virtual reality interaction scene is completed according to the indoor layout information and the position information of the indoor objects. 2. The method according to claim 1, wherein the preprocessing of the indoor image comprises: The indoor image is adjusted to a predetermined resolution. 3. The method according to claim 1, wherein the detecting the information of the straight line in the feature information comprises: The position of the straight line in the feature information is detected by a straight line detection algorithm, and the position information of the straight line is stored. 4. The method according to claim 3, wherein, inputting the information of the straight line into a convolutional neural network, and outputting the information of the connected part of the wall in the indoor image, comprising: Using the convolutional neural network as a discriminator, the stored position of the straight line is scored, and the straight line of the connecting part of the wall is obtained by scoring. 5. The method according to claim 1, wherein the inputting the feature information of the indoor image into a pre-trained target detection network, and outputting the indoor object information in the indoor image, comprises: Using a target detection network to detect indoor objects on the indoor image, and output the center coordinates, length and width information of the indoor objects in the indoor image. 6. The method according to claim 5, wherein the construction of the indoor and virtual reality interaction scene is completed according to the indoor layout information and the position information of the indoor objects, comprising: cropping the indoor image; Detecting the position coordinates of the wall connection part in the indoor image through the information of the wall connection part output by the convolutional neural network; extracting center coordinates of indoor objects in the indoor image; Obtain the position of the indoor object through the relationship between the position coordinates of the wall connection portion in the indoor image and the center coordinates of the indoor object; Based on the position of the indoor object and the information of the connecting part of the wall, the construction of the indoor and virtual reality interaction scene is completed. 7. An apparatus for constructing an indoor VR scene with joint constraints of image semantics and scene geometry, comprising: an acquisition unit configured to acquire an indoor image and the height of the camera from the ground when the indoor image is captured; a processing unit configured to preprocess the indoor image; a first input-output unit, configured to input the pre-processed indoor image into a pre-trained deep residual network, and output feature information of the indoor image; a detection unit, configured to detect information of straight lines in the feature information; The second input and output unit is configured to input the information of the straight line into the convolutional neural network, and output the information of the connected part of the wall in the indoor image, wherein the connected part of the wall is in the indoor image is the straight line connecting the walls; a first determining unit, configured to obtain indoor layout information based on the height of the camera from the ground and information on the connecting portion of the wall; a third input-output unit, configured to input the feature information into a pre-trained target detection network, and output indoor object information in the indoor image; a second determining unit, configured to obtain the position information of the indoor object based on the indoor object information and the information of the wall connecting part; The third determining unit is configured to complete the construction of the indoor and virtual reality interaction scene according to the indoor layout information and the position information of the indoor objects. 8. An electronic device comprising: one or more processors; a storage device for storing one or more programs; The one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method of any of claims 1-6. 9. A computer-readable medium having stored thereon a computer program, wherein the program, when executed by a processor, implements the method of any one of claims 1-6.

Images