JP7001138B2 - How to determine the location of items, equipment and computer readable storage media - Google Patents

Description

The present invention relates to the field of image processing technology, and specifically relates to a method for determining an item placement position, an apparatus, and a computer-readable storage medium.

Panorama, also known as 3D real-world, is an emerging rich media technology that is "manipulable and interactive", the biggest difference from traditional streaming media such as video, audio and images. Panorama virtual reality (also called virtual reality) is a real scene virtual reality technology based on panoramic images, and computer technology enables all-round interactive viewing of the restored display of the real scene. With the support of the playback plug-in, you can adjust the direction of looking around to give the user the feeling of being in a live environment and get an experience close to the real scene.

As one of the application scenes of panoramic virtual reality, virtual goods can be placed in the panoramic image. For example, a panoramic camera can take an indoor image of a vacant room, and then virtual furniture can be placed in the vacant room to make the room look more attractive. For example, many of the properties currently on sale or waiting for rent taken by real estate agents are vacant rooms, and if you place some virtual furniture in these rooms and then display them to the customer, you can imagine that the customer also moved in. As a result, the transaction rate can be increased and the transaction cycle can be shortened.

The business of arranging virtual furniture in panoramic images is welcomed by many real estate companies, but the current business is mainly done by hand, so it takes a lot of labor and time, and the business scale is further increased. Expansion is very difficult. Therefore, a solution that can automatically place virtual goods in the panoramic image is required, and the first problem to be solved in the above solution is to determine a suitable position for item placement.

The technical problem to be solved by the present invention provides a method for determining an item placement position, a device and a computer-readable storage medium, can determine a suitable position for placing an item in space, and is a virtual goods. It is to provide some background information on the automatic placement of.

In order to solve the technical problem, in the embodiment of the present invention, the step of acquiring the first image of the interior space and the vertical correction of the first image are performed, and the intersection between the adjacent wall surfaces in the first image is performed. By adjusting the line in the vertical direction to acquire the second image and extracting the features of the second image and analyzing the features, the position information of the ground-wall boundary line in the interior space and the wall-wall Steps for acquiring boundary line existence possibility information and item placement possibility information, position information of the ground-wall boundary line in the interior space, wall-wall boundary line existence possibility information, and item placement possibility information. Provides a step of determining the position of the ground-wall boundary suitable for item placement, and a method of determining the placement position of the item, including.

Preferably, based on the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement, the position of the ground-wall boundary line suitable for the item arrangement is determined. The steps to determine are a step of finding the position of the peak based on the possibility information of the existence of the wall-wall boundary line in the interior space and acquiring the position information of the wall-wall boundary line in the interior space, and the indoor space. Based on the position information of the ground-wall boundary line in the space and the possibility information of the item placement, the step of determining the reliability indicating whether the ground-wall boundary line is suitable for the item placement and the wall in the interior space- Includes a step of determining the location of the ground-wall border suitable for item placement based on the location information of the wall border and the reliability indicating whether the ground-wall border is suitable for item placement. ..

Preferably, the ground-wall boundary line suitable for item placement is based on the position information of the wall-wall boundary line in the interior space and the reliability indicating whether or not the ground-wall boundary line is suitable for item placement. The step of determining the position is a step of dividing the ground-wall boundary line in the indoor space into the ground-wall boundary line of each wall surface by using the position information of the wall-wall boundary line in the indoor space, and the step of dividing each wall surface. From the ground-wall boundary line, select a continuous portion whose length is longer than the predetermined length threshold and whose reliability is larger than the predetermined reliability threshold, and obtain the position of the ground-wall boundary line suitable for item placement. Including steps to do.

Preferably, by extracting the features of the second image and analyzing the features, the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement are obtained. In the step of acquiring, the second image is input to a pre-trained neural network model to extract features, and the neural network model is used to provide position information of a ground-wall boundary line and a wall-wall boundary in the interior space. Includes a step of predicting and acquiring line existence possibility information and item placement possibility information.

Preferably, the position information of the ground-wall boundary line, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement in the indoor space are all one-dimensional arrays whose width is equal to the width of the second image. The k-th element in the one-dimensional array indicates a value corresponding to the k-th image column of the second image.

Preferably, before the step of acquiring the first image of the interior space, a step of acquiring a plurality of vertically corrected sample images and tags thereof, wherein the tag is the ground in the interior space corresponding to the sample image. Trained and trained neural networks and weights using steps and the sample images and their tags, including wall boundary location information, wall-wall boundary existence potential information, and item placement potential information. Further includes steps to obtain a neural network model of the.

Preferably, in the tag, the item placement possibility information is labeled based on whether or not the wall in the corresponding k-th column of image is cut.

Preferably, the item is furniture.

In an embodiment of the present invention, an image acquisition unit that acquires a first image of an interior space and a vertical correction are performed on the first image to vertically adjust an intersection line between adjacent wall surfaces in the first image. By extracting the features of the second image and analyzing the features of the image correction unit that acquires the second image, the position information of the ground-wall boundary line in the interior space and the existence of the wall-wall boundary line are present. Based on the feature analysis unit that acquires the possibility information and the possibility information of item placement, the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of item placement. Further, a positioning unit for determining the position of the ground-wall boundary line suitable for item placement, and a device for determining the placement position of the item including the item are further provided.

Preferably, the positioning unit finds the position of the peak based on the possibility information of the existence of the wall-wall boundary line in the interior space, and acquires the position information of the wall-wall boundary line in the interior space. Based on the position information of the ground-wall boundary line in the interior space and the possibility information of the item arrangement, the reliability indicating whether the ground-wall boundary line is suitable for the item arrangement is determined, and the wall in the interior space- Based on the position information of the wall boundary line and the reliability indicating whether or not the ground-wall boundary line is suitable for the item placement, the position of the ground-wall boundary line suitable for the item placement is determined.

Preferably, the positioning unit uses the position information of the wall-wall boundary line in the interior space to divide the ground-wall boundary line in the interior space into the ground-wall boundary line of each wall surface, and the position of each wall surface. From the ground-wall boundary line, select a continuous portion whose length is longer than the predetermined length threshold and whose reliability is larger than the predetermined reliability threshold, and obtain the position of the ground-wall boundary line suitable for item placement. do.

Preferably, the feature analysis unit inputs the second image into a pre-trained neural network model to extract features, and the neural network model provides ground-wall boundary position information, walls in the interior space. -Predict and acquire information on the possibility of existence of wall boundaries and information on the possibility of item placement.

An embodiment of the present invention is a computer-readable storage medium, in which a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the arrangement position of the item is described. Further provides a computer-readable storage medium that implements the steps of the determination method.

Compared with the prior art, the method for determining the item placement position, the device, and the computer-readable storage medium according to the embodiment of the present invention determine a specific position suitable for placing the item in a certain interior space. And provided some background on the automatic placement of virtual goods. Further, in the embodiment of the present invention, when the position of the item arrangement is determined, the specific position suitable for the arrangement of the specific item can be selected in consideration of the limitation of the width of the item.

In order to clarify the technical proposal of the embodiment of the present invention, the drawings used for the description of the embodiment of the present invention will be briefly described below. Obviously, the drawings described below are merely embodiments of the present invention, and it is self-evident to those skilled in the art that other drawings obtained from these drawings are possible, provided that no creative work is performed. be.
It is a flowchart of the method of determining the item arrangement position of the Example of this invention. It is an application example figure of the position information of the ground-wall by the Example of this invention. It is an application example figure of the possibility information of the existence of the wall-wall boundary line by the Example of this invention. It is an application example figure of the possibility information of item arrangement by an Example of this invention. It is an application example figure of the wall-wall boundary line determined by the Example of this invention. It is an application example figure which fused the position information of the ground-wall boundary line and the possibility information of item arrangement by the Example of this invention. It is an application example figure of the position determination of the item arrangement by the Example of this invention. It is a structural application example figure of the neural network model adopted in the Example of this invention. It is another structural schematic diagram of the device which determines the item arrangement position of the Example of this invention. It is another structural schematic diagram of the device for determining the item arrangement position of the Example of this invention.

In order to further clarify the problems, technical proposals and advantages to be solved by the present invention, the drawings and specific examples will be described in detail below in combination. In the drawings below, the specific arrangement and specific details of the module are only to help you understand the whole embodiment of the invention. Therefore, those skilled in the art should be aware that various modifications and modifications to the examples described herein do not deviate from the scope and spirit of the present invention. Also, for clarity and brevity, the description of known functions and structures is omitted.

The "one embodiment" or "one embodiment" referred to throughout the specification means that a particular feature, structure or property associated with an embodiment is included in at least one embodiment of the invention. Should be understood. Therefore, "in one embodiment" or "in one embodiment" appearing in various parts of the specification does not always refer to the same embodiment. In addition, these and specific features, structures, and properties can be combined into one or more embodiments in any suitable manner.

In various embodiments of the present invention, the magnitude of each process number below does not mean before or after the execution order. The order of execution of each process depends on its function and internal logic, and is not limited to the process of carrying out the embodiments of the present invention.

An embodiment of the present invention provides a method for determining a position suitable for arranging an item in a space, and the method can determine a specific position suitable for arranging an item in a specific interior space. This can provide prior knowledge for the automatic placement of virtual goods. In the embodiment of the present invention, the item has a certain volume, and when it is arranged in an indoor space, it is necessary to dedicate a certain space. The item can be specifically furniture (sofa, table or chair, closet, etc.), for example, equipment such as a copier or printer often placed in an office, various products placed in an exhibition hall, or a specific room. It can also be other equipment that can be placed in the space, and the embodiments of the present invention do not limit this in detail.

As shown in FIG. 1, the sequence labeling method according to the embodiment of the present invention includes the following steps.

In step 101, the first image of the interior space is acquired.

Here, the first image can be a panoramic image (for example, a regular-distance cylindrical diagram) obtained by photographing a certain indoor space with a panoramic camera, and is a perspective view taken with a general angle-of-view camera. There can be. In order to facilitate the subsequent processing, the items to be arranged in the interior space are reduced as much as possible, for example, the interior space is an empty room, and furniture or other items are not arranged.

In step 102, vertical correction is performed on the first image, and the line of intersection between adjacent wall surfaces in the first image is adjusted in the vertical direction to acquire the second image.

In a building structure, the lines of intersection between adjacent walls (the surfaces of walls that are perpendicular to each other in the horizontal direction) are usually located in the vertical direction (ie, in the direction perpendicular to the horizontal plane). In the captured image, there is a possibility that the line of intersection deviates from the vertical direction due to factors such as the imaging angle and deformation. In an embodiment of the present invention, in step 102, the first image is vertically corrected, and the line of intersection between adjacent wall surfaces is adjusted in the vertical direction to acquire the corrected image, which is referred to as a second image here. In order to specifically realize the vertical correction, the vertical / distortion correction algorithm of the existing technique can be referred to, and the embodiment of the present invention does not limit this in detail. Vertical correction allows the wall-wall boundaries of two adjacent walls in the second image to be vertical vertical lines in the image coordinate system.

In step 103, by extracting the features of the second image and analyzing the features, the position information of the ground-wall boundary line in the indoor space, the possibility information of the existence of the wall-wall boundary line, and the possibility of item placement are possible. Get information.

In the embodiment of the present invention, the above step 103 can be executed by utilizing the pre-trained neural network model. For example, the second image is input to the neural network model to extract features, and the neural network model provides information on the position of the ground-wall boundary line in the interior space and information on the possibility of the existence of the wall-wall boundary line. And the possibility information of item placement can be predicted and acquired. Among them, the ground-wall boundary line refers to the line of intersection between the ground and the wall surface in the interior space, and the wall-wall boundary line refers to the line of intersection between the wall surface and the wall surface in the interior space.

Here, the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement are all one-dimensional arrays whose widths are equal to the width of the second image. The k-th element in the one-dimensional array indicates a value corresponding to the k-th image column of the second image. For example, the second image is an image having a height × width of 512 × 1024, and includes 1024 pixel units in the width direction. In this case, the position information of the ground-wall boundary line in the indoor space is one 1 × 1024-dimensional array, and the k-th element in the array is the ground-wall boundary corresponding to the k-th column image column in the second image. The position information of the line is shown, and here, the k-th column image string refers to an image composed of pixels in the k-th column in the second image. Similarly, the possibility information of the existence of the wall-wall boundary line in the interior space is also one 1 × 1024-dimensional array, and the k-th element in the array is the wall corresponding to the k-th column image column in the second image. -Indicates the possibility information of the existence of the wall boundary line, and the possibility information of the item arrangement in the interior space is also one 1 × 1024 dimensional array, and the kth element in the array is the kth column image string in the second image. Indicates the possibility information of the item arrangement corresponding to.

In FIGS. 2 to 4, the second image is input to a pre-trained neural network model to extract features, and the neural network model is used to position information on the ground-wall boundary in the interior space and the wall. -Examples of predicting and acquiring information on the existence of wall boundaries and information on the possibility of item placement are shown. Among them, the horizontal axis of FIGS. 2 to 4 shows the image sequence, and the vertical axis shows, for example, the position information of the ground-wall boundary line, the possibility information of the existence of the wall-wall boundary line, and the possibility information of item placement. Shows the value of each prediction result, such as the value. Here, the position information of the ground-wall boundary line is already normalized to [-π / 2, π / 2], and the possibility information of the existence of the wall-wall boundary line is already normalized to [0, 1]. The item placement possibility information has already been normalized to [0, 1].

In step 104, the position of the ground-wall boundary line suitable for item placement is determined by the position information of the ground-wall boundary line in the indoor space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item placement. do.

Here, considering that the item has a certain width, the area corresponding to the wall surface that is too narrow is not suitable for the arrangement of the item due to the limitation of space. Therefore, in step 104, the embodiment of the present invention can determine a ground-wall boundary line suitable for the arrangement of items based on the plurality of parameters acquired in step 103, and specifically, the next step. (A) and (b) can be included.

In step (a), the step of finding the position of the peak based on the possibility information of the existence of the wall-wall boundary line in the interior space and acquiring the position information of the wall-wall boundary line in the interior space, and the interior space. Based on the position information of the ground-wall boundary line and the possibility of item placement in the above, the reliability that the ground-wall boundary line is suitable for the item placement is determined.

Here, in the embodiment of the present invention, the larger the value of the possibility information of the existence of the wall-wall boundary line corresponding to a certain image sequence, the greater the possibility that the wall-wall boundary line exists in the image sequence. Therefore, it is possible to determine the position of the wall-wall boundary line by finding the peak position of the possibility information of the existence of the wall-wall boundary line in the indoor space. Figure 5 shows an example of determining the position of the wall-to-wall boundary based on the peak position. The wall-wall boundary lines 51 to 57 in FIG. 5 are wall-wall boundary lines determined based on the peak position of the possibility information of the existence of the wall-wall boundary line.

In addition, in the embodiment of the present invention, the position information of the ground-wall boundary line in the interior space and the possibility information of item placement are fused to determine the reliability that the ground-wall strengthening line is suitable for the placement of items. can do. For example, the confidence that the ground-wall boundary line corresponding to the image string is given to the item placement possibility information corresponding to each image string and the ground-wall boundary line corresponding to the image string is suitable for the item placement. Be sex. Figure 6 shows an example in which the position information of the ground-wall boundary line in the interior space and the possibility information of item placement are fused. In this example, the ground-wall boundaries whose reliability is less than the preset reliability threshold are not displayed, only the ground-wall boundaries whose reliability is greater than the preset reliability threshold are shown.

In step (b), the position of the ground-wall boundary line suitable for item placement is determined by the position information of the wall-wall boundary line in the interior space and the reliability that the ground-wall boundary line is suitable for item placement. do.

Here, the position information of the wall-wall boundary line in the indoor space can be used to divide the ground-wall boundary line in the indoor space into the ground-wall boundary line of each wall surface. That is, the wall-wall boundary line is used to divide the ground-wall boundary line and divide it into the ground-wall boundary line of each wall surface. Further, from the ground-wall boundary line of each wall surface, a continuous portion whose length is longer than the preset length threshold value and whose reliability is larger than the preset reliability threshold value is selected, and the ground suitable for the placement of items is selected. -Get the position of the wall boundary.

FIG. 7 shows an example of the above step (b). Using the position information of the wall-wall boundary lines 51 to 57 in the indoor space, the ground-wall boundary line in the indoor space is divided into a plurality of wall surfaces, and the wall surfaces are 701 to 708, respectively. Since the ground-wall boundary line shown in FIG. 7 is a boundary line whose reliability is larger than the preset reliability threshold value, the length is longer than the ground-wall boundary line 711 to 716 of each displayed wall surface. By simply finding a continuous portion that is larger than the preset length threshold, you can obtain the position of the ground-wall boundary line suitable for item placement. On the wall surface 705, there are three consecutive ground-wall boundary lines 712 to 714 whose reliability is larger than the preset reliability threshold value, but all of them are truncated because their lengths are smaller than the preset threshold value. You can see that it has been done. On the other hand, the ground-wall boundary line 711 on the wall surface 704 and the ground-wall boundary line 715 on the wall surface 706 both meet the above-mentioned length requirements and are therefore regarded as suitable ground-wall boundary lines for the placement of items. become.

Through the above steps, the embodiments of the present invention were able to determine a specific position suitable for item placement in a certain interior space, and provided preliminary knowledge for automatic placement of virtual goods. In addition, in the embodiment of the present invention, in the above step, considering that the item usually has a certain width, a specific position suitable for the arrangement of the item is selected according to the width of the specific item. be able to.

In the embodiment of the present invention, the relevant information was acquired by using the pre-trained neural network model in step 103. Below, an embodiment of the present invention further provides a realization method for acquiring the neural network model by training. Specifically, prior to step 101, the embodiment of the present invention can acquire a plurality of vertically corrected sample images and tags thereof, and the tags referred to here can be labeled by an artificial method. The tag can include position information of the ground-wall boundary line in the interior space corresponding to the sample image, information on the possibility of existence of the wall-wall boundary line, and information on the possibility of item placement. Then, using the sample image and its tag, the neural network and weights are trained to obtain a trained neural network model.

As one of the realization methods, in the embodiment of the present invention, the position information of the ground-wall boundary line and the possibility information of the existence of the wall-wall boundary line can be the result of normalization processing. For example, the position information of the ground-wall boundary line uses the coordinate system of the image as the reference coordinate system, and then is normalized to [-π / 2, π / 2] collectively. The possibility information of the wall-wall boundary can be normalized to [0, 1]. The embodiments of the present invention further provide a realization method for normalizing the possibility information of the existence of a wall-wall boundary line. Specifically, the possibility information of the existence of the wall-wall boundary line of the current image string is 0.96 ^d , in which d is the wall-wall boundary closest to the column index of the image string and the image string. The distance from the image sequence in which the line exists.

Also, in the item placement possibility information, the value of the kth element usually depends on whether or not the wall in the kth column of the image is cut, and whether or not this is cut is based on the preset rules. Can be judged. For example, if a specific preset target (for example, a window, a closet, a door, etc.) exists in the k-th image column, it is determined that the wall is cut. If the target does not exist in the k-th image column, it is determined that the wall is not cut. In this case, there may be no items on the wall, or there may be other items that do not affect the placement of the items, such as switches or air conditioners. If the wall is cut, the value of the kth element can be set to 0 and vice versa. Of course, the values 0 and 1 above can be swapped, and only the meanings represented by the different values need to be defined in advance.

FIG. 8 shows an application example diagram of the structure of the neural network model that can be adopted in the embodiment of the present invention. Needless to say, the embodiments of the present invention do not limit the structure of the neural network model in detail, and FIG. 8 provides only an exemplary explanation using a convolutional neural network (ResNet-101 network).

In this example, the size of the input image is 512 x 1024 (height x width). The ResNet-101 network includes unsampling and transforming the four output characteristics of conv2_x, conv3_x, conv4_x and conv5_x, respectively, unsampling processing includes continuous 4-layer convolution and unsampling, and convolution operation is characteristic. The height of the convolution is compressed to 1/16, the size of the convolution kernel is 3 × 3, the step width is 2 × 1 (h × w), and one ReLU activation function is attached to each layer after the convolution operation. ing. Unsampling samples the width of the characteristic to the same width of 256. The transformation process unifies the size of the characteristic to 1 of the same height. Then, the four processed characteristics are connected along the dimension of the channel, and the size of the characteristics is newly adjusted to obtain the characteristics with a size of 1 × 256 × 1024 (c × h × w). The feature extraction task is complete.

After this, a fully connected layer is still required to complete the prediction of ground-wall boundary location, wall-wall boundary existence potential and item placement potential. The characteristic is 3 x 1024 (h x w) in size after two linear operations and can be regarded as three 1 x 1024 arrays. For the first 1 × 1024 array, the position information of the ground-wall boundary can be obtained by sending it to the L1 loss function layer. The second 1 × 1024 array can be sent to the binary cross-entropy loss function layer to obtain information on the possibility of wall-wall boundary existence. By sending the third 1 × 1024 array to the binary cross entropy loss function layer, the possibility information of item placement can be obtained.

Based on the above method, an embodiment of the present invention provided an apparatus for carrying out the above method, as shown in FIG. The item placement position determination device 900 provided by the embodiment of the present invention includes an image acquisition unit 901, an image correction unit 902, a feature analysis unit 903, and a position determination unit 904.

The image acquisition unit 901 acquires the first image of the interior space.

The image correction unit 902 performs vertical correction on the first image, adjusts the line of intersection between adjacent wall surfaces in the first image in the vertical direction, and acquires the second image.

The feature analysis unit 903 extracts the features of the second image and analyzes the features to obtain information on the position of the ground-wall boundary line in the interior space, information on the possibility of existence of the wall-wall boundary line, and item arrangement. Get the possibility information.

The position determination unit 904 is based on the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement, and the position of the ground-wall boundary line suitable for the item arrangement. To determine.

Through the above unit, the item placement position determination device 900 according to the embodiment of the present invention can determine a position suitable for item placement in space, and provides preliminary knowledge for automatic placement of virtual goods.

Preferably, the positioning unit 904 finds the position of the peak based on the possibility information of the existence of the wall-wall boundary line in the interior space, and acquires the position information of the wall-wall boundary line in the interior space. Based on the position information of the ground-wall boundary line in the interior space and the possibility information of item placement, the reliability that the ground-wall boundary line is suitable for the item placement is determined, and the position information of the wall-wall boundary line in the interior space and the position information of the wall-wall boundary line. With the reliability that the ground-wall boundary line is suitable for item placement, the position of the ground-wall boundary line suitable for item placement can be determined.

Preferably, the positioning unit 904 also utilizes the position information of the wall-wall boundary in the interior space to divide the ground-wall boundary in the interior space into the ground-wall boundary of each wall surface. From the ground-wall boundary of each wall, select a continuous portion whose length is longer than the preset length threshold and whose reliability is greater than the preset reliability threshold, and the ground-wall suitable for the placement of items. You can get the position of the border.

Preferably, the feature analysis unit 903 also inputs the second image into a pre-trained neural network model to extract features and, by the neural network model, the ground-wall boundary in the interior space. Position information, wall-wall boundary line existence possibility information, and item placement possibility information can be predicted and acquired.

Preferably, the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement are all one-dimensional arrays whose width is equal to the width of the second image. The k-th element in the one-dimensional array indicates a value corresponding to the k-th image column of the second image.

Preferably, the item placement position determination device 900 further includes the following units.

The model training unit (not shown in FIG. 9) acquires a plurality of vertically corrected sample images and their tags, and the tags are the position information of the ground-wall boundary line and the wall-in the interior space corresponding to the sample images. The sample image and its tag, which include the possibility information of the existence of the wall boundary line and the possibility information of the item arrangement, train the neural network and the weight, and acquire the trained neural network model.

Preferably, in the tag, the item placement possibility information is labeled according to whether or not the wall in the corresponding k-th column of images is cut. Whether or not the wall here is cut can be determined based on preset rules. For example, if a specific preset target exists in the k-th image column, it is determined that the wall is cut. If the target does not exist in the k-th image column, it is determined that the wall is not cut. If the wall is cut, the value of the kth element can be set to 0 and vice versa. Of course, the values 0 and 1 above can be swapped, and only the meanings represented by the different values need to be defined in advance.

As can be seen with reference to FIG. 10, an embodiment of the present invention further provides a hardware structure block diagram of an item placement position determination device. As shown in FIG. 10, the item placement position determination device 1000 includes a processor 1002 and a memory 1004 for storing a program.

Here, when the program is executed by the processor, the processor 1002 performs a step of acquiring a first image of the interior space and a vertical correction on the first image, and the adjacent wall surface in the first image is performed. The step of acquiring the second image by adjusting the intersection line between them in the vertical direction, and the position information of the ground-wall boundary line in the interior space by extracting the features of the second image and analyzing the features. Steps to acquire wall-wall boundary line existence possibility information and item placement possibility information, ground-wall boundary line position information in the interior space, wall-wall boundary line existence possibility information and item placement With the possibility information, the step of determining the position of the ground-wall boundary line suitable for the item placement is executed.

Further, as shown in FIG. 10, the item placement position determination device 1000 includes a network interface 1001, an input device 1003, a hard disk 1005, and a display device 1006.

Each of the interfaces and the device are connected to each other by a bus architecture. The bus architecture can include any number of interconnected buses and bridges. Specifically, one or a plurality of central processors (CPUs) represented by the processor 1002, and various circuits of the one or a plurality of memories represented by the memory 1004 are connected to one. The bus architecture can also be connected with peripherals and various other circuits such as voltage regulators and power management circuits. It is understandable that the bus architecture is for realizing connection communication between these components. The bus architecture includes a power bus, a control bus, and a status signal bus in addition to the data bus, all of which are known in the art and are not described in detail herein.

The network interface 1001 can be connected to a network (eg, the Internet, Ethernet, etc.) and collects / or receives sample data (including sample images and their tags) from the network and collects the first image. The data can be stored in the hard disk 1005.

The input device 1003 can receive various commands input by the operator and send them to the processor 1002 for execution. The input device 1003 may include a keyboard or click device (eg, a mouse, trackball), a touchpad, a touch panel, and the like.

The display device 1006 can display the result obtained by executing a command by the processor 1002, such as the position of the ground-wall boundary line suitable for item placement.

The memory 1004 is used to store programs and data necessary for executing the operation system and data such as intermediate results of the calculation process of the processor 1002.

It can be understood that the memory 1004 of the embodiment of the present invention may be a volatile memory or a non-volatile memory, or may include both a volatile memory and a non-volatile memory. The non-volatile memory can be electrically erased regardless of whether it is a read-only memory (ROM), a programmable read-only memory (PROM), or an erasable programmable read-only memory (EPROM). It may be a programmable read-only memory (EEPROM) or a flash memory. The volatile memory may be a random access memory (RAM) and is used as an external cache. The device and method memory 1004 described herein includes, but is not limited to, these and any other compatible type of memory.

In some embodiments, the memory 1004 stores the elements of an executable module or data structure, a subset or extension set thereof, an operating system 10041, an application program 10042.

The operating system 10041 includes various system programs such as a framework layer, a core layer, and a drive layer so as to realize various basic tasks and tasks based on hardware. The application program 10024 includes various application programs such as a browser so as to realize various application tasks. A program that implements the methods of the embodiments of the present invention can be included in the application program 10024.

The method according to the above embodiment of the present invention can be applied to or realized by the processor 1002. Processor 1002 may be an integrated circuit chip capable of processing signals. In the implementation process, each step of the above method can be completed by a hardware integrated logic circuit in processor 1002 or a software-type command. The processor 1002 can be a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device. It may be an individual gate or transistor logic device, or an individual hardware unit, and each method, step, and logic block diagram disclosed in the embodiments of the present invention can be realized or executed. The general purpose processor may be a microprocessor or any conventional processor. What can be done directly in combination with the steps of the method published from the embodiments of the present invention can be performed and completed by a hardware decoder processor, or by a combination of hardware and software modules within the decoder processor. To be completed. Software modules can be located on recording media mature in the art such as random memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers. The recording medium is in memory 1004, and processor 1002 reads the information in memory 1004 and combines it with its hardware to complete the steps of the above method.

It is understandable that these embodiments described herein can be achieved by hardware, software, firmware, middleware, microcode or a combination thereof. For hardware realization, process units are one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), and field programmable gate arrays. It can be realized by (FPGA), a general-purpose processor, a controller, a microcontroller, a microprocessor, another electronic module for realizing the above-mentioned function of the present application, or a combination.

With respect to the realization of software, the techniques described herein can be realized by modules (eg, processes, functions, etc.) that perform the functions described herein. The software code is stored in memory and can be executed by the processor. Memory can be implemented inside or outside the processor.

Preferably, when the program is executed by the processor 1002, the position of the peak is found by the possibility information of the existence of the wall-wall boundary line in the interior space, and the position information of the wall-wall boundary line in the interior space is obtained. The step to acquire, the step to determine the reliability that the ground-wall boundary line is suitable for the item arrangement based on the position information of the ground-wall boundary line in the indoor space and the possibility information of the item arrangement, and the wall in the indoor space. -The step of determining the position of the ground-wall boundary line suitable for the item placement by the position information of the wall boundary line and the reliability that the ground-wall boundary line is suitable for the item placement may be executed.

Preferably, when the program is executed by the processor 1002, the position information of the wall-wall boundary line in the interior space is used to divide the ground-wall boundary line in the interior space into the ground-wall boundary line of each wall surface. From the ground-wall boundary of each wall, select a continuous portion whose length is longer than the preset length threshold and whose reliability is greater than the preset reliability threshold, which is suitable for item placement. You may perform the step of getting the position of the ground-wall boundary line.

Preferably, when the program is executed by processor 1002, the second image is input to a pre-trained neural network model to extract features, and the neural network model allows the ground-wall in the interior space. You may perform the step of predicting and acquiring the position information of the boundary line, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement.

Preferably, the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement are all one-dimensional arrays whose width is equal to the width of the second image. The k-th element in the one-dimensional array indicates a value corresponding to the k-th image column of the second image.

Preferably, when the program is executed by the processor 1002, a plurality of vertically corrected sample images and tags thereof are acquired, and the tags are the position information of the ground-wall boundary line in the interior space corresponding to the sample image, the wall-. A step including information on the possibility of existence of a wall boundary line and information on the possibility of item placement, and a step of training a neural network and weights using the sample image and its tag to obtain a trained neural network model. , May be executed.

Preferably, in the tag, the item placement possibility information is labeled according to whether or not the wall in the corresponding k-th column of images is cut.

One of ordinary skill in the art can combine the units and algorithmic steps of the various examples described in the embodiments disclosed herein to be realized by electronic hardware or a combination of computer software and electronic hardware. I can understand. Whether these functions are executed by hardware or software is determined by the specific application of the technical plan and design constraints. Those skilled in the art can realize the above-mentioned functions by using different methods for specific applications, but such realization should not be regarded as beyond the scope of the present invention.

For convenience and brevity of description, one of ordinary skill in the art can clearly understand that the corresponding process in the embodiment of the above method can be referred to for the specific working process of the above system, apparatus and unit. do not do.

It should be understood that the methods and devices disclosed in the examples provided by the present invention are also feasible in other embodiments. For example, the device described above is only exemplary. For example, the division of the unit is only a logical functional division, and another division method may be adopted when actually realizing the unit. For example, multiple units or modules may be combined, aggregated into another system, some functions may be omitted, or they may not be executed. Further, the above-mentioned displayed or disclosed mutual connection or direct connection or communicable connection may be a connection via an interface, and an indirect connection or a communicable connection between devices or units may be used. It may be an electrical or mechanical or other form of connection.

The unit described as the separating member may or may not be physically separated. The member displayed as a unit may or may not be a physical unit. That is, they may be located in the same place or may be distributed on a plurality of network units. The object of the embodiment of the present invention can be realized by selecting some or all the units thereof according to the actual needs.

Further, each functional unit in each embodiment of the present invention may be assembled into one process unit, each unit may be physically present independently, or two or more units may be present. May be combined into one unit.

The function is realized in the form of a software functional unit and, when sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the essence of the technical plan of the present invention, the part that contributes to the prior art, or a part of the technical plan can be realized in the form of a software product. The computer software product is stored on a recording medium and may be a few commands that cause a computer device (which may be a personal computer, server, network equipment, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. Includes. The recording medium includes various media such as a USB memory, a removable disk, a ROM, a RAM, a magnetic disk, or an optical disk, which can store a program code.

The above description is merely a specific embodiment of the present invention, and the scope of protection of the present invention is not limited to this. Within the technical scope disclosed in the present invention, changes or substitutions easily conceived by those skilled in the art should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should conform to the scope of protection of the right requirement.

Claims (13)

Steps to get the first image of the interior space,
A step of performing vertical correction on the first image and vertically adjusting the line of intersection between adjacent wall surfaces in the first image to acquire a second image.
By extracting the features of the second image and analyzing the features, the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement are acquired. Steps and
A step of determining the position of the ground-wall boundary line suitable for item placement based on the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item placement. And, a method of determining the placement position of an item, characterized by including. A step of determining the position of the ground-wall boundary line suitable for item placement based on the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item placement. teeth,
A step of finding the position of the peak based on the possibility information of the existence of the wall-wall boundary line in the interior space and acquiring the position information of the wall-wall boundary line in the interior space.
Based on the position information of the ground-wall boundary line in the interior space and the possibility information of the item placement, the step of determining the reliability indicating whether the ground-wall boundary line is suitable for the item placement, and the step.
The position of the ground-wall boundary suitable for item placement is determined based on the position information of the wall-wall boundary line in the interior space and the reliability indicating whether or not the ground-wall boundary line is suitable for item placement. The method for determining the placement position of an item according to claim 1, wherein the item is included in the step. The position of the ground-wall boundary suitable for item placement is determined based on the position information of the wall-wall boundary line in the interior space and the reliability indicating whether or not the ground-wall boundary line is suitable for item placement. Steps to do
A step of dividing the ground-wall boundary line in the indoor space into the ground-wall boundary line of each wall surface by using the position information of the wall-wall boundary line in the indoor space.
From the ground-wall boundary line of each wall surface, a continuous portion whose length is longer than a predetermined length threshold value and whose reliability is larger than the predetermined reliability threshold value is selected, and the ground-wall boundary line suitable for item placement is selected. The method for determining the placement position of an item according to claim 2, further comprising a step of acquiring a position. By extracting the features of the second image and analyzing the features, the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement are acquired. The step is
The second image is input to a pre-trained neural network model to extract features, and the neural network model provides information on the position of the ground-wall boundary line in the interior space and information on the possibility of the existence of the wall-wall boundary line. The method for determining an item placement position according to any one of claims 1 to 3, further comprising a step of predicting and acquiring item placement possibility information. The position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement are all displayed in a one-dimensional array whose width is equal to the width of the second image. ,
The method for determining an arrangement position of an item according to claim 4, wherein the k-th element in the one-dimensional array indicates a value corresponding to the k-th image column of the second image. Before the step of getting the first image of the interior space,
It is a step of acquiring a plurality of vertically corrected sample images and their tags, and the tags are information on the position of the ground-wall boundary line in the interior space corresponding to the sample image and information on the possibility of the existence of the wall-wall boundary line. And steps, including potential item placement information,
The determination of the placement position of the item according to claim 4, further comprising a step of training a neural network and weights using the sample image and its tag to obtain a trained neural network model. Method. The item placement position according to claim 6, wherein in the tag, the item placement possibility information is labeled based on whether or not the wall in the corresponding k-th column of the image row is cut. How to determine. The method for determining an arrangement position of an item according to any one of claims 1 to 3, wherein the item is furniture. An image acquisition unit that acquires the first image of the interior space,
An image correction unit that performs vertical correction on the first image and vertically adjusts the line of intersection between adjacent wall surfaces in the first image to acquire a second image.
By extracting the features of the second image and analyzing the features, the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item arrangement are acquired. Feature analysis unit and
A position that determines the position of the ground-wall boundary line suitable for item placement based on the position information of the ground-wall boundary line in the interior space, the possibility information of the existence of the wall-wall boundary line, and the possibility information of the item placement. A determination unit and a device for determining the placement position of an item, including. The position-fixing unit is
Based on the possibility information of the existence of the wall-wall boundary line in the indoor space, the position of the peak is found, and the position information of the wall-wall boundary line in the indoor space is acquired.
Based on the position information of the ground-wall boundary line in the interior space and the possibility information of the item arrangement, the reliability indicating whether the ground-wall boundary line is suitable for the item arrangement is determined.
The position of the ground-wall boundary suitable for item placement is determined based on the position information of the wall-wall boundary line in the interior space and the reliability indicating whether or not the ground-wall boundary line is suitable for item placement. The device for determining the placement position of an item according to claim 9. The position-fixing unit is
Using the position information of the wall-wall boundary line in the interior space, the ground-wall boundary line in the interior space is divided into the ground-wall boundary line of each wall surface.
From the ground-wall boundary line of each wall surface, a continuous portion whose length is longer than a predetermined length threshold value and whose reliability is larger than the predetermined reliability threshold value is selected, and the ground-wall boundary line suitable for item placement is selected. The device for determining an arrangement position of an item according to claim 10, wherein the position is acquired. The feature analysis unit inputs the second image into a pre-trained neural network model to extract features, and the neural network model uses the ground-wall boundary line position information and wall-wall boundary in the interior space. The device for determining an item placement position according to any one of claims 9 to 11, wherein the line existence possibility information and the item placement possibility information are predicted and acquired. A computer-readable storage medium
A computer program is stored in the computer-readable storage medium, and the computer program is stored in the computer-readable storage medium.
A computer-readable storage medium, characterized in that, when the computer program is executed by a processor, the steps of the method of determining the placement position of any one of claims 1 to 8 are realized.

Images