WO2019196871A1

WO2019196871A1 - Modeling method and related device

Info

Publication number: WO2019196871A1
Application number: PCT/CN2019/082082
Authority: WO
Inventors: 叶青
Original assignee: Oppo广东移动通信有限公司
Priority date: 2018-04-12
Filing date: 2019-04-10
Publication date: 2019-10-17
Also published as: CN110378993A

Abstract

Disclosed in the present application are a modeling method and a related device, the method comprising: acquiring N depth images of a target space; according to a point of interest algorithm, determining at least one point of interest in each depth image so as to obtain a plurality of points of interest; determining relevant information of each point of interest, the relevant information comprising depth information; according to the relevant information corresponding to the plurality of points of interest and a space modeling algorithm, establishing a model of the target space. By using an embodiment of the present application, a three-dimensional space may be modeled merely by means of a mobile terminal.

Description

Modeling method and related device

Technical field

The present application relates to the field of electronic technologies, and in particular, to a modeling method and related devices.

Background technique

With the increasing popularity of mobile terminals such as smart phones, smart phones have become an inseparable part of the lives of smartphone users. Users can take photos, image processing, etc. through their smartphones. At present, modeling a three-dimensional space (such as a closed space and a semi-enclosed space) usually requires a combination of a mobile terminal and an external device, and the operation is complicated.

Summary of the invention

The embodiment of the present application provides a modeling method and related apparatus for providing a convenient three-dimensional space modeling manner.

In a first aspect, an embodiment of the present application provides a modeling method, including:

Obtaining N depth images of the target space, the N being an integer greater than 1;

Determining at least one point of interest in each of the depth images according to a point of interest algorithm to obtain a plurality of points of interest;

Determining related information of each of the points of interest, the related information including depth information;

And establishing a model of the target space according to related information and a spatial modeling algorithm corresponding to the plurality of points of interest.

In a second aspect, an embodiment of the present application provides a modeling apparatus, including:

An acquiring unit, configured to acquire N depth images of the target space, where N is an integer greater than 1;

a first determining unit, configured to determine, according to the point of interest algorithm, at least one point of interest in each of the depth images to obtain a plurality of points of interest;

a second determining unit, configured to determine related information of each of the points of interest, where the related information includes depth information;

And a modeling unit, configured to establish a model of the target space according to related information and a spatial modeling algorithm corresponding to the plurality of points of interest.

In a third aspect, an embodiment of the present application provides a mobile terminal, including a processor, a memory, a communication interface, and one or more programs, where the one or more programs are stored in the foregoing memory, and are configured by the foregoing processing. Executing, the above program includes instructions for performing the steps in any of the methods of the first aspect of the embodiments of the present application.

In a fourth aspect, the embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program for electronic data exchange, wherein the computer program causes the computer to execute the embodiment of the present application. Some or all of the steps described in either method.

In a fifth aspect, an embodiment of the present application provides a computer program product, where the computer program product includes a non-transitory computer readable storage medium storing a computer program, the computer program being operative to cause a computer to perform the implementation as in the present application. Some or all of the steps described in any of the methods of the first aspect. The computer program product can be a software installation package.

It can be seen that, in the embodiment of the present application, first, the mobile terminal acquires N depth images of the target space, and then determines a plurality of interest points in the N depth images, and finally builds according to depth information and space corresponding to the multiple interest points. The modulo algorithm establishes a model of the target space. It realizes that the stereo space can be modeled only by the mobile terminal, and the operation is convenient and the user experience is improved.

These and other aspects of the present application will be more readily apparent from the following description of the embodiments.

DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background art, the drawings to be used in the embodiments of the present application or the background art will be described below.

1 is a schematic flow chart of a modeling method provided by an embodiment of the present application;

2 is a schematic diagram of a cutoff distance function of a voxel grid provided by an embodiment of the present application;

3 is a schematic flowchart of another modeling method provided by an embodiment of the present application;

4 is a schematic structural diagram of a mobile terminal according to an embodiment of the present application;

FIG. 5 is a schematic structural diagram of a modeling apparatus according to an embodiment of the present disclosure;

FIG. 6 is a schematic structural diagram of a mobile terminal according to an embodiment of the present application.

detailed description

The technical solutions in the embodiments of the present application are clearly and completely described in the following with reference to the accompanying drawings in the embodiments of the present application. It is an embodiment of the present application, and not all of the embodiments. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope shall fall within the scope of the application.

The details are described below separately.

The terms "first", "second", "third", and "fourth" and the like in the specification and claims of the present application and the drawings are used to distinguish different objects, and are not used to describe a specific order. . Furthermore, the terms "comprises" and "comprising" and "comprising" are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device that comprises a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units not listed, or alternatively Other steps or units inherent to these processes, methods, products or equipment.

References to "an embodiment" herein mean that a particular feature, structure, or characteristic described in connection with the embodiments can be included in at least one embodiment of the present application. The appearances of the phrases in various places in the specification are not necessarily referring to the same embodiments, and are not exclusive or alternative embodiments that are mutually exclusive. Those skilled in the art will understand and implicitly understand that the embodiments described herein can be combined with other embodiments.

The mobile terminal may include various handheld devices having wireless communication functions, in-vehicle devices, wearable devices, computing devices, or other processing devices connected to the wireless modem, and various forms of User Equipment (UE), mobile stations ( Mobile Station, MS), terminal device, etc.

At present, modeling stereo space is usually a combination of mobile terminals and external devices. The external device can accurately measure the distance between any two points by using a laser. Since the external device combines the augmented reality technology AR, the user can select the place where the measurement is needed on the display screen of the mobile terminal to complete the measurement; After the distance information, the 3D image is automatically generated by the software built in the external device, and the generated 3D image is presented on the display screen of the mobile terminal, so that the mobile terminal can model the stereo space by means of the external device.

In view of the above problems, the present application provides a modeling method, and the specific implementation manner of the modeling method is as follows.

Referring to FIG. 1 , FIG. 1 is a schematic flowchart of a modeling method provided by an embodiment of the present application, where the method includes:

Step 101: The mobile terminal acquires N depth images of the target space, where N is an integer greater than 1.

The target space refers to a three-dimensional space, and the three-dimensional space may be a closed space or a semi-closed space, which is not limited herein.

Among them, the depth image refers to a three-dimensional image reflecting the geometry of the visible surface in the space.

Each depth image corresponds to an angle, and an angle corresponding to each depth image is different, and an angle of each depth image is an average of a plurality of angles corresponding to the plurality of two-dimensional images that synthesize the depth image.

In an embodiment of the present application, the mobile terminal acquires N depth images of the target space, including:

The mobile terminal acquires M two-dimensional images of the target space by using a camera of the mobile terminal, each of the two-dimensional images corresponding to an angle, and the angles corresponding to the M two-dimensional images are different from each other. The angle corresponding to the two-dimensional image is determined when the two-dimensional image is acquired, and the M is an integer greater than one;

The mobile terminal converts the M two-dimensional images into N three-dimensional images according to a three-dimensional reconstruction algorithm, and each of the three-dimensional images is converted by a plurality of two-dimensional images, and the plurality of two corresponding to the three-dimensional images The angle difference corresponding to any two of the two-dimensional images in the dimension image is smaller than the first threshold, and the M is greater than the N;

The mobile terminal uses the N three-dimensional images as N depth images, and the N three-dimensional images are in one-to-one correspondence with the N depth images.

The number of cameras may be one or multiple. In the case where the number of cameras is one, the one camera may be a front camera, a rear camera or a depth camera; in the case where the number of cameras is two, the two cameras may be front dual cameras, Can be a rear dual camera.

The angle of each two-dimensional image includes three angles of the X-axis, the Y-axis, and the Z-axis of the mobile terminal and the three-dimensional coordinate system when the mobile terminal captures the two-dimensional image by using an angle detecting device (such as a gyroscope);

The angle of the two-dimensional image i is three angles of the X-axis, the Y-axis and the Z-axis of the two-dimensional image i mobile terminal and the three-dimensional coordinate system 1, and the angle of the two-dimensional image j is a two-dimensional image j mobile terminal Three angles with the X-axis, the Y-axis, and the Z-axis of the three-dimensional coordinate system 2, wherein the three-dimensional coordinate system 1 and the three-dimensional coordinate system 2 are the same three-dimensional coordinate system, and the two-dimensional image i and the two-dimensional image j are both M One of the two-dimensional images.

Specifically, in a case where the target space is approximately a rectangular parallelepiped, the plurality of two-dimensional images include a two-dimensional image of a front surface, an upper surface, a rear surface, and a lower surface of the rectangular parallelepiped photographed by the camera in the first direction, and the camera is in the second direction a two-dimensional image of the left, upper, right, and lower surfaces of the cuboid, each surface corresponding to a plurality of two-dimensional images, the first direction being different from the second direction; since the camera is photographed in both the first direction and the second direction There are upper and lower surfaces of the rectangular parallelepiped, and thus a plurality of two-dimensional images have overlapping regions.

Among them, in the case that the camera is a non-depth camera, the three-dimensional reconstruction algorithm has Disney new image algorithm, Structure From Motion (SFM) algorithm, etc.; in the case that the camera is a depth camera, the three-dimensional reconstruction algorithm has Kinect Fusion algorithm .

Wherein, the angle difference is the sum of the angles of the angle i and the angle j of the three angles of the X-axis, the Y-axis and the Z-axis, wherein the angle i and the angle j are each one of M angles.

The first threshold may be set by the user, or may be set by the mobile terminal, and is not limited herein.

Specifically, in a case where the target space is approximately a rectangular parallelepiped and the camera is a depth camera, the mobile terminal converts the plurality of two-dimensional images whose M-angle images have an angle difference smaller than the first threshold into a three-dimensional image according to the three-dimensional reconstruction algorithm. The mobile terminal converts multiple two-dimensional images corresponding to the front surface of the rectangular parallelepiped captured in the first direction into a three-dimensional image corresponding to the front surface according to the Kinect Fusion algorithm, and the angle difference between the plurality of two-dimensional images corresponding to the front surface is smaller than a first threshold; the mobile terminal repeats the above operations on the upper surface, the lower surface, and the rear surface of the rectangular parallelepiped; the mobile terminal sequentially converts the plurality of two-dimensional images corresponding to the left surface of the rectangular parallelepiped captured in the second direction into corresponding ones of the left surface according to the Kinect Fusion algorithm A three-dimensional image, the angle difference of the plurality of two-dimensional images corresponding to the left surface is smaller than the first threshold; and the mobile terminal repeats the above operations on the upper surface, the right surface, and the lower surface of the rectangular parallelepiped.

Step 102: The mobile terminal determines at least one point of interest in each of the depth images according to a point of interest algorithm to obtain a plurality of points of interest.

Among them, the point of interest refers to a pixel having a special property in the depth image. For example, in the case where the target space is a room, the point of interest may be a corner point of the room.

Among them, the interest point algorithm has an edge extraction method, a corner point detection method, an interest operator method, and the like.

Step 103: The mobile terminal determines related information of each of the points of interest, where the related information includes depth information.

The related information further includes color information and an angle, and the color information refers to red R, green G, and blue B information of the point of interest.

The depth information of the point of interest includes the three-dimensional coordinates of the point of interest in the target space.

Specifically, the mobile terminal determines the depth information and the color information of each point of interest. The specific implementation manner is: the mobile terminal directly converts each depth image into a point cloud of the point cloud library PCL; the mobile terminal determines each point of interest in the PCL. Corresponding point cloud data; the mobile terminal uses point cloud data corresponding to each point of interest as depth information and color information corresponding to each point of interest.

Wherein, in the case where the point of interest i and the point of interest j are points of interest in the same depth image, the angle of the point of interest i and the angle of the point of interest j are the same; the point of interest i and the point of interest j are different depth images In the case of a point of interest, the angle of the point of interest i and the angle of the point of interest j are different.

Step 104: The mobile terminal establishes a model of the target space according to related information and a spatial modeling algorithm corresponding to the plurality of points of interest.

The spatial modeling algorithm may be a 3D block matching algorithm.

In an embodiment of the present application, the mobile terminal establishes a model of the target space according to related information and a spatial modeling algorithm corresponding to the multiple points of interest, including:

The mobile terminal determines a local descriptor of each of the points of interest according to the three-dimensional convolution model and related information of each of the points of interest;

The mobile terminal divides the plurality of points of interest into a plurality of sets of points of interest, each set of points of interest includes two points of interest, and the depth images corresponding to the two points of interest are different;

The mobile terminal determines an Euler distance of each set of points of interest according to a Euler formula and a local descriptor of the points of interest included in each set of points of interest;

The mobile terminal removes one of the points of interest in the set of points of interest whose Euler distance is less than a second threshold;

The mobile terminal determines a model of the target space according to remaining points of interest among the plurality of points of interest.

Specifically, the three-dimensional convolution model is a pre-trained model, and the specific implementation manner of constructing the three-dimensional convolution model according to the reference sample data includes: inputting the reference sample data into the initial three-dimensional convolution model, and performing N-layer forward operation to obtain an output. As a result, the gradient corresponding to the output result is obtained according to the output result, and the gradient corresponding to the output result is subjected to the N-layer inverse operation to obtain the weight gradient of each layer, and the weight of each layer is updated according to the weight gradient of each layer. The final iterative calculation results in the final weight, and the final weight is used as the convolution kernel of the initial three-dimensional convolution model to obtain a three-dimensional convolution model. The reference sample data may be derived from data in an image taken by the mobile terminal, an open source data set, or data of an image obtained in other modeling processes; the reference sample data has diversity, thereby improving the three-dimensional convolution model. The accuracy of the operation.

The specific implementation manner of the local terminal determining the local descriptor of each interest point according to the three-dimensional convolution model and the related information of each interest point is: the mobile terminal expresses the depth information of the interest point i as 30*30*30 voxels The truncation distance function of the grid (as shown in Fig. 2); the mobile terminal inputs the truncation distance function of the 30*30*30 voxel grid corresponding to the point of interest i into the three-dimensional convolution model to obtain the local descriptor of the point of interest i, The point of interest i is one of a plurality of points of interest.

Among them, Euler's formula is:

e ^ix =cos x+i sin x

Where e is the base of the natural logarithm and i is the imaginary unit.

Wherein, the second threshold value represents a maximum value corresponding to the Euler distances of the two points of interest when the two points of interest in the different depth images are the same point of interest in the target space, that is, when two points of interest in the different depth images When the Euler distance is less than the second threshold, the two points of interest are the same point of interest in the target space; when the Euler distances of the two points of interest in the different depth images are greater than or equal to the second threshold, the two points of interest are targets Different points of interest in space.

Specifically, the mobile terminal determines a model of the target space according to the remaining points of interest in the plurality of points of interest. The mobile terminal connects the remaining points of interest in the plurality of points of interest to form a model of the target space.

For example, suppose the target space is approximated as a rectangular parallelepiped, each surface corresponds to a depth image, each surface has 4 points of interest, and the second threshold is 0.2. The mobile terminal according to the three-dimensional convolution model and the depth information of each point of interest Determining the local descriptor of each point of interest, the mobile terminal divides the 24 points of interest into 80 sets of points of interest, each set of points of interest includes two points of interest corresponding to different surfaces of the cuboid, and the mobile terminal according to the Euler formula e ^ix =cos x+i sin x and 80 sets of interest points in each set of points of interest include local descriptors of two points of interest to determine the Euler distance of each set of points of interest, and the mobile terminal sets the same two points of interest The point of interest (ie, the Euler distance of the set of points of interest is less than 0.2) is removed, and the mobile terminal connects the remaining points of interest of the 24 points of interest to form a model of the target space.

In an embodiment of the present application, before the mobile terminal acquires the N depth images of the target space, the method further includes:

The mobile terminal performs a setting prompt by the prompting device, and the setting prompt is used to prompt the user to acquire N depth images of the target space.

Wherein, the prompting device comprises a display screen and a microphone.

Specifically, the manner in which the mobile terminal performs setting prompts through the prompting device includes: a text prompt, a voice prompt, a comic reminder, a short video prompt, and the like.

The mobile terminal determines that the mobile terminal satisfies a setting condition, the setting condition includes at least one of: detecting a click operation for a virtual button, the virtual button is used to trigger spatial modeling; the mobile terminal is in a setting Environment; a voice instruction that is modeled for the current target space is detected.

Specifically, the virtual button may be represented by an application icon set on the desktop of the mobile terminal. In the case that the virtual button adopts the application icon, if the user needs to model the current space, if the mobile terminal is in the lock screen, the user needs to unlock the screen of the mobile terminal, and click the virtual button corresponding to the desktop of the mobile terminal. The application icon, which in turn quickly launches spatial modeling capabilities. The virtual button can also be set in the interface of the modeling application of the mobile terminal, which is not limited herein.

Specifically, the setting environment is that the user is in a closed space, or the user is in a semi-enclosed space, such as the user is in the house, or the user is in the car. In the case where the user is in a non-enclosed space, such as a playground in which the user is in the open air and surrounded by a wall, since the playground is open-air and there is no surrounding wall, it is meaningless for the mobile terminal to spatially model the playground.

Specifically, if the user needs to model the current target space, the user may interact with the mobile terminal in the form of voice, for example, the voice is “modeling for the current target space”, and the mobile terminal detects that the current target space is built. The voice command of the mode, the space modeling function is started to model the current target space, which is convenient and quick, and enhances the user experience.

In an embodiment of the present application, after the mobile terminal establishes the model of the target space according to the related information and the spatial modeling algorithm corresponding to the multiple points of interest, the method further includes:

In the case where a confirmation instruction for displaying the model of the target space is detected, the mobile terminal displays a model of the target space through a display screen of the mobile terminal.

It can be seen that, in the embodiment of the present application, in the case that the confirmation instruction for displaying the model of the target space is detected, the mobile terminal displays the model of the target space through the display screen. In this way, the user can use the model of the target space to display and explain to others, thereby improving the user experience.

The embodiment of the present application further provides another more detailed method flow, as shown in FIG. 3, the method includes:

Step 301: The mobile terminal determines that the mobile terminal satisfies a setting condition, where the setting condition includes at least one of: detecting a click operation for a virtual button, the virtual button is used to trigger spatial modeling; the mobile terminal In a setting environment; a voice instruction that models the current target space is detected.

Step 302: The mobile terminal acquires M two-dimensional images of the target space by using a camera of the mobile terminal, each of the two-dimensional images corresponding to an angle, and the angles corresponding to the M two-dimensional images are different from each other. The angle corresponding to each of the two-dimensional images is determined when the two-dimensional image is acquired, and the M is an integer greater than one.

Step 303: The mobile terminal converts the M two-dimensional images into N three-dimensional images according to a three-dimensional reconstruction algorithm, and each of the three-dimensional images is converted by a plurality of two-dimensional images, and each of the three-dimensional images corresponds to the The angle difference corresponding to any two of the plurality of two-dimensional images is smaller than a first threshold, and the M is greater than the N.

Step 304: The mobile terminal uses the N three-dimensional images as N depth images, and the N three-dimensional images are in one-to-one correspondence with the N depth images.

Step 305: The mobile terminal determines at least one point of interest in each of the depth images according to a point of interest algorithm to obtain a plurality of points of interest.

Step 306: The mobile terminal determines related information of each of the points of interest, where the related information includes depth information.

Step 307: The mobile terminal determines a local descriptor of each of the points of interest according to the three-dimensional convolution model and related information of each of the points of interest.

Step 308: The mobile terminal divides the plurality of points of interest into a plurality of sets of points of interest, each set of points of interest includes two points of interest, and the depth images corresponding to the two points of interest are different.

Step 309: The mobile terminal determines the Euler distance of each set of points of interest according to the Euler formula and the local descriptor of the point of interest included in each set of points of interest.

Step 310: The mobile terminal removes one of the points of interest in the set of points of interest whose Euler distance is less than a second threshold.

Step 311: The mobile terminal determines a model of the target space according to remaining points of interest in the plurality of points of interest.

It should be noted that the specific implementation process of each step of the method shown in FIG. 3 can be referred to the specific implementation process of the foregoing method, and is not described herein.

FIG. 4 is a schematic structural diagram of a mobile terminal according to an embodiment of the present disclosure. The mobile terminal includes a processor, a memory, a communication interface, and a Or a plurality of programs, wherein the one or more programs are stored in the memory and configured to be executed by the processor, the program comprising instructions for performing the following steps:

In an embodiment of the present application, before acquiring the N depth images of the target space, the program further includes instructions for performing the following steps:

Determining that the mobile terminal satisfies a setting condition, the setting condition includes at least one of: detecting a click operation for a virtual button, the virtual button is used to trigger spatial modeling; the mobile terminal is in a setting environment; A voice instruction that models the current target space is detected.

A setting prompt is provided by the prompting means for prompting to acquire N depth images of the target space.

In an embodiment of the present application, in acquiring N depth images of the target space, the above program includes instructions specifically for performing the following steps:

Obtaining M two-dimensional images of the target space, each of the two-dimensional images corresponding to an angle, and the angles of the M two-dimensional images are different from each other, and the angle corresponding to each of the two-dimensional images is acquired When the two-dimensional image is determined, the M is an integer greater than one;

Converting the M two-dimensional images into N three-dimensional images according to a three-dimensional reconstruction algorithm, each of the three-dimensional images being converted by a plurality of two-dimensional images, the plurality of two-dimensional images corresponding to each of the three-dimensional images The angle difference corresponding to any two of the two-dimensional images is less than a first threshold, and the M is greater than the N;

The N three-dimensional images are used as N depth images, and the N three-dimensional images are in one-to-one correspondence with the N depth images.

In an embodiment of the present application, in establishing a model of the target space according to related information and a spatial modeling algorithm corresponding to the plurality of points of interest, the program includes instructions specifically for performing the following steps:

Determining a local descriptor of each of the points of interest according to a three-dimensional convolution model and related information of each of the points of interest;

Dividing the plurality of points of interest into a plurality of sets of points of interest, each set of points of interest comprising two points of interest, wherein the depth images corresponding to the two points of interest are different;

Determining the Euler distance of each set of points of interest according to the Euler formula and the local descriptor of the points of interest included in each set of points of interest;

Removing one of the points of interest in the set of points of interest whose Euler distance is less than a second threshold;

Determining a model of the target space based on remaining points of interest among the plurality of points of interest.

In an embodiment of the present application, in determining a model of the target space according to remaining points of interest in the plurality of points of interest, the program includes instructions specifically for performing the following steps:

The remaining points of interest of the plurality of points of interest are joined to form a model of the target space.

In an embodiment of the present application, the above program further includes instructions for performing the following steps:

In the case where a confirmation instruction for displaying the model of the target space is detected, the model of the target space is displayed.

In an embodiment of the present application, each depth image corresponds to an angle, and the angle corresponding to each depth image is different.

In an embodiment of the present application, the point of interest is a pixel having a special property in the depth image.

It should be noted that the specific implementation process of the embodiment may be referred to the specific implementation process described in the foregoing method embodiment, and is not described herein.

The above description mainly introduces the solution of the embodiment of the present application from the perspective of the method side execution process. It can be understood that, in order to implement the above functions, the mobile terminal includes corresponding hardware structures and/or software modules for performing various functions. Those skilled in the art will readily appreciate that the present application can be implemented in a combination of hardware or hardware and computer software in combination with the elements and algorithm steps of the various examples described in the embodiments disclosed herein. Whether a function is implemented in hardware or computer software to drive hardware depends on the specific application and design constraints of the solution. A person skilled in the art can use different methods for each particular application to implement the described functionality, but such implementation should not be considered to be beyond the scope of the application.

The embodiment of the present application may divide the functional unit into the modeling device according to the foregoing method example. For example, each functional unit may be divided according to each function, or two or more functions may be integrated into one processing unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. It should be noted that the division of the unit in the embodiment of the present application is schematic, and is only a logical function division. In actual implementation, there may be another division manner.

In the case of employing an integrated unit, FIG. 5 shows a block diagram of one possible functional unit composition of the modeling apparatus involved in the above embodiment. The modeling apparatus 500 includes a processing unit 501, a storage unit 502, and a communication unit 503. The processing unit 501 includes an acquisition unit, a first determination unit, a second determination unit, and a modeling unit, where:

In an embodiment of the present application, the processing unit 501 further includes:

a third determining unit, configured to determine that the mobile terminal satisfies a setting condition before acquiring the N depth images of the target space, where the setting condition includes at least one of: detecting a click operation for the virtual button, A virtual button is used to trigger spatial modeling; the mobile terminal is in a setting environment; a voice instruction that is modeled for the current target space is detected.

The prompting unit is configured to perform a setting prompt by using a prompting device, and the setting prompt is used to prompt to acquire N depth images of the target space before acquiring the N depth images of the target space.

In an embodiment of the present application, in acquiring N depth images of the target space, the acquiring unit is specifically configured to:

Obtaining M two-dimensional images of the target space, each of the two-dimensional images corresponding to an angle, and the angles corresponding to the M two-dimensional images are different from each other, and the angle corresponding to each of the two-dimensional images is Determined when the two-dimensional image is acquired, the M is an integer greater than one;

Converting the M two-dimensional images into N three-dimensional images according to a three-dimensional reconstruction algorithm, each of the three-dimensional images being converted by a plurality of two-dimensional images, the plurality of two-dimensional images corresponding to each of the three-dimensional images The angle difference corresponding to any two of the two-dimensional graphics is less than a first threshold, and the M is greater than the N;

In an embodiment of the present application, the modeling unit is specifically configured to: according to related information and a spatial modeling algorithm corresponding to the plurality of points of interest to establish a model of the target space:

In an embodiment of the present application, in determining a model of the target space according to remaining points of interest in the plurality of points of interest, the modeling unit is specifically configured to:

And a display unit configured to display a model of the target space in a case where a confirmation instruction for displaying the model of the target space is detected.

The processing unit 501 may be a processor or a controller, and may be, for example, a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), and a dedicated integrated controller (Application). -Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA) or other programmable logic device, transistor logic device, hardware component, or any combination thereof. The storage unit 502 may be a memory, and the communication unit 503 may It is a transceiver, a transceiver controller, a radio frequency chip, a communication interface, and the like.

When the processing unit 501 is a processor, the storage unit 502 is a memory, and the communication unit 503 is a communication interface, the modeling apparatus according to the embodiment of the present application may be the mobile terminal shown in FIG.

The embodiment of the present application further provides a schematic structural diagram of a mobile terminal. As shown in FIG. 6 , the mobile terminal includes: a shell, a circuit board, a camera, a display screen, an angle detecting device, a control camera, a display screen, and an angle detecting device. The controller; the camera, the angle detecting device and the controller are disposed on the circuit board, the display screen is connected to the controller, and the angle detecting device comprises a gyroscope, an angle sensor, etc., wherein:

a controller, configured to acquire N depth images of the target space, where N is an integer greater than 1; determining at least one interest point in each of the depth images according to a point of interest algorithm to obtain a plurality of points of interest; determining each Relevant information of the points of interest, the related information includes depth information; and a model of the target space is established according to related information and a spatial modeling algorithm corresponding to the plurality of points of interest.

The mobile terminal further includes at least one functional component, and the controller is coupled to the at least one functional component, and the at least one functional component includes at least one of the following: a speaker, a microphone, a face recognition device, a proximity sensor, and an ambient light sensor.

Wherein, the display screen comprises a touch screen and a display screen, and the display screen comprises an organic light emitting diode display screen OLED.

Wherein, the controller may comprise a processor and a memory, the control center of the mobile terminal, connecting various parts of the entire mobile terminal by using various interfaces and lines, by running or executing software programs and/or modules stored in the memory And invoking data stored in the memory, performing various functions of the mobile terminal and processing data, thereby performing overall monitoring of the mobile terminal. Optionally, the processor can integrate an application processor and a modem processor, wherein the application processor mainly processes an operating system, a user interface, an application, etc., and the modem processor mainly processes wireless communication. It will be appreciated that the modem processor may also not be integrated into the processor.

The memory can be used to store software programs and modules, and the processor executes various functional applications and data processing of the mobile terminal by running software programs and modules stored in the memory. The memory may mainly include a storage program area and an storage data area, wherein the storage program area may store an operating system, an application required for at least one function, and the like; the storage data area may store data created according to usage of the mobile terminal, and the like. Further, the memory may include a high speed random access memory, and may also include a nonvolatile memory such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.

It can be seen that, in the embodiment of the present application, first, the mobile terminal acquires N depth images of the target space, and then determines a plurality of interest points in the N depth images, and finally builds related information and space corresponding to the plurality of interest points. The modulo algorithm establishes a model of the target space. It realizes that the stereo space can be modeled only by the mobile terminal, and the operation is convenient and the user experience is improved.

In an embodiment of the present application, the controller is further configured to:

In an embodiment of the present application, in acquiring N depth images of a target space,

a camera for acquiring M two-dimensional images of the target space, where M is an integer greater than one;

An angle detecting device, configured to determine an angle of the two-dimensional image when the camera acquires the two-dimensional image, each of the two-dimensional images corresponding to an angle, and the angles corresponding to the M two-dimensional images are different from each other;

The controller is specifically configured to: convert the M two-dimensional images into N three-dimensional images according to a three-dimensional reconstruction algorithm, where each of the three-dimensional images is converted by a plurality of two-dimensional images, and each of the three-dimensional images corresponds to An angle difference corresponding to any two of the two-dimensional images is smaller than a first threshold, the M is greater than the N; the N three-dimensional images are N depth images, and the N The three-dimensional image is in one-to-one correspondence with the N depth images.

In an embodiment of the present application, in establishing a model of the target space according to related information and a spatial modeling algorithm corresponding to the plurality of points of interest, the controller is specifically configured to:

In an embodiment of the present application, the display screen is specifically configured to:

It should be noted that the specific implementation process of this embodiment can be referred to the specific implementation process described in the following method, and is not described herein.

The embodiment of the present application further provides a computer storage medium, wherein the computer storage medium stores a computer program for electronic data exchange, the computer program causing the computer to perform some or all of the steps of any of the methods described in the foregoing method embodiments. The above computer includes a mobile terminal.

The embodiment of the present application further provides a computer program product, where the computer program product comprises a non-transitory computer readable storage medium storing a computer program, the computer program being operative to cause the computer to execute any one of the methods described in the foregoing method embodiments. Part or all of the steps of the method. The computer program product can be a software installation package, and the computer includes a mobile terminal.

It should be noted that, for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations, but those skilled in the art should understand that the present application is not limited by the described action sequence. Because certain steps may be performed in other sequences or concurrently in accordance with the present application. In the following, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

In the above embodiments, the descriptions of the various embodiments are different, and the details that are not detailed in a certain embodiment can be referred to the related descriptions of other embodiments.

In the several embodiments provided herein, it should be understood that the disclosed apparatus may be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the above units is only a logical function division. In actual implementation, there may be another division manner. For example, multiple units or components may be combined or integrated. Go to another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be electrical or otherwise.

The units described above as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit.

The above integrated unit, if implemented in the form of a software functional unit and sold or used as a standalone product, may be stored in a computer readable memory. Based on such understanding, the technical solution of the present application, in essence or the contribution to the prior art, or all or part of the technical solution may be embodied in the form of a software product stored in a memory. A number of instructions are included to cause a computer device (which may be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-described methods of various embodiments of the present application. The foregoing memory includes: a U disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic disk, or an optical disk, and the like, which can store program codes.

A person skilled in the art can understand that all or part of the steps of the foregoing embodiments can be completed by a program to instruct related hardware, and the program can be stored in a computer readable memory, and the memory can include: a flash drive , read-only memory (English: Read-Only Memory, referred to as: ROM), random accessor (English: Random Access Memory, referred to as: RAM), disk or CD.

The embodiments of the present application have been described in detail above. The principles and implementations of the present application are described in the specific examples. The description of the above embodiments is only used to help understand the method and core ideas of the present application. A person skilled in the art will have a change in the specific implementation and application scope according to the idea of the present application. In the above, the content of the present specification should not be construed as limiting the present application.

Claims

A modeling method, comprising:

Obtaining N depth images of the target space, the N being an integer greater than 1;

Determining at least one point of interest in each of the depth images according to a point of interest algorithm to obtain a plurality of points of interest;

Determining related information of each of the points of interest, the related information including depth information;

And establishing a model of the target space according to related information and a spatial modeling algorithm corresponding to the plurality of points of interest.
The method according to claim 1, wherein before the acquiring the N depth images of the target space, the method further comprises:

Determining that the mobile terminal satisfies a setting condition, the setting condition includes at least one of: detecting a click operation for a virtual button, the virtual button is used to trigger spatial modeling; the mobile terminal is in a setting environment; A voice instruction that models the current target space is detected.
The method according to claim 1, wherein before the acquiring the N depth images of the target space, the method further comprises:

A setting prompt is provided by the prompting means for prompting to acquire N depth images of the target space.
The method according to any one of claims 1-3, wherein the acquiring the N depth images of the target space comprises:

Acquiring, by the camera of the mobile terminal, M two-dimensional images of the target space, each of the two-dimensional images corresponding to an angle, and the angles corresponding to the M two-dimensional images are different from each other, and each of the two The angle corresponding to the dimension image is determined when the two-dimensional image is acquired, and the M is an integer greater than one;

Converting the M two-dimensional images into N three-dimensional images according to a three-dimensional reconstruction algorithm, each of the three-dimensional images being converted by a plurality of two-dimensional images, the plurality of two-dimensional images corresponding to each of the three-dimensional images The angle difference corresponding to any two of the two-dimensional images is less than a first threshold, and the M is greater than the N;

The N three-dimensional images are used as N depth images, and the N three-dimensional images are in one-to-one correspondence with the N depth images.
The method according to any one of claims 1-4, wherein the establishing a model of the target space according to related information and a spatial modeling algorithm corresponding to the plurality of points of interest comprises:

Determining a local descriptor of each of the points of interest according to a three-dimensional convolution model and related information of each of the points of interest;

Dividing the plurality of points of interest into a plurality of sets of points of interest, each set of points of interest comprising two points of interest, wherein the depth images corresponding to the two points of interest are different;

Determining the Euler distance of each set of points of interest according to the Euler formula and the local descriptor of the points of interest included in each set of points of interest;

Removing one of the points of interest in the set of points of interest whose Euler distance is less than a second threshold;

Determining a model of the target space based on remaining points of interest among the plurality of points of interest.
The method according to claim 5, wherein the determining the model of the target space according to the remaining points of interest in the plurality of points of interest comprises:

The remaining points of interest of the plurality of points of interest are joined to form a model of the target space.
The method according to any one of claims 1-6, wherein after the establishing the model of the target space according to the related information and the spatial modeling algorithm corresponding to the plurality of points of interest, the method further comprises :

In the case where a confirmation instruction for displaying the model of the target space is detected, the model of the target space is displayed by the display screen of the mobile terminal.
The method according to any one of claims 1 to 7, wherein each depth image corresponds to an angle, and the angle corresponding to each depth image is different.
The method according to any one of claims 1-8, wherein the point of interest is a pixel having a special property in the depth image.
A modeling device, comprising:

An acquiring unit, configured to acquire N depth images of the target space, where N is an integer greater than 1;

a first determining unit, configured to determine, according to the point of interest algorithm, at least one point of interest in each of the depth images to obtain a plurality of points of interest;

a second determining unit, configured to determine related information of each of the points of interest, where the related information includes depth information;

And a modeling unit, configured to establish a model of the target space according to related information and a spatial modeling algorithm corresponding to the plurality of points of interest.
The modeling apparatus according to claim 10, wherein the apparatus further comprises:

a third determining unit, configured to determine that the mobile terminal satisfies a setting condition, where the setting condition includes at least one of: detecting a click operation for a virtual button, the virtual button is used to trigger spatial modeling; The mobile terminal is in a setting environment; a voice instruction for modeling the current target space is detected.
The modeling apparatus according to claim 10, wherein the apparatus further comprises:

a prompting unit, configured to perform a setting prompt by using a prompting device, where the setting prompt is used to prompt to acquire N depth images of the target space.
The modeling apparatus according to any one of claims 10 to 12, wherein the acquiring unit is specifically configured to:

Obtaining M two-dimensional images of the target space, each of the two-dimensional images corresponding to an angle, and the angles corresponding to the M two-dimensional images are different from each other, and the angle corresponding to each of the two-dimensional images is Determined when the two-dimensional image is acquired, the M is an integer greater than one;

Converting the M two-dimensional images into N three-dimensional images according to a three-dimensional reconstruction algorithm, each of the three-dimensional images being converted by a plurality of two-dimensional images, the plurality of two-dimensional images corresponding to each of the three-dimensional images The angle difference corresponding to any two of the two-dimensional images is less than a first threshold, and the M is greater than the N;

The N three-dimensional images are used as N depth images, and the N three-dimensional images are in one-to-one correspondence with the N depth images.
The modeling apparatus according to any one of claims 10 to 13, wherein the modeling is performed on a model of the target space based on correlation information and a spatial modeling algorithm corresponding to the plurality of points of interest The unit is specifically used to:

Determining a local descriptor of each of the points of interest according to a three-dimensional convolution model and related information of each of the points of interest;

Dividing the plurality of points of interest into a plurality of sets of points of interest, each set of points of interest comprising two points of interest, wherein the depth images corresponding to the two points of interest are different;

Determining the Euler distance of each set of points of interest according to the Euler formula and the local descriptor of the points of interest included in each set of points of interest;

Removing one of the points of interest in the set of points of interest whose Euler distance is less than a second threshold;

Determining a model of the target space based on remaining points of interest among the plurality of points of interest.
The modeling apparatus according to claim 14, wherein the modeling unit is specifically configured to: determine a model of the target space according to remaining points of interest in the plurality of points of interest:

The remaining points of interest of the plurality of points of interest are joined to form a model of the target space.
The modeling apparatus according to claim 14, wherein the apparatus further comprises:

a display unit, configured to display a model of the target space through a display screen of the mobile terminal in a case where a confirmation instruction for displaying a model of the target space is detected.
The modeling apparatus according to any one of claims 10-16, wherein each depth image corresponds to an angle, and each of the depth images corresponds to an angle different.
The modeling apparatus according to any one of claims 10-17, wherein the point of interest is a pixel having a special property in the depth image.
A mobile terminal, comprising a processor, a memory, a communication interface, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, The program comprises instructions for performing the steps in the method of any of claims 1-9.
A computer readable storage medium, characterized by storing a computer program for electronic data exchange, wherein the computer program causes a computer to perform the method of any one of claims 1-9.