WO2020108125A1

WO2020108125A1 - Information pushing method, information pushing device and terminal device

Info

Publication number: WO2020108125A1
Application number: PCT/CN2019/110876
Authority: WO
Inventors: 刘耀勇; 陈岩
Original assignee: Oppo广东移动通信有限公司
Priority date: 2018-11-27
Filing date: 2019-10-12
Publication date: 2020-06-04
Also published as: CN109492607A; CN109492607B

Abstract

An information pushing method, an information pushing device and a terminal device, the method comprising: acquiring an image to be detected, and carrying out target detection on the image to be detected (S101); determining whether one or a plurality of target objects are detected in the image to be detected (S102); if a target object is detected in the image to be detected, pushing three-dimensional information corresponding to the one or plurality of target objects respectively in the image to be detected to a user, wherein the three-dimensional information corresponding to each target object is information which is used to display a three-dimensional structure of the target object and which is obtained after carrying out three-dimensional reconstruction of the target object (S103). The present method may, to a certain extent, solve the current technical problem of a terminal device being unable to allow a user to more quickly and effectively understand an external object.

Description

Information pushing method, information pushing device and terminal equipment

This application requires the priority of the Chinese patent application with the application number CN201811423310.9 and the invention titled "An Information Push Method, Information Push Device, and Terminal Equipment" submitted on November 28, 2018. The entire content of the Chinese patent application Incorporated by reference in this application.

Technical field

The present application belongs to the technical field of terminals, and particularly relates to an information pushing method, an information pushing device, a terminal device, and a computer-readable storage medium.

Background technique

At present, when users want to know an object from the outside world (for example, want to know a certain building), they usually use terminal equipment to find the object through network search and other means. Obviously, this traditional method requires users Only after performing more operations on the terminal device can the desired information be obtained.

Therefore, current terminal devices cannot enable users to quickly and efficiently understand the outside world.

technical problem

One of the purposes of the embodiments of the present application is to provide an information pushing method, information pushing device, terminal device and computer-readable storage medium, aiming to solve to a certain extent that current terminal devices cannot enable users to understand the outside world more quickly and efficiently Technical issues.

Technical solution

To solve the above technical problems, the technical solutions adopted in the embodiments of the present application are:

In the first aspect, an information push method is provided, including:

Obtain the image to be detected, and perform target detection on the image to be detected;

Determine whether one or more target objects are detected in the image to be detected;

If a target object is detected in the above-mentioned image to be detected, then:

Push the three-dimensional information corresponding to the one or more target objects in the above-mentioned image to be detected to the user, wherein the three-dimensional information corresponding to each target object is obtained by performing three-dimensional reconstruction on the target object to display the target Information about the three-dimensional structure of the object.

In a second aspect, an information pushing device is provided, including:

The image acquisition module is used to acquire the image to be detected and perform target detection on the image to be detected;

The target judgment module is used to judge whether one or more target objects are detected in the image to be detected;

The three-dimensional information pushing module is used to push the three-dimensional information corresponding to one or more target objects in the above-mentioned image to the user if the target object is detected in the above-mentioned image to be detected, wherein each target object corresponds to The three-dimensional information is the information used to display the three-dimensional structure of the target object after three-dimensional reconstruction of the target object.

In a third aspect, a terminal device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method of the first aspect is implemented. step.

According to a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect described above are implemented.

According to a fifth aspect, a computer program product is provided. The computer program product includes a computer program, and when the computer program is executed by one or more processors, the steps of the method of the first aspect described above are implemented.

BRIEF DESCRIPTION

In order to more clearly explain the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments or exemplary technical descriptions. Obviously, the drawings in the following description are only for the application For some embodiments, those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

1 is a schematic diagram of an implementation process of an information pushing method provided in Embodiment 1 of the present application;

2 is a schematic diagram of performing target detection on an image to be detected by using a target detection model provided by Embodiment 1 of the present application;

3 is a schematic diagram of an implementation process of another information pushing method provided in Embodiment 2 of the present application;

4 is a schematic diagram of an implementation process of a method for determining a target object of interest to a user provided in Embodiment 2 of the present application;

5 is a schematic diagram of determining a target object of interest to a user provided by Embodiment 2 of the present application;

6 is a schematic structural diagram of an information pushing device provided in Embodiment 2 of the present application;

7 is a schematic structural diagram of a terminal device according to Embodiment 4 of the present application.

detailed description

In the following description, for the purpose of illustration rather than limitation, specific details such as specific system structure and technology are proposed to thoroughly understand the embodiments of the present application. However, those skilled in the art should understand that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid unnecessary details hindering the description of the present application.

The information pushing method provided in the embodiment of the present application is applicable to a terminal device. Exemplarily, the terminal device includes but is not limited to: a smart phone, a palmtop computer, a notebook, a desktop computer, a smart wearable device, and the like.

It should be understood that when used in this specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements, and/or components, but does not exclude one or more other features , Wholes, steps, operations, elements, components and/or their existence or addition.

It should also be further understood that the term "and/or" used in the specification of the present application and the appended claims refers to any and all possible combinations of one or more of the associated listed items and includes these combinations .

As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "once" or "in response to determination" or "in response to detection" depending on the context . Similarly, the phrase "if determined" or "if [described condition or event] is detected" can be interpreted in the context to mean "once determined" or "in response to a determination" or "once detected [described condition or event ]" or "In response to detection of [the described condition or event]".

In addition, in the description of the present application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

In order to explain the technical solutions described in this application, the following detailed description will be made in conjunction with specific drawings and embodiments.

The following describes the information pushing method provided in Embodiment 1 of the present application. The steps described in Embodiment 1 of the present application are applied to a terminal device. Please refer to FIG. 1. The information pushing method in Embodiment 1 of the present application includes:

In step S101, obtain an image to be detected, and perform target detection on the image to be detected;

The image to be detected in step S101 may be an image captured by the user through the camera application (Application, APP) of the terminal device. For example, the user launches the camera APP in the terminal device and uses the camera APP to capture an image. The terminal device The image may be determined as the image to be detected; or, the image to be detected may be a frame of preview image in the preview screen collected by the camera APP or camera APP in the terminal device, for example, after the user starts the camera APP of the terminal device, the The terminal device may use a certain frame of image displayed on the display screen of the terminal device as the image to be detected; or, the image to be detected may also be an image saved locally in the terminal device, for example, the terminal device may use a The image to be detected is an image to be detected; alternatively, the image to be detected may also be a certain frame image in a video watched online or a locally saved video, for example, a certain frame image in a cartoon watched online by a user is determined to be Check the image. This application does not limit the source of the image to be detected.

In the embodiments of the present application, the trained target detection model (that is, the neural network model for target detection) can be used to detect the target object in the acquired image to be detected; or, other target detection commonly used in the art can also be used Method to detect the target object in the acquired image to be detected. Here, the detection method of the target object is not limited. The method for detecting the target object using the target detection model is described in detail below.

To use the target detection model to perform target detection on the image to be detected obtained in step S101, a target detection model needs to be trained in advance to obtain the trained target detection model, so that the trained target detection model can be used to detect the image to be detected in step S101 Perform target detection. After obtaining a trained target detection model, the target objects that the trained target detection model can detect are also fixed. For example, the trained target detection model A can be used to detect the Eiffel Tower, Lei Feng Tower and Taj Ling, the target detection model B after training can be used to detect dogs, cats, monkeys and elephants. Therefore, in the embodiment of the present application, the terminal device may download and update different trained target detection models to the user's terminal device according to the different users of the information push method provided in the present application, for example, the terminal The device can determine which type or types of target detection models need to be downloaded by obtaining information such as the user’s work type, age range, or objects of interest, for example, if the terminal device X obtains the user’s age is 3-5 Years old, because children in this age range are generally more interested in small animals, then the terminal device X can download the trained target detection model for detecting animals to the terminal device X; if the terminal device Y obtains the user The objects of interest are animals and buildings, and the trained target detection model for detecting animals and the trained target detection model for detecting buildings can be downloaded to the terminal device Y.

As shown in FIG. 2, using the trained target detection model 201 to perform target detection on the image to be detected 202, wherein the target objects that the trained target detection model 201 can detect are the Eiffel Tower, Leifeng Tower and the Taj Mahal, then Those skilled in the art can easily obtain that the trained target detection model 201 can detect the Eiffel Tower in the image to be detected 202, and the detection result 203 output by the trained target detection model 201 can be used to indicate The position of the Eiffel Tower in the image to be detected 202 and the image area indicated by the position are the confidence of the Eiffel Tower.

In step S102, it is determined whether one or more target objects are detected in the image to be detected;

In the embodiment of the present application, it is necessary to determine whether the target object is included in the image to be detected acquired in step S101, and if the target object is included, the subsequent step S103 is executed.

For example, if the to-be-detected image acquired in step S101 is the image 202 in FIG. 2, in step S101, the trained target detection model 201 is used to perform target detection on the acquired to-be-detected image 202, then the judgment in step S102 The result is that one or more target objects are detected in the image to be detected; if the image to be detected acquired in step S101 is the image 203 in FIG. 2, in step S101, the trained target detection model 201 is used to acquire If the image to be detected 203 performs target detection, then the judgment result of step S102 is that one or more target objects are not detected in the image to be detected.

In step S103, if a target object is detected in the above-mentioned image to be detected, the three-dimensional information corresponding to one or more target objects in the above-mentioned image to be detected respectively is pushed to the user, wherein the three-dimensional information corresponding to each target object The information is information for displaying the three-dimensional structure of the target object after three-dimensional reconstruction of the target object;

In the embodiment of the present application, if the judgment result in step S102 is affirmative, step S103 is executed to display the three-dimensional information corresponding to one or more target objects in the image to be detected obtained in step S101 to the user. For example, if a target object is detected in the image to be detected, in step S103, the three-dimensional information corresponding to the target object contained in the image to be detected is displayed. If more than one object is detected in the image to be detected Target objects, for example, if 3 target objects are detected, in this step S103, three-dimensional information of one of the three target objects may be pushed, or two of the three target objects may be pushed separately Corresponding three-dimensional information, or the three-dimensional information corresponding to the three target objects can be pushed to the user, that is, if multiple target objects are detected in the image to be detected, the embodiment of the present application The number of objects is not limited.

In the embodiment of the present application, the three-dimensional information of each target object may be a .obj file or a .ply file generated after three-dimensional reconstruction of the target object. The .obj file and the .ply file can not only support dynamic playback to display the target The three-dimensional structure of the object can also support the user to manually zoom the sliding mobile phone screen to display the three-dimensional structure of the target object. It is more intuitive to display the three-dimensional structure in these two ways, and when the user of the information push method defined in this application is As a child, it can make it easier for the child to understand the three-dimensional structure of the target object, which can greatly improve the child's knowledge to a certain extent and better develop the child's intelligence.

In addition, in the embodiment of the present application, the three-dimensional information of the target object in the image to be detected may be stored in advance on the server or locally, for example, the server or the local pre-stored three-dimensional information of the Eiffel Tower, once the Eiffel Tower is detected in the image To obtain the 3D information of the Eiffel Tower directly from the server or locally and push it to the user, there is no need to spend time on the 3D reconstruction of the Eiffel Tower, so the 3D structure of the Eiffel Tower can be pushed to the user more quickly. In the embodiment of the present application, in order to quickly show the three-dimensional structure of the target object to the user, the three-dimensional structure of some target objects may be stored locally in the terminal device according to different users, for example, if the user is a child, because the child is small Animals and some living items are more interested, so you can save the three-dimensional information of some small animals and living items (such as dogs, cats, monkeys, tea cups and spoons, etc.) to the local.

Or, the three-dimensional information of the target object in the image to be detected is not pre-stored, but obtained after performing three-dimensional reconstruction on the target object after detecting that the image to be detected contains the target object. For example, when it is detected that there is an apple in the image to be detected, three-dimensional reconstruction is performed on the apple to obtain the three-dimensional information of the apple, and then the three-dimensional information of the apple is pushed to the user. At this time, in order to be able to perform three-dimensional reconstruction of the target object in the detected image, it is necessary to obtain multiple images of the target object at different angles. Therefore, in this case, the user may be prompted to take multiple images of the target object at different angles In this way, the target object can be three-dimensionally reconstructed. At this time, although it takes a certain amount of time to perform three-dimensional reconstruction, the three-dimensional structure of the target object itself contained in the image to be detected can be shown to the user. For example, the target object in the image to be detected is a dog, and the breed of the dog is a German shepherd. The terminal device pre-stores the three-dimensional information corresponding to the dog, but the pre-saved three-dimensional information shows the Pomeranian The three-dimensional structure, therefore, if the pre-saved three-dimensional information corresponding to the dog is pushed to the user, it may not be what the user wants.

In addition, in the embodiment of the present application, if the judgment result of step S102 is affirmative, the information pushing method defined in the embodiment of the present application may further include the following steps:

The production process, introduction information, video information, image information, and/or commodity information corresponding to one or more target objects in the image to be detected obtained in step S101 are pushed to the user.

That is, in addition to pushing the three-dimensional information of the target object as described in step S103, it is also possible to push the production process of one or more target objects in the image to be detected (for example, if the target object Eiffel is detected in the image to be detected Tower, you can push the construction process of the Eiffel Tower to the user), introduction information (for example, if the target object Taj Mahal is detected in the image to be detected, you can push the introduction information of the Ej Mahal, such as building the Taj The historical background of the mausoleum, the artistic value of the Taj Mahal and/or the architectural style of the Taj Mahal, etc.), video information (for example, if a target monkey is detected in the image to be detected, the video information corresponding to the monkey can be pushed to the user , Such as the cartoon "Large Heaven", the TV series "Journey to the West" and/or the cartoon "Monkey Fishing", etc., image information (for example, if the target dog is detected in the image to be detected, it can be pushed to the user Image information corresponding to dogs, such as Husky photos, sketch images of Chihuahuas, and/or oil paintings of German shepherds, etc. and/or product information (for example, if a cat is detected in the image to be detected, you can push the purchase to the user Links to pet cats, links to buy cat-shaped dolls, links to buy a picture book "Cat and Mouse" and/or the location of a nearby cat cafe, etc.).

In addition, in the embodiment of the present application, in addition to pushing the three-dimensional information of the target object described in step S103, the remaining information to be pushed can be determined according to the type of user, for example, if the user of the information pushing method defined in the present application is Children 3-5 years old, in addition to pushing the three-dimensional information of the target object to the children, you can also push the animation and/or toy purchase links corresponding to the target object to the children. When the user is 3-5 children, you can Try to avoid pushing all text messages to children. Personalized recommendations based on the type of user can increase the viscosity of the user's use of the information push method defined in this application to a certain extent.

In addition, in the embodiment of the present application, if the judgment result of step S102 is affirmative, the information pushing method defined in the embodiment of the present application may also include the following steps:

Output evaluation request information, which is used to indicate whether the user is satisfied with this three-dimensional information push.

That is, after showing the three-dimensional information of the target object to the user, the user can be requested to evaluate the push, and the terminal device can send it to the backend server after receiving the user's evaluation, so that the The developer of the information push method can continuously update and update the software that implements the information push method of the present application (such as the update of the three-dimensional reconstruction method used) according to user evaluation, so as to better serve users.

In addition, in the embodiment of the present application, if the judgment result in step S102 is negative, that is, the target object is not detected in the image to be detected, a message for prompting the user that the three-dimensional structure cannot be displayed can be output; or, Prompt the user to circle the object to be viewed in the three-dimensional structure in the image to be detected, and transmit the image area circled by the user to the background server, so that the developer can push the application information according to the circled image area The software of the method is upgraded and updated (so that objects in the image area circled by the user can be detected), so as to better serve the user.

It can be seen from the above that in the technical solution provided in the first embodiment of the present application, the user can be pushed to display the relevant information for displaying the three-dimensional structure of the target object in the image based on only one image, which can avoid the user's online search operation. , To a certain extent, it can solve the technical problem that the current terminal equipment cannot make the user understand the external things more quickly and efficiently.

The following describes another information pushing method provided by Embodiment 2 of the present application. Similar to Embodiment 1, the information pushing method of Embodiment 2 of the present application is also applied to a terminal device. Please refer to FIG. 3 for the embodiment 2 of the present application. Information push methods include:

In step S301, obtain an image to be detected, and perform target detection on the image to be detected;

In step S302, it is determined whether one or more target objects are detected in the image to be detected;

In the second embodiment of the present application, the above steps S301-S302 are performed in the same way as the steps S101-S102 in the first embodiment. For details, please refer to the description of the first embodiment, which will not be repeated here.

In step S303, if a target object is detected in the above-mentioned image to be detected, then among the target objects of the above-mentioned image to be detected, a target object of interest to the user is determined;

In the second embodiment of the present application, the target objects of interest to the user can be determined from the detected target objects, and then only the three-dimensional information corresponding to the target objects of interest to the user is pushed to the user, wherein the step S303 The target object that the user is interested in can be multiple or one.

The method for determining the target object that the user is interested in may be: pushing the detected target object to the user, and the user selects the target object of interest.

Or, it may be: calculating the area intersection ratio of the image area occupied by each target object and the preset image area (the preset image area may be the middle area of the image to be detected) (Intersection-over-Union, IOU) Value, the target object corresponding to the IOU value greater than the IOU threshold and the maximum corresponding IOU value is determined as the target object of interest to the user, if there is no target object corresponding to the IOU value greater than the IOU threshold in the image to be detected, you can select any target The object serves as the target object of interest to the user.

Alternatively, the method shown in FIG. 4 may also be used to determine the target object of interest to the user.

As shown in FIG. 4, the target objects of interest to the user are determined through steps S401-S404.

In step S401, it is determined whether the number of target objects detected in the image to be detected is plural, if not, step S402 is executed, and if so, step S403 is executed;

In step S402, the target object in the image to be detected is determined as the target object of interest to the user;

If only one target object is detected in the image to be detected acquired in step S301, the target object is directly determined as the target object of interest to the user;

In step S403, the position information of each target object in the above-mentioned image to be detected, the image area indicated by each position information occupy the area ratio information of the above-mentioned image to be detected and/or the position information of each target object are indicated respectively The image area of is the confidence level of the corresponding target object;

In step S404, according to the position information of each target object in the image to be detected, the image area indicated by each position information occupies the area ratio information of the image to be detected and/or indicated by the position information of each target object The image area of is the confidence level of the corresponding target object, which determines the target object of interest to the user;

If multiple target objects are detected in the image to be detected acquired in step S301, the target objects of interest to the user need to be determined among the multiple target objects. In this step S403-S404, it is based on the position of each target object, the area ratio of the image area indicated by each position to the image to be detected and/or the confidence degree of the image area indicated by each position as the corresponding target object , To determine the target object of interest to the user. Wherein, the position of each target object and the confidence that the image area indicated by each position is the corresponding target object can be obtained in the detection result obtained after performing target detection on the image to be detected in step S301.

As shown in FIG. 5, assume that two target objects are detected in the image to be detected 501, namely a goat and a monkey. The position information of the goat is composed of the position information of each pixel in the image area 502, and the position information of the monkey is composed of the image area The position information of each pixel in 503 is composed of the confidence of the image area 502 for the goat is 0.8, and the confidence of the image area 503 for the monkey is 0.9, we can calculate the area ratio of the image area 502 and the image 501 to be detected The area ratio of the area 503 to the image to be detected 501, and the target object with the largest corresponding area ratio can be determined as the target object of interest to the user. Therefore, the goat in FIG. 5 can be determined as the target object of interest to the user.

In addition, the above step S403 may include the following steps:

S4031: Obtain the position information of each target object in the image to be detected;

S4032: Calculate the area ratio information of the image area indicated by the position information of each target object to the image to be detected respectively according to the position information of each target object, to obtain the area ratio information corresponding to each target object;

S4033: Obtain the confidence level of the image area indicated by the position information of each target object as the corresponding target object, and obtain the confidence level corresponding to each target object;

Accordingly, the above step S404 includes:

S4041: According to the position information of each target object, calculate the IOU value of the image area indicated by the position information of each target object and the area of the preset image area, and obtain the IOU value corresponding to each target object, where The preset image area is a preset image area in the image to be detected, and the preset image area may be a middle area of the image to be detected, such as the semi-transparent area 504 in FIG. 5;

S4042: Determine whether there is a target object corresponding to the IOU value greater than the IOU threshold, the corresponding area proportion information greater than the area proportion threshold, and the corresponding confidence greater than the confidence threshold among the target objects of the image to be detected;

S4043: If it exists, determine the target object whose corresponding IOU value is greater than the aforementioned IOU threshold, the corresponding area proportion information is greater than the aforementioned area proportion threshold, and the corresponding confidence degree is greater than the aforementioned confidence threshold value as the target object of interest to the user. In addition, if the result of the determination in step S4042 is not present, any target object may be selected as the target object of interest to the user.

In addition, the above step S4043 may further include:

If there is, it is determined whether the corresponding IOU value is greater than the above-mentioned IOU threshold, the corresponding area proportion information is greater than the above-mentioned area proportion threshold, and the corresponding confidence is greater than the above-mentioned confidence threshold.

If there is only one, the target object with the corresponding IOU value greater than the above-mentioned IOU threshold, the corresponding area proportion information greater than the above-mentioned area proportion threshold, and the corresponding confidence greater than the above-mentioned confidence threshold is determined as the target object of interest to the user;

If there are multiple, each target object with a corresponding IOU value greater than the aforementioned IOU threshold, corresponding area proportion information greater than the aforementioned area proportion threshold, and a corresponding confidence greater than the aforementioned confidence threshold is pushed to the user;

A target object selected by the user whose corresponding IOU value is greater than the aforementioned IOU threshold, corresponding area proportion information is greater than the aforementioned area proportion threshold, and whose corresponding confidence is greater than the aforementioned confidence threshold is determined as the target object of interest to the user.

In step S304, the three-dimensional information corresponding to the target object of interest is pushed to the user, wherein the three-dimensional information corresponding to each target object is obtained after the three-dimensional reconstruction of the target object to obtain the three-dimensional representation of the target object Structured information;

In this step S304, only the three-dimensional information of the target object of interest to the user is displayed to the user. This step S304 may specifically include the following steps:

S3041: Search for three-dimensional information corresponding to all the target objects of interest in the local and/or preset server;

S3042: determine whether all the three-dimensional information corresponding to the above-mentioned target objects of interest are found;

S3043: If the 3D information corresponding to all the target objects of interest is found, the 3D information corresponding to all the target objects of interest found from the local and/or the preset server is pushed to the user;

S3044: If all the three-dimensional information corresponding to the above-mentioned target objects of interest are not found, for each target object of interest that does not find the three-dimensional information, multiple images of different angles of the target object of interest are obtained, and Perform three-dimensional reconstruction based on the images of different angles, obtain three-dimensional information of the target object of interest, and push the three-dimensional information corresponding to each of the target objects of interest to the user.

In order to understand the above steps S3041-S3044 more clearly, the following uses FIG. 5 to illustrate:

Assuming that Figure 5 includes two target objects of interest to users, namely monkeys and goats, first find the 3D information corresponding to the monkey and the 3D information corresponding to the goat in the local and/or preset server, if the monkey and the 3D information corresponding to goats respectively, then push the found 3D information corresponding to monkeys and 3D information corresponding to goats; if only the 3D information corresponding to monkeys is found in the local and/or preset server, no goat corresponding 3D information, you can remind the user to take multiple images of the goat at different angles, and then reconstruct the goat based on the multiple images taken by the user to obtain the 3D information corresponding to the goat, and then push the 3D corresponding to the goat to the user Information and three-dimensional information corresponding to monkeys.

In addition, in the embodiment of the present application, the target object of interest to the user determined in step S303 may be saved in a preset interest list, which stores the target object of interest to the user within a preset time period And can push the target objects of interest of each user saved in the preset interest list to the user. For example, if the target interest list stored in the preset interest list contains the target objects that the user is interested in within one year of this year, the target interest objects of the year may be pushed to the user at 24:00 at the end of the year.

It can be seen from the above that the technical solution provided by the second embodiment of the present application is only to push the three-dimensional information of the target object of interest to the user, therefore, compared with the first embodiment, it can better meet the needs of the user. The second embodiment is the same as the first embodiment, and can also solve the technical problem that the current terminal device cannot make the user more quickly and efficiently understand the outside world.

It should be understood that the size of the sequence numbers of the steps in the above method embodiments does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application .

Embodiment 3 of the present application provides an information pushing device. As shown in FIG. 6, the information pushing device 600 includes:

The image acquisition module 601 is used to acquire an image to be detected and perform target detection on the image to be detected;

The target judgment module 602 is used to judge whether one or more target objects are detected in the image to be detected;

The three-dimensional information pushing module 603 is used to push the three-dimensional information corresponding to one or more target objects in the above-mentioned to-be-detected image to the user if a target object is detected in the above-mentioned to-be-detected image. The corresponding three-dimensional information is the information used to display the three-dimensional structure of the target object after three-dimensional reconstruction of the target object.

Optionally, the above three-dimensional information pushing module 603 includes:

The interest object determination submodule is used to determine the target object of interest to the user among the target objects of the image to be detected;

The three-dimensional information pushing submodule is used to push the three-dimensional information corresponding to the target object of interest to the user.

Optionally, the above information pushing device 600 further includes:

An interest object saving module, configured to save the determined target object of interest to a preset interest list, where the preset interest list stores target objects that the user is interested in within a preset time period;

The interest list display module is used to display the target objects of interest of each user saved in the preset interest list.

Optionally, the above three-dimensional information pushing submodule includes:

A searching unit, used to search for the three-dimensional information corresponding to all the above-mentioned target objects of interest in the local and/or preset server;

The search and judgment unit is used to judge whether the 3D information corresponding to all the above-mentioned target objects of interest are found;

The first pushing unit is configured to: if the 3D information corresponding to all the above-mentioned target objects of interest is found, then the 3D information corresponding to all the above-mentioned target objects of interest found from the local and/or the preset server respectively Push to users;

The second push unit is used to obtain multiple different target objects of interest for each target object of interest that does not find the three-dimensional information if all the three-dimensional information corresponding to the target objects of interest are not found An image at an angle, and perform three-dimensional reconstruction based on the images at different angles to obtain the three-dimensional information of the target object of interest, and push the three-dimensional information corresponding to each of the target objects of interest to the user.

Optionally, the above-mentioned interest object determination submodule includes:

The object number judging unit is used to judge whether the number of target objects detected in the image to be detected is more than one;

A first interest object determination unit, configured to determine the target object in the image to be detected as the target object of interest to the user if the number of detected target objects is only one;

The information acquisition unit is configured to acquire the position information of each target object in the image to be detected and the image area indicated by the position information of each target object respectively occupy the The image area indicated by the area ratio information of the detection image and/or the position information of each target object is the confidence of the corresponding target object;

A second interest object determination unit, according to the location information of each target object in the image to be detected, the image area indicated by the location information of each target object, respectively, occupying the area ratio information of the image to be detected and/or each The image area indicated by the position information of the target object is the confidence of the corresponding target object, and the target object of interest to the user is determined.

Optionally, the above information acquisition unit includes:

The position information subunit is used to obtain the position information of each target object in the image to be detected;

The area ratio subunit is used to calculate the area ratio information of the image area indicated by the position information of each target object to the above-mentioned image to be detected according to the position information of each target object, to obtain the area ratio information corresponding to each target object ;

The confidence subunit is used to obtain the confidence level of the corresponding target object in the image area indicated by the position information of each target object, and obtain the confidence level corresponding to each target object;

Accordingly, the above second interest object determination unit includes:

The IOU sub-unit is used to calculate the area of each target object indicated by the position information of each target object and the area of the preset image area intersect with the IOU value according to the position information of each target object to obtain the corresponding IOU value, wherein the preset image area is the preset image area in the image to be detected;

The judgment subunit is used for judging whether each target object of the above-mentioned image to be detected has a corresponding IOU value greater than the IOU threshold, corresponding area proportion information is greater than the area proportion threshold, and the corresponding confidence level is greater than the confidence threshold target object;

The second interest object determination subunit is used to determine if the corresponding IOU value is greater than the IOU threshold, the corresponding area ratio information is greater than the area ratio threshold, and the corresponding target object with a confidence greater than the confidence threshold is determined to be of interest Target audience.

Optionally, the above second interest object determination subunit includes:

The number judging unit is used to judge whether the corresponding IOU value is greater than the above-mentioned IOU threshold, the corresponding area proportion information is greater than the above-mentioned area proportion threshold, and the corresponding confidence level is greater than the above-mentioned confidence threshold if there are multiple target objects;

The first interest object determination cell is used to determine as the user a target object whose corresponding IOU value is greater than the aforementioned IOU threshold, corresponding area proportion information is greater than the aforementioned area proportion threshold, and whose corresponding confidence is greater than the aforementioned confidence threshold Interested target object;

Pushing a small unit is used to push each target object whose corresponding IOU value is greater than the above-mentioned IOU threshold, corresponding area proportion information is greater than the above-mentioned area proportion threshold, and whose corresponding confidence is greater than the above-mentioned confidence threshold if there are multiple units;

The second interest object determination cell is used to determine that the user selects a target object whose corresponding IOU value is greater than the above-mentioned IOU threshold, corresponding area proportion information is greater than the above-mentioned area proportion threshold, and the corresponding confidence is greater than the above-mentioned confidence threshold Target audience.

Optionally, the above information pushing device 600 further includes:

The remaining information pushing module is used to push the production process, introduction information, video information, image information and/or commodity information corresponding to one or more target objects in the image to be detected to the user;

and / or

The evaluation request module is used to output evaluation request information, and the evaluation request information is used to indicate whether the user is satisfied with the current three-dimensional information push.

It should be noted that the information interaction and execution process between the above devices/units are based on the same concept as the method embodiments of the present application, and their specific functions and technical effects can be referred to the method embodiments. I will not repeat them here.

7 is a schematic diagram of a terminal device provided in Embodiment 4 of the present application. As shown in FIG. 7, the terminal device 7 of this embodiment includes: a processor 70, a memory 71, and a computer program 72 stored in the memory 71 and executable on the processor 70. When the processor 70 executes the computer program 72, the steps in the above method embodiments are implemented, for example, steps S101 to S103 shown in FIG. 1. Alternatively, when the processor 70 executes the computer program 72, the functions of the modules/units in the foregoing device embodiments are realized, for example, the functions of the modules 601 to 603 shown in FIG. 6.

Exemplarily, the computer program 72 may be divided into one or more modules/units, and the one or more modules/units are stored in the memory 71 and executed by the processor 70 to complete the application. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions. The instruction segments are used to describe the execution process of the computer program 72 in the terminal device 7. For example, the above-mentioned computer program 72 can be divided into an image acquisition module, a target judgment module, and a three-dimensional information push module. The specific functions of each module are as follows:

If a target object is detected in the above-mentioned image to be detected, the three-dimensional information corresponding to one or more target objects in the above-mentioned image to be detected is pushed to the user, wherein the three-dimensional information corresponding to each target object is the target After the object is three-dimensionally reconstructed, the obtained information for displaying the three-dimensional structure of the target object is obtained.

The above terminal device may include, but is not limited to, the processor 70 and the memory 71. Those skilled in the art may understand that FIG. 7 is only an example of the terminal device 7 and does not constitute a limitation on the terminal device 7, and may include more or less components than those illustrated, or a combination of certain components, or different components. For example, the above terminal device may further include an input and output device, a network access device, a bus, and the like.

The so-called processor 70 may be a central processing unit (Central Processing Unit, CPU), or other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), Field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The above-mentioned memory 71 may be an internal storage unit of the above-mentioned terminal device 7, such as a hard disk or a memory of the terminal device 7. The memory 71 may also be an external storage device of the terminal device 7, for example, a plug-in hard disk equipped on the terminal device 7, a smart memory card (Smart) Media (SMC), a secure digital (SD) card, and a flash memory Card (Flash), etc. Further, the memory 71 may include both the internal storage unit of the terminal device 7 and the external storage device. The memory 71 is used to store the computer program and other programs and data required by the terminal device. The above-mentioned memory 71 may also be used to temporarily store data that has been output or will be output.

Those skilled in the art can clearly understand that, for convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example for illustration. In practical applications, the above-mentioned functions may be allocated by different functional units, Module completion means that the internal structure of the above device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above integrated unit may use hardware It can also be implemented in the form of software functional units. In addition, the specific names of each functional unit and module are only for the purpose of distinguishing each other, and are not used to limit the protection scope of the present application. For the specific working processes of the units and modules in the above system, reference may be made to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

In the above embodiments, the description of each embodiment has its own emphasis. For a part that is not detailed or recorded in an embodiment, you can refer to the related descriptions of other embodiments.

Those of ordinary skill in the art may realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application of the technical solution and design constraints. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

In the embodiments provided in this application, it should be understood that the disclosed device/terminal device and method may be implemented in other ways. For example, the device/terminal device embodiments described above are only schematic. For example, the above-mentioned division of modules or units is only a division of logical functions. In actual implementation, there may be other divisions, such as multiple units or Components can be combined or integrated into another system, or some features can be ignored or not implemented.

The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units may be integrated into one unit. The above integrated unit can be implemented in the form of hardware or software function unit.

If the above integrated module/unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application can implement all or part of the processes in the methods of the above embodiments, and can also be completed by instructing relevant hardware through a computer program. The above computer program can be stored in a computer-readable storage medium, and the computer program When executed by the processor, the steps of the foregoing method embodiments may be implemented. Wherein, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code may be in the form of source code, object code, executable file or some intermediate form. The aforementioned computer readable medium may include: any entity or device capable of carrying the aforementioned computer program code, recording medium, U disk, removable hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random Access memory (RAM, Random Access Memory), electrical carrier signals, telecommunications signals, and software distribution media, etc. It should be noted that the content contained in the above computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in jurisdictions. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media are not Including electrical carrier signals and telecommunications signals.

The above-mentioned embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, persons of ordinary skill in the art should understand that they can still perform the foregoing embodiments The recorded technical solutions are modified, or some of the technical features are equivalently replaced; and these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of this application, and should be included in this Within the scope of protection applied for.

Claims

An information push method, characterized in that it includes:

Acquiring an image to be detected, and performing target detection on the image to be detected;

Determine whether one or more target objects are detected in the image to be detected;

If a target object is detected in the image to be detected, then:

Pushing the three-dimensional information corresponding to one or more target objects in the image to be detected to the user respectively, wherein the three-dimensional information corresponding to each target object is obtained after performing three-dimensional reconstruction of the target object to display the Information about the three-dimensional structure of the target object.
The information pushing method according to claim 1, wherein the pushing the three-dimensional information corresponding to one or more target objects in the image to be detected to the user respectively comprises:

Among the target objects of the image to be detected, determine the target objects of interest to the user;

Push the three-dimensional information corresponding to the target object of interest to the user.
The information pushing method according to claim 2, wherein if a target object is detected in the image to be detected, the information pushing method further comprises:

Saving the determined target object of interest to a preset interest list, where the target object of interest to the user within a preset time period is stored in the preset interest list;

Display target objects of interest of each user saved in the preset interest list.
The information pushing method according to claim 2, wherein the pushing the three-dimensional information corresponding to the target object of interest to the user includes:

Searching for the three-dimensional information corresponding to all the target objects of interest in the local and/or preset server;

Determine whether to find the three-dimensional information corresponding to all the target objects of interest;

If three-dimensional information corresponding to all the target objects of interest is found, then:

Pushing the 3D information corresponding to all the target objects of interest found from the local and/or the preset server to the user;

If the 3D information corresponding to all the target objects of interest is not found, then:

For each target object of interest that does not find three-dimensional information, obtain multiple images of different angles of the target object of interest, and perform three-dimensional reconstruction based on the images of different angles to obtain the target object of interest Three-dimensional information;

Push the three-dimensional information corresponding to each of the interested target objects to the user.
The information pushing method according to claim 2, wherein, among the target objects of the image to be detected, determining the target objects of interest to the user includes:

Determine whether the number of target objects detected in the image to be detected is more than one;

If there is only one detected target object, the target object in the image to be detected is determined as the target object of interest to the user;

If the number of detected target objects is multiple, then:

Acquiring the position information of each target object in the image to be detected, the image area indicated by the position information of each target object to the area ratio information of the image to be detected and/or the position information of each target object The image area of is the confidence level of the corresponding target object;

According to the position information of each target object in the image to be detected, the image area indicated by the position information of each target object occupies area ratio information of the image to be detected, and/or the The image area indicated by the position information is the confidence of the corresponding target object, and the target object of interest to the user is determined.
The information pushing method according to claim 5, characterized in that the image areas indicated by the acquiring position information of each target object in the image to be detected and the position information of each target object respectively occupy the to-be-detected The image area indicated by the area ratio information of the image and/or the position information of each target object is the confidence of the corresponding target object, including:

Acquiring the position information of each target object in the image to be detected;

Calculating, according to the position information of each target object, the area ratio information of the image area indicated by the position information of each target object to the image to be detected respectively, to obtain the area ratio information corresponding to each target object;

Acquiring the image area indicated by the position information of each target object as the confidence level of the corresponding target object, and obtaining the confidence level corresponding to each target object;

Correspondingly, the image area indicated by the position information of each target object in the image to be detected and the image area indicated by the position information of each target object respectively occupy the area ratio information of the image to be detected and/or the The image area indicated by the position information of each target object is the confidence level of the corresponding target object, and the target object of interest to the user is determined, including:

According to the position information of each target object, calculate the IOU value of the image area indicated by the position information of each target object and the area of the preset image area, respectively, and obtain the IOU value corresponding to each target object. The preset image area is a preset image area in the image to be detected;

Judging whether each target object of the image to be detected has a corresponding IOU value greater than the IOU threshold, corresponding area proportion information is greater than the area proportion threshold, and the corresponding confidence level is greater than the confidence threshold target object;

If it exists, the target object with the corresponding IOU value greater than the IOU threshold, the corresponding area proportion information greater than the area proportion threshold, and the corresponding confidence greater than the confidence threshold is determined as the target object of interest to the user.
The information pushing method according to claim 6, wherein, if it exists, the corresponding IOU value is greater than the IOU threshold, the corresponding area ratio information is greater than the area ratio threshold, and the corresponding confidence is greater than the confidence The target object with a degree threshold is determined as the target object of interest to the user, including:

If there is, it is determined whether the corresponding IOU value is greater than the IOU threshold, the corresponding area ratio information is greater than the area ratio threshold, and the corresponding confidence level is greater than the confidence threshold target object;

If there is only one, the target object with the corresponding IOU value greater than the IOU threshold, the corresponding area proportion information greater than the area proportion threshold, and the corresponding confidence greater than the confidence threshold is determined as the target object of interest to the user;

If there are multiple, each target object with a corresponding IOU value greater than the IOU threshold, corresponding area proportion information greater than the area proportion threshold, and a corresponding confidence greater than the confidence threshold is pushed to the user;

A target object selected by the user for which the corresponding IOU value is greater than the IOU threshold, the corresponding area ratio information is greater than the area ratio threshold, and the corresponding confidence level is greater than the confidence threshold value is determined as the target object of interest to the user.
The information pushing method according to any one of claims 1 to 7, wherein, if a target object is detected in the image to be detected, the information pushing method further includes:

Push the production process, introduction information, video information, image information and/or commodity information corresponding to one or more target objects in the image to be detected to the user;

and / or

Output evaluation request information, which is used to indicate whether the user is satisfied with the current three-dimensional information push.
An information pushing device, characterized in that it includes:

The image acquisition module is used to acquire the image to be detected and perform target detection on the image to be detected;

The target judgment module is used to judge whether one or more target objects are detected in the image to be detected;

The three-dimensional information pushing module is used to push the three-dimensional information corresponding to one or more target objects in the to-be-detected image to the user if a target object is detected in the to-be-detected image, wherein each target object The corresponding three-dimensional information is the information used to display the three-dimensional structure of the target object after three-dimensional reconstruction of the target object.
The information pushing device according to claim 9, wherein the three-dimensional information pushing module includes:

The interest object determination submodule is used to determine the target object of interest to the user among the target objects of the image to be detected;

The three-dimensional information pushing submodule is used to push the three-dimensional information corresponding to the target object of interest to the user.
The information pushing device according to claim 10, further comprising:

An interest object saving module, configured to save the determined target object of interest to a preset interest list, where the preset interest list stores target objects that the user is interested in within a preset time period;

The interest list display module is used to display the target objects of interest of each user saved in the preset interest list.
A terminal device includes a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, when the processor executes the computer program, the following steps are implemented:

Acquiring an image to be detected, and performing target detection on the image to be detected;

Determine whether one or more target objects are detected in the image to be detected;

If a target object is detected in the image to be detected, then:

Pushing the three-dimensional information corresponding to one or more target objects in the image to be detected to the user respectively, wherein the three-dimensional information corresponding to each target object is obtained after performing three-dimensional reconstruction of the target object to display the Information about the three-dimensional structure of the target object.
The terminal device according to claim 12, wherein the pushing the three-dimensional information corresponding to one or more target objects in the image to be detected respectively to the user includes:

Among the target objects of the image to be detected, determine the target objects of interest to the user;

Push the three-dimensional information corresponding to the target object of interest to the user.
The terminal device according to claim 13, wherein if a target object is detected in the image to be detected, the processor further implements the following steps when executing the computer program:

Saving the determined target object of interest to a preset interest list, where the target object of interest to the user within a preset time period is stored in the preset interest list;

Display target objects of interest of each user saved in the preset interest list.
The terminal device according to claim 13, wherein the pushing the three-dimensional information corresponding to the target object of interest to the user includes:

Searching for the three-dimensional information corresponding to all the target objects of interest in the local and/or preset server;

Determine whether to find the three-dimensional information corresponding to all the target objects of interest;

If three-dimensional information corresponding to all the target objects of interest is found, then:

Pushing the 3D information corresponding to all the target objects of interest found from the local and/or the preset server to the user;

If the 3D information corresponding to all the target objects of interest is not found, then:

For each target object of interest that does not find three-dimensional information, obtain multiple images of different angles of the target object of interest, and perform three-dimensional reconstruction based on the images of different angles to obtain the target object of interest Three-dimensional information;

Push the three-dimensional information corresponding to each of the interested target objects to the user.
The terminal device according to claim 13, wherein, among the target objects of the image to be detected, determining the target objects of interest to the user includes:

Determine whether the number of target objects detected in the image to be detected is more than one;

If there is only one detected target object, the target object in the image to be detected is determined as the target object of interest to the user;

If the number of detected target objects is multiple, then:

Acquiring the position information of each target object in the image to be detected, the image area indicated by the position information of each target object to the area ratio information of the image to be detected and/or the position information of each target object The image area of is the confidence level of the corresponding target object;

According to the position information of each target object in the image to be detected, the image area indicated by the position information of each target object occupies area ratio information of the image to be detected, and/or the The image area indicated by the position information is the confidence of the corresponding target object, and the target object of interest to the user is determined.
The terminal device according to claim 16, wherein the image area indicated by the acquisition of the position information of each target object in the image to be detected and the position information of each target object respectively occupy the image to be detected The image area indicated by the area ratio information and/or the location information of each target object is the confidence of the corresponding target object, including:

Acquiring the position information of each target object in the image to be detected;

Calculating, according to the position information of each target object, the area ratio information of the image area indicated by the position information of each target object to the image to be detected respectively, to obtain the area ratio information corresponding to each target object;

Acquiring the image area indicated by the position information of each target object as the confidence level of the corresponding target object, and obtaining the confidence level corresponding to each target object;

Correspondingly, the image area indicated by the position information of each target object in the image to be detected and the image area indicated by the position information of each target object respectively occupy the area ratio information of the image to be detected and/or the The image area indicated by the position information of each target object is the confidence level of the corresponding target object, and the target object of interest to the user is determined, including:

According to the position information of each target object, calculate the IOU value of the image area indicated by the position information of each target object and the area of the preset image area, respectively, and obtain the IOU value corresponding to each target object. The preset image area is a preset image area in the image to be detected;

Judging whether each target object of the image to be detected has a corresponding IOU value greater than the IOU threshold, corresponding area proportion information is greater than the area proportion threshold, and the corresponding confidence level is greater than the confidence threshold target object;

If it exists, the target object with the corresponding IOU value greater than the IOU threshold, the corresponding area proportion information greater than the area proportion threshold, and the corresponding confidence greater than the confidence threshold is determined as the target object of interest to the user.
The terminal device according to claim 17, characterized in that, if present, the corresponding IOU value is greater than the IOU threshold, the corresponding area ratio information is greater than the area ratio threshold, and the corresponding confidence is greater than the confidence The threshold target object is determined as the target object of interest to the user, including:

If there is, it is determined whether the corresponding IOU value is greater than the IOU threshold, the corresponding area ratio information is greater than the area ratio threshold, and the corresponding confidence level is greater than the confidence threshold target object;

If there is only one, the target object with the corresponding IOU value greater than the IOU threshold, the corresponding area proportion information greater than the area proportion threshold, and the corresponding confidence greater than the confidence threshold is determined as the target object of interest to the user;

If there are multiple, each target object with a corresponding IOU value greater than the IOU threshold, corresponding area proportion information greater than the area proportion threshold, and a corresponding confidence greater than the confidence threshold is pushed to the user;

A target object selected by the user for which the corresponding IOU value is greater than the IOU threshold, the corresponding area ratio information is greater than the area ratio threshold, and the corresponding confidence level is greater than the confidence threshold value is determined as the target object of interest to the user.
The device terminal according to any one of claims 12 to 18, wherein, if a target object is detected in the image to be detected, the processor further implements the following steps when executing the computer program:

Push the production process, introduction information, video information, image information and/or commodity information corresponding to one or more target objects in the image to be detected to the user;

and / or

Output evaluation request information, which is used to indicate whether the user is satisfied with the current three-dimensional information push.
A computer-readable storage medium storing a computer program, characterized in that, when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.