CN114419292A - Image processing method, device, equipment and storage medium - Google Patents

Abstract

The embodiment of the disclosure discloses an image processing method, an image processing device, image processing equipment and a storage medium. Acquiring a target object detected in a current key frame and a rendered virtual object corresponding to the current key frame; wherein the key frame is a frame for triggering object detection; determining a virtual object to be newly added according to the detected target object and the visual field space queue; determining a virtual object to be deleted according to the rendered virtual object and the view space queue; and updating the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted. According to the image processing method provided by the embodiment of the disclosure, the virtual object to be newly added is determined according to the detected target object and the visual field space queue, and the virtual object to be deleted is determined according to the rendered virtual object and the visual field space queue, so that the calculation amount during updating of the virtual object can be reduced, and the image processing efficiency is improved.

Description

Image processing method, device, equipment and storage medium

technical field

The embodiments of the present disclosure relate to the field of augmented reality technologies, and in particular, to an image processing method, apparatus, device, and storage medium.

Background technique

Overloading virtual objects on detected target objects is a common application scenario in augmented reality. In this application scenario, object detection in space is required. Usually, after multiple detections of the same target object in space, it is necessary to perform overlapping judgment and remove the target object outside the field of view. In the prior art, it is possible to realize the filtering of repeated objects and the deletion of out-of-view objects by means of target tracking and feature matching, but this method requires a large amount of calculation and is inefficient.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure provide an image processing method, apparatus, device, and storage medium, which can reduce the amount of calculation when virtual objects are updated, thereby improving image processing efficiency.

In a first aspect, an embodiment of the present disclosure provides an image processing method, including:

Obtain the target object detected in the current key frame and the rendered virtual object corresponding to the current key frame; wherein, the key frame is the frame that triggers object detection;

Determine the virtual object to be added according to the detected target object and the field of view space queue; wherein, the field of view space queue is composed of the field of view space corresponding to the N nearest key frames, and the current key frame is the Nth in the queue Keyframe;

Determine the virtual object to be deleted according to the rendered virtual object and the view space queue;

The virtual object corresponding to the current key frame is updated according to the virtual object to be added and the virtual object to be deleted.

In a second aspect, an embodiment of the present disclosure further provides an image processing apparatus, including:

a target object detection module, configured to acquire the target object detected in the current key frame and the rendered virtual object corresponding to the current key frame; wherein the key frame is the frame that triggers object detection;

The virtual object to be added determination module is used to determine the virtual object to be added according to the detected target object and the field of view space queue; wherein, the field of view space queue is composed of the field of view space corresponding to the N nearest key frames, and the The current key frame is the Nth key frame in the queue;

a virtual object determination module to be deleted, configured to determine the virtual object to be deleted according to the rendered virtual object and the field of view space queue;

The virtual object updating module is configured to update the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted.

In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

one or more processing devices;

a storage device for storing one or more programs;

When the one or more programs are executed by the one or more processing apparatuses, the one or more processing apparatuses implement the image processing method according to the embodiments of the present disclosure.

In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processing apparatus, implements the image processing method according to the embodiment of the present disclosure.

Embodiments of the present disclosure disclose an image processing method, apparatus, device, and storage medium. Obtain the target object detected in the current key frame and the rendered virtual object corresponding to the current key frame; the key frame is the frame that triggers object detection; the virtual object to be added is determined according to the detected target object and the field of view space queue; wherein , the field of view space queue consists of the field of view space corresponding to the N nearest key frames, and the current key frame is the Nth key frame in the queue; the virtual object to be deleted is determined according to the rendered virtual object and the field of view space queue; the virtual object to be added is determined according to the virtual object to be added. The virtual object corresponding to the current key frame is updated with the virtual object to be deleted. The image processing method provided by the embodiment of the present disclosure determines the virtual object to be added according to the detected target object and the field of view space queue, and determines the virtual object to be deleted according to the rendered virtual object and the field of view space queue, which can reduce the calculation when the virtual object is updated. to improve image processing efficiency.

Description of drawings

1 is a flowchart of an image processing method in an embodiment of the present disclosure;

2 is a schematic diagram of an optic frustum in an embodiment of the present disclosure;

3 is an example diagram of a newly added target object in a current key frame in an embodiment of the present disclosure;

4 is a schematic structural diagram of an image processing apparatus in an embodiment of the present disclosure;

FIG. 5 is a schematic structural diagram of an electronic device in an embodiment of the present disclosure.

Detailed ways

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for the purpose of A more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes, and are not intended to limit the protection scope of the present disclosure.

It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in different orders and/or in parallel. Furthermore, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this regard.

As used herein, the term "including" and variations thereof are open-ended inclusions, ie, "including but not limited to". The term "based on" is "based at least in part on." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below.

It should be noted that concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or interdependence.

It should be noted that the modifications of "a" and "a plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, they should be understood as "one or a plurality of". multiple".

The names of messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of these messages or information.

FIG. 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure. This embodiment can be applied to a situation where a virtual object in a screen in a three-dimensional space is updated. The method can be executed by an image processing apparatus. It can be composed of hardware and/or software, and can generally be integrated into a device with image processing functions, which can be an electronic device such as a server, a mobile terminal, or a server cluster. As shown in Figure 1, the method specifically includes the following steps:

S110: Acquire the target object detected in the current key frame and the rendered virtual object corresponding to the current key frame.

The key frame is a frame that triggers object detection. When the set trigger condition is met, the terminal device triggers the object detection module to detect the target object on the current screen. The set trigger condition may be: every set time length, every set number of video frames, or the number of currently rendered virtual objects is less than a set value, and the like. In this embodiment, the existing target object detection algorithm may be used to detect the target object on the current picture, which is not limited here.

The rendered virtual object may be understood as a virtual object that has been displayed and mounted on an object in a three-dimensional space, and the virtual object is a 3D object. The virtual objects can be of any theme category, for example: "Lantern Festival" theme, the virtual objects can be: virtual lanterns, virtual dumplings, etc., which are not limited here.

S120: Determine the virtual object to be added according to the detected target object and the field of view space queue.

Among them, the view space queue consists of the view space corresponding to the N key frames closest to the current moment, the current key frame is the N th key frame in the queue, the previous key frame is the N-1 th frame, and so on. The field of view space can be understood as the area visible on the screen in the three-dimensional world, that is, the field of view of the virtual camera. That is, the field of view is a three-dimensional space that takes the shooting angle of the camera as the viewing angle and falls within the shooting range of the camera. Optionally, the visual field space is represented by an optic frustum, and the optic frustum is surrounded by a plurality of boundary surfaces. In this embodiment, the optic frustum is enclosed by 6 boundary surfaces, which are a near plane (Near plane) and a distance plane respectively. (Far plane) and four side interfaces. Exemplarily, FIG. 2 is a schematic diagram of a viewing frustum in this embodiment, and in the figure, Oc represents a camera. The set of platform spaces formed by the cone i in space is Qi. For the view space queue, it is the set of regions formed by the view space corresponding to N key frames, expressed as: Q=Q ₁ ∪Q ₂ ∪...∪Q _N , if the three-dimensional space point p is located in the space Q, it means is p∈Q. Due to the memory limitation of the computing unit and the increase of the time difference, the reference value of the information of the old keyframes for the new keyframes decreases, so the old keyframes can be removed at an appropriate time. Therefore, the N recent keyframes are stored in the field of view space queue. The keyframe corresponding to the field of view space.

The virtual object to be added is determined by the target object added in the current key frame. If a target object is added in the current key frame, a new virtual object needs to be mounted on the newly added target object. Specifically, the method of determining the virtual object to be added according to the detected target object and the field of view space queue may be: determining the new target object according to the detected target object and the field of view space queue; determining the virtual object to be added according to the newly added target object object.

Among them, the newly added target object can be understood as the target object newly detected in the current key frame, that is, not detected in the previous key frame. The process of determining the virtual object to be added according to the newly added target object may be: determining the virtual object frame according to the detection frame of the newly added target object, obtaining the corresponding material in the virtual object, and adding the material to the virtual object frame for rendering, Get new virtual objects.

In this embodiment, the method for determining the newly added target object according to the detected target object and the field of view space queue may be: judging whether the detected target object falls into at least the first to N-1th field of view space in the field of view space queue One; if not, the detected target object is a new target object; if so, the detected object is an old target object.

则表明该检测到的目标物体为新增目标物体。示例性的，图3是本实施例中在当前关键帧新增的目标物体示例图，如图3所示，检测到的目标物体未落入前1到N-1帧的任一视野空间，则为新增目标物体。Specifically, if none of the detected target objects fall into the first to N-1th view spaces in the view space queue, the detected target object is a newly added target object. Assuming that the target object is represented by the center point p of the target object, if

It indicates that the detected target object is a new target object. Exemplarily, FIG. 3 is an example diagram of a target object newly added in the current key frame in this embodiment. As shown in FIG. 3 , the detected target object does not fall into any visual field of the previous 1 to N-1 frames, It is a new target object.

Optionally, the way of judging whether the detected target object falls into the first to N-1th view spaces in the view space queue may be: for each view space in the 1st to N-1th view spaces, set the Project the field of view space to the camera plane to obtain the two-dimensional area corresponding to the field of view space; project the three-dimensional center point of the detected target object to the camera plane to obtain the two-dimensional center point; determine whether the two-dimensional center point is in the two-dimensional area; , the detected target object falls into the visual field.

In this embodiment, the field of view space can be projected onto the camera plane and the detected three-dimensional center point of the target object can be projected onto the camera plane based on the projection principle. The way of judging whether the two-dimensional center point is in the two-dimensional area may be to judge whether the coordinates of the two-dimensional center satisfy the function set corresponding to the two-dimensional area, and if so, the two-dimensional center point is in the two-dimensional area. If the two-dimensional center point is in the two-dimensional area, it means that the three-dimensional center point of the target object is inside the viewing frustum, and the detected target object at this time falls into the visual field space. Representing the target object by the three-dimensional center point can greatly reduce the computational complexity.

Optionally, the method of judging whether the detected target object falls into the first to N-1th viewing spaces in the viewing space queue may be: for each viewing space in the first to N-1th viewing spaces, obtain The detected three-dimensional center point of the target object is a polygonal pyramid composed of multiple boundary surfaces corresponding to the visual field space; the volume sum of the multiple polygonal pyramids is calculated; if the volume is equal to the volume of the visual cone corresponding to the visual field space, the detection The arriving target object falls into the field of view space.

In this embodiment, the viewing cone corresponding to the visual field space is surrounded by 6 boundary surfaces, and each surface includes four sides. Therefore, the three-dimensional center point and each boundary surface form a quadrangular pyramid, thereby obtaining 6 quadrangular pyramids. Calculate the volume of each quadrangular pyramid separately, and sum the volumes of the 6 quadrangular pyramids. If the volume sum is equal to the volume of the viewing cone, it means that the three-dimensional center point of the target object is inside the viewing cone, and the target detected at this time is Objects fall into this field of view.

S130: Determine the virtual object to be deleted according to the rendered virtual object and the view space queue.

In this embodiment, if the rendered virtual object has not been detected for a period of time, in order to save resources and memory, the rendered virtual object needs to be deleted.

Specifically, the method of determining the virtual object to be deleted according to the rendered virtual object and the field of view space queue may be: judging whether the rendered virtual object falls into the recently set number of field of view spaces in the field of view space queue; The virtual object is determined to be the virtual object to be deleted.

Wherein, the set number is less than or equal to N, for example, N/2 may be taken. For example, if N is 10, the set number may be five. In this embodiment, if the rendered virtual objects all fall into the field of view space of the most recently set number, the rendered virtual objects are virtual objects to be deleted.

Optionally, the method of judging whether the rendered virtual object falls into the most recently set number of view spaces in the view space may be: for each view space of the most recently set number in the view space queue, project the view space to the camera plane. , obtain the two-dimensional area corresponding to the field of view space; project the three-dimensional center point of the rendered virtual object to the camera plane to obtain the two-dimensional center point; determine whether the two-dimensional center point is in the two-dimensional area; if so, the rendered virtual object falls into the field of vision.

In this embodiment, the visual field space can be projected to the camera plane and the three-dimensional center point of the rendered virtual object can be projected to the camera plane based on the projection principle. The way of judging whether the two-dimensional center point is in the two-dimensional area may be to judge whether the coordinates of the two-dimensional center satisfy the function set corresponding to the two-dimensional area, and if so, the two-dimensional center point is in the two-dimensional area. If the 2D center point is in the 2D area, it means that the 3D center point of the rendered virtual object is inside the viewing frustum, and the detected rendered virtual object at this time falls into the visual field space. Representing rendered virtual objects by 3D center points can greatly reduce the amount of computation.

Optionally, the method of judging whether the rendered virtual object falls into the most recently set number of view spaces in the view space may be: for each view space of the most recently set number in the view space queue, obtain the three-dimensional view of the rendered virtual object. The center point is a polygonal pyramid surrounded by multiple boundary surfaces corresponding to the visual field; the volume sum of the multiple polygonal pyramids is calculated; if the volume is equal to the volume of the visual cone corresponding to the visual field, the rendered virtual object falls into the visual field space.

In this embodiment, the viewing cone corresponding to the visual field space is surrounded by 6 boundary surfaces, and each surface includes four sides. Therefore, the three-dimensional center point and each boundary surface form a quadrangular pyramid, thereby obtaining 6 quadrangular pyramids. Calculate the volume of each quadrangular pyramid separately, and sum the volumes of the 6 quadrangular pyramids. If the volume sum is equal to the volume of the viewing frustum, it means that the 3D center point of the rendered virtual object is inside the viewing frustum, and the rendering has been done at this time. The virtual object falls into the field of view.

S140: Update the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted.

Specifically, the process of updating the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted may be as follows: mount the virtual object to be added at the position corresponding to the newly added target object; Object deletion.

其中，P_k表示更新后当前关键帧对应的已渲染虚拟物体，D_k表示新挂载的虚拟物体。In the current keyframe, the set of rendered virtual objects after adding virtual objects is:

Among them, P _k represents the rendered virtual object corresponding to the current key frame after the update, and D _k represents the newly mounted virtual object.

In the current key frame, the set of rendered virtual objects after deleting the virtual objects is: P _k =P _k-1 ∩(Q ₁ ∪Q ₂ ∪…∪Q _N ), where P _k represents the updated virtual objects corresponding to the current key frame after the update. Render virtual objects.

According to the technical solution of the embodiment of the present disclosure, the target object detected in the current key frame and the rendered virtual object corresponding to the current key frame are acquired; wherein, the key frame is the frame that triggers object detection; according to the detected target object and the field of view space queue Determine the virtual object to be added; wherein, the field of view space queue consists of the field of view space corresponding to the N nearest key frames, and the current key frame is the Nth key frame in the queue; determine the virtual object to be deleted according to the rendered virtual object and the field of view space queue Object; the virtual object corresponding to the current key frame is updated according to the virtual object to be added and the virtual object to be deleted. The image processing method provided by the embodiment of the present disclosure determines the virtual object to be added according to the detected target object and the field of view space queue, and determines the virtual object to be deleted according to the rendered virtual object and the field of view space queue, which can reduce the calculation when the virtual object is updated. to improve image processing efficiency.

FIG. 4 is a schematic structural diagram of an image processing apparatus disclosed in an embodiment of the present disclosure. As shown in FIG. 4 , the apparatus includes:

The target object detection module 210 is used to obtain the target object detected in the current key frame and the rendered virtual object corresponding to the current key frame; wherein, the key frame is a frame that triggers object detection;

The virtual object to be added determination module 220 is configured to determine the virtual object to be added according to the detected target object and the field of view space queue; wherein, the field of view space queue is composed of the fields of view corresponding to the N key frames closest to the current moment Space composition, the current key frame is the Nth key frame in the queue;

The virtual object to be deleted determination module 230 is configured to determine the virtual object to be deleted according to the rendered virtual object and the field of view space queue;

The virtual object updating module 240 is configured to update the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted.

Optionally, the virtual object determination module 220 to be added is further configured to:

Determine the newly added target object according to the detected target object and the field of view space queue;

Determine the virtual object to be added according to the newly added target object;

Optionally, the virtual object update module 240 is further configured to:

Mount the virtual object to be added at a position corresponding to the newly added target object.

Optionally, the virtual object determination module 220 to be added is further configured to:

Determine whether the detected target object falls into at least one of the first to N-1th view spaces in the view space queue;

If not, the detected target object is a new target object; if so, the detected object is an old target object.

Optionally, the virtual object determination module 230 to be deleted is further configured to:

Judging whether the rendered virtual object falls into the most recently set number of view spaces in the view space queue;

If not, the rendered virtual object is determined as the virtual object to be deleted; wherein, the set number is less than or equal to N.

Optionally, the visual field space is represented by a visual cone, and the visual cone is surrounded by a plurality of boundary surfaces.

Optionally, the virtual object determination module 220 to be added is further configured to:

For each of the first to N-1th view spaces, project the view space to the camera plane to obtain a two-dimensional area corresponding to the view space;

Projecting the three-dimensional center point of the detected target object to the camera plane to obtain a two-dimensional center point;

Determine whether the two-dimensional center point is within the two-dimensional area;

If so, the detected target object falls into the visual field space.

Optionally, the virtual object determination module 220 to be added is further configured to:

For each field of view space from the 1st to the N-1th, obtain a polygonal pyramid formed by the three-dimensional center point of the detected target object and a plurality of boundary surfaces corresponding to the field of view space;

Calculate the volume sum of multiple polygons;

If the volume is equal to the volume of the viewing cone corresponding to the viewing space, the detected target object falls into the viewing space.

Optionally, the virtual object determination module 230 to be deleted is further configured to:

For each field of view space of the most recently set number in the field of view space queue, project the field of view space to the camera plane to obtain a two-dimensional area corresponding to the field of view space;

Projecting the three-dimensional center point of the rendered virtual object to the camera plane to obtain a two-dimensional center point;

Determine whether the two-dimensional center point is within the two-dimensional area;

If so, the rendered virtual object falls into the view space.

Optionally, the virtual object determination module 230 to be deleted is further configured to:

For each view space of the most recently set number in the view space queue, obtain a polygonal pyramid formed by the three-dimensional center point of the rendered virtual object and a plurality of boundary surfaces corresponding to the view space respectively;

Calculate the volume sum of multiple polygons;

If the volume is equal to the volume of the frustum corresponding to the view space, the rendered virtual object falls into the view space.

The above-mentioned apparatus can execute the methods provided by all the foregoing embodiments of the present disclosure, and has corresponding functional modules and beneficial effects for executing the above-mentioned methods. For technical details not described in detail in this embodiment, reference may be made to the methods provided by all the foregoing embodiments of the present disclosure.

Referring next to FIG. 5 , it shows a schematic structural diagram of an electronic device 300 suitable for implementing an embodiment of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablets), PMPs (portable multimedia players), vehicle-mounted terminals (eg, Mobile terminals such as in-vehicle navigation terminals), etc., and stationary terminals such as digital TVs, desktop computers, etc., or various forms of servers, such as stand-alone servers or server clusters. The electronic device shown in FIG. 5 is only an example, and should not impose any limitation on the function and scope of use of the embodiments of the present disclosure.

As shown in FIG. 5, the electronic device 300 may include a processing device (eg, a central processing unit, a graphics processor, etc.) 301, which may be loaded into a random Various appropriate actions and processes are executed by accessing the program in the storage device (RAM) 303 . In the RAM 303, various programs and data necessary for the operation of the electronic device 300 are also stored. The processing device 301 , the ROM 302 , and the RAM 303 are connected to each other through a bus 304 . An input/output (I/O) interface 305 is also connected to bus 304 .

Typically, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; including, for example, a liquid crystal display (LCD), speakers, vibration An output device 307 of a computer, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 309. Communication means 309 may allow electronic device 300 to communicate wirelessly or by wire with other devices to exchange data. Although FIG. 5 illustrates electronic device 300 having various means, it should be understood that not all of the illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.

In particular, according to embodiments of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing a recommended method of a word. In such an embodiment, the computer program may be downloaded and installed from the network via the communication device 309 , or from the storage device 305 , or from the ROM 302 . When the computer program is executed by the processing device 301, the above-mentioned functions defined in the methods of the embodiments of the present disclosure are executed.

It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or a combination of any of the above. More specific examples of computer readable storage media may include, but are not limited to, electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable Programmable read only memory (EPROM or flash memory), fiber optics, portable compact disk read only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing. In this disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present disclosure, however, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave with computer-readable program code embodied thereon. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device . Program code embodied on a computer readable medium may be transmitted using any suitable medium including, but not limited to, electrical wire, optical fiber cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the client and server can communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and can communicate with digital data in any form or medium (eg, a communications network) interconnected. Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet (eg, the Internet), and peer-to-peer networks (eg, ad hoc peer-to-peer networks), as well as any currently known or future development network of.

The above-mentioned computer-readable medium may be included in the above-mentioned electronic device; or may exist alone without being assembled into the electronic device.

The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by the electronic device, the electronic device is made to: acquire the target object detected in the current key frame and the rendered rendered corresponding to the current key frame A virtual object; wherein, the key frame is a frame that triggers object detection; the virtual object to be added is determined according to the detected target object and the field of view space queue; wherein, the field of view space queue consists of N closest to the current moment The view space corresponding to the key frame is composed, and the current key frame is the Nth key frame in the queue; the virtual object to be deleted is determined according to the rendered virtual object and the view space queue; according to the virtual object to be added and the The virtual object to be deleted updates the virtual object corresponding to the current key frame.

Computer program code for performing operations of the present disclosure may be written in one or more programming languages, including but not limited to object-oriented programming languages—such as Java, Smalltalk, C++, and This includes conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (eg, using an Internet service provider through Internet connection).

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code that contains one or more logical functions for implementing the specified functions executable instructions. It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented in dedicated hardware-based systems that perform the specified functions or operations , or can be implemented in a combination of dedicated hardware and computer instructions.

The units involved in the embodiments of the present disclosure may be implemented in a software manner, and may also be implemented in a hardware manner. Among them, the name of the unit does not constitute a limitation of the unit itself under certain circumstances.

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), Systems on Chips (SOCs), Complex Programmable Logical Devices (CPLDs) and more.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

According to one or more of the embodiments of the present disclosure, the embodiments of the present disclosure disclose an image processing method, including:

Obtain the target object detected in the current key frame and the rendered virtual object corresponding to the current key frame; wherein, the key frame is the frame that triggers object detection;

Determine the virtual object to be added according to the detected target object and the field of view space queue; wherein, the field of view space queue is composed of the field of view space corresponding to the N key frames closest to the current moment, and the current key frame is in the queue Nth keyframe;

Determine the virtual object to be deleted according to the rendered virtual object and the view space queue;

The virtual object corresponding to the current key frame is updated according to the virtual object to be added and the virtual object to be deleted.

Further, determining the virtual object to be added according to the detected target object and the field of view space queue, including:

Determine the newly added target object according to the detected target object and the field of view space queue;

Determine the virtual object to be added according to the newly added target object;

Update the virtual object corresponding to the current key frame according to the virtual object to be added, including:

Mount the virtual object to be added at a position corresponding to the newly added target object.

Further, determining the newly added target object according to the detected target object and the field of view space queue, including:

Determine whether the detected target object falls into the first to N-1th view spaces in the view space queue;

If not, the detected target object is a new target object; otherwise, the detected object is an old target object.

Further, determining the virtual object to be deleted according to the rendered virtual object and the field of view space queue, including:

Judging whether the rendered virtual object falls into the most recently set number of view spaces in the view space queue;

If not, the rendered virtual object is determined as the virtual object to be deleted; wherein, the set number is less than or equal to N.

Further, the visual field space is represented by an optic cone, and the optic cone is surrounded by a plurality of boundary surfaces.

Further, judging whether the detected target object falls into the first to N-1th view spaces in the view space queue, including:

For each of the first to N-1th view spaces, project the view space to the camera plane to obtain a two-dimensional area corresponding to the view space;

Projecting the three-dimensional center point of the detected target object to the camera plane to obtain a two-dimensional center point;

Determine whether the two-dimensional center point is within the two-dimensional area;

If so, the detected target object falls into the visual field space.

Further, judging whether the detected target object falls into the first to N-1th view spaces in the view space queue, including:

For each field of view space from the 1st to the N-1th, obtain a polygonal pyramid formed by the three-dimensional center point of the detected target object and a plurality of boundary surfaces corresponding to the field of view space;

Calculate the volume sum of multiple polygons;

If the volume is equal to the volume of the viewing cone corresponding to the viewing space, the detected target object falls into the viewing space.

Further, judging whether the rendered virtual object falls into the most recently set number of view spaces in the view space, including:

For each field of view space of the most recently set number in the field of view space queue, project the field of view space to the camera plane to obtain a two-dimensional area corresponding to the field of view space;

Projecting the three-dimensional center point of the rendered virtual object to the camera plane to obtain a two-dimensional center point;

Determine whether the two-dimensional center point is within the two-dimensional area;

If so, the rendered virtual object falls into the view space.

Further, judging whether the rendered virtual object falls into the most recently set number of view spaces in the view space, including:

For each view space of the recently set number in the view space queue, obtain a polygonal pyramid formed by the three-dimensional center point of the rendered virtual object and a plurality of boundary surfaces corresponding to the view space respectively;

Calculate the volume sum of multiple polygons;

If the volume is equal to the volume of the frustum corresponding to the view space, the rendered virtual object falls into the view space.

Note that the above are only preferred embodiments of the present disclosure and applied technical principles. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure has been described in detail through the above embodiments, the present disclosure is not limited to the above embodiments, and can also include more other equivalent embodiments without departing from the concept of the present disclosure. The scope is determined by the scope of the appended claims.

Claims (12)

1. an image processing method, is characterized in that, comprises: Obtain the target object detected in the current key frame and the rendered virtual object corresponding to the current key frame; wherein, the key frame is the frame that triggers object detection; Determine the virtual object to be added according to the detected target object and the field of view space queue; wherein, the field of view space queue is composed of the field of view space corresponding to the N key frames closest to the current moment, and the current key frame is in the queue Nth keyframe; Determine the virtual object to be deleted according to the rendered virtual object and the view space queue; The virtual object corresponding to the current key frame is updated according to the virtual object to be added and the virtual object to be deleted. 2. The method according to claim 1, wherein determining the virtual object to be added according to the detected target object and the field of view space queue, comprising: Determine the newly added target object according to the detected target object and the field of view space queue; Determine the virtual object to be added according to the newly added target object; Update the virtual object corresponding to the current key frame according to the virtual object to be added, including: The virtual object to be added is rendered at a position corresponding to the newly added target object. 3. The method according to claim 2, wherein determining a new target object according to the detected target object and the field of view space queue, comprising: Determine whether the detected target object falls into at least one of the first to N-1th view spaces in the view space queue; If not, the detected target object is a new target object; if so, the detected object is an old target object. 4. The method according to claim 1, wherein determining the virtual object to be deleted according to the rendered virtual object and the view space queue, comprising: Judging whether the rendered virtual object falls into the most recently set number of view spaces in the view space queue; If not, the rendered virtual object is determined as the virtual object to be deleted; wherein, the set number is less than or equal to N. 5. The method according to claim 3 or 4, wherein the field of view space is a three-dimensional space that takes a camera shooting angle as a viewing angle and falls within the shooting range of the camera; the field of view space is defined by a viewing frustum represented, and the frustum is bounded by a plurality of boundary surfaces. 6. The method according to claim 5, wherein judging whether the detected target object falls into the first to N-1th viewing spaces in the viewing space queue, comprising: For each of the first to N-1th view spaces, project the view space to the camera plane to obtain a two-dimensional area corresponding to the view space; Projecting the three-dimensional center point of the detected target object to the camera plane to obtain a two-dimensional center point; Determine whether the two-dimensional center point is within the two-dimensional area; If so, the detected target object falls into the visual field space. 7. The method according to claim 5, wherein judging whether the detected target object falls into the first to N-1th viewing spaces in the viewing space queue, comprising: For each field of view space from the 1st to the N-1th, obtain a polygonal pyramid formed by the three-dimensional center point of the detected target object and a plurality of boundary surfaces corresponding to the field of view space; Calculate the volume sum of multiple polygons; If the volume is equal to the volume of the viewing cone corresponding to the viewing space, the detected target object falls into the viewing space. 8. The method according to claim 5, wherein judging whether the rendered virtual object falls into a recently set number of view spaces in the view space, comprising: For each field of view space of the most recently set number in the field of view space queue, project the field of view space to the camera plane to obtain a two-dimensional area corresponding to the field of view space; Projecting the three-dimensional center point of the rendered virtual object to the camera plane to obtain a two-dimensional center point; Determine whether the two-dimensional center point is within the two-dimensional area; If so, the rendered virtual object falls into the view space. 9 . The method according to claim 5 , wherein judging whether the rendered virtual object falls into a recently set number of view spaces in the view space, comprising: 10 . For each view space of the recently set number in the view space queue, obtain a polygonal pyramid formed by the three-dimensional center point of the rendered virtual object and a plurality of boundary surfaces corresponding to the view space respectively; Calculate the volume sum of multiple polygons; If the volume is equal to the volume of the frustum corresponding to the view space, the rendered virtual object falls into the view space. 10. An image processing device, comprising: a target object detection module, configured to acquire the target object detected in the current key frame and the rendered virtual object corresponding to the current key frame; wherein the key frame is a frame that triggers object detection; The virtual object to be added determination module is used to determine the virtual object to be added according to the detected target object and the field of view space queue; wherein, the field of view space queue consists of the field of view space corresponding to the N key frames closest to the current moment composition, the current key frame is the Nth key frame in the queue; a virtual object determination module to be deleted, configured to determine the virtual object to be deleted according to the rendered virtual object and the field of view space queue; The virtual object updating module is configured to update the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted. 11. An electronic device, characterized in that the electronic device comprises: one or more processing devices; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processing apparatuses, the one or more processing apparatuses implement the image processing method according to any one of claims 1-9. 12. A computer-readable medium on which a computer program is stored, characterized in that, when the program is executed by a processing device, the image processing method according to any one of claims 1-9 is implemented.

Images