CN114419292A - Image processing method, device, equipment and storage medium - Google Patents
Image processing method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114419292A CN114419292A CN202210072428.1A CN202210072428A CN114419292A CN 114419292 A CN114419292 A CN 114419292A CN 202210072428 A CN202210072428 A CN 202210072428A CN 114419292 A CN114419292 A CN 114419292A
- Authority
- CN
- China
- Prior art keywords
- virtual object
- view
- target object
- space
- queue
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 18
- 230000000007 visual effect Effects 0.000 claims abstract description 61
- 238000012545 processing Methods 0.000 claims abstract description 30
- 238000001514 detection method Methods 0.000 claims abstract description 17
- 238000000034 method Methods 0.000 claims description 24
- 238000004590 computer program Methods 0.000 claims description 9
- 238000009877 rendering Methods 0.000 claims description 2
- 238000004364 calculation method Methods 0.000 abstract description 6
- 238000010586 diagram Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 10
- 238000004891 communication Methods 0.000 description 6
- 230000003287 optical effect Effects 0.000 description 6
- 230000003190 augmentative effect Effects 0.000 description 2
- 239000000463 material Substances 0.000 description 2
- 239000013307 optical fiber Substances 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 235000007164 Oryza sativa Nutrition 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 235000009566 rice Nutrition 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/006—Mixed reality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
- G06T15/005—General purpose rendering architectures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Computer Graphics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Geometry (AREA)
- Processing Or Creating Images (AREA)
Abstract
The embodiment of the disclosure discloses an image processing method, an image processing device, image processing equipment and a storage medium. Acquiring a target object detected in a current key frame and a rendered virtual object corresponding to the current key frame; wherein the key frame is a frame for triggering object detection; determining a virtual object to be newly added according to the detected target object and the visual field space queue; determining a virtual object to be deleted according to the rendered virtual object and the view space queue; and updating the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted. According to the image processing method provided by the embodiment of the disclosure, the virtual object to be newly added is determined according to the detected target object and the visual field space queue, and the virtual object to be deleted is determined according to the rendered virtual object and the visual field space queue, so that the calculation amount during updating of the virtual object can be reduced, and the image processing efficiency is improved.
Description
Technical Field
The embodiments of the present disclosure relate to the field of augmented reality technologies, and in particular, to an image processing method, an image processing apparatus, an image processing device, and a storage medium.
Background
Application scenarios common in augmented reality are achieved by overloading virtual objects on detected target objects. In this application scenario, target detection needs to be performed on the space. Generally, after detecting the same target object in space for multiple times, the determination of the overlap and the removal of the target object outside the field of view need to be performed. In the prior art, filtering of repeated objects of an object and deleting of objects outside a field of view can be realized through a target tracking and feature matching mode, but the mode has large calculation amount and low efficiency.
Disclosure of Invention
The embodiment of the disclosure provides an image processing method, an image processing device, an image processing apparatus and a storage medium, which can reduce the amount of calculation when a virtual object is updated, thereby improving the image processing efficiency.
In a first aspect, an embodiment of the present disclosure provides an image processing method, including:
acquiring a target object detected in a current key frame and a rendered virtual object corresponding to the current key frame; wherein the key frame is a frame for triggering object detection;
determining a virtual object to be newly added according to the detected target object and the visual field space queue; the view space queue consists of view spaces corresponding to N nearest key frames, and the current key frame is the Nth key frame in the queue;
determining a virtual object to be deleted according to the rendered virtual object and the view space queue;
and updating the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted.
In a second aspect, an embodiment of the present disclosure further provides an image processing apparatus, including:
the target object detection module is used for acquiring a target object detected in the current key frame and a rendered virtual object corresponding to the current key frame; wherein the key frame is a frame for triggering object detection;
the virtual object to be newly added determining module is used for determining a virtual object to be newly added according to the detected target object and the visual field space queue; the view space queue consists of view spaces corresponding to N nearest key frames, and the current key frame is the Nth key frame in the queue;
the virtual object to be deleted determining module is used for determining the virtual object to be deleted according to the rendered virtual object and the view space queue;
and the virtual object updating module is used for updating the virtual object corresponding to the current key frame according to the virtual object to be newly added and the virtual object to be deleted.
In a third aspect, an embodiment of the present disclosure further provides an electronic device, where the electronic device includes:
one or more processing devices;
storage means for storing one or more programs;
when the one or more programs are executed by the one or more processing devices, the one or more processing devices are caused to implement the image processing method according to the embodiment of the present disclosure.
In a fourth aspect, the disclosed embodiments also provide a computer readable medium, on which a computer program is stored, which when executed by a processing apparatus, implements an image processing method according to the disclosed embodiments.
The embodiment of the disclosure discloses an image processing method, an image processing device, image processing equipment and a storage medium. Acquiring a target object detected in a current key frame and a rendered virtual object corresponding to the current key frame; wherein, the key frame is a frame for triggering object detection; determining a virtual object to be newly added according to the detected target object and the visual field space queue; the visual field space queue consists of visual field spaces corresponding to N nearest key frames, and the current key frame is the Nth key frame in the queue; determining a virtual object to be deleted according to the rendered virtual object and the visual field space queue; and updating the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted. According to the image processing method provided by the embodiment of the disclosure, the virtual object to be newly added is determined according to the detected target object and the visual field space queue, and the virtual object to be deleted is determined according to the rendered virtual object and the visual field space queue, so that the calculation amount during updating of the virtual object can be reduced, and the image processing efficiency is improved.
Drawings
FIG. 1 is a flow chart of an image processing method in an embodiment of the present disclosure;
FIG. 2 is a schematic view of a viewing cone in an embodiment of the present disclosure;
FIG. 3 is a diagram illustrating an example of a target object newly added to a current keyframe in an embodiment of the present disclosure;
fig. 4 is a schematic structural diagram of an image processing apparatus in an embodiment of the present disclosure;
fig. 5 is a schematic structural diagram of an electronic device in an embodiment of the present disclosure.
Detailed Description
Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.
It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.
The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.
It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.
It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.
The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.
Fig. 1 is a flowchart of an image processing method provided in an embodiment of the present disclosure, where the embodiment is applicable to a case where a virtual object in a picture of a three-dimensional space is updated, and the method may be executed by an image processing apparatus, where the apparatus may be composed of hardware and/or software, and may be generally integrated in a device with an image processing function, where the device may be an electronic device such as a server, a mobile terminal, or a server cluster. As shown in fig. 1, the method specifically includes the following steps:
s110, a target object detected in the current key frame and a rendered virtual object corresponding to the current key frame are obtained.
And when the set triggering condition is met, the terminal equipment triggers the object detection module to detect the target object on the current picture. Wherein, the setting triggering condition may be: every set time length, every set number of video frames or the number of the currently rendered virtual objects is less than a set value, and the like. In this embodiment, the current target object detection algorithm may be used to detect the target object on the current frame, which is not limited herein.
Here, the rendered virtual object may be understood as a virtual object that has been displayed and mounted on an object in a three-dimensional space, and the virtual object is a 3D object. The virtual objects may be of any subject matter category, for example: the "sweet-and-sour" theme, the virtual objects may be: virtual lanterns, virtual rice dumplings, etc., without limitation.
And S120, determining a virtual object to be newly added according to the detected target object and the visual field space queue.
The view space queue consists of view spaces corresponding to N key frames nearest to the current moment, the current key frame is the Nth key frame in the queue, the last key frame is the (N-1) th key frame, and the like. The field of view space is understood to be the area of the three-dimensional world that is visible on the screen, i.e. the field of view of the virtual camera. Namely, the visual field space is a three-dimensional space which takes the camera shooting angle as the visual angle and falls into the camera shooting range. Optionally, the visual field space is characterized by a visual cone body, the visual cone body is enclosed by a plurality of boundary surfaces, in this embodiment, the visual cone body is enclosed by 6 boundary surfaces, which are a Near plane (Near plane), a Far plane (Far plane), and four side boundary surfaces respectively. Illustratively, fig. 2 is a schematic view of a view cone in the present embodiment, in which Oc represents a camera. The set of landing spaces formed by the view vertebral body i in space is Qi. For the view space queue, a region set formed by the view spaces corresponding to the N key frames is expressed as: q ═ Q1∪Q2∪...∪QNIf the three-dimensional space point p is located in the space Q, p ∈ Q is expressed. Since the memory constraint of the computing unit and the reference value of the information of the old key frame to the new key frame decrease with the increase of the time difference, the old key frame can be removed at a proper time, and thus, the view space corresponding to the N nearest key frames is saved in the view space queue.
The virtual object to be newly added is determined by the target object newly added in the current key frame, and if the target object is newly added in the current key frame, the new virtual object needs to be mounted on the newly added target object. Specifically, the method for determining the virtual object to be newly added according to the detected target object and the view space queue may be: determining a newly added target object according to the detected target object and the visual field space queue; and determining a virtual object to be newly added according to the newly added target object.
Wherein, the new target object can be understood as a target object newly detected in the current key frame, i.e. not detected in the previous key frame. The process of determining the virtual object to be newly added according to the newly added target object may be: and determining a virtual object frame according to the detection frame of the newly added target object, acquiring a corresponding material in the virtual object, and adding the material into the virtual object frame for rendering to obtain the newly added virtual object.
In this embodiment, the manner of determining the new target object according to the detected target object and the view space queue may be: judging whether the detected target object falls into at least one of the 1 st to the N-1 st view spaces in the view space queue; if not, the detected target object is a newly added target object; if yes, the detected object is an old target object.
Specifically, if it is detected that none of the target objects falls into the 1 st to the N-1 st view spaces in the view space queue, the detected target object is a newly added target object. Assuming that the target object is characterized by its center point p, if
It indicates that the detected target object is the new target object. For example, fig. 3 is an exemplary diagram of a target object newly added in the current key frame in the embodiment, and as shown in fig. 3, a detected target object does not fall into any view space of the previous 1 to N-1 frames, and is a newly added target object.
Optionally, the manner of determining whether the detected target object falls into the 1 st to the N-1 st view spaces in the view space queue may be: for each view space from 1 st to N-1 st, projecting the view space to a camera plane to obtain a two-dimensional region corresponding to the view space; projecting the detected three-dimensional central point of the target object to a camera plane to obtain a two-dimensional central point; judging whether the two-dimensional central point is in a two-dimensional area; if yes, the detected target object falls into the visual field space.
In this embodiment, the view space may be projected to the camera plane and the detected three-dimensional center point of the target object may be projected to the camera plane based on the projection principle. The manner of determining whether the two-dimensional center point is located in the two-dimensional region may be determining whether the coordinate of the two-dimensional center satisfies a function set corresponding to the two-dimensional region, and if so, determining that the two-dimensional center point is located in the two-dimensional region. If the two-dimensional center point is in the two-dimensional area, the three-dimensional center point of the target object is in the visual cone, and the detected target object falls into the visual field space. The representation of the target object by the three-dimensional central point can greatly reduce the calculation amount.
Optionally, the manner of determining whether the detected target object falls into the 1 st to the N-1 st view spaces in the view space queue may be: for each visual field space from 1 st to N-1 st, acquiring a polygonal pyramid formed by a plurality of boundary surfaces, wherein the three-dimensional center points of the detected target objects respectively correspond to the visual field spaces; calculating the volume sum of a plurality of polygonal pyramids; if the volume is equal to the volume of the view cone corresponding to the view space, the detected target object falls into the view space.
In this embodiment, the view cone corresponding to the view space is surrounded by 6 boundary surfaces, and each surface includes four sides, so that the three-dimensional center point and each boundary surface form a rectangular pyramid, thereby obtaining 6 rectangular pyramids. And respectively calculating the volume of each rectangular pyramid, summing the volumes of the 6 rectangular pyramids, and if the volume is equal to the volume of the view cone, indicating that the three-dimensional central point of the target object is inside the view cone, wherein the detected target object falls into the view space.
And S130, determining the virtual object to be deleted according to the rendered virtual object and the view space queue.
In this embodiment, if the rendered virtual object has not been detected for a period of time, the rendered virtual object needs to be deleted in order to save resources and memory.
Specifically, the method for determining the virtual object to be deleted according to the rendered virtual object and the view space queue may be: judging whether the rendered virtual objects fall into the view space with the latest set number in the view space queue; if not, determining the rendered virtual object as the virtual object to be deleted.
The set number is less than or equal to N, for example, N/2 may be taken, and for example, assuming that N is 10, the set number may be 5. In this embodiment, if the rendered virtual objects all fall into the view space with the latest set number, the rendered virtual objects are the virtual objects to be deleted.
Optionally, the manner of determining whether the rendered virtual object falls into the view space with the latest set number in the view space may be: for each view space with the number set recently in the view space queue, projecting the view space to a camera plane to obtain a two-dimensional area corresponding to the view space; projecting the three-dimensional central point of the rendered virtual object to a camera plane to obtain a two-dimensional central point; judging whether the two-dimensional central point is in a two-dimensional area; if yes, the rendered virtual object falls into the visual field space.
In this embodiment, the view space may be projected to the camera plane and the three-dimensional center point of the rendered virtual object may be projected to the camera plane based on the projection principle. The manner of determining whether the two-dimensional center point is located in the two-dimensional region may be determining whether the coordinate of the two-dimensional center satisfies a function set corresponding to the two-dimensional region, and if so, determining that the two-dimensional center point is located in the two-dimensional region. If the two-dimensional center point is in the two-dimensional area, the three-dimensional center point of the rendered virtual object is in the view cone, and the rendered virtual object detected at the moment falls into the view space. Characterizing the rendered virtual object by the three-dimensional center point can greatly reduce the computational effort.
Optionally, the manner of determining whether the rendered virtual object falls into the view space with the latest set number in the view space may be: for each view space with the latest set number in the view space queue, acquiring a polygonal pyramid formed by the three-dimensional center point of the rendered virtual object and a plurality of boundary surfaces corresponding to the view space; calculating the volume sum of a plurality of polygonal pyramids; if the volume is equal to the volume of the view volume corresponding to the view space, the rendered virtual object falls into the view space.
In this embodiment, the view cone corresponding to the view space is surrounded by 6 boundary surfaces, and each surface includes four sides, so that the three-dimensional center point and each boundary surface form a rectangular pyramid, thereby obtaining 6 rectangular pyramids. And respectively calculating the volume of each rectangular pyramid, summing the volumes of the 6 rectangular pyramids, and if the volume is equal to the volume of the view cone, indicating that the three-dimensional central point of the rendered virtual object is inside the view cone, and at the moment, the rendered virtual object falls into the view space.
And S140, updating the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted.
Specifically, the process of updating the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted may be: and hanging the virtual object to be newly added at the position corresponding to the newly added target object, and deleting the virtual object to be deleted.
In the current key frame, the rendered virtual object set after the virtual object is newly added is
Wherein, PkRepresenting the rendered virtual object, D, corresponding to the current keyframe after the updatekRepresenting the newly mounted virtual object.
In the current key frame, the rendered virtual object set after the virtual object is deleted is as follows: pk＝Pk-1∩(Q1∪Q2∪…∪QN)，PkAnd representing the rendered virtual object corresponding to the current key frame after updating.
According to the technical scheme of the embodiment of the disclosure, a target object detected in a current key frame and a rendered virtual object corresponding to the current key frame are obtained; wherein, the key frame is a frame for triggering object detection; determining a virtual object to be newly added according to the detected target object and the visual field space queue; the visual field space queue consists of visual field spaces corresponding to N nearest key frames, and the current key frame is the Nth key frame in the queue; determining a virtual object to be deleted according to the rendered virtual object and the visual field space queue; and updating the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted. According to the image processing method provided by the embodiment of the disclosure, the virtual object to be newly added is determined according to the detected target object and the visual field space queue, and the virtual object to be deleted is determined according to the rendered virtual object and the visual field space queue, so that the calculation amount during updating of the virtual object can be reduced, and the image processing efficiency is improved.
Fig. 4 is a schematic structural diagram of an image processing apparatus disclosed in an embodiment of the present disclosure, and as shown in fig. 4, the apparatus includes:
a target object detection module 210, configured to obtain a target object detected in a current key frame and a rendered virtual object corresponding to the current key frame; wherein the key frame is a frame for triggering object detection;
a to-be-added virtual object determining module 220, configured to determine a to-be-added virtual object according to the detected target object and the view space queue; the view space queue consists of view spaces corresponding to N key frames nearest to the current moment, and the current key frame is the Nth key frame in the queue;
a to-be-deleted virtual object determining module 230, configured to determine a to-be-deleted virtual object according to the rendered virtual object and the view space queue;
and a virtual object updating module 240, configured to update the virtual object corresponding to the current key frame according to the virtual object to be newly added and the virtual object to be deleted.
Optionally, the module for determining a virtual object to be newly added 220 is further configured to:
determining a newly added target object according to the detected target object and the visual field space queue;
determining a virtual object to be newly added according to the newly added target object;
optionally, the virtual object updating module 240 is further configured to:
and mounting the virtual object to be newly added at the position corresponding to the newly added target object.
Optionally, the module for determining a virtual object to be newly added 220 is further configured to:
judging whether the detected target object falls into at least one of the 1 st to the N-1 st view spaces in the view space queue;
if not, the detected target object is a newly added target object; if yes, the detected object is an old target object.
Optionally, the to-be-deleted virtual object determining module 230 is further configured to:
judging whether the rendered virtual objects fall into the view space with the latest set number in the view space queue;
if not, determining the rendered virtual object as a virtual object to be deleted; wherein the set number is less than or equal to N.
Optionally, the viewing volume is characterized by a viewing volume, the viewing volume being bounded by a plurality of boundary surfaces.
Optionally, the module for determining a virtual object to be newly added 220 is further configured to:
for each view space from 1 st to N-1 st, projecting the view space to a camera plane to obtain a two-dimensional region corresponding to the view space;
projecting the detected three-dimensional central point of the target object to a camera plane to obtain a two-dimensional central point;
judging whether the two-dimensional central point is in the two-dimensional area;
and if so, the detected target object falls into the visual field space.
Optionally, the module for determining a virtual object to be newly added 220 is further configured to:
for each view space from 1 st to N-1 st, acquiring a polygonal pyramid formed by a plurality of boundary surfaces, corresponding to the view space, of the three-dimensional central point of the detected target object respectively;
calculating the volume sum of a plurality of polygonal pyramids;
and if the volume is equal to the volume of the visual cone corresponding to the visual field space, the detected target object falls into the visual field space.
Optionally, the to-be-deleted virtual object determining module 230 is further configured to:
for each view space with the number set recently in the view space queue, projecting the view space to a camera plane to obtain a two-dimensional region corresponding to the view space;
projecting the three-dimensional central point of the rendered virtual object to a camera plane to obtain a two-dimensional central point;
judging whether the two-dimensional central point is in the two-dimensional area;
if yes, the rendered virtual object falls into the view space.
Optionally, the to-be-deleted virtual object determining module 230 is further configured to:
for each view space with the latest set number in the view space queue, acquiring a polygonal pyramid formed by the three-dimensional center point of the rendered virtual object and a plurality of boundary surfaces corresponding to the view space;
calculating the volume sum of a plurality of polygonal pyramids;
and if the volume is equal to the volume of the visual cone corresponding to the visual field space, the rendered virtual object falls into the visual field space.
The device can execute the methods provided by all the embodiments of the disclosure, and has corresponding functional modules and beneficial effects for executing the methods. For technical details that are not described in detail in this embodiment, reference may be made to the methods provided in all the foregoing embodiments of the disclosure.
Referring now to FIG. 5, a block diagram of an electronic device 300 suitable for use in implementing embodiments of the present disclosure is shown. The electronic device in the embodiments of the present disclosure may include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a PDA (personal digital assistant), a PAD (tablet computer), a PMP (portable multimedia player), a vehicle terminal (e.g., a car navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like, or various forms of servers such as a stand-alone server or a server cluster. The electronic device shown in fig. 5 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.
As shown in fig. 5, electronic device 300 may include a processing means (e.g., central processing unit, graphics processor, etc.) 301 that may perform various appropriate actions and processes in accordance with a program stored in a read-only memory device (ROM)302 or a program loaded from a storage device 305 into a random access memory device (RAM) 303. In the RAM 303, various programs and data necessary for the operation of the electronic apparatus 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input/output (I/O) interface 305 is also connected to bus 304.
Generally, the following devices may be connected to the I/O interface 305: input devices 306 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 307 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage devices 308 including, for example, magnetic tape, hard disk, etc.; and a communication device 309. The communication means 309 may allow the electronic device 300 to communicate wirelessly or by wire with other devices to exchange data. While fig. 5 illustrates an electronic device 300 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided.
In particular, according to an embodiment of the present disclosure, the processes described above with reference to the flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer-readable medium, the computer program containing program code for performing a method for recommending words. In such an embodiment, the computer program may be downloaded and installed from a network through the communication means 309, or installed from the storage means 305, or installed from the ROM 302. The computer program, when executed by the processing device 301, performs the above-described functions defined in the methods of the embodiments of the present disclosure.
It should be noted that the computer readable medium in the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In contrast, in the present disclosure, a computer readable signal medium may comprise a propagated data signal with computer readable program code embodied therein, either in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.
In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.
The computer readable medium may be embodied in the electronic device; or may exist separately without being assembled into the electronic device.
The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: acquiring a target object detected in a current key frame and a rendered virtual object corresponding to the current key frame; wherein the key frame is a frame for triggering object detection; determining a virtual object to be newly added according to the detected target object and the visual field space queue; the view space queue consists of view spaces corresponding to N key frames nearest to the current moment, and the current key frame is the Nth key frame in the queue; determining a virtual object to be deleted according to the rendered virtual object and the view space queue; and updating the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted.
Computer program code for carrying out operations for the present disclosure may be written in any combination of one or more programming languages, including but not limited to an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).
The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units described in the embodiments of the present disclosure may be implemented by software or hardware. Where the name of an element does not in some cases constitute a limitation on the element itself.
The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
According to one or more embodiments of the disclosed embodiments, the disclosed embodiments disclose an image processing method, including:
acquiring a target object detected in a current key frame and a rendered virtual object corresponding to the current key frame; wherein the key frame is a frame for triggering object detection;
determining a virtual object to be newly added according to the detected target object and the visual field space queue; the view space queue consists of view spaces corresponding to N key frames nearest to the current moment, and the current key frame is the Nth key frame in the queue;
determining a virtual object to be deleted according to the rendered virtual object and the view space queue;
and updating the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted.
Further, determining a virtual object to be newly added according to the detected target object and the visual field space queue, including:
determining a newly added target object according to the detected target object and the visual field space queue;
determining a virtual object to be newly added according to the newly added target object;
updating the virtual object corresponding to the current key frame according to the virtual object to be newly added, including:
and mounting the virtual object to be newly added at the position corresponding to the newly added target object.
Further, determining a new target object according to the detected target object and the view space queue, including:
judging whether the detected target object falls into the 1 st to the N-1 st view spaces in the view space queue or not;
if not, the detected target object is a newly added target object; otherwise, the detected object is an old target object.
Further, determining a virtual object to be deleted according to the rendered virtual object and the view space queue includes:
judging whether the rendered virtual objects fall into the view space with the latest set number in the view space queue;
if not, determining the rendered virtual object as a virtual object to be deleted; wherein the set number is less than or equal to N.
Further, the viewing volume is characterized by a viewing volume, which is bounded by a plurality of boundary surfaces.
Further, judging whether the detected target object falls into the 1 st to the N-1 st view spaces in the view space queue includes:
for each view space from 1 st to N-1 st, projecting the view space to a camera plane to obtain a two-dimensional region corresponding to the view space;
projecting the detected three-dimensional central point of the target object to a camera plane to obtain a two-dimensional central point;
judging whether the two-dimensional central point is in the two-dimensional area;
and if so, the detected target object falls into the visual field space.
Further, judging whether the detected target object falls into the 1 st to the N-1 st view spaces in the view space queue includes:
for each view space from 1 st to N-1 st, acquiring a polygonal pyramid formed by a plurality of boundary surfaces, corresponding to the view space, of the three-dimensional central point of the detected target object respectively;
calculating the volume sum of a plurality of polygonal pyramids;
and if the volume is equal to the volume of the visual cone corresponding to the visual field space, the detected target object falls into the visual field space.
Further, determining whether the rendered virtual object falls within a view space of a most recently set number of the view spaces includes:
for each view space with the number set recently in the view space queue, projecting the view space to a camera plane to obtain a two-dimensional region corresponding to the view space;
projecting the three-dimensional central point of the rendered virtual object to a camera plane to obtain a two-dimensional central point;
judging whether the two-dimensional central point is in the two-dimensional area;
if yes, the rendered virtual object falls into the view space.
Further, determining whether the rendered virtual object falls within a view space of a most recently set number of the view spaces includes:
for each view space with the latest set number in the view space queue, acquiring a polygonal pyramid formed by the three-dimensional center point of the rendered virtual object and a plurality of boundary surfaces corresponding to the view space;
calculating the volume sum of a plurality of polygonal pyramids;
and if the volume is equal to the volume of the visual cone corresponding to the visual field space, the rendered virtual object falls into the visual field space.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present disclosure and the technical principles employed. Those skilled in the art will appreciate that the present disclosure is not limited to the particular embodiments described herein, and that various obvious changes, adaptations, and substitutions are possible, without departing from the scope of the present disclosure. Therefore, although the present disclosure has been described in greater detail with reference to the above embodiments, the present disclosure is not limited to the above embodiments, and may include other equivalent embodiments without departing from the spirit of the present disclosure, the scope of which is determined by the scope of the appended claims.
Claims (12)
1. An image processing method, comprising:
acquiring a target object detected in a current key frame and a rendered virtual object corresponding to the current key frame; wherein the key frame is a frame for triggering object detection;
determining a virtual object to be newly added according to the detected target object and the visual field space queue; the view space queue consists of view spaces corresponding to N key frames nearest to the current moment, and the current key frame is the Nth key frame in the queue;
determining a virtual object to be deleted according to the rendered virtual object and the view space queue;
and updating the virtual object corresponding to the current key frame according to the virtual object to be added and the virtual object to be deleted.
2. The method of claim 1, wherein determining a virtual object to be added according to the detected target object and the view space queue comprises:
determining a newly added target object according to the detected target object and the visual field space queue;
determining a virtual object to be newly added according to the newly added target object;
updating the virtual object corresponding to the current key frame according to the virtual object to be newly added, including:
and rendering the virtual object to be newly added at the position corresponding to the newly added target object.
3. The method of claim 2, wherein determining new target objects from the detected target objects and the spatial queue of fields of view comprises:
judging whether the detected target object falls into at least one of the 1 st to the N-1 st view spaces in the view space queue;
if not, the detected target object is a newly added target object; if yes, the detected object is an old target object.
4. The method of claim 1, wherein determining virtual objects to delete from the rendered virtual objects and the view space queue comprises:
judging whether the rendered virtual objects fall into the view space with the latest set number in the view space queue;
if not, determining the rendered virtual object as a virtual object to be deleted; wherein the set number is less than or equal to N.
5. The method according to claim 3 or 4, wherein the visual field space is a three-dimensional space which is viewed from a camera shooting angle and falls within the camera shooting range; the viewing volume is characterized by a viewing volume, and the viewing volume is bounded by a plurality of boundary surfaces.
6. The method of claim 5, wherein determining whether the detected target object falls within the 1 st to the N-1 st view spaces of the view space queue comprises:
for each view space from 1 st to N-1 st, projecting the view space to a camera plane to obtain a two-dimensional region corresponding to the view space;
projecting the detected three-dimensional central point of the target object to a camera plane to obtain a two-dimensional central point;
judging whether the two-dimensional central point is in the two-dimensional area;
and if so, the detected target object falls into the visual field space.
7. The method of claim 5, wherein determining whether the detected target object falls within the 1 st to the N-1 st view spaces of the view space queue comprises:
for each view space from 1 st to N-1 st, acquiring a polygonal pyramid formed by a plurality of boundary surfaces, corresponding to the view space, of the three-dimensional central point of the detected target object respectively;
calculating the volume sum of a plurality of polygonal pyramids;
and if the volume is equal to the volume of the visual cone corresponding to the visual field space, the detected target object falls into the visual field space.
8. The method of claim 5, wherein determining whether the rendered virtual object falls within a most recently set number of view spaces in the view space comprises:
for each view space with the number set recently in the view space queue, projecting the view space to a camera plane to obtain a two-dimensional region corresponding to the view space;
projecting the three-dimensional central point of the rendered virtual object to a camera plane to obtain a two-dimensional central point;
judging whether the two-dimensional central point is in the two-dimensional area;
if yes, the rendered virtual object falls into the view space.
9. The method of claim 5, wherein determining whether the rendered virtual object falls within a most recently set number of view spaces in the view space comprises:
for each view space with the latest set number in the view space queue, acquiring a polygonal pyramid formed by the three-dimensional center point of the rendered virtual object and a plurality of boundary surfaces corresponding to the view space;
calculating the volume sum of a plurality of polygonal pyramids;
and if the volume is equal to the volume of the visual cone corresponding to the visual field space, the rendered virtual object falls into the visual field space.
10. An image processing apparatus characterized by comprising:
the target object detection module is used for acquiring a target object detected in the current key frame and a rendered virtual object corresponding to the current key frame; wherein the key frame is a frame for triggering object detection;
the virtual object to be newly added determining module is used for determining a virtual object to be newly added according to the detected target object and the visual field space queue; the view space queue consists of view spaces corresponding to N key frames nearest to the current moment, and the current key frame is the Nth key frame in the queue;
the virtual object to be deleted determining module is used for determining the virtual object to be deleted according to the rendered virtual object and the view space queue;
and the virtual object updating module is used for updating the virtual object corresponding to the current key frame according to the virtual object to be newly added and the virtual object to be deleted.
11. An electronic device, characterized in that the electronic device comprises:
one or more processing devices;
storage means for storing one or more programs;
when executed by the one or more processing devices, cause the one or more processing devices to implement the image processing method of any one of claims 1-9.
12. A computer-readable medium, on which a computer program is stored, characterized in that the program, when being executed by processing means, carries out the image processing method according to any one of claims 1-9.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210072428.1A CN114419292A (en) | 2022-01-21 | 2022-01-21 | Image processing method, device, equipment and storage medium |
| PCT/CN2023/071878 WO2023138469A1 (en) | 2022-01-21 | 2023-01-12 | Image processing method and apparatus, device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210072428.1A CN114419292A (en) | 2022-01-21 | 2022-01-21 | Image processing method, device, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114419292A true CN114419292A (en) | 2022-04-29 |
Family
ID=81274451
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210072428.1A Pending CN114419292A (en) | 2022-01-21 | 2022-01-21 | Image processing method, device, equipment and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN114419292A (en) |
| WO (1) | WO2023138469A1 (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023138469A1 (en) * | 2022-01-21 | 2023-07-27 | 北京字跳网络技术有限公司 | Image processing method and apparatus, device, and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106056534A (en) * | 2016-05-31 | 2016-10-26 | 中国科学院深圳先进技术研究院 | Obstruction perspective method and device based on smart glasses |
| CN106484085A (en) * | 2015-08-31 | 2017-03-08 | 北京三星通信技术研究有限公司 | Method and its head mounted display of real-world object is shown in head mounted display |
| US20190114841A1 (en) * | 2017-10-17 | 2019-04-18 | Colopl, Inc. | Method, program and apparatus for providing virtual experience |
| CN109840947A (en) * | 2017-11-28 | 2019-06-04 | 广州腾讯科技有限公司 | Implementation method, device, equipment and the storage medium of augmented reality scene |
| US20190279407A1 (en) * | 2018-03-07 | 2019-09-12 | Samsung Electronics Co., Ltd | System and method for augmented reality interaction |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6719945B2 (en) * | 2016-03-31 | 2020-07-08 | キヤノン株式会社 | Information processing apparatus, information processing method, information processing system, and program |
| US11024088B2 (en) * | 2016-05-27 | 2021-06-01 | HoloBuilder, Inc. | Augmented and virtual reality |
| WO2019213392A1 (en) * | 2018-05-02 | 2019-11-07 | Fyusion, Inc. | System and method for generating combined embedded multi-view interactive digital media representations |
| CN111127615A (en) * | 2019-12-26 | 2020-05-08 | 四川航天神坤科技有限公司 | Data scheduling method and device of three-dimensional model and electronic equipment |
| CN111127637A (en) * | 2019-12-26 | 2020-05-08 | 四川航天神坤科技有限公司 | Data processing method and device of three-dimensional model and electronic equipment |
| CN111476911B (en) * | 2020-04-08 | 2023-07-25 | Oppo广东移动通信有限公司 | Virtual image realization method, device, storage medium and terminal equipment |
| CN112365397A (en) * | 2020-11-20 | 2021-02-12 | 天津七所精密机电技术有限公司 | Method for fusing two-dimensional video information and three-dimensional geographic information |
| CN114419292A (en) * | 2022-01-21 | 2022-04-29 | 北京字跳网络技术有限公司 | Image processing method, device, equipment and storage medium |
-
2022
- 2022-01-21 CN CN202210072428.1A patent/CN114419292A/en active Pending
-
2023
- 2023-01-12 WO PCT/CN2023/071878 patent/WO2023138469A1/en unknown
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106484085A (en) * | 2015-08-31 | 2017-03-08 | 北京三星通信技术研究有限公司 | Method and its head mounted display of real-world object is shown in head mounted display |
| CN106056534A (en) * | 2016-05-31 | 2016-10-26 | 中国科学院深圳先进技术研究院 | Obstruction perspective method and device based on smart glasses |
| US20190114841A1 (en) * | 2017-10-17 | 2019-04-18 | Colopl, Inc. | Method, program and apparatus for providing virtual experience |
| CN109840947A (en) * | 2017-11-28 | 2019-06-04 | 广州腾讯科技有限公司 | Implementation method, device, equipment and the storage medium of augmented reality scene |
| US20190279407A1 (en) * | 2018-03-07 | 2019-09-12 | Samsung Electronics Co., Ltd | System and method for augmented reality interaction |
Non-Patent Citations (2)
| Title |
|---|
| 孙超;张明敏;李扬;汤兴;潘志庚;: "增强现实环境下的人手自然交互", 计算机辅助设计与图形学学报, no. 04, 15 April 2011 (2011-04-15) * |
| 范广斌;: "虚拟现实技术在传媒业的应用与未来发展", 传媒, no. 06, 25 March 2020 (2020-03-25) * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023138469A1 (en) * | 2022-01-21 | 2023-07-27 | 北京字跳网络技术有限公司 | Image processing method and apparatus, device, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2023138469A1 (en) | 2023-07-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110222775B (en) | Image processing method, image processing device, electronic equipment and computer readable storage medium | |
| CN110728622B (en) | Fisheye image processing method, device, electronic equipment and computer readable medium | |
| CN115908679A (en) | Texture mapping method, device, equipment and storage medium | |
| CN110796144B (en) | License plate detection method, device, equipment and storage medium | |
| CN114742934A (en) | Image rendering method and device, readable medium and electronic equipment | |
| WO2023138469A1 (en) | Image processing method and apparatus, device, and storage medium | |
| CN111915532B (en) | Image tracking method and device, electronic equipment and computer readable medium | |
| CN111862342B (en) | Augmented reality texture processing method and device, electronic equipment and storage medium | |
| WO2023138467A1 (en) | Virtual object generation method and apparatus, device, and storage medium | |
| CN111833459A (en) | Image processing method and device, electronic equipment and storage medium | |
| CN113274735B (en) | Model processing method and device, electronic equipment and computer readable storage medium | |
| CN114419298A (en) | Virtual object generation method, device, equipment and storage medium | |
| CN111680754B (en) | Image classification method, device, electronic equipment and computer readable storage medium | |
| CN113034570A (en) | Image processing method and device and electronic equipment | |
| CN113744259B (en) | Forest fire smoke detection method and equipment based on gray value increasing number sequence | |
| CN112884794B (en) | Image generation method, device, electronic equipment and computer readable medium | |
| CN111368015B (en) | Method and device for compressing map | |
| CN114359673B (en) | Small sample smoke detection method, device and equipment based on metric learning | |
| CN113808050B (en) | Denoising method, device and equipment for 3D point cloud and storage medium | |
| CN111461966B (en) | Image processing method, image processing device, electronic equipment and computer readable medium | |
| CN111489286B (en) | Picture processing method, device, equipment and medium | |
| CN112215789B (en) | Image defogging method, device, equipment and computer readable medium | |
| CN110838132B (en) | Object segmentation method, device and equipment based on video stream and storage medium | |
| CN112395826B (en) | Text special effect processing method and device | |
| CN112668474A (en) | Plane generation method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |