WO2018014601A1 - Procédé et appareil associé de suivi d'orientation, procédé et dispositif destinés à la mise en œuvre de la réalité augmentée - Google Patents

Procédé et appareil associé de suivi d'orientation, procédé et dispositif destinés à la mise en œuvre de la réalité augmentée Download PDF

Info

Publication number
WO2018014601A1
WO2018014601A1 PCT/CN2017/080276 CN2017080276W WO2018014601A1 WO 2018014601 A1 WO2018014601 A1 WO 2018014601A1 CN 2017080276 W CN2017080276 W CN 2017080276W WO 2018014601 A1 WO2018014601 A1 WO 2018014601A1
Authority
WO
WIPO (PCT)
Prior art keywords
target entity
video frame
visible surface
globe
direction vector
Prior art date
Application number
PCT/CN2017/080276
Other languages
English (en)
Chinese (zh)
Inventor
刘钢
熊剑明
陈健
方堃
韦晓宁
Original Assignee
央数文化(上海)股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 央数文化(上海)股份有限公司 filed Critical 央数文化(上海)股份有限公司
Publication of WO2018014601A1 publication Critical patent/WO2018014601A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/006—Mixed reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/40—Extraction of image or video features
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B27/00—Planetaria; Globes
    • G09B27/08—Globes

Definitions

  • the present application relates to the field of computers, and in particular, to an azimuth tracking method, a method for implementing augmented reality, and related devices and devices.
  • the globe In the process of people knowing the world, the globe has always been an indispensable geographic teaching aid. It allows people to understand the concept of space and location in a more vivid and interesting way, and to understand the vastness of the land, the vastness of the ocean, and the change of the four seasons. Especially for children to form a correct scientific and worldview is also of great significance.
  • traditional globes can only express very limited geographic information, such as the boundary between land and ocean, the border between countries, etc., and for richer geographic information, such as the internal structure of the Earth, the Earth in the solar system. The position of the earth, the distribution of animals and plants on the surface of the earth, etc., the traditional globe is powerless.
  • the purpose of the application is to provide an orientation tracking method, a method for implementing augmented reality, and related devices and devices.
  • the present application provides a method for tracking an orientation of a target entity, the method comprising:
  • the feature set includes a plurality of surface regions of the target entity and image feature information of each surface region;
  • a center point coordinate and a direction vector of the visible surface of the target entity are respectively determined according to a center point coordinate and a direction vector of the surface area matching the visible surface.
  • the method further includes:
  • the target entity is preprocessed to construct a prefabricated feature set of the target entity, including:
  • determining a center point coordinate of the visible surface of the target entity according to a center point coordinate of the surface area matching the visible surface comprises:
  • the center point coordinates of the plurality of surface regions are subjected to weighted averaging processing, and the center point coordinates of the target entity are determined.
  • determining the target entity visible surface direction vector according to a direction vector of the surface area matching the visible surface comprises:
  • the direction vectors of the plurality of surface regions are weighted and added and subjected to vector normalization processing to obtain a direction vector of the visible surface of the target entity.
  • the application also provides a method for implementing augmented reality, the method comprising:
  • the display portion is rendered into the video frame based on the center point coordinates such that a visible surface of the target entity is covered by the display portion.
  • the method further includes:
  • the virtual model is provided with a trigger point
  • the method further includes:
  • the trigger effect of the trigger point is rendered into the video frame.
  • the target entity is a physical globe
  • the virtual model is a virtual globe matching the physical globe.
  • an orientation tracking device for a target entity comprising:
  • An image acquisition module configured to acquire a video frame that includes a target entity
  • a feature extraction module configured to perform image feature extraction on a target entity in the video frame, and acquire image feature information of a visible surface of the target entity
  • a feature matching module configured to perform feature recognition on the acquired image feature information of the visible surface of the target entity based on the set of prefabricated features of the target entity, and acquire a surface region matching the visible surface in the prefabricated feature set
  • the set of prefabricated features includes a plurality of surface regions of the target entity and image feature information of each surface region;
  • an integrated processing module configured to respectively determine a center point coordinate and a direction vector of the visible surface of the target entity according to a center point coordinate and a direction vector of the surface area matching the visible surface.
  • the device further includes:
  • a pre-processing module configured to pre-process the target entity before acquiring the video frame that includes the target entity, and construct a pre-made feature set of the target entity.
  • the preprocessing module is configured to control the image acquisition module to acquire a target real An image of a plurality of surface regions, wherein the plurality of surface regions cover at least all surface regions of the target entity; controlling the feature extraction module to perform image feature extraction on images of the plurality of surface regions to obtain the Image feature information for each surface region; and constructing a set of prefabricated features of the target entity based on image feature information for each of the surface regions.
  • the integrated processing module is configured to perform weighted averaging processing on center point coordinates of the plurality of surface regions when there are multiple surface regions matching the visible surface, and determine a center point coordinate of the target entity .
  • the integrated processing module is configured to weight-add the direction vectors of the plurality of surface regions and perform vector normalization processing to obtain the target when there are multiple surface regions matching the visible surface The direction vector of the solid visible surface.
  • the application also provides a device for realizing augmented reality, the device comprising:
  • An orientation tracking device configured to acquire a center point coordinate and a direction vector of the target entity
  • a rendering device configured to acquire a display portion corresponding to the visible surface from the virtual model according to the direction vector, and render the display portion into the video frame according to the center point coordinate, so that the The visible surface of the target entity is covered by the display portion;
  • An output device for outputting a finished video frame.
  • the rendering device is further configured to acquire an information superposition point in the display portion, and render virtual information corresponding to the information superimposition point into the video frame to make a corresponding position of the display portion The virtual information is displayed.
  • the virtual model is provided with a trigger point
  • the rendering device is further configured to determine whether a trigger point of the display portion is located in a trigger region of the video frame, and when the trigger point is in a trigger region of the video frame, triggering the trigger point Rendered into the video frame.
  • the target entity is a physical globe
  • the virtual model is a virtual globe matching the physical globe.
  • the solution of the present application can superimpose some virtual graphics, images, information, etc. into a real scene based on augmented reality technology, and enhances the real scene by using the physical globe as a target entity. It enables it to combine the advantages of both physical globes and virtual globes in the prior art.
  • Get the central coordinate point of the target entity in real time and The direction vector realizes the azimuth tracking, and the real-time rendering of the output video frame based on the central coordinate point and the direction vector, so that the virtual globe rotates at the same angle and speed during the manipulation of the physical globe, which can greatly improve the user.
  • the direction vector realizes the azimuth tracking, and the real-time rendering of the output video frame based on the central coordinate point and the direction vector, so that the virtual globe rotates at the same angle and speed during the manipulation of the physical globe, which can greatly improve the user.
  • the virtual globe rotates at the same angle and speed during the manipulation of the physical globe, which can greatly improve the user.
  • the image feature extraction is performed by the video frame containing the target entity acquired in real time, and the image feature matching is performed with the pre-built pre-made feature set, and the calculation is performed based on the matching result to obtain the center of the visible surface of the target entity in the current video frame.
  • the point coordinates and the direction vector can quickly and accurately achieve the azimuth tracking of the target entity, ensuring that the virtual globe displayed on the screen and the physical globe can move synchronously, giving the user a real sense of space presence and an actual operational experience.
  • FIG. 1 is a flowchart of a method for tracking an azimuth of a target entity according to an embodiment of the present application
  • FIG. 2( a ) is a schematic diagram of a feasible projection of a surface area of a globe in an equatorial plane provided by the present application.
  • FIG. 2(b) to (e) are schematic views showing the projection of the four hemispherical planes divided by the plane of Fig. 2(a) on the equatorial plane;
  • FIG. 3 is a schematic diagram showing the spatial orientation of two surface regions matching the visible surface in a reference coordinate system in the embodiment of the present application;
  • FIG. 5 is a flowchart of a method for implementing augmented reality according to an embodiment of the present application
  • FIG. 6 is a schematic structural diagram of an azimuth tracking device of a target entity according to an embodiment of the present application.
  • FIG. 7 is a schematic structural diagram of an apparatus for implementing augmented reality according to an embodiment of the present disclosure.
  • the terminal, the device of the service network, and the trusted party are both included.
  • One or more processors CPUs
  • input/output interfaces network interfaces
  • memory volatile and non-volatile memory
  • the memory may include non-persistent memory, random access memory (RAM), and/or non-volatile memory in a computer readable medium, such as read only memory (ROM) or flash memory.
  • RAM random access memory
  • ROM read only memory
  • Memory is an example of a computer readable medium.
  • Computer readable media includes both permanent and non-persistent, removable and non-removable media, and information storage can be implemented by any method or technology.
  • the information can be computer readable instructions, data structures, modules of programs, or other data.
  • Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory.
  • PRAM phase change memory
  • SRAM static random access memory
  • DRAM dynamic random access memory
  • RAM random access memory
  • ROM read only memory
  • EEPROM electrically erasable programmable read only memory
  • flash memory or other memory technology
  • compact disk read only memory CD-ROM
  • DVD digital versatile disk
  • a magnetic tape cartridge magnetic tape storage or other magnetic storage device or any other non-transportable medium that can be used to store information that can be accessed by a computing device.
  • computer readable media does not include non-transitory computer readable media, such as modulated data signals and carrier waves.
  • FIG. 1 is a schematic diagram of an azimuth tracking method for a target entity according to an embodiment of the present application.
  • the method is used to implement azimuth tracking of a visible surface of a physical target during processing of AR (Augmented Reality). Specifically, the following steps are included:
  • Step S101 Acquire a video frame including a target entity.
  • the target entity may be a physical globe, or may be other regular or irregularly shaped entities such as spheres, ellipsoids, polyhedra, and the like.
  • a target entity can be photographed by a device with an image capture device such as a camera to obtain a video frame containing the target entity.
  • an image capture device such as a camera
  • a video of a physical globe is taken by a camera of a smartphone, and any frame in the video can be used for subsequent steps to obtain a center coordinate point and a direction vector of the visible surface of the physical globe in the video frame.
  • Step S102 performing image feature extraction on the target entity in the video frame, and acquiring image feature information of the visible surface of the target entity.
  • a certain target entity only the surface of a certain part of the surface can be seen in the video frame.
  • the visible surface of the physical globe in the video frame is at most half a spherical surface, and when image feature extraction is performed, Also only need to mention Take the visual half-spherical image feature information.
  • the visible surface may include one or more of the sides depending on the angle of the camera lens.
  • image feature extraction various mature image feature extraction techniques in the prior art may be used, for example, based on any one or more combinations of image feature information such as color features, texture features, shape features, and spatial relationship features. Extracting one or more image feature information of the surface pattern of the globe.
  • Step S103 performing feature recognition on the acquired image feature information of the visible surface of the target entity based on the pre-made feature set of the target entity, and acquiring a surface area matching the visible surface in the pre-made feature set.
  • the prefabricated feature set includes a plurality of surface regions of the target entity and image feature information of each surface region
  • the physical globe is still taken as an example, and a plurality of surface regions included in the prefabricated feature set are feasible.
  • the division is as follows: dividing the globe by the direction of the warp, dividing the whole globe into four complete hemispheres, the center points of the four hemispheres (ie, the center of the sphere) are completely coincident, and each adjacent two hemispheres There is a 90° overlap between the two.
  • the projection of this division on the equatorial plane is shown in Fig. 2(a), and the projection of each hemisphere on the equatorial plane is shown in Fig. 2(b) to (e).
  • the preset feature set may directly adopt the existing set data, or may be obtained by offline pre-processing before processing.
  • the image feature information corresponding to the four hemispheres will also be completely different.
  • the image feature information corresponding to each hemispherical surface may also adopt a combination of any one or more of image feature information such as a color feature, a texture feature, a shape feature, and a spatial relationship feature.
  • the two can be matched, thereby obtaining the prefabricated feature. All surface areas in the collection that match the visible surface.
  • the visible surface of the acquired physical globe may contain a complete hemisphere in the set of prefabricated features, or may be composed of multiple A part of the hemisphere.
  • the lens direction coincides with any point on the equator and the geocentric line.
  • the visible surface of the physical globe may contain a complete hemisphere in the set of prefabricated features, or a part of two hemispheres; If the earth axes coincide, then the visible surface of the physical globe may be composed of portions of the four hemispheres in the set of prefabricated features described above.
  • Step S104 determining a center point coordinate and a direction vector of the visible surface of the target entity according to a center point coordinate and a direction vector of the surface area matched with the visible surface.
  • the position of the current camera is set as the origin of the reference coordinate system. In this reference coordinate system, whether the camera moves or the target entity moves, the position of the target entity can be regarded as occurring. The change, while the position of the camera remains the same. Since the spatial position of the visible surface of the target entity can be determined in the video frame, the center point coordinates and the direction vector of the surface area that matches the visible surface can thereby be determined.
  • the center point coordinate of the surface area is the coordinates of the center of the sphere in the reference coordinate system whose origin is the position where the camera is located, and the direction vector can be specified as the point from the center of the sphere under the same reference frame.
  • the unit vector to the longitude and latitude are 0° points.
  • the center point coordinate and the direction vector of the surface area are the center point coordinates and the direction vector of the visible surface of the target entity.
  • the center point coordinates and the direction vector of the surface area acquired by the video frame may exist due to factors unavoidable such as processing precision, camera shake, and the like. error.
  • the coordinates of the center points corresponding to the surface areas of the same target entity should be consistent, and in the presence of errors, the center point coordinates of the multiple surface areas may not be the same, and the corresponding direction vectors may also be different.
  • the spatial orientation of the two surface regions matching the visible surface in the reference coordinate system is as shown in FIG. 3, the coordinates of the center point are respectively with The direction vector is with
  • a center point coordinate of a visible surface of the target entity according to a center point coordinate of a surface area matching the visible surface, if there are multiple surface areas matching the visible surface, a plurality of surface areas
  • the center point coordinates are subjected to a weighted averaging process to determine the center point coordinates of the target entity.
  • the setting of the weights may be specifically set according to different scenarios to achieve an optimal calculation precision, and as a feasible implementation manner, the weights of all surface regions may be set. Set to the same, you can calculate according to the following formula:
  • the coordinates of the center point of the visible surface of the target entity Represents the center point coordinates of any surface area that matches the visible surface, where n is the number of matched surface areas.
  • the target entity visible surface direction vector is determined according to the direction vector of the surface area matching the visible surface, if there are multiple surface areas matching the visible surface, the direction of the plurality of surface areas
  • the vector weights are added and subjected to vector normalization processing to obtain a direction vector of the visible surface of the target entity.
  • the weight setting can be specifically set according to different scenarios to achieve optimal calculation accuracy, and as a feasible implementation manner, the weights of all surface areas can be set to be the same. It can be calculated according to the following formula:
  • the spatial orientation of the physical globe at each moment can be dynamically obtained. Whether changing the relative position of the camera and the physical globe, or rotating the physical globe, the physical globe can be accurately acquired relative to the center of the camera. The point coordinates and the direction vector, and based on this, the virtual globe is rendered to ensure that the virtual globe and the physical globe rotate or move in synchronization.
  • the target entity may be pre-processed before the video frame containing the target entity is acquired to construct a pre-made feature set of the target entity.
  • a corresponding set of prefabricated features can be constructed for a target entity of any shape, thereby realizing real-time azimuth tracking and AR implementation of the target entity of any shape.
  • the main process of the preprocessing is to scan the image features on the surface of the sphere offline to obtain a prefabricated feature set on the surface of the sphere.
  • the prefabricated feature set is pre-stored and can be collected in real time.
  • the image features of the visible surface of the physical globe are compared and matched.
  • the specific process includes the steps shown in Figure 4:
  • Step S401 acquiring an image of multiple surface areas of the target entity, wherein the plurality of surface areas cover at least all surface areas of the target entity;
  • Step S402 performing image feature extraction on the images of the plurality of surface regions, and acquiring image feature information of each surface region;
  • Step S403 construct a pre-made feature set of the target entity according to the image feature information of each surface area.
  • the prefabricated feature set is composed of a plurality of surface regions of the surface of the sphere (relative to the entire sphere surface, the surface region being a partial curved surface), and the coverage area of each surface region is no more than one complete hemisphere;
  • each two adjacent surface areas may have a certain overlap area
  • the embodiment of the present application further provides a method for implementing augmented reality.
  • the processing flow of the method is as shown in FIG. 5, and includes the following steps:
  • Step S501 Acquire a center point coordinate and a direction vector of the target entity by using the foregoing azimuth tracking method.
  • Step S502 obtaining a display portion corresponding to the visible surface from the virtual model according to the direction vector.
  • the virtual model may be a virtual globe that matches the physical globe.
  • a display portion corresponding to the visible surface may be determined based on a center of the virtual globe, and the display portion is a hemispherical surface pointed by the direction vector.
  • Step S503 rendering the display portion into the video frame according to the center point coordinate, so that a visible surface of the target entity is covered by the display portion.
  • the display portion Based on the coordinates of the center point, it can be determined at which position in the video frame the display portion needs to be rendered.
  • the hemispherical surface of the display portion can be determined in the video frame based on the coordinates of the center point and the radius of the virtual globe. The spatial position that should be in, thereby completing the synthesis of the final picture, such that the visible surface of the original physical globe in the video frame is covered by the corresponding hemisphere of the virtual globe.
  • Step S504 outputting the completed video frame.
  • the physical globe can not be seen in the picture presented to the user, but the virtual globe replaces the original position of the physical globe, because the virtual globe can display a large amount of information, and in the process of manipulating the physical globe, The virtual globe rotates at the same angle and speed, greatly improving the user experience.
  • virtual globes can be used to carry large amounts of information.
  • virtual information of different topics can be added, for example, in correspondence.
  • the geographical location presents three-dimensional models of landmark buildings, rare animals, and geological wonders.
  • the specific treatment method is:
  • an information superposition point in the display portion is acquired.
  • the display part contains a preset information superposition point, for example, an information superposition point about the giant panda is preset in the position of Sichuan, when the virtual globe is set.
  • the superposition point of the information can be detected.
  • the virtual information corresponding to the information superposition point is rendered into the video frame, so that the corresponding position of the display portion displays the virtual information.
  • the virtual information can be a three-dimensional model of a giant panda, so that the user can directly see a giant panda in Sichuan on the screen, and more intuitive and clear understanding of various geographical knowledge.
  • some trigger points can be set on the virtual model.
  • the trigger area can be set according to actual needs. For example, for a touch screen, you can The trigger area is set in the area currently touched by the user, so that the user clicks to implement the trigger; or can be statically set in a specific area of the screen, such as the center of the screen; and can be further marked by a cursor, an aperture, etc., to facilitate user identification. .
  • the triggering effect may be a voice, a video, a three-dimensional animation effect, etc., for example, setting the trigger point to a display position of the giant panda. When the three-dimensional model of the giant panda is in the triggering region, the three-dimensional model of the giant panda is Zoom in and perform corresponding actions at the same time, such as walking, standing, etc.
  • the triggering effect can also be a highlighting effect, for example, setting the trigger point to the range of any administrative area, when an administrative area enters the triggering area, the administrative The area will be highlighted.
  • a highlighting effect for example, setting the trigger point to the range of any administrative area, when an administrative area enters the triggering area, the administrative The area will be highlighted.
  • the triggering area, the triggering effect, and the setting manner of the triggering point are only examples, and other existing or future possible manners may be applied to the present application, and should also be included in the present application. It is within the scope of protection and is hereby incorporated by reference.
  • an orientation tracking device 60 for a target entity is also provided.
  • the device 60 is configured to implement azimuth tracking of a visible surface of a physical target during processing of implementing the AR.
  • 6 shows an image acquisition module 610, a feature extraction module 620, a feature matching module 630, and an integrated processing module 640.
  • the image collection module 610 is configured to acquire a video frame that includes a target entity.
  • the target entity may be a physical globe, or may be other regular or irregularly shaped entities such as spheres, ellipsoids, polyhedra, and the like.
  • a target entity can be photographed by a device with an image capture device such as a camera to obtain a video frame containing the target entity.
  • a video of a physical globe is taken by a camera of a smartphone, and any frame in the video can be used for subsequent processing to obtain a central coordinate point and a direction vector of the visible surface of the physical globe in the video frame.
  • the feature extraction module 620 is configured to perform image feature extraction on a target entity in the video frame to acquire image feature information of a visible surface of the target entity.
  • a target entity For a certain target entity, only the surface of a certain part of the surface can be seen in the video frame.
  • the visible surface of the physical globe in the video frame is at most half a spherical surface, and when image feature extraction is performed, It is also only necessary to extract visual half-spherical image feature information.
  • the visible surface may include one or more of the sides depending on the angle of the camera lens.
  • image feature extraction various mature image feature extraction techniques in the prior art can be used, for example, based on color features, texture features, and shapes.
  • image feature information such as signs and spatial relationship features, and extracting one or more image feature information of the surface pattern of the globe.
  • the feature matching module 630 is configured to perform feature recognition on the acquired image feature information of the target entity visible surface based on the set of pre-made features of the target entity, and acquire a matching match with the visible surface in the pre-made feature set.
  • Surface area Wherein the prefabricated feature set includes a plurality of surface regions of the target entity and image feature information of each surface region, and the physical globe is still taken as an example, and a plurality of surface regions included in the prefabricated feature set are feasible
  • the division is as follows: dividing the globe by the direction of the warp, dividing the whole globe into four complete hemispheres, the center points of the four hemispheres (ie, the center of the sphere) are completely coincident, and each adjacent two hemispheres There is a 90° overlap between the two.
  • the projection of this division on the equatorial plane is shown in Fig. 2(a), and the projection of each hemisphere on the equatorial plane is shown in Fig. 2(b) to (e).
  • the preset feature set may directly adopt the existing set data, or may be obtained by offline pre-processing before processing.
  • the image feature information corresponding to the four hemispheres will also be completely different.
  • the image feature information corresponding to each hemispherical surface may also adopt a combination of any one or more of image feature information such as a color feature, a texture feature, a shape feature, and a spatial relationship feature.
  • the two can be matched, thereby obtaining the prefabricated feature. All surface areas in the collection that match the visible surface.
  • the visible surface of the acquired physical globe may contain a complete hemisphere in the set of prefabricated features, or may be composed of multiple A part of the hemisphere.
  • the lens direction coincides with any point on the equator and the geocentric line
  • the visible surface of the physical globe may contain a complete hemisphere in the set of prefabricated features, or a part of two hemispheres;
  • the direction coincides with the earth axis, at which point the visible surface of the physical globe may consist of portions of the four hemispheres in the set of prefabricated features described above.
  • the comprehensive processing module 640 is configured to determine a center point coordinate and a direction vector of the visible surface of the target entity according to a center point coordinate and a direction vector of the surface area matched with the visible surface.
  • the position of the current camera is set as the origin of the reference coordinate system. In this reference coordinate system, whether the camera moves or the target entity moves, the position of the target entity can be regarded as occurring. The change, while the position of the camera remains the same. Since the spatial position of the visible surface of the target entity can be determined in the video frame, the center point coordinates and the direction vector of the surface area that matches the visible surface can thereby be determined.
  • the center point coordinate of the surface area is the coordinates of the center of the sphere in the reference coordinate system whose origin is the position where the camera is located, and the direction vector can be specified as the point from the center of the sphere under the same reference frame.
  • the unit vector to the longitude and latitude are 0° points.
  • the center point coordinate and the direction vector of the surface area are the center point coordinates and the direction vector of the visible surface of the target entity.
  • the center point coordinates and the direction vector of the surface area acquired by the video frame may exist due to factors unavoidable such as processing precision, camera shake, and the like. error.
  • the coordinates of the center points corresponding to the surface areas of the same target entity should be consistent, and in the presence of errors, the center point coordinates of the multiple surface areas may not be the same, and the corresponding direction vectors may also be different.
  • the spatial orientation of the two surface regions matching the visible surface in the reference coordinate system is as shown in FIG. 3, the coordinates of the center point are respectively with The direction vector is with
  • the integrated processing module 640 determines the center point coordinates of the visible surface of the target entity according to the center point coordinates of the surface area matching the visible surface, if there are multiple surface areas matching the visible surface And performing weighted averaging processing on center point coordinates of the plurality of surface regions to determine a center point coordinate of the target entity.
  • the weight setting may be specifically set according to different scenarios to achieve an optimal calculation precision, and as a feasible implementation manner, the weights of all surface regions may be set to be the same, and the calculation may be performed according to the following formula:
  • the coordinates of the center point of the visible surface of the target entity Represents the coordinates of the center point of any surface area that matches the visible surface, where n is the number of matched surface areas.
  • the integrated processing module 640 determines the target entity visible surface direction vector according to the direction vector of the surface area matching the visible surface, if there are multiple surface areas matching the visible surface, The direction vectors of the plurality of surface regions are weighted and added and subjected to vector normalization processing to obtain a direction vector of the visible surface of the target entity.
  • the weight setting can be specifically set according to different scenarios to achieve optimal calculation accuracy, and as a feasible implementation manner, the weights of all surface areas can be set to be the same. It can be calculated according to the following formula:
  • the spatial orientation of the physical globe at each moment can be dynamically obtained. Whether changing the relative position of the camera and the physical globe, or rotating the physical globe, the physical globe can be accurately acquired relative to the center of the camera. The point coordinates and the direction vector, and based on this, the virtual globe is rendered to ensure that the virtual globe and the physical globe rotate or move in synchronization.
  • the target entity may be pre-processed by the pre-processing module to obtain a pre-made feature set of the target entity before acquiring the video frame including the target entity.
  • a corresponding set of prefabricated features can be constructed for a target entity of any shape, thereby realizing real-time azimuth tracking and AR implementation of the target entity of any shape.
  • the main process of the preprocessing is to scan the image features on the surface of the sphere offline to obtain a prefabricated feature set on the surface of the sphere.
  • the prefabricated feature set is pre-stored and can be collected in real time.
  • the physical features of the physical globe are aligned and matched to the image features of the visible surface.
  • the pre-processing module is specifically configured to control the image acquisition module to acquire an image of a plurality of surface regions of the target entity, wherein the plurality of surface regions cover at least all surface regions of the target entity; and the feature extraction module is controlled Image of the plurality of surface regions is subjected to image feature extraction Taking image feature information of each surface area; and constructing a set of prefabricated features of the target entity according to image feature information of each surface area.
  • the prefabricated feature set is composed of a plurality of surface regions of the surface of the sphere (relative to the entire sphere surface, the surface region being a partial curved surface), and the coverage area of each surface region is no more than one complete hemisphere;
  • each two adjacent surface areas may have a certain overlap area
  • the embodiment of the present application further provides an apparatus for implementing augmented reality.
  • the structure of the apparatus is as shown in FIG. 7, and includes an azimuth tracking device 60, a rendering device 710, and an output device 720.
  • the azimuth tracking device 60 is configured to acquire a center point coordinate and a direction vector of the target entity.
  • the rendering device 710 is configured to acquire a display portion corresponding to the visible surface from a virtual model according to the direction vector, and render the display portion into the video frame according to the center point coordinate, so that The visible surface of the target entity is covered by the display portion.
  • the virtual model may be a virtual globe that matches the physical globe.
  • a display portion corresponding to the visible surface may be determined based on a center of the virtual globe, and the display portion is a hemispherical surface pointed by the direction vector. Based on the coordinates of the center point, it can be determined at which position in the video frame the display portion needs to be rendered. Still taking the virtual globe as an example, the hemispherical surface of the display portion can be determined in the video frame based on the coordinates of the center point and the radius of the virtual globe. The spatial position that should be in, thereby completing the synthesis of the final picture, such that the visible surface of the original physical globe in the video frame is covered by the corresponding hemisphere of the virtual globe.
  • the output device 720 is configured to output a video frame that is finished rendering.
  • the output device 720 is configured to output a video frame that is finished rendering.
  • the device may include, but is not limited to, a user terminal or a device formed by integrating a user terminal and a network device through a network.
  • the user terminal includes, but is not limited to, a personal computer, a touch terminal, and the like, and specifically may be a smart phone, a tablet computer, a PDA, an AR glasses, or other electronic device having an image acquisition, processing, and output function; Implementations are not limited to, for example, a network host, a single network server, multiple network server sets, or a cloud-based computer collection.
  • the cloud is composed of a large number of host or network servers based on Cloud Computing, which is a kind of distributed computing, a virtual computer composed of a group of loosely coupled computers.
  • the device may also be an electronic device running an application (APP) containing associated algorithms and graphical user interfaces, or the application itself.
  • APP application
  • the camera of the smart phone can be used to implement the related functions of the image capturing module 610
  • the processor can be used to implement the feature extraction module 620, the feature matching module 630, the comprehensive processing module 640, and
  • the related functions of the device 710 are rendered while their screens enable the associated functions of the output device 720.
  • the processor can also upload relevant data to the network device through the communication module, and the network device completes calculation and processing of related data.
  • virtual globes can be used to carry large amounts of information.
  • virtual information of different topics can be added, for example, in correspondence.
  • the geographical location presents three-dimensional models of landmark buildings, rare animals, and geological wonders.
  • the rendering device 710 is further configured to perform the following processing:
  • an information superposition point in the display portion is acquired.
  • the display part contains a pre-set information superposition point, for example, a superposition point about the giant panda is preset in the position of Sichuan, when the virtual globe is set.
  • a superposition point about the giant panda is preset in the position of Sichuan, when the virtual globe is set.
  • the hemisphere of the display part contains the geographical area where Sichuan is located, it can be detected. Go to the superposition point of this information.
  • the virtual information corresponding to the information superposition point is rendered into the video frame, so that the corresponding position of the display portion displays the virtual information.
  • the virtual information can be a three-dimensional model of a giant panda, so that the user can directly see a giant panda in Sichuan on the screen, and more intuitive and clear understanding of various geographical knowledge.
  • some trigger points can be set on the virtual model.
  • the rendering device 710 determines whether the trigger point of the display portion is located in a trigger region of the video frame; if the trigger point is in a trigger region of the video frame, triggering the trigger point Rendered into the video frame.
  • the trigger area can be set according to actual needs. For example, for a touch screen, the trigger area may be set to an area currently touched by the user to implement a user click to implement triggering; or may be statically set in a certain specific area of the screen, such as the center of the screen; and may further pass a cursor, an aperture, etc. Mark it for user identification.
  • the triggering effect may be audio, video, three-dimensional animation effect, etc., for example, setting the trigger point to the display position of the giant panda. When the three-dimensional model of the giant panda is in the triggering area, the three-dimensional model of the giant panda will be Zoom in and perform corresponding actions at the same time, such as walking, standing, etc.
  • the triggering effect can also be a highlighting effect, for example, setting the trigger point to the range of any administrative area, when an administrative area enters the triggering area, the administrative The area will be highlighted.
  • a highlighting effect for example, setting the trigger point to the range of any administrative area, when an administrative area enters the triggering area, the administrative The area will be highlighted.
  • the triggering area, the triggering effect, and the setting manner of the triggering point are only examples, and other existing or future possible manners may be applied to the present application, and should also be included in the present application. It is within the scope of protection and is hereby incorporated by reference.
  • the solution of the present application can superimpose some virtual graphics, images, information, etc. into a real scene based on augmented reality technology, and enhances the real scene by using the physical globe as a target entity. It combines the advantages of both physical globes and virtual globes in the prior art.
  • Azimuth tracking is realized by acquiring the central coordinate point and direction vector of the target entity in real time, and the output video frame is rendered in real time based on the central coordinate point and the direction vector, so that the virtual globe will be the same in the process of manipulating the physical globe.
  • Angle and speed rotation can greatly improve the user experience.
  • image feature extraction is performed by a video frame containing the target entity acquired in real time
  • Image feature matching is performed with the pre-built pre-made feature set, and calculation is performed based on the matching result to obtain the center point coordinate and the direction vector of the visible surface of the target entity in the current video frame, thereby enabling fast and accurate orientation tracking of the target entity.
  • the present application can be implemented in software and/or a combination of software and hardware, for example, using an application specific integrated circuit (ASIC), a general purpose computer, or any other similar hardware device.
  • the software program of the present application can be executed by a processor to implement the steps or functions described above.
  • the software programs (including related data structures) of the present application can be stored in a computer readable recording medium such as a RAM memory, a magnetic or optical drive or a floppy disk and the like.
  • some of the steps or functions of the present application may be implemented in hardware, for example, as a circuit that cooperates with a processor to perform various steps or functions.
  • a portion of the present application can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide a method and/or technical solution in accordance with the present application.
  • the program instructions for invoking the method of the present application may be stored in a fixed or removable recording medium, and/or transmitted by a data stream in a broadcast or other signal bearing medium, and/or stored in a The working memory of the computer device in which the program instructions are run.
  • an embodiment in accordance with the present application includes a device including a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, triggering
  • the apparatus operates based on the aforementioned methods and/or technical solutions in accordance with various embodiments of the present application.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Graphics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Astronomy & Astrophysics (AREA)
  • Business, Economics & Management (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • Processing Or Creating Images (AREA)

Abstract

La présente invention concerne un procédé et un appareil pertinent de suivi d'orientation et un procédé et un dispositif destinés à la mise en œuvre de la réalité augmentée, lesquels, sur la base d'une technologie de réalité augmentée, peuvent superposer certains graphiques, images, informations, etc. virtuels dans une scène réelle, augmentant ainsi la scène réelle. La présente invention prend un globe terrestre physique en tant qu'entité cible et permet au globe terrestre physique de posséder les avantages à la fois d'un globe terrestre physique selon l'état de la technique et d'un globe terrestre virtuel. La présente invention permet de mettre en œuvre le suivi d'orientation par acquisition, en temps réel, du point de coordonnées central et d'un vecteur de direction d'une entité cible et de fournir, en temps réel, à partir du point de coordonnées central et du vecteur de direction, des trames vidéo émises de sorte que lorsqu'un utilisateur manipule un globe terrestre physique, un globe terrestre virtuel tourne selon le même angle et selon la même vitesse, ce qui améliore considérablement l'expérience de l'utilisateur.
PCT/CN2017/080276 2016-07-19 2017-04-12 Procédé et appareil associé de suivi d'orientation, procédé et dispositif destinés à la mise en œuvre de la réalité augmentée WO2018014601A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201610570465.X 2016-07-19
CN201610570465.XA CN106251404B (zh) 2016-07-19 2016-07-19 方位跟踪方法、实现增强现实的方法及相关装置、设备

Publications (1)

Publication Number Publication Date
WO2018014601A1 true WO2018014601A1 (fr) 2018-01-25

Family

ID=57613380

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/080276 WO2018014601A1 (fr) 2016-07-19 2017-04-12 Procédé et appareil associé de suivi d'orientation, procédé et dispositif destinés à la mise en œuvre de la réalité augmentée

Country Status (2)

Country Link
CN (1) CN106251404B (fr)
WO (1) WO2018014601A1 (fr)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111093301A (zh) * 2019-12-14 2020-05-01 安琦道尔(上海)环境规划建筑设计咨询有限公司 一种灯光控制方法及其系统
CN113269810A (zh) * 2018-04-11 2021-08-17 深圳市瑞立视多媒体科技有限公司 一种捕捉球的运动姿态识别方法及其装置
CN115057242A (zh) * 2022-03-07 2022-09-16 上海箱云物流科技有限公司 一种基于虚拟现实和增强现实结合的集装箱管理方法和系统

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106251404B (zh) * 2016-07-19 2019-02-01 央数文化(上海)股份有限公司 方位跟踪方法、实现增强现实的方法及相关装置、设备
CN108540824B (zh) * 2018-05-15 2021-01-19 北京奇虎科技有限公司 一种视频渲染方法和装置
CN109582147B (zh) * 2018-08-08 2022-04-26 亮风台（上海）信息科技有限公司 一种用于呈现增强交互内容的方法以及用户设备
CN109656363B (zh) * 2018-09-04 2022-04-15 亮风台（上海）信息科技有限公司 一种用于设置增强交互内容的方法与设备
CN110084204B (zh) * 2019-04-29 2020-11-24 北京字节跳动网络技术有限公司 基于目标对象姿态的图像处理方法、装置和电子设备
CN110335292B (zh) * 2019-07-09 2021-04-30 北京猫眼视觉科技有限公司 一种基于图片跟踪实现模拟场景跟踪的方法、系统及终端
CN110727350A (zh) * 2019-10-09 2020-01-24 武汉幻石佳德数码科技有限公司 一种基于增强现实的物体识别方法、终端设备及存储介质
CN112051746B (zh) * 2020-08-05 2023-02-07 华为技术有限公司 获取服务的方法以及装置
CN112565605B (zh) * 2020-12-02 2022-11-25 维沃移动通信有限公司 图像显示方法、装置和电子设备
US20240169582A1 (en) * 2021-03-08 2024-05-23 Hangzhou Taro Positioning Technology Co., Ltd. Scenario triggering and interaction based on target positioning and identification
CN118112606A (zh) * 2024-04-30 2024-05-31 中孚安全技术有限公司 一种车辆gps跟踪器精确排查系统、方法及介质

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120243739A1 (en) * 2011-03-25 2012-09-27 Masaki Fukuchi Information processing device, object recognition method, program, and terminal device
CN103049729A (zh) * 2012-12-30 2013-04-17 成都理想境界科技有限公司 基于二维码的增强现实方法、系统及终端
CN104050475A (zh) * 2014-06-19 2014-09-17 樊晓东 基于图像特征匹配的增强现实的系统和方法
CN104966307A (zh) * 2015-07-10 2015-10-07 成都品果科技有限公司 一种基于实时跟踪的ar算法
CN105069754A (zh) * 2015-08-05 2015-11-18 意科赛特数码科技(江苏)有限公司 基于在图像上无标记增强现实的系统和方法
CN105338117A (zh) * 2015-11-27 2016-02-17 亮风台（上海）信息科技有限公司 用于生成ar应用和呈现ar实例的方法、设备与系统
CN105491365A (zh) * 2015-11-25 2016-04-13 罗军 基于移动终端的图像处理方法、装置及系统
CN106251404A (zh) * 2016-07-19 2016-12-21 央数文化(上海)股份有限公司 方位跟踪方法、实现增强现实的方法及相关装置、设备

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102142055A (zh) * 2011-04-07 2011-08-03 上海大学 基于增强现实交互技术的真三维设计方法
US8855366B2 (en) * 2011-11-29 2014-10-07 Qualcomm Incorporated Tracking three-dimensional objects
CN103366610B (zh) * 2013-07-03 2015-07-22 央数文化(上海)股份有限公司 基于增强现实的三维互动学习系统及方法
CN104134229A (zh) * 2014-08-08 2014-11-05 李成 实时交互的增强现实系统以及方法
CN104463108B (zh) * 2014-11-21 2018-07-31 山东大学 一种单目实时目标识别及位姿测量方法

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120243739A1 (en) * 2011-03-25 2012-09-27 Masaki Fukuchi Information processing device, object recognition method, program, and terminal device
CN103049729A (zh) * 2012-12-30 2013-04-17 成都理想境界科技有限公司 基于二维码的增强现实方法、系统及终端
CN104050475A (zh) * 2014-06-19 2014-09-17 樊晓东 基于图像特征匹配的增强现实的系统和方法
CN104966307A (zh) * 2015-07-10 2015-10-07 成都品果科技有限公司 一种基于实时跟踪的ar算法
CN105069754A (zh) * 2015-08-05 2015-11-18 意科赛特数码科技(江苏)有限公司 基于在图像上无标记增强现实的系统和方法
CN105491365A (zh) * 2015-11-25 2016-04-13 罗军 基于移动终端的图像处理方法、装置及系统
CN105338117A (zh) * 2015-11-27 2016-02-17 亮风台（上海）信息科技有限公司 用于生成ar应用和呈现ar实例的方法、设备与系统
CN106251404A (zh) * 2016-07-19 2016-12-21 央数文化(上海)股份有限公司 方位跟踪方法、实现增强现实的方法及相关装置、设备

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113269810A (zh) * 2018-04-11 2021-08-17 深圳市瑞立视多媒体科技有限公司 一种捕捉球的运动姿态识别方法及其装置
CN111093301A (zh) * 2019-12-14 2020-05-01 安琦道尔(上海)环境规划建筑设计咨询有限公司 一种灯光控制方法及其系统
CN115057242A (zh) * 2022-03-07 2022-09-16 上海箱云物流科技有限公司 一种基于虚拟现实和增强现实结合的集装箱管理方法和系统
CN115057242B (zh) * 2022-03-07 2023-08-01 上海箱云物流科技有限公司 一种基于虚拟现实和增强现实结合的集装箱管理方法

Also Published As

Publication number Publication date
CN106251404B (zh) 2019-02-01
CN106251404A (zh) 2016-12-21

Similar Documents

Publication Publication Date Title
WO2018014601A1 (fr) Procédé et appareil associé de suivi d'orientation, procédé et dispositif destinés à la mise en œuvre de la réalité augmentée
US11481923B2 (en) Relocalization method and apparatus in camera pose tracking process, device, and storage medium
KR102506959B1 (ko) 깊이 추정 시스템의 자가 감독 훈련
CN109887003B (zh) 一种用于进行三维跟踪初始化的方法与设备
CN110910486B (zh) 室内场景光照估计模型、方法、装置、存储介质以及渲染方法
US11748906B2 (en) Gaze point calculation method, apparatus and device
Ventura et al. Wide-area scene mapping for mobile visual tracking
CN103839277B (zh) 一种户外大范围自然场景的移动增强现实注册方法
US10573060B1 (en) Controller binding in virtual domes
CN111046125A (zh) 一种视觉定位方法、系统及计算机可读存储介质
WO2019238114A1 (fr) Procédé, appareil et dispositif de reconstruction de modèle dynamique tridimensionnel, et support de stockage
CN115690382B (zh) 深度学习模型的训练方法、生成全景图的方法和装置
WO2021093679A1 (fr) Procédé et dispositif de positionnement visuel
WO2023280038A1 (fr) Procédé de construction d'un modèle tridimensionnel de scène réelle et appareil associé
CN112927362A (zh) 地图重建方法及装置、计算机可读介质和电子设备
WO2022088819A1 (fr) Procédé de traitement vidéo, appareil de traitement vidéo et support de stockage
US10740957B1 (en) Dynamic split screen
CN112766215A (zh) 人脸融合方法、装置、电子设备及存储介质
CN113724391A (zh) 三维模型构建方法、装置、电子设备和计算机可读介质
US11100617B2 (en) Deep learning method and apparatus for automatic upright rectification of virtual reality content
CN112270709A (zh) 地图构建方法及装置、计算机可读存储介质和电子设备
CN114882106A (zh) 位姿确定方法和装置、设备、介质
WO2022126921A1 (fr) Procédé et dispositif de détection d'image panoramique, terminal et support de stockage
CN110956571A (zh) 基于slam进行虚实融合的方法及电子设备
CN108205820B (zh) 平面的重建方法、融合方法、装置、设备及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17830239

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17830239

Country of ref document: EP

Kind code of ref document: A1