CN117930978A - AR interaction method, system, equipment and storage medium for museum cultural relics - Google Patents
AR interaction method, system, equipment and storage medium for museum cultural relics Download PDFInfo
- Publication number
- CN117930978A CN117930978A CN202410113363.XA CN202410113363A CN117930978A CN 117930978 A CN117930978 A CN 117930978A CN 202410113363 A CN202410113363 A CN 202410113363A CN 117930978 A CN117930978 A CN 117930978A
- Authority
- CN
- China
- Prior art keywords
- cultural relic
- virtual
- cultural
- museum
- model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000003993 interaction Effects 0.000 title claims abstract description 66
- 238000000034 method Methods 0.000 title claims abstract description 45
- 230000006870 function Effects 0.000 claims description 11
- 230000008859 change Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 8
- 238000013523 data management Methods 0.000 claims description 6
- 238000006243 chemical reaction Methods 0.000 claims description 4
- 210000002478 hand joint Anatomy 0.000 claims description 4
- 230000002452 interceptive effect Effects 0.000 claims description 4
- 238000006073 displacement reaction Methods 0.000 claims description 3
- 238000011176 pooling Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 238000012935 Averaging Methods 0.000 claims description 2
- 230000004931 aggregating effect Effects 0.000 claims description 2
- 230000003190 augmentative effect Effects 0.000 abstract description 5
- 238000001514 detection method Methods 0.000 abstract description 4
- 230000004927 fusion Effects 0.000 abstract description 3
- 230000000694 effects Effects 0.000 description 12
- 238000010586 diagram Methods 0.000 description 8
- 238000005516 engineering process Methods 0.000 description 8
- 230000008569 process Effects 0.000 description 7
- 238000007726 management method Methods 0.000 description 3
- 230000019771 cognition Effects 0.000 description 2
- 240000002234 Allium sativum Species 0.000 description 1
- 240000005979 Hordeum vulgare Species 0.000 description 1
- 235000007340 Hordeum vulgare Nutrition 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 235000004611 garlic Nutrition 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000003307 slaughter Methods 0.000 description 1
- 239000004575 stone Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Abstract
The invention discloses a method, a system, equipment and a storage medium for AR interaction of cultural relics in a museum, which relate to the technical fields of augmented reality and intelligent museums and comprise the steps of collecting image information of the cultural relics in the museum and carrying out three-dimensional reconstruction to obtain a virtual cultural relic model; acquiring a cultural relic photo in a museum, and calling a virtual cultural relic model corresponding to the cultural relic photo; acquiring a real scene, identifying a real scene plane, and placing a virtual cultural relic model on the identified plane to realize AR placement; and acquiring the key point information of the hand of the user, determining the gesture information of the hand of the user, and displaying the interaction information of the corresponding gesture. According to the method, the virtual cultural relic model is placed on the identified real scene plane by combining with the plane detection algorithm, so that the plane detection performance is improved, the accuracy of the virtual cultural relics in placement is promoted, the fusion degree of the virtual cultural relics and the real space scene is better realized, and the reality of interaction between visitors and the cultural relics is improved.
Description
Technical Field
The invention relates to the technical field of augmented reality and intelligent museums, in particular to a museum cultural relic AR interaction method, a system, equipment and a storage medium.
Background
Augmented reality technology is a technology intended to add digitized information resources in the real physical world (hereinafter collectively referred to as AR technology) to enhance human perception and cognition in a completely new way. Along with the high-speed development of mobile information technology and intelligent equipment, the AR technology is widely applied to various fields in recent years, has the advantages of real-time interaction, virtual-real combination, strong entertainment and the like, and can bring more real experience to users.
Starting from the 80 s of the 20 th century, the digitization of the national museum starts, and the processes of informatization, digitization, wisdom and the like are started. At present, domestic museums open up intelligent museums including intelligent management, intelligent protection, intelligent service and the like. In the content display service, most museums adopt an AR technical means to dynamically display resources, so that users can clearly watch and interact with a high-precision three-dimensional model of the cultural relics, and therefore, AR is a main technical means for the digital display of the cultural relics and the intelligent construction of the museums in the current and future.
In 2018, the large slaughter commemorative museum in washington, d.c. has shown an AR application that uses AR to describe Eisiskes civilian life; zhang Shaobo and the like propose an augmented reality museum display system based on an object 6D gesture estimation algorithm, which can meet the timeliness requirement of AR application and effectively improve the interaction mode of a user and cultural relics.
However, when the conventional museum displays the cultural relics through the AR technology, the cultural relics are only virtually displayed purely, the general conditions of the cultural relics are described, and the virtual cultural relics are placed randomly in the scene, so that the integration of the conventional museum and the space scene in the display process cannot be unified, the AR display mode is single and unreal, the sightedness of visitors is poor, and the popularization of agricultural cultural relics in China is not facilitated.
Disclosure of Invention
The invention aims to overcome the defects of the prior art, and provides a method, a system, equipment and a storage medium for AR interaction of museum cultural relics, so as to solve the problems that virtual cultural relics are placed randomly in the scene in the prior art, so that fusion with a space scene in the exhibition process cannot be unified, the AR exhibition mode is single and unreal, and the sightseeing of visitors is poor.
The invention specifically provides the following technical scheme: the museum cultural relic AR interaction method comprises the following steps of:
acquiring various cultural relic image information in a museum, and carrying out three-dimensional reconstruction according to various cultural relic image information to obtain a virtual cultural relic model;
after a user side obtains a cultural relic photo in a museum, a virtual cultural relic model corresponding to the cultural relic photo is called; real scenes are acquired in real time through a user side, a real scene plane is identified, and the virtual cultural relic model is placed on the real scene plane to be placed by AR;
Performing various gesture interactions through the hands of the user and the virtual cultural relic model placed by the AR;
The user side obtains the hand key point information of the user, identifies the hand gesture information of the user, and displays interaction information of the corresponding gesture and the virtual cultural relic model through the user side.
Preferably, the method for acquiring image information of various cultural relics in the museum and performing three-dimensional reconstruction according to the image information of various cultural relics to obtain a virtual cultural relic model comprises the following steps:
Acquiring a plurality of cultural relics from a museum exhibition hall, and shooting the cultural relics in different dimensions by using shooting equipment to acquire a profile of each cultural relic;
measuring cultural relics in the showcase by using a ranging APP to obtain related length, width and height structural data of each cultural relic;
inputting the profile of each cultural relic and the measured structural data into modeling software, carrying out three-dimensional reconstruction on the cultural relic, obtaining a virtual cultural relic model and exporting the virtual cultural relic model in an OBJ format;
Converting the virtual cultural relic models in the OBJ format into GLTF format by using a format conversion tool, and creating a cultural relic digital resource library;
And storing the cultural relic images and the virtual cultural relic models which are used as AR identification in the cultural relic digital resource library in a cloud server.
Preferably, the retrieving a virtual cultural relic model corresponding to the cultural relic photograph includes AR image recognition and AR landmark recognition, and specifically includes the following steps:
the AR image recognition includes the steps of:
an AR image recognition component is established, an image recognition mode is set according to a WeChat applet xr-frame framework, and scene light and camera following basic settings are set;
taking the cultural relic pictures in the cultural relic digital resource library as image sources for AR identification;
after a user side is used for scanning the cultural relic picture, a virtual cultural relic model corresponding to the cultural relic appears, and the user side is used for carrying out scaling and rotation control on the virtual cultural relic model;
the AR landmark identification comprises the following steps:
An AR road sign recognition component is established, an OSD recognition mode is set according to a WeChat applet xr-frame framework, and scene light and shooting equipment follow-up settings are set;
Taking a real photo of a road sign in the museum as a feature recognition source, and simultaneously adding child node position information;
and scanning a specific road sign of the museum for AR identification, and displaying AR indication information.
Preferably, the scaling and rotation control of the virtual cultural relic model by the user terminal includes the following steps:
Whether the operation is a single-finger operation or a double-finger operation is judged by event.
If the single-finger operation is judged, the virtual cultural relic model is rotated during the single-finger operation, and the specific operation comprises the following steps:
The current canvas information width and height are acquired, and the rotation radius R is determined, wherein the solving formula is as follows:
Acquiring a clicking position startX of a last triggering event and a current clicking position x of a finger, setting a rotating speed rSpeed as a fixed value, and setting an angle change difference value theta in the x direction according to a rotating radius, wherein a solving formula is as follows:
Acquiring an angle change difference value in the y direction in the same way, and determining a rotation result of the virtual cultural relic model according to the angle change in the x and y directions;
if the operation is judged to be double-finger operation, the virtual cultural relic model is scaled during the double-finger operation, and the specific operation comprises the following steps:
the method comprises the steps of obtaining the position information of the current double fingers in the x direction and the y direction, calculating the difference value of the two fingers in the x direction and the y direction, calculating the distance dis between the double fingers, and solving the following formula:
The distance startDis between the two fingers when the last scaling event occurs is obtained, the scaling ratio S is calculated, and the solving formula is as follows:
S＝(dis-startDis)×p+q
wherein p and q are scaling parameters for converting the distance difference into a scaling;
and multiplying the model values of the virtual cultural relic model in the x, y and z directions by the scaling ratio to finish scaling.
Preferably, placing the virtual cultural relic model on the real scene plane for AR placement, including the following steps:
Using WebRTC to acquire a real scene video stream of the museum, and using FFmpeg to extract key frames of the acquired video;
defining the space surrounding all the key frames as Fi, and extracting image features in the key frames by adopting 2D CNN;
back-projecting the image features into Fi to obtain voxel features and three-dimensional feature bodies;
Aggregating features from the same voxel in different keyframes using averaging pooling;
3D sparse convolution is used for processing the three-dimensional feature, constructing a three-dimensional sparse feature, and determining the occupation condition of voxels belonging to a plane;
Extracting normal parameters and plane offset belonging to the plane voxels according to the three-dimensional sparse feature;
Calculating the displacement between the voxel center and the centroid of the plane, and moving the voxel center to the plane centroid;
Determining final plane parameters after the voxel center moves to the plane centroid by using a Mean-Shift clustering algorithm;
and placing the virtual cultural relic model on the detected plane, and carrying out AR placement.
Preferably, the user side obtains the hand key point information of the user, identifies the hand gesture information of the user, and displays the interaction information of the corresponding gesture and the virtual cultural relic model through the user side, and the method comprises the following steps:
Creating a gesture recognition component, and setting a gesture recognition mode according to a WeChat applet xr-frame framework;
capturing hand joint point information of a user through shooting equipment, and determining hand gesture information;
And customizing basic gesture information, and displaying interaction information corresponding to the virtual cultural relic model according to the identified hand gesture information.
Preferably, the method further comprises the step of connecting the user side with the cultural relic digital resource library in the cloud server to perform interaction between multiple users and the cultural relics, and comprises the following steps:
storing personal information authorized by a user in a MySQL database of a cloud server;
And selecting a message interaction module, establishing connection with the MySQL database, watching the message content of other users, inputting the message content of the users, and synchronously updating the new message content in the database.
Preferably, the present invention also includes a museum cultural relic AR interactive system, comprising:
the data management module is used for collecting various cultural relics image information in the museum and carrying out three-dimensional reconstruction according to the various cultural relics image information to obtain a virtual cultural relics model;
the AR function module is used for calling a virtual cultural relic model corresponding to the cultural relic photo after the user side obtains the cultural relic photo in the museum; real scenes are acquired in real time through a user side, a real scene plane is identified, and the virtual cultural relic model is placed on the real scene plane to be placed by AR;
the interaction module is used for carrying out various gesture interactions through the hands of the user and the virtual cultural relic model placed by the AR;
The gesture recognition module is used for acquiring the key point information of the hand of the user by the user end, recognizing the gesture information of the hand of the user and displaying the interaction information of the corresponding gesture and the virtual cultural relic model by the user end.
The invention also provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, causes the processor to execute the steps of the museum cultural relics AR interaction method.
The present invention also provides a storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the museum relic AR interaction method.
Compared with the prior art, the invention has the following remarkable advantages:
According to the method, three-dimensional reconstruction is carried out on each type of cultural relic information through the collected cultural relic information, so that respective virtual cultural relic models are obtained, the real detail information of the cultural relics observed subsequently can be effectively improved, and the scanning and recognition pressure of a subsequent user side is reduced; meanwhile, during AR recognition, virtual cultural relics corresponding to the cultural relic photographs are directly obtained through obtaining the cultural relic photographs, real scenes are collected in real time, a virtual cultural relic model is placed on a recognized real scene plane through a corresponding algorithm, the plane detection performance is improved, the accuracy of the virtual cultural relics in placement is promoted, the fusion degree of the virtual cultural relics and the real space scenes is better achieved, and the reality of interaction between visitors and the cultural relics is improved.
Drawings
FIG. 1 is a diagram of the overall architecture of the system of the present invention;
FIG. 2 is a three-dimensional model diagram of a part of agricultural historic cultural relics; wherein (a) is a stone axe pattern; (b) is a garlic bulb pot; (c) a curved beam plow map; (d) is a lift map; (e) is an agricultural small implement-small hook agricultural implement graph; (f) barley fork diagram;
FIG. 3 is a flowchart of an AR image recognition operation;
FIG. 4 is an AR image recognition effect diagram, wherein (a) is an AR recognized image, (b) is an image scaled to a virtual cultural relic model, and (c) is a rotated image to a virtual cultural relic model;
FIG. 5 is a flowchart of an AR landmark recognition operation;
FIG. 6 is an AR roadmap effect map;
FIG. 7 is a flowchart of AR placement operation;
FIG. 8 is an AR placement effect diagram;
FIG. 9 is a flowchart of an AR gesture recognition operation;
FIG. 10 is an AR gesture effect diagram;
FIG. 11 is a flow chart of a multi-user message interaction operation;
fig. 12 is a diagram of a multi-user message interaction effect.
Detailed Description
The following description of the embodiments of the present invention, taken in conjunction with the accompanying drawings, will clearly and completely describe the embodiments of the present invention, and it is evident that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.
The invention provides a museum cultural relic AR interaction method, a system, equipment and a storage medium, which specifically comprise the following steps:
The AR technology can provide a more visual and vivid display mode, better store and display the original appearance of the cultural relics, not only can enable spectators to conduct virtual exhibition, but also can enable spectators to conduct interaction in various forms with the cultural relic model directly, and therefore agricultural cultural relics are effectively protected and inherited.
In summary, the use of the AR technology to construct the Chinese agricultural history museum not only can effectively protect and inherit agricultural cultural relics and improve the visiting experience of audiences, but also can provide diversified display modes and enhance communication and interaction among the audiences. This has important significance for promoting the propagation and promotion of agricultural culture and improving the cognition and understanding of the public to agricultural history. To facilitate an understanding of the features and advantages of the invention, further details of the invention are set forth in the accompanying block flow diagrams.
Referring to fig. 1, an embodiment of the present application provides a museum cultural relic AR interaction method, including the following steps:
Step S1: and acquiring image information of various cultural relics in the museum, and carrying out three-dimensional reconstruction according to the image information of the various cultural relics to obtain a virtual cultural relic model.
Wherein, the museum is a Chinese agricultural history museum.
In the step, various cultural relics image information in the museum is collected, and three-dimensional reconstruction is carried out according to the various cultural relics image information to obtain a virtual cultural relics model, and the method specifically comprises the following steps of:
Step S11: a plurality of (100 more representative agricultural histories) cultural relics are obtained from a museum (six) exhibition halls, different dimension shooting (360-degree surrounding shooting from the upper dimension, the middle dimension and the lower dimension respectively) is carried out on the cultural relics by using shooting equipment, and a profile of each cultural relic is obtained.
Step S12: and measuring the cultural relics in the showcase by using the ranging APP to obtain the related length, width, height and other structural data of each cultural relic.
Step S13: and (3 ds MAX modeling) modeling software is used for carrying out three-dimensional reconstruction on the measured cultural relics, and a three-dimensional model (virtual cultural relic model) is obtained and exported in an OBJ format.
In order to enable the three-dimensional model of the cultural relic to be normally displayed in the system, the cultural relic model in the OBJ format is converted into GLTF format by using an OBJ2gltf format conversion tool, so that various AR interactions between the user and the three-dimensional model of the cultural relic in the whole experience process can be ensured, and the interaction experience is ensured.
And storing the cultural relic image and the cultural relic three-dimensional model which are used as AR identification in a messenger cloud server for unified management, so that the related images and models can be conveniently called by various subsequent AR functions. The three-dimensional model display effect of part of the agricultural historical relics is shown in fig. 2. The method for creating the cultural relic digital resource library and uploading the cultural relic digital resource library to the cloud server comprises the following steps:
Step S14: and converting the three-dimensional model in the OBJ format into GLTF format by using a format conversion tool, and creating the cultural relic digital resource library.
Step S15: and storing the cultural relic images and the virtual cultural relic models which are used as AR identification in the cultural relic digital resource library in a cloud server for unified management.
In order to facilitate unified data management, the personal information of the user, including information such as a nickname, an id and the like of the user, is stored in a Tencent cloud MySQL database and is used for storing various data of the user, so that interaction operation between subsequent users is facilitated.
Step S2: after the user side obtains the cultural relic photos in the museum, a virtual cultural relic model corresponding to the cultural relic photos is called; real scenes are acquired in real time through the user side, the real scene plane is identified, the virtual cultural relic model is placed on the real scene plane, and AR placement is conducted.
In this step, a virtual cultural relic model corresponding to a cultural relic photo is called, including AR image recognition, specifically as shown in fig. 3, including the following steps:
step S21: an AR image recognition component is created, an image recognition mode is set according to a WeChat applet xr-frame framework, and basic settings such as scene lighting, camera following and the like are set.
Step S22: and taking the cultural relic pictures in the cultural relic digital resource library as image sources for AR identification.
The cultural relic picture selection information of the acquired page is transmitted to an AR image recognition component; the AR image recognition component determines the feature pictures and modules that need to be recognized.
Step S23: after the mobile phone is used for scanning the cultural relic pictures, a corresponding three-dimensional model (virtual cultural relic model) of the cultural relic appears, and the three-dimensional model is used for carrying out zooming and rotation control on the cultural relic.
Namely, scanning cultural relic pictures in real life, identifying and tracking a three-dimensional model of the cultural relic, performing operations such as rotation, amplification or reduction on the three-dimensional model, judging whether to select to replace cultural relic picture selection information, and re-acquiring if yes, otherwise, completing AR identification.
The method is used for carrying out scaling and rotation control on the cultural relics and comprises the following steps of:
Whether a single-finger operation or a double-finger operation is determined by event.
If the single-finger operation is determined, rotating the model during the single-finger operation, wherein the specific operation comprises the following steps:
The current canvas information width and height are acquired, and the rotation radius R is determined, wherein the solving formula is as follows:
Acquiring a clicking position startX of a last triggering event and a current clicking position x of a finger, setting a rotating speed rSpeed as a fixed value, and setting an angle change difference value theta in the x direction according to a rotating radius, wherein a solving formula is as follows:
And obtaining an angle change difference value in the y direction in the same way, and determining a rotation result of the cultural relic three-dimensional model according to the angle change in the x and y directions.
If the operation is judged to be double-finger operation, the model is scaled during the double-finger operation, and the specific operation comprises the following steps:
the method comprises the steps of obtaining the position information of the current double fingers in the x direction and the y direction, calculating the difference value of the two fingers in the x direction and the y direction, calculating the distance dis between the double fingers, and solving the following formula:
The distance startDis between the two fingers when the last scaling event occurs is obtained, the scaling ratio S is calculated, and the solving formula is as follows:
S＝(dis-startDis)×p+q
Where p and q are scaling parameters for converting the distance difference into a scaling.
Model values in the x, y and z directions of the three-dimensional model of the cultural relic (virtual cultural relic model) are multiplied by scaling ratios, and scaling is completed.
The AR image recognition effect and the effect of rotating and scaling the virtual cultural relic model are shown in fig. 4 (a), (b) and (c).
The invention designs an AR road sign recognition function, and particularly is shown in fig. 5. The user uses the rear camera of the mobile phone to scan the specific road sign with the AR mark in the Chinese agricultural history museum, after the scanning is successful, the AR indication information can be displayed, the user can select an ornamental route according to the own requirement, and the AR road sign recognition effect is shown in figure 6.
The method comprises the steps of calling a virtual cultural relic model corresponding to a cultural relic photo in the step, wherein the virtual cultural relic model comprises AR (augmented reality) landmark recognition, and specifically comprises the following steps of:
Step S24: an AR road sign recognition component is created, an OSD recognition mode is set according to a WeChat applet xr-frame framework, and scene lights and camera follow settings are set.
Step S25: and taking the real photo of the road sign in the museum as a feature recognition source, and adding the child node position information.
Step S26: and (3) scanning a specific road sign of the museum in real life to conduct AR identification, and displaying AR indication information.
After a specific road sign in the museum is scanned, judging whether the specific road sign is consistent with an identification source in an AR road sign identification component, if so, calculating child node position information, displaying road sign information, and if not, failing to identify.
In this step, the virtual cultural relic model is placed on the identified real scene plane, and AR placement is performed, as shown in fig. 7. The method comprises the following steps:
When a user scans a real life scene by using a camera arranged behind the mobile phone, a WebRTC is used for obtaining a real scene video stream of the museum, and the video stream of the camera of the mobile phone is pushed to a messenger cloud server.
Performing key frame extraction on an acquired video (processing a real scene by using a plane detection algorithm), defining the space surrounding all key frames as Fi, adopting a 2D CNN to extract image features in the key frames, back projecting the image features into (local feature integral) Fi to obtain voxel features and three-dimensional feature bodies, using average pooling to aggregate the features of the same voxels from different key frames, using 3D sparse convolution to process the three-dimensional feature bodies, (judging whether the distance between a voxel and a surface is smaller than lambda, if yes, the voxel belongs to a plane (wherein lambda is larger than lambda and is not a plane), judging whether the distance between the voxel and the surface is smaller than theta, if yes, the voxel does not belong to the plane, otherwise, constructing a three-dimensional sparse feature body), and determining the occupation condition of the voxels belonging to the plane.
And extracting normal parameters and plane offset (estimation) belonging to the plane voxels according to the three-dimensional sparse feature, calculating the displacement (estimation) of the voxel center and the centroid of the plane, and moving the voxel center to the plane centroid.
And determining final plane parameters by using a Mean-Shift clustering algorithm, placing a virtual model of the cultural relics on the detected plane (the plane in Fi is detected, plane data information is returned, and the virtual cultural relics are placed on the center position of the plane), and carrying out AR placement to achieve the effect shown in figure 8.
The gesture recognition function is designed in the invention, and particularly shown in fig. 9. The user uses the rear camera of the mobile phone to scan the single-hand gestures according to the indication information in the screen, the hand joint point information of the user is captured through the camera, the hand gesture information is determined, the corresponding interaction information is displayed, the interestingness of the user in viewing the cultural relics is improved, and the gesture recognition effect is shown in fig. 10. The current customized basic gestures mainly comprise six interactive gestures such as a comparison gesture, a praise gesture, an ok gesture, a first interaction gesture, a fifth interaction gesture and a sixth interaction gesture, and gesture types can be added according to requirements. The method is not limited by a user side system, downloading is not needed, any user can start to use by scanning the two-dimensional code, and convenience of user operation is improved.
Step S3: and performing various gesture interactions through the hands of the user and the virtual cultural relic model placed by the AR.
Step S4: and acquiring the hand key point information of the user by adopting the user terminal, identifying the hand gesture information of the user, and displaying the interaction information of the corresponding gesture and the virtual cultural relic model.
The method specifically comprises the following steps:
step S41: a gesture recognition component is created that sets a gesture recognition mode according to the WeChat applet xr-frame framework.
Step S42: and capturing hand joint point information of a user through a shooting device, and determining hand gesture information.
Step S43: and customizing basic gesture information, and displaying corresponding interaction information according to the identified hand gesture information.
The invention designs a multi-user message interaction function, which is shown in fig. 11. The user can select the message board in the process of browsing the museum, and can check the mind expressed by other users in the browsing process before, and meanwhile, the user can input own message content after authorizing personal information to the system, and the new message content is synchronously updated in the database and displayed in the message board, so that all users in the museum can communicate conveniently, and the effect is shown in figure 12.
Further comprising step S5: and connecting the user side with the cultural relic digital resource library in the cloud server to perform interaction between multiple users and the cultural relics.
The method specifically comprises the following steps:
Step S51: and storing the personal information authorized by the user in a MySQL database of the cloud server.
Step S52: and selecting a message interaction module, establishing connection with the MySQL database, watching the message content of other users, inputting the message content of the users, and synchronously updating the new message content in the database.
According to the above method and statement, the present invention further includes a museum relic AR interactive system, comprising: the device comprises a data management module, an AR function module, a gesture recognition module and a multi-user interaction module.
The data management module is used for collecting various cultural relics image information in the museum and carrying out three-dimensional reconstruction according to the various cultural relics image information to obtain a virtual cultural relics model; the AR function module is used for calling a virtual cultural relic model corresponding to the cultural relic photo after the user side obtains the Chinese cultural relic photo of the museum; real scenes are acquired in real time through a user side, and a real scene plane is identified; placing the virtual cultural relic model on the identified real scene plane, and performing AR placement; the gesture recognition module is used for acquiring the hand key point information of the user, recognizing the hand gesture information of the user and displaying the interaction information of the corresponding gesture; the multi-user interaction module is used for connecting the user side with the cultural relic digital resource library in the cloud server to perform interaction between multiple users and the cultural relics. The data management module comprises cultural relic image information and a three-dimensional model of the Chinese agricultural history museum.
According to the invention, the Chinese agricultural historical relic digital resource library is created, the purpose that in a Chinese agricultural historical museum, the details of the short-distance ornamental relics are watched by using an AR image recognition mode, the browsing route can be customized by using an AR road sign function, diversified interaction is carried out with the relics by using an AR placement function and a gesture recognition function, the distance between a user and the relics is shortened, and the unique charm of the relics is felt.
The invention also provides an electronic device, which comprises a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, causes the processor to execute the steps of the museum cultural relics AR interaction method.
The invention also provides a storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of a museum relic AR interaction method.
The present invention has been described in further detail with reference to specific preferred embodiments, and it should be understood by those skilled in the art that the present invention may be embodied with several simple deductions or substitutions without departing from the spirit of the invention.
Claims (10)
1. The museum cultural relic AR interaction method is characterized by comprising the following steps of:
acquiring various cultural relic image information in a museum, and carrying out three-dimensional reconstruction according to various cultural relic image information to obtain a virtual cultural relic model;
after a user side obtains a cultural relic photo in a museum, a virtual cultural relic model corresponding to the cultural relic photo is called; real scenes are acquired in real time through a user side, a real scene plane is identified, and the virtual cultural relic model is placed on the real scene plane to be placed by AR;
Performing various gesture interactions through the hands of the user and the virtual cultural relic model placed by the AR;
The user side obtains the hand key point information of the user, identifies the hand gesture information of the user, and displays interaction information of the corresponding gesture and the virtual cultural relic model through the user side.
2. The method for AR interaction of museum cultural relics according to claim 1, wherein the steps of collecting image information of various cultural relics in the museum, and performing three-dimensional reconstruction according to the image information of various cultural relics to obtain a virtual cultural relics model comprise the following steps:
Acquiring a plurality of cultural relics from a museum exhibition hall, and shooting the cultural relics in different dimensions by using shooting equipment to acquire a profile of each cultural relic;
measuring cultural relics in the showcase by using a ranging APP to obtain related length, width and height structural data of each cultural relic;
inputting the profile of each cultural relic and the measured structural data into modeling software, carrying out three-dimensional reconstruction on the cultural relic, obtaining a virtual cultural relic model and exporting the virtual cultural relic model in an OBJ format;
Converting the virtual cultural relic models in the OBJ format into GLTF format by using a format conversion tool, and creating a cultural relic digital resource library;
And storing the cultural relic images and the virtual cultural relic models which are used as AR identification in the cultural relic digital resource library in a cloud server.
3. The method for AR interaction of museum cultural relics according to claim 1, wherein said retrieving a virtual cultural relic model corresponding to said cultural relic photograph includes AR image recognition and AR landmark recognition, specifically comprising the steps of:
the AR image recognition includes the steps of:
an AR image recognition component is established, an image recognition mode is set according to a WeChat applet xr-frame framework, and scene light and camera following basic settings are set;
taking the cultural relic pictures in the cultural relic digital resource library as image sources for AR identification;
after a user side is used for scanning the cultural relic picture, a virtual cultural relic model corresponding to the cultural relic appears, and the user side is used for carrying out scaling and rotation control on the virtual cultural relic model;
the AR landmark identification comprises the following steps:
An AR road sign recognition component is established, an OSD recognition mode is set according to a WeChat applet xr-frame framework, and scene light and shooting equipment follow-up settings are set;
Taking a real photo of a road sign in the museum as a feature recognition source, and simultaneously adding child node position information;
and scanning a specific road sign of the museum for AR identification, and displaying AR indication information.
4. The method for AR interaction of museum relics as in claim 3, wherein said scaling and rotation control of the virtual relic model by the user side comprises the steps of:
Whether the operation is a single-finger operation or a double-finger operation is judged by event.
If the single-finger operation is judged, the virtual cultural relic model is rotated during the single-finger operation, and the specific operation comprises the following steps:
The current canvas information width and height are acquired, and the rotation radius R is determined, wherein the solving formula is as follows:
Acquiring a clicking position startX of a last triggering event and a current clicking position x of a finger, setting a rotating speed rSpeed as a fixed value, and setting an angle change difference value theta in the x direction according to a rotating radius, wherein a solving formula is as follows:
Acquiring an angle change difference value in the y direction in the same way, and determining a rotation result of the virtual cultural relic model according to the angle change in the x and y directions;
if the operation is judged to be double-finger operation, the virtual cultural relic model is scaled during the double-finger operation, and the specific operation comprises the following steps:
the method comprises the steps of obtaining the position information of the current double fingers in the x direction and the y direction, calculating the difference value of the two fingers in the x direction and the y direction, calculating the distance dis between the double fingers, and solving the following formula:
The distance startDis between the two fingers when the last scaling event occurs is obtained, the scaling ratio S is calculated, and the solving formula is as follows:
S＝(dis-startDis)×p+q
wherein p and q are scaling parameters for converting the distance difference into a scaling;
and multiplying the model values of the virtual cultural relic model in the x, y and z directions by the scaling ratio to finish scaling.
5. The museum cultural relic AR interaction method according to claim 1, wherein the virtual cultural relic model is placed on the real scene plane for AR placement, comprising the following steps:
Using WebRTC to acquire a real scene video stream of the museum, and using FFmpeg to extract key frames of the acquired video;
defining the space surrounding all the key frames as Fi, and extracting image features in the key frames by adopting 2D CNN;
back-projecting the image features into Fi to obtain voxel features and three-dimensional feature bodies;
Aggregating features from the same voxel in different keyframes using averaging pooling;
3D sparse convolution is used for processing the three-dimensional feature, constructing a three-dimensional sparse feature, and determining the occupation condition of voxels belonging to a plane;
Extracting normal parameters and plane offset belonging to the plane voxels according to the three-dimensional sparse feature;
Calculating the displacement between the voxel center and the centroid of the plane, and moving the voxel center to the plane centroid;
Determining final plane parameters after the voxel center moves to the plane centroid by using a Mean-Shift clustering algorithm;
and placing the virtual cultural relic model on the detected plane, and carrying out AR placement.
6. The method for AR interaction of museum cultural relics according to claim 1, wherein the steps of obtaining user hand key point information, identifying user hand gesture information, and displaying interaction information of corresponding gestures and the virtual cultural relics model through the user side are as follows:
Creating a gesture recognition component, and setting a gesture recognition mode according to a WeChat applet xr-frame framework;
capturing hand joint point information of a user through shooting equipment, and determining hand gesture information;
And customizing basic gesture information, and displaying interaction information corresponding to the virtual cultural relic model according to the identified hand gesture information.
7. The method for AR interaction of museum relics according to claim 2, further comprising the step of connecting a user side with a relic digital resource library in a cloud server to perform interaction between multiple users and relics, comprising the steps of:
storing personal information authorized by a user in a MySQL database of a cloud server;
And selecting a message interaction module, establishing connection with the MySQL database, watching the message content of other users, inputting the message content of the users, and synchronously updating the new message content in the database.
8. A museum cultural relic AR interactive system, comprising:
the data management module is used for collecting various cultural relics image information in the museum and carrying out three-dimensional reconstruction according to the various cultural relics image information to obtain a virtual cultural relics model;
the AR function module is used for calling a virtual cultural relic model corresponding to the cultural relic photo after the user side obtains the cultural relic photo in the museum; real scenes are acquired in real time through a user side, a real scene plane is identified, and the virtual cultural relic model is placed on the real scene plane to be placed by AR;
the interaction module is used for carrying out various gesture interactions through the hands of the user and the virtual cultural relic model placed by the AR;
The gesture recognition module is used for acquiring the key point information of the hand of the user by the user end, recognizing the gesture information of the hand of the user and displaying the interaction information of the corresponding gesture and the virtual cultural relic model by the user end.
9. An electronic device comprising a memory and a processor, wherein the memory stores a computer program which, when executed by the processor, causes the processor to perform the steps of a museum relic AR interaction method as claimed in any one of claims 1 to 7.
10. A storage medium having stored thereon a computer program, which when executed by a processor, implements the steps of a museum relic AR interaction method as claimed in any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410113363.XA CN117930978A (en) | 2024-01-26 | 2024-01-26 | AR interaction method, system, equipment and storage medium for museum cultural relics |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410113363.XA CN117930978A (en) | 2024-01-26 | 2024-01-26 | AR interaction method, system, equipment and storage medium for museum cultural relics |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117930978A true CN117930978A (en) | 2024-04-26 |
Family
ID=90753600
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410113363.XA Pending CN117930978A (en) | 2024-01-26 | 2024-01-26 | AR interaction method, system, equipment and storage medium for museum cultural relics |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117930978A (en) |
-
2024
- 2024-01-26 CN CN202410113363.XA patent/CN117930978A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110147231B (en) | Combined special effect generation method and device and storage medium | |
| CN110276840B (en) | Multi-virtual-role control method, device, equipment and storage medium | |
| US9424255B2 (en) | Server-assisted object recognition and tracking for mobile devices | |
| CN103270537B (en) | Image processing equipment, image processing method and program | |
| US8391615B2 (en) | Image recognition algorithm, method of identifying a target image using same, and method of selecting data for transmission to a portable electronic device | |
| US20140248950A1 (en) | System and method of interaction for mobile devices | |
| CN106816077A (en) | Interactive sandbox methods of exhibiting based on Quick Response Code and augmented reality | |
| CN104508680B (en) | Improved video signal is tracked | |
| CN108114471B (en) | AR service processing method and device, server and mobile terminal | |
| US20240078703A1 (en) | Personalized scene image processing method, apparatus and storage medium | |
| WO2022088819A1 (en) | Video processing method, video processing apparatus and storage medium | |
| CN104081307A (en) | Image processing apparatus, image processing method, and program | |
| CN105427369A (en) | Mobile terminal and method for generating three-dimensional image of mobile terminal | |
| WO2022048373A1 (en) | Image processing method, mobile terminal, and storage medium | |
| CN108134945B (en) | AR service processing method, AR service processing device and terminal | |
| CN114387445A (en) | Object key point identification method and device, electronic equipment and storage medium | |
| CN113766296A (en) | Live broadcast picture display method and device | |
| CN116261850A (en) | Bone tracking for real-time virtual effects | |
| CN115147554A (en) | Three-dimensional scene construction method, device, equipment and storage medium | |
| KR102464271B1 (en) | Pose acquisition method, apparatus, electronic device, storage medium and program | |
| CN112070901A (en) | AR scene construction method and device for garden, storage medium and terminal | |
| WO2023006011A1 (en) | Wallpaper generation method and apparatus and electronic device | |
| CN117930978A (en) | AR interaction method, system, equipment and storage medium for museum cultural relics | |
| WO2022016803A1 (en) | Visual positioning method and apparatus, electronic device, and computer readable storage medium | |
| JP2017168132A (en) | Virtual object display system, display system program, and display method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination |