CN111770450B - Workshop production monitoring server, mobile terminal and application - Google Patents
Workshop production monitoring server, mobile terminal and application Download PDFInfo
- Publication number
- CN111770450B CN111770450B CN202010442353.2A CN202010442353A CN111770450B CN 111770450 B CN111770450 B CN 111770450B CN 202010442353 A CN202010442353 A CN 202010442353A CN 111770450 B CN111770450 B CN 111770450B
- Authority
- CN
- China
- Prior art keywords
- information
- workshop
- positioning
- real
- module
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/02—Services making use of location information
- H04W4/029—Location-based management or tracking services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W16/00—Network planning, e.g. coverage or traffic planning tools; Network deployment, e.g. resource partitioning or cells structures
- H04W16/18—Network planning tools
- H04W16/20—Network planning tools for indoor coverage or short range network deployment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W16/00—Network planning, e.g. coverage or traffic planning tools; Network deployment, e.g. resource partitioning or cells structures
- H04W16/22—Traffic simulation tools or models
- H04W16/225—Traffic simulation tools or models for indoor or short range network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
- H04W4/30—Services specially adapted for particular environments, situations or purposes
- H04W4/33—Services specially adapted for particular environments, situations or purposes for indoor environments, e.g. buildings
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04W—WIRELESS COMMUNICATION NETWORKS
- H04W64/00—Locating users or terminals or network equipment for network management purposes, e.g. mobility management
Abstract
The invention relates to a workshop production monitoring server, a mobile terminal and application, wherein the workshop production monitoring server comprises: the indoor map building module is used for building an indoor three-dimensional map in a hierarchical mode based on the positioning coordinates of interest points in the workshop; the characteristic information base module is used for storing characteristic information of positioning coordinates of interest points in a workshop; the interface module is used for realizing data interaction with the mobile terminal, the physical workshop and an external system; the virtual visual information generation module is used for constructing a corresponding virtual visual model according to external business logic information and realizing the superposition of visual information of a target object in workshop production monitoring; and the AR technology module is used for matching the real-time positioning information of the moving object, acquiring pose estimation information of the moving object and pushing a corresponding virtual visualization model according to the pose estimation information. Compared with the prior art, the invention has the advantages of comprehensively, accurately pushing the workshop information in real time, improving the workshop production efficiency and the like.
Description
Technical Field
The invention relates to the technical field of digital workshop production monitoring, in particular to a workshop production monitoring server, a mobile terminal and application.
Background
With the rapid development of the internet of things technology and the digital manufacturing technology, the modern manufacturing industry also makes obvious progress, and the production workshops of more and more modeling enterprises are changed to digital workshops so as to achieve the purposes of cost reduction, quality improvement, efficiency improvement and rapid response to the market. The digital workshop is a new generation manufacturing system which is based on hardware facilities such as production equipment, production facilities and the like, and performs planning, management, simulation, optimization and other works on production resources and production processes in a computer virtual environment through means of digitization, networking, intellectualization and the like on the basis of optimized management of links such as process design, production organization, process control and the like. Due to the fact that the phenomenon of order insertion in workshop production generally exists, logistics emergencies occur frequently and are difficult to effectively deal with. In the face of the emergencies of equipment failure, line side warehouse material backlog, warehouse material shortage and the like in the production and logistics operation process of a workshop, real-time visual guidance information is lacked, and production scheduling personnel cannot accurately adjust the information. Meanwhile, visual human-computer interaction equipment is lacked, the traditional application program based on the PC can obtain a retrieval result only by inputting retrieval information, effective information cannot be obtained in time, and production efficiency is influenced.
Patent application CN108415386A discloses a method for superimposing information onto an existing scene using an augmented reality system, the augmented reality system of an intelligent workshop being used comprising: an augmented reality system server in communication with the field data server; the portable AR viewfinder is connected with the augmented reality system server through an augmented reality system service network; the networked AR interactive terminal establishes a pairing relationship with the unique portable AR viewfinder through the identity of the networked equipment to form complete system data input and AR image output. The reference method for realizing AR is provided, but the flexibility and the universality in the actual application process are not enough by adopting the method of real scene shooting and networking AR interactive terminal pairing.
At present, a practical, effective and reliable method is not available in the aspect of workshop production monitoring.
Disclosure of Invention
The invention aims to overcome the defects in the prior art and provides a workshop production monitoring server, a mobile terminal and application, which can comprehensively, timely and accurately push workshop information and improve workshop production efficiency.
The purpose of the invention can be realized by the following technical scheme:
a shop production monitoring server comprising:
the indoor map building module is used for building an indoor three-dimensional map in a hierarchical mode based on the positioning coordinates of interest points in the workshop;
the characteristic information base module is used for storing characteristic information of positioning coordinates of interest points in a workshop;
the interface module is used for realizing data interaction with the mobile terminal, the physical workshop and an external system;
the virtual visual information generation module is used for constructing a corresponding virtual visual model according to the external business logic information, the virtual visual model is fused with equipment production information and the characteristic information, and the superposition of the visual information of the target object in the workshop production monitoring is realized;
and the AR technology module is used for matching the real-time positioning information of the moving object, acquiring pose estimation information of the moving object and pushing a corresponding virtual visualization model according to the pose estimation information.
Further, the positioning coordinates of the interest points in the workshop are obtained based on the UWB tags and the UWB base stations.
Further, the construction of the indoor three-dimensional map specifically comprises:
obtaining the positioning coordinates of interest points in each workshop;
drawing indoor maps of different levels, binding the names of the interest points in each workshop and the corresponding positioning coordinates, and calculating the distance between the interest points in adjacent workshops;
setting a map icon to form the indoor three-dimensional map;
the indoor three-dimensional map contains all destinations allocated by the logistics in the workshop.
Further, the pose estimation is obtained by:
the method comprises the steps of carrying out feature extraction and normalization processing on real-time positioning information of a moving object to obtain a corresponding feature parameter vector, carrying out feature matching on the real-time positioning information of the moving object and positioning coordinates of interest points in a workshop in an indoor three-dimensional map based on the feature parameter vector, calculating a rotation matrix after matching is successful, and realizing pose estimation of the moving object.
Preferably, the feature matching is implemented using a nearest neighbor decision rule with a reject decision.
The invention also provides a workshop production monitoring mobile terminal which is arranged corresponding to the mobile object and comprises the following components:
the azimuth tracking sensing module is used for tracking, acquiring and sending the pose information of the mobile object in real time, namely the pose information of the camera of the mobile terminal;
the position tracking sensing module is used for acquiring and sending real-time positioning coordinates of the moving object;
the AR fusion display module is used for receiving the virtual visualization model and the pose estimation of the AR fusion display module, carrying out perspective conversion based on the pose estimation, and fusing and displaying the virtual visualization model and the real environment;
and the human-computer interaction module is used for realizing multi-channel human-computer interaction.
Further, the orientation tracking sensing module comprises an acceleration sensor and an electronic compass.
Further, the location tracking awareness module comprises a UWB tag.
Further, the information displayed in a fusion mode comprises basic equipment information, operation information, fault early warning information, equipment preventive maintenance information and the production progress condition of the production work order on the equipment.
Further, the human-computer interaction module comprises a touch display screen, a microphone and a loudspeaker.
Further, the virtual visualization model carries push information corresponding to pose estimation, which can be presented by voice, vibration or image.
Furthermore, a trigger is arranged in the mobile terminal, and an alarm mechanism in the production process is triggered according to a set threshold value.
Further, the fault early warning information comprises equipment fault early warning information and production progress late warning information.
The invention also provides a workshop production monitoring mobile application device based on indoor positioning and augmented reality technology, which comprises a workshop production monitoring server and a plurality of workshop production monitoring mobile terminals, wherein each workshop production monitoring mobile terminal is arranged corresponding to a mobile object, wherein,
the workshop production monitoring server comprises:
the indoor map building module is used for building an indoor three-dimensional map in a hierarchical mode based on the positioning coordinates of interest points in the workshop;
the characteristic information base module is used for storing characteristic information of positioning coordinates of interest points in a workshop;
the interface module is used for realizing data interaction with the mobile terminal, the physical workshop and an external system;
the virtual visual information generation module is used for constructing a corresponding virtual visual model according to the external business logic information, the virtual visual model is fused with equipment production information and the characteristic information, and the superposition of the visual information of the target object in the workshop production monitoring is realized;
the AR technology module is used for matching real-time positioning information of a moving object, acquiring pose estimation information of the moving object and pushing a corresponding virtual visualization model according to the pose estimation information;
the workshop production monitoring mobile terminal comprises:
the azimuth tracking sensing module is used for tracking, acquiring and sending pose information of the moving object in real time;
the position tracking sensing module is used for acquiring and sending real-time positioning coordinates of the moving object;
the AR fusion display module is used for receiving the virtual visualization model and the pose estimation of the AR fusion display module, carrying out perspective conversion based on the pose estimation, and fusing and displaying the virtual visualization model and the real environment;
and the human-computer interaction module is used for realizing multi-channel human-computer interaction.
Compared with the prior art, the invention has the following beneficial effects:
1. aiming at the characteristics of workshop production management, the invention applies the related technology to realize the workshop production monitoring mobile application device based on the mobile equipment, so as to support accurate information pushing of a digital workshop and the change of a production management flow, comprehensively and accurately push the workshop information to an operator in real time, and improve the workshop production efficiency.
2. According to the invention, the indoor positioning and augmented reality technology is utilized, and the 'active push type' workshop monitoring realized by combining the mobile equipment can trigger the corresponding production and logistics monitoring information push mechanism according to the real-time position of personnel, so that the visual management and guidance of the workshop product production flow are realized.
3. The invention adopts the ultra-wideband positioning technology, the positioning precision can reach within 20cm, the three-dimensional positioning can be realized to a certain degree, and the current indoor position information of the moving object can be sensed conveniently.
4. The invention adopts a positioning tracking registration method, integrates a plurality of sensing sensors, and each sensor takes the best of its own functions to make up for the weakness, thereby obtaining more reliable azimuth information of the mobile object.
5. The invention can effectively realize the monitoring of the production process of the workshop equipment through the effective integration of the production information.
6. The invention can expand the traditional man-machine interface by utilizing the interaction modes of voice, vibration and the like of the mobile equipment, and is convenient for field personnel to apply.
Drawings
FIG. 1 is a schematic diagram of a mobile application device for monitoring production in a workshop according to the present invention;
FIG. 2 is a schematic flow chart of the overall apparatus for applying the present invention;
FIG. 3 is a schematic view of a process for constructing virtual information according to the present invention;
FIG. 4 is a schematic diagram of a construction process of an indoor three-dimensional map;
FIG. 5 is a schematic view of a plant production monitoring system using the apparatus of the present invention;
FIG. 6 is a flow diagram of a nearest neighbor decision algorithm with a reject decision;
FIG. 7 is a schematic view of production monitoring of a mobile terminal according to the present invention;
FIG. 8 is a schematic diagram of a coordinate transformation process;
FIG. 9 is a UI diagram illustrating dynamic device operating parameters of the mobile terminal;
FIG. 10 is a diagram illustrating an AR display of a mobile terminal according to the present invention.
Detailed Description
The invention is described in detail below with reference to the figures and specific embodiments. The present embodiment is implemented on the premise of the technical solution of the present invention, and a detailed implementation manner and a specific operation process are given, but the scope of the present invention is not limited to the following embodiments.
Example 1
Referring to fig. 1, the embodiment provides a workshop production monitoring server (AR-LC server a), which includes an indoor map building module a1, a characteristic information base module a2, an interface module A3, a virtual visual information generating module a4, and an AR technology module, where the indoor map building module a1 is used to build an indoor three-dimensional map in a hierarchical level based on location coordinates of points of interest in a workshop; the characteristic information base module A2 is used for storing characteristic information of positioning coordinates of interest points in a workshop and is used for reference comparison of real-time positioning information of the mobile object; the interface module A3 is used for realizing data interaction with the mobile terminal, the physical workshop and the external system; the virtual visual information generation module A4 is used for constructing a corresponding virtual visual model according to the external business logic information, the virtual visual model is fused with the equipment production information and the characteristic information, and visual realization and 3D rendering are carried out on the virtual visual model, so that the superposition of the visual information of the target object in the workshop production monitoring is realized; the AR technical module is used for matching real-time positioning information of the moving object, acquiring pose estimation information of the moving object and pushing a corresponding virtual visual model according to the pose estimation information, wherein the real-time positioning information comprises real-time positioning coordinates and pose information of the moving object, and the pose information comprises an acceleration value and geomagnetic parameters.
The interest points in the workshop comprise a work center, equipment and a warehouse shelf, and positioning coordinates are obtained through the UWB indoor positioning technology. The indoor three-dimensional map is used for determining the position of the interest point and serving as a standard reference position for real-time positioning of the mobile terminal.
The construction of the indoor three-dimensional map specifically comprises the following steps: obtaining the positioning coordinates of interest points in each workshop; drawing indoor maps of different levels, binding the names of the interest points in each workshop and the corresponding positioning coordinates, and calculating the distance between the interest points in adjacent workshops; setting a map icon to form the indoor three-dimensional map; the indoor three-dimensional map contains all destinations allocated by the logistics in the workshop.
The interface module comprises an external system interface, a perception characteristic information interface and a workshop production information interface. The external system interface is mainly used for calling an optimization model of an external system; the perception characteristic information interface is mainly used for the server to acquire real-time perception information of a built-in sensor of the mobile terminal in real time; the workshop production information interface is mainly used for importing production information.
The AR technical module has the functions of recognizing and extracting real-time sensing characteristic information, matching the extracted characteristic information with positioning characteristic information in a characteristic library and estimating the pose of the mobile terminal. The AR technology module includes a feature recognition unit a5, a feature matching unit a6, and a pose estimation unit a7, wherein,
feature identification module a 5: preprocessing a real-time positioning coordinate (mainly referring to camera positioning coordinate information of a mobile terminal) of a mobile object (a person, a material or equipment), extracting a corresponding characteristic region, and carrying out standardization processing to obtain a corresponding characteristic parameter vector;
feature matching module a 6: matching the characteristic parameter vector with the original characteristic information in the characteristic information base by using a matching algorithm, and obtaining a matching result;
pose estimation module a 7: and judging the position and the direction of the camera in the real environment in real time according to the pose information (the acceleration value and the geomagnetic parameters) of the moving object, establishing the relation between the pose information and the world coordinate, calculating a rotation matrix, and realizing the pose estimation of the moving object.
Example 2
Referring to fig. 2, the embodiment provides a mobile terminal B for monitoring workshop production, which is arranged corresponding to a mobile object, and includes an orientation tracking sensing module B1, a position tracking sensing module B2, an AR fusion display module B3, and a human-computer interaction module B4, where the orientation tracking sensing module B1 is configured to track and acquire pose information of the mobile object in real time, and is used as a basis for tracking the position and posture of the head of a user in real time; the position tracking sensing module B2 is used for obtaining real-time positioning coordinates of the moving object; the AR fusion display module B3 is used for receiving the virtual visualization model and the pose estimation of the virtual visualization model, carrying out perspective conversion based on the pose estimation, and carrying out fusion display on the virtual visualization model and the real environment, wherein the virtual visualization model is provided with push information corresponding to the pose estimation, and the push information can be presented through voice, vibration or images; the human-computer interaction module B4 is used for realizing multi-channel human-computer interaction.
The orientation tracking sensing module comprises a built-in acceleration sensor and an electronic compass. The electronic compass has the capability of azimuth tracking and measures the angle change of three rotational degrees of freedom; the acceleration sensor measures the displacement of three translational degrees of freedom.
The mobile terminal B is provided with a camera. The position tracking sensing module comprises a UWB (ultra wide band) tag, real-time positioning of a moving object is realized through an indoor positioning system, and the obtained real-time positioning coordinate mainly refers to a three-dimensional positioning coordinate of a mobile terminal camera indoors.
In the AR fusion display module B3, the visual information displayed in fusion includes basic device information, operation information, failure warning information, preventive maintenance information of the device, and a production progress situation of the production work order on the device, and the failure warning information includes device failure warning information and production progress delay warning information.
The human-computer interaction module B4 comprises a touch display screen, a microphone and a loudspeaker. The multi-channel human-computer interaction comprises vibration interaction, gesture interaction, voice interaction and the like, so that a user can be more naturally integrated into a scene, and the user experience is enhanced.
The mobile terminal is internally provided with a trigger which triggers an alarm mechanism in the production process according to a set threshold value.
Example 3
As shown in fig. 1, the embodiment provides a workshop production monitoring mobile application device based on indoor positioning and augmented reality technology, including a workshop production monitoring server and a plurality of workshop production monitoring mobile terminals, where each workshop production monitoring mobile terminal is set corresponding to a mobile object, a specific structure of the workshop production monitoring server is the same as that in embodiment 1, and a specific structure of the workshop production monitoring mobile terminal is the same as that in embodiment 2. The workshop production monitoring mobile terminal is an AR mobile terminal.
(1) General application flow
As shown in fig. 2, the overall application flow of the application device of the present embodiment includes two major parts, namely "virtual information construction" and "system application", and specifically,
s01: the virtual information construction comprises indoor map construction, feature library construction and virtual visualization model construction;
s02: the mobile terminal performs indoor positioning and realizes production monitoring based on augmented reality.
(2) Virtual information construction process
And virtual information construction, namely constructing an indoor positioning map, a characteristic information comparison base and a virtual visual model before the mobile terminal produces monitoring application.
And the indoor positioning map provides positioning characteristic measurement information of interest points in the workshop and provides standard direction reference for a built-in sensor of the mobile terminal.
And the characteristic information base extracts the characteristics of the positioning coordinates of the interest points of the workshop and stores the characteristics as reference comparison of real-time positioning information of the mobile object.
And the virtual visualization model is used for visualizing information superposed on the target object in the workshop production monitoring.
The process of virtual information construction is shown in fig. 3, and the detailed steps are as follows:
s101: the positioning coordinates of the interest points are measured in the workshop through a UWB positioning technology, an indoor three-dimensional positioning reference map formed by the positioning coordinates of the interest points in the workshop is realized, and the indoor three-dimensional positioning reference map is used as characteristic information for realizing augmented reality according to positioning information and is also used as a reference point of other production elements in the workshop.
As shown in fig. 4, the algorithm for constructing the indoor three-dimensional positioning reference map is as follows:
s111: one or more devices are arranged in the work center, the devices in the work center in the workshop are selected as positioning interest points, and fixed positioning interest points in the position range of each device are specified.
S112: and positioning the fixed equipment interest points by using the 3D positioning system in the workshop to obtain the coordinate positions of the equipment interest points.
S113: drawing indoor maps of different levels, binding the names of the interest points with the positioning coordinates thereof, naming uniformly, calculating the distance between adjacent interest points,
s114: setting the north-pointing direction of the map and a scale for reducing the map. The map should contain all destinations for the logistics within the plant.
S102: and establishing a characteristic information base of workshop production elements, and performing tracking registration according to the characteristic information by using the augmented reality technology. In practical applications, features are generally classified into visual feature information and non-visual feature information. The non-visual characteristic information is positioning characteristic information in a workshop positioning system. These feature information needs to be preprocessed before storage, to remove noise in the features, to enhance useful information, and to recover degraded information.
S103: and constructing a corresponding visual monitoring logic model in the AR-LC server A according to the optimization model logic imported through the external system interface, and logically programming the corresponding visual monitoring logic model by using an object-oriented programming language, such as C #, Java and the like.
S104: and performing visual script development on the visual monitoring logic model, such as displaying the statistical analysis in a form of column statistics, displaying the production progress in a form of progress bars, and developing corresponding operation animations to complete the guidance operation process.
S105: and binding the developed virtual visualization models with the feature information in the workshop feature information base in the AR-LC server, so that each virtual visualization model has corresponding feature information.
(3) Workshop production monitoring application process
The workshop production monitoring application process realizes the production monitoring function of the mobile terminal equipment on the production elements within a certain range through matching of indoor positioning characteristic information and pose evaluation of a camera.
As shown in fig. 5, the specific steps of monitoring the application flow include:
s201: when a mobile object with a mobile terminal is close to production equipment, the mobile terminal respectively senses an acceleration value and a geomagnetic parameter through a built-in acceleration sensor and an electronic compass, and when the mobile object is at a certain place in a workshop, the mobile object is positioned by utilizing an indoor 3D positioning technology, so that real-time positioning information of the mobile object is obtained. These information are uploaded to the AR-LC server through the perceptual information interface in interface module a 3.
S202: the interest point (three-dimensional) in the positioning information is detected through a feature detection algorithm, in the embodiment, the three-dimensional coordinate of the interest point is transformed through a K-L standard basis according to a minimum mean square error criterion to obtain dimension-reduced feature information (two-dimensional), and then normalization processing is performed (the two-dimensional coordinate values are all mapped to a [0,1] interval) to obtain a corresponding feature parameter vector.
S203: feature point matching refers to evaluating similarity between features, and is generally measured by using some cost function or distance function. The present embodiment matches the position coordinates of the moving object with the position coordinates of the point of interest in the map using a nearest neighbor decision rule with a rejection decision.
As shown in fig. 6, the nearest neighbor decision algorithm with a rejection decision specifically includes:
s231: dividing each working area of the workshop into a category omegaiI 1, 2, 3 … … C, the equipment in the work center being a sample p in this categoryk，k＝1，2，3……N。
S232: calculating real-time positioning coordinate p and category omega of mobile objectiWithin each sample xkAnd find the minimum distance gi(p) defining the distance as a real-time location coordinate and a category ωiThe distance between them.
gi(x)＝min||p-pk||k＝1，2，3……N (1)
S233: finding out positioning coordinate and category omegaiMinimum distance g betweenj(p)。
gj(p)＝mingi(p) (2)
S234: determining:
gj(p)≤l (3)
if the distance is less than a certain distance l, the positioning coordinate x is classified into the category, the matching is successful, and otherwise, the decision is rejected.
S204: if the positioning characteristic information is successfully matched and the direction tracking information of the mobile terminal sensor (the acceleration sensor and the electronic compass) is obtained, the acceleration sensor measures the displacement of three translational degrees of freedom, the electronic compass measures the angle change of three rotational degrees of freedom, and a rotation matrix R is calculated. And calculating an azimuth angle (azimuth), a pitch angle (pitch) and a rotation angle (roll) according to the values in the rotation matrix, and estimating the pose of the camera.
S205: if the matching of the positioning characteristic information is unsuccessful, the mobile terminal further approaches the production equipment, and the step S201 is carried out.
S206: and the AR-LC server A calls the real-time information of the production equipment through a workshop production information interface and is fused with the virtual visual model.
S207: and pushing a virtual visualization model bound with the positioning feature information in the feature library according to the posture estimation of the camera, as shown in fig. 7. The spatial localization (MX, MY, MZ) of the point of interest is obtained from the indoor map, and when the AR device camera is directed towards the target point of interest, the localization coordinates are converted into camera coordinates (X, Y, Z) by the corresponding rotation matrix. In this case, it is also necessary to convert the point of the camera coordinate, which is a coordinate system representing a point in a three-dimensional space (obtained by the camera pose described above), into a coordinate system represented in a screen coordinate system representing a point P (x, y) of a two-dimensional plane, so that the three-dimensional to two-dimensional conversion needs to be solved through perspective projection. The last step is to convert the image plane coordinate system into a pixel coordinate system, namely, to convert the point P in the conversion process into a point Z (u, v) in the digital image, so that the virtual visualization model is subjected to image fusion in the pixel coordinate system of the mobile terminal and the interest point in the real environment and is displayed through a display of the mobile terminal.
The coordinate transformation relationship is shown in fig. 8.
S208: a manufacturer carries the AR mobile terminal to enter a workshop production device within a certain range, the AR device camera faces the production device, and basic information, operation information, fault early warning information and device preventive maintenance information of the device can be displayed on the displayer. If the device fails, the AR mobile device will vibrate for early warning.
S209: meanwhile, the display of the AR mobile device can display the production progress condition of the production worksheet on the AR mobile device. If the progress of the production work order is lagged behind, the AR equipment can also carry out vibration early warning through a built-in vibration device.
Example 4
The embodiment applies the mobile workshop production monitoring application device based on the indoor positioning and augmented reality technology to a typical management and control flow of specific equipment in a workshop.
The workshop visual monitoring technology based on the indoor positioning technology and the AR is applied to the equipment management and control process, and the processes of equipment fault early warning, production work order management and control and the like of a work center can be achieved in a multi-channel man-machine interaction mode. When a production person approaches a certain work center, the AR equipment carried with the AR equipment can display the equipment which needs preventive maintenance in the work center, and when the production person approaches a certain piece of equipment and the camera of the AR equipment faces the equipment, the mobile AR equipment can display the basic information, the fault early warning information and the production work order progress information of the equipment.
The specific process for realizing the production monitoring of the mobile terminal by taking a certain work area W-DMC in a workshop and a certain device DMC80H in a work center as an example is as follows:
s10: and (3) indoor map construction, namely positioning interest points (a work center and equipment) in the workshop through a UWB indoor positioning technology, constructing indoor map construction by utilizing the positioning information of the work center and the equipment, and extracting characteristic information from the positioning information of the work center and the equipment in the workshop.
S20: the AR-LC system server program is built in C # language in the Unity3D engine and the preventive maintenance model is imported through the external system interface. And then constructing a visual logic program for monitoring equipment production in the server program, such as visual logic for preventive maintenance, visual logic for equipment failure alarm and visual logic for production work order progress.
S30: and calling the equipment production information through a workshop production information interface program (Web Service, JDBC and the like) in the equipment monitoring visual logic program to generate a data logic model for equipment production monitoring.
S40: the visualization implementation and 3D rendering of the production monitored data logical model in Unity3D generates a virtual visualization model as illustrated in fig. 9. The historical health statistical data of the equipment is represented by a fan, and corresponding to colors, blue represents the historical statistical data of equipment operation, red represents the historical statistical data of equipment failure, and gray represents the historical statistical data of equipment standby. And the color representation mode of the health state of the current equipment also corresponds to the color of the historical health of the equipment. The order completion status is represented by a progress bar as a percentage of its completion.
S50: and binding the rendered virtual visualization model with the positioning characteristic information of the equipment.
S60: when a producer wears AR equipment to be close to a working area W-DMC or the equipment DMC80H in a working center within a certain range, indoor positioning coordinate information of the mobile terminal is acquired by using a UWB base station and a UWB label, and a built-in sensor of the mobile terminal AR equipment senses an acceleration value and an electronic compass value of an equipment camera in real time. And the acceleration value, the electronic compass value and the indoor positioning coordinate information are called by the server program through the interface.
S70: and carrying out normalized processing on the mobile terminal positioning information uploaded to the server, and carrying out feature extraction through a feature transformation algorithm to obtain a parameter vector of the mobile terminal positioning information.
S80: and performing matching calculation on the feature parameter vector in the S80 and the feature parameter vector in the feature information base by using a nearest neighbor decision algorithm with a rejection decision. If the matching is successful, entering S90; otherwise, the AR device is moved again, relocation is performed and S70 is turned on.
S90: the AR-LC system server A calls data from an external auxiliary system, such as an MES system, through a corresponding production information calling interface, such as a calling interface program Web Service, JDBC, and the like, and fuses with the virtual visualization model.
S100: a sensor root arranged in the AR equipment acquires coordinate values in a standard sensor coordinate system, a coordinate vector acquired by the acceleration sensor is A, and a coordinate vector acquired by the electronic compass is B. And calculating a rotation matrix R by using the coordinate vector A and the vector B, and then calculating an azimuth angle (azimuth), a pitch angle (pitch) and a rotation angle (roll) of the sensor according to the rotation matrix R, so as to estimate the pose of the AR equipment camera.
S110: the spatial positioning (MX, MY, MZ) of the production device is obtained from the indoor map, and when the AR device camera is directed towards the production device (as the target point of interest), the positioning coordinates are converted into camera coordinates by the corresponding rotation matrix. In this case, it is also necessary to convert the point of the camera coordinate, which is a coordinate system representing a point in a three-dimensional space (obtained by the camera pose described above), into a coordinate system represented in a screen coordinate system representing a point P (x, y) of a two-dimensional plane, so that the three-dimensional to two-dimensional conversion needs to be solved through perspective projection. The last step is to convert from the image plane coordinate system to the pixel coordinate system, i.e. to convert the P-point in the conversion process to a point in the digital image. Meanwhile, the conversion process of the coordinates of the indoor production equipment into the coordinates of the mobile terminal pixels is also the conversion process of the virtual visual model into the coordinates of the mobile terminal pixels, so that the virtual guide information (the rendered virtual visual model in the step S50) superimposed on the production equipment can be displayed on the screen of the mobile terminal.
S120: fig. 10 shows production monitoring information superimposed on production equipment, which is displayed by the AR equipment, and includes equipment basic information, equipment historical health information, equipment failure warning information, and production order completion information. The left information in the figure is basic information of the equipment, and the right information in the figure is operation information and production order completion information of the equipment. If the equipment fails or the progress of the production work order exceeds a delay threshold, the AR equipment can give an alarm through the vibration device.
The foregoing detailed description of the preferred embodiments of the invention has been presented. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concepts. Therefore, the technical solutions that can be obtained by a person skilled in the art through logic analysis, reasoning or limited experiments based on the prior art according to the concept of the present invention should be within the protection scope determined by the present invention.
Claims (8)
1. A shop production monitoring server, comprising:
the indoor map building module is used for building an indoor three-dimensional map in a hierarchical mode based on the positioning coordinates of the interest points in the workshop, selecting equipment in a working center in the workshop as the positioning interest points, and appointing the fixed positioning interest points in the position range of each equipment;
the characteristic information base module is used for storing characteristic information of positioning coordinates of interest points in the workshop, and comprises visual characteristic information and non-visual characteristic information, wherein the non-visual characteristic information comprises positioning characteristic information;
the interface module is used for realizing data interaction with the mobile terminal, the physical workshop and an external system;
the virtual visual information generation module is used for constructing a corresponding virtual visual model according to the external business logic information, the virtual visual model is fused with equipment production information and the characteristic information, and the superposition of the visual information of the target object in the workshop production monitoring is realized;
the AR technical module is used for matching the real-time positioning information of the moving object, acquiring pose estimation information of the moving object and pushing a corresponding virtual visualization model according to the pose estimation;
the pose estimation is obtained by:
performing feature extraction and normalization processing on the real-time positioning information of the moving object to obtain a corresponding feature parameter vector, performing feature matching on the real-time positioning information of the moving object and positioning coordinates of interest points in a workshop in an indoor three-dimensional map based on the feature parameter vector, and calculating a rotation matrix after matching is successful to realize pose estimation on the moving object;
the feature matching is realized by adopting a nearest neighbor decision rule with a rejection decision, which specifically comprises the following steps:
s231: dividing each working area of the workshop into a category omegai1, 2, 3.. C, the equipment in the work center is a sample p in this categoryk，k＝1，2，3......N；
S232: calculating real-time positioning coordinate p and category omega of mobile objectiWithin each sample xkAnd find the minimum distance gi(p) defining the distance as a real-time location coordinate and a category ωiThe distance between:
gi(x)＝min||p-pk||k＝1，2，3……N；
s233: finding out positioning coordinate and category omegaiMinimum distance g betweenj(p)：
gj(p)＝mingi(p)；
S234: determining:
gj(p)≤l
if the distance is less than a certain distance l, the positioning coordinate x is classified into the category, the matching is successful, and otherwise, the decision is rejected.
2. The shop production monitoring server of claim 1, wherein the location coordinates of the point of interest within the shop are obtained based on a UWB tag and a UWB base station.
3. The workshop production monitoring server of claim 1, wherein the indoor three-dimensional map is specifically constructed by:
obtaining the positioning coordinates of interest points in each workshop;
drawing indoor maps of different levels, binding the names of the interest points in each workshop and the corresponding positioning coordinates, and calculating the distance between the interest points in adjacent workshops;
setting a map icon to form the indoor three-dimensional map;
the indoor three-dimensional map contains all destinations allocated by the logistics in the workshop.
4. The utility model provides a workshop production control mobile terminal which characterized in that, corresponds the setting with the mobile object, includes:
the azimuth tracking sensing module is used for tracking, acquiring and sending pose information of the moving object in real time;
the position tracking sensing module is used for acquiring and sending real-time positioning coordinates of the moving object;
the AR fusion display module is used for receiving the virtual visualization model and the pose estimation of the AR fusion display module, carrying out perspective conversion based on the pose estimation, and fusing and displaying the virtual visualization model and the real environment;
the human-computer interaction module is used for realizing multi-channel human-computer interaction;
the virtual visualization model is constructed according to external business logic information, the virtual visualization model is fused with equipment production information and characteristic information, the superposition of the visualization information of a target object in workshop production monitoring is realized, the characteristic information is the characteristic information of a positioning coordinate for storing interest points in a workshop and comprises the visualization characteristic information and non-visualization characteristic information, the non-visualization characteristic information comprises the positioning characteristic information, equipment in a working center in the workshop is selected as the positioning interest points, and the fixed positioning interest points in the position range of each equipment are designated;
the pose estimation is obtained by:
performing feature extraction and normalization processing on the real-time positioning information of the moving object to obtain a corresponding feature parameter vector, performing feature matching on the real-time positioning information of the moving object and positioning coordinates of interest points in a workshop in an indoor three-dimensional map based on the feature parameter vector, and calculating a rotation matrix after matching is successful to realize pose estimation on the moving object;
the feature matching is realized by adopting a nearest neighbor decision rule with a rejection decision, which specifically comprises the following steps:
s231: dividing each working area of the workshop into a category omegai1, 2, 3.. C, the equipment in the work center is a sample p in this categoryk，k＝1，2，3......N；
S232: calculating real-time positioning coordinate p and category omega of mobile objectiWithin each sample xkAnd find the minimum distance gi(p) defining the distance as a real-time location coordinate and a category ωiThe distance between:
gi(x)＝min||p-pk||k＝1，2，3……N；
s233: finding out positioning coordinate and category omegaiMinimum distance g betweenj(p)：
gj(p)＝mingi(p)；
S234: determining:
gj(p)≤l
if the distance is less than a certain distance l, the positioning coordinate x is classified into the category, the matching is successful, and otherwise, the decision is rejected.
5. The plant production monitoring mobile terminal of claim 4, wherein the orientation tracking sensing module comprises an acceleration sensor and an electronic compass, and the location tracking sensing module comprises a UWB tag.
6. The workshop production monitoring mobile terminal according to claim 4, wherein the fusion displayed information includes basic equipment information, operation information, fault early warning information, preventive equipment maintenance information and production progress of a production work order on the equipment.
7. The workshop production monitoring mobile terminal of claim 4, wherein a trigger is built in the mobile terminal, and an alarm mechanism of a production process is triggered according to a set threshold.
8. A workshop production monitoring mobile application device based on indoor positioning and augmented reality technology is characterized by comprising a workshop production monitoring server and a plurality of workshop production monitoring mobile terminals, wherein each workshop production monitoring mobile terminal is arranged corresponding to a mobile object,
the workshop production monitoring server comprises:
the indoor map building module is used for building an indoor three-dimensional map in a hierarchical mode based on the positioning coordinates of the interest points in the workshop, selecting equipment in a working center in the workshop as the positioning interest points, and appointing the fixed positioning interest points in the position range of each equipment;
the characteristic information base module is used for storing characteristic information of positioning coordinates of interest points in the workshop, and comprises visual characteristic information and non-visual characteristic information, wherein the non-visual characteristic information comprises positioning characteristic information;
the interface module is used for realizing data interaction with the mobile terminal, the physical workshop and an external system;
the virtual visual information generation module is used for constructing a corresponding virtual visual model according to the external business logic information, the virtual visual model is fused with equipment production information and the characteristic information, and the superposition of the visual information of the target object in the workshop production monitoring is realized;
the AR technology module is used for matching the real-time positioning information of the moving object, acquiring pose estimation information of the moving object and pushing a corresponding virtual visualization model according to the pose estimation information;
the workshop production monitoring mobile terminal comprises:
the azimuth tracking sensing module is used for tracking, acquiring and sending pose information of the moving object in real time;
the position tracking sensing module is used for acquiring and sending real-time positioning coordinates of the moving object;
the AR fusion display module is used for receiving the virtual visualization model and the pose estimation of the AR fusion display module, carrying out perspective conversion based on the pose estimation, and fusing and displaying the virtual visualization model and the real environment;
the human-computer interaction module is used for realizing multi-channel human-computer interaction;
the pose estimation is obtained by:
performing feature extraction and normalization processing on the real-time positioning information of the moving object to obtain a corresponding feature parameter vector, performing feature matching on the real-time positioning information of the moving object and positioning coordinates of interest points in a workshop in an indoor three-dimensional map based on the feature parameter vector, and calculating a rotation matrix after matching is successful to realize pose estimation on the moving object;
the feature matching is realized by adopting a nearest neighbor decision rule with a rejection decision, which specifically comprises the following steps:
s231: dividing each working area of the workshop into a category omegai1, 2, 3.. C, the equipment in the work center is a sample p in this categoryk，k＝1，2，3......N；
S232: calculating real-time positioning coordinate p and category omega of mobile objectiWithin each sample xkAnd find the minimum distance gi(p) defining the distance as a real-time location coordinate and a category ωiThe distance between:
gi(x)＝min||p-pk||k＝1，2，3……N；
s233: finding out positioning coordinate and category omegaiMinimum distance g betweenj(p)：
gj(p)＝mingi(p)；
S234: determining:
gj(p)≤l
if the distance is less than a certain distance l, the positioning coordinate x is classified into the category, the matching is successful, and otherwise, the decision is rejected.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010442353.2A CN111770450B (en) | 2020-05-22 | 2020-05-22 | Workshop production monitoring server, mobile terminal and application |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010442353.2A CN111770450B (en) | 2020-05-22 | 2020-05-22 | Workshop production monitoring server, mobile terminal and application |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111770450A CN111770450A (en) | 2020-10-13 |
| CN111770450B true CN111770450B (en) | 2022-02-18 |
Family
ID=72719657
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010442353.2A Active CN111770450B (en) | 2020-05-22 | 2020-05-22 | Workshop production monitoring server, mobile terminal and application |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111770450B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113050501A (en) * | 2021-03-24 | 2021-06-29 | 上海海事大学 | Workshop virtual monitoring system and service terminal |
| CN115561704B (en) * | 2022-09-30 | 2023-05-16 | 中国测绘科学研究院 | Positioning method and system for high-precision map enhanced closed space sparse UWB base station network |
| CN116740301B (en) * | 2023-08-11 | 2024-01-30 | 深圳麦格米特电气股份有限公司 | Three-dimensional virtual monitoring system and method and monitoring equipment |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107885336A (en) * | 2017-11-30 | 2018-04-06 | 成都飞机工业（集团）有限责任公司 | Map the virtual workshop model and information fusion display method of the true condition of production |
| CN109240237A (en) * | 2018-10-10 | 2019-01-18 | 中国恩菲工程技术有限公司 | Mineral Processing Enterprises Three-Dimensional Dynamic monitoring system |
| CN109947052A (en) * | 2017-12-20 | 2019-06-28 | 中国科学院沈阳自动化研究所 | A kind of virtual factory manufacturing execution system based on digitlization AR technology |
| CN110599603A (en) * | 2019-09-20 | 2019-12-20 | 上海大学 | Mechanical equipment visual interaction and equipment state monitoring system and method based on augmented reality |
| CN110825234A (en) * | 2019-11-11 | 2020-02-21 | 江南大学 | Projection type augmented reality tracking display method and system for industrial scene |
-
2020
- 2020-05-22 CN CN202010442353.2A patent/CN111770450B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107885336A (en) * | 2017-11-30 | 2018-04-06 | 成都飞机工业（集团）有限责任公司 | Map the virtual workshop model and information fusion display method of the true condition of production |
| CN109947052A (en) * | 2017-12-20 | 2019-06-28 | 中国科学院沈阳自动化研究所 | A kind of virtual factory manufacturing execution system based on digitlization AR technology |
| CN109240237A (en) * | 2018-10-10 | 2019-01-18 | 中国恩菲工程技术有限公司 | Mineral Processing Enterprises Three-Dimensional Dynamic monitoring system |
| CN110599603A (en) * | 2019-09-20 | 2019-12-20 | 上海大学 | Mechanical equipment visual interaction and equipment state monitoring system and method based on augmented reality |
| CN110825234A (en) * | 2019-11-11 | 2020-02-21 | 江南大学 | Projection type augmented reality tracking display method and system for industrial scene |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111770450A (en) | 2020-10-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111770450B (en) | Workshop production monitoring server, mobile terminal and application | |
| CN108764048B (en) | Face key point detection method and device | |
| CN109978409B (en) | Multi-engineering enterprise-level intelligent construction site block chain management method | |
| CN112950773A (en) | Data processing method and device based on building information model and processing server | |
| CN109918977B (en) | Method, device and equipment for determining idle parking space | |
| WO2023093217A1 (en) | Data labeling method and apparatus, and computer device, storage medium and program | |
| CN110049121B (en) | Data center inspection system based on augmented reality technology | |
| KR102371336B1 (en) | Smart factory monitoring system using 3 Dimention integrated monitoring and Augmented Reality and Virtual Reality | |
| CN112929602B (en) | Data monitoring method and device based on image processing and related equipment | |
| CN110579191A (en) | target object inspection method, device and equipment | |
| CN111192321A (en) | Three-dimensional positioning method and device for target object | |
| CN115578433B (en) | Image processing method, device, electronic equipment and storage medium | |
| CN113326964B (en) | Data-based composite multi-stage comprehensive pipe gallery risk management and control method | |
| CN115330734A (en) | Automatic robot repair welding system based on three-dimensional target detection and point cloud defect completion | |
| CN115659452B (en) | Intelligent patrol method, intelligent patrol system and computer readable storage medium | |
| CN116127821A (en) | Three-dimensional visual presentation method and platform for operation and maintenance data | |
| CN112667139A (en) | Robot operation method, device, equipment and storage medium based on augmented reality | |
| CN112733641A (en) | Object size measuring method, device, equipment and storage medium | |
| CN113752264A (en) | Mechanical arm intelligent equipment control method and system based on digital twins | |
| CN114898055A (en) | Virtual reality-based remote intelligent inspection method and system for extra-high voltage converter station | |
| CN113627005B (en) | Intelligent vision monitoring method | |
| US20220150406A1 (en) | Multi-camera positioning and dispatching system, and method thereof | |
| CN115982824A (en) | Construction site worker space management method and device, electronic equipment and storage medium | |
| CN114266876B (en) | Positioning method, visual map generation method and device | |
| CN107391710B (en) | Machine operation monitoring system and server |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |