CN113469378B

CN113469378B - Maintenance method and maintenance equipment

Info

Publication number: CN113469378B
Application number: CN202110600950.8A
Authority: CN
Inventors: 孙明华; 王宗文; 沈建良; 杨浩
Original assignee: Yantai Jereh Oilfield Services Group Co Ltd
Current assignee: Yantai Jereh Oilfield Services Group Co Ltd
Priority date: 2021-05-31
Filing date: 2021-05-31
Publication date: 2023-11-24
Anticipated expiration: 2041-05-31
Also published as: CN113469378A

Abstract

The application discloses an overhaul method and overhaul equipment, relates to the field of equipment overhaul, and aims to solve the problem of low equipment overhaul efficiency in the related art. The overhaul method comprises the following steps: acquiring an overhaul operation image; judging whether the overhaul operation corresponding to the overhaul operation image is correct or not based on the overhaul operation image; and if the overhaul operation is wrong, acquiring first prompt information, and enabling the terminal to display the first prompt information. The application is used for equipment overhaul.

Description

Maintenance method and maintenance equipment

Technical Field

The application relates to the field of equipment overhaul, in particular to an overhaul method and overhaul equipment.

Background

Augmented reality (Augmented Reality, AR) technology is newer technology that facilitates the integration of real world information and virtual world information content, and can implement simulated simulation processing of entity information that is otherwise difficult to experience in the spatial domain of the real world on the basis of scientific technology such as computers, and the like, superimpose virtual information content in the real world, and in the process can be perceived by human senses, thereby realizing a sensory experience that exceeds reality. Further, after the real environment and the virtual object are overlapped by the AR technique, they can coexist in the same screen and space.

At present, most enterprises only rely on manual maintenance when overhauling equipment, however, when overhauling, an maintainer sometimes has the condition of operation errors, and if errors are not found in time, the subsequent operation errors can be caused. This results in inefficient equipment servicing.

Disclosure of Invention

The embodiment of the application provides an overhaul method and overhaul equipment, which can solve the problem of low overhaul efficiency of equipment in the related art. In order to achieve the above object, the embodiment of the present application adopts the following technical scheme:

in a first aspect, a method of servicing is provided, the method comprising:

acquiring an overhaul operation image;

judging whether the overhaul operation corresponding to the overhaul operation image is correct or not based on the overhaul operation image;

and if the overhaul operation is wrong, acquiring first prompt information, and enabling the terminal to display the first prompt information.

In a second aspect, a method of servicing is provided, the method comprising:

the augmented reality equipment acquires an overhaul operation image;

the augmented reality device sends the overhaul operation image to a server;

the augmented reality equipment receives first prompt information sent by the server;

The augmented reality device displays the first prompt message.

In a third aspect, an inspection apparatus is provided, the inspection apparatus comprising:

the acquisition module is used for acquiring an overhaul operation image;

the judging module is used for judging whether the overhaul operation corresponding to the overhaul operation image is correct or not based on the overhaul operation image;

the acquisition module is further used for acquiring first prompt information if the overhaul operation is wrong;

and the control module is used for enabling the terminal to display the first prompt information.

In a fourth aspect, there is provided an augmented reality device comprising:

the acquisition module is used for acquiring an overhaul operation image;

the sending module is used for sending the overhaul operation image to a server;

the receiving module is used for receiving the first prompt information sent by the server;

and the display module is used for displaying the first prompt information.

In a fifth aspect, a computer readable storage medium is provided, characterized in that the computer readable storage medium has stored thereon a computer program which, when executed by a processor, implements the steps of the method according to the first aspect described above.

In a sixth aspect, there is provided a network device comprising: a memory, a processor and a computer program stored on the memory and executable on the processor, which when executed by the processor performs the steps of the method as described in the first aspect above.

In the embodiment of the application, when an overhauling staff overhauls equipment to be overhauled, an overhauling operation image of the overhauling staff during overhauling operation can be obtained; judging whether the overhaul operation corresponding to the overhaul operation image is correct or not based on the overhaul operation image; and if the overhaul operation is wrong, acquiring first prompt information, and enabling the terminal to display the first prompt information. Therefore, when an operator operates the error, the operator can timely find the error and prompt the error, so that the error of subsequent operation is avoided, and the equipment maintenance efficiency is improved.

Drawings

The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the application and do not constitute a limitation on the application. In the drawings:

fig. 1 is a schematic diagram of an implementation environment of an inspection system according to an embodiment of the present application.

Fig. 2 is a flowchart of an overhaul method provided in an embodiment of the present application.

Fig. 3 is a flow chart of another maintenance method provided by an embodiment of the present application.

Fig. 4 is a flow chart of yet another service method provided by an embodiment of the present application.

Fig. 5 is a block diagram of an overhaul equipment according to an embodiment of the present application.

Fig. 6 is a block diagram illustrating an AR device according to an embodiment of the present application.

Fig. 7 is a block diagram of an overhaul system according to an embodiment of the present application.

Detailed Description

For the purpose of promoting an understanding of the principles of the application, reference will now be made in detail to specific embodiments of the application and the accompanying drawings, in which it is apparent that some, but not all embodiments of the application will be illustrated. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.

The application provides a solution to the problem of low equipment maintenance efficiency in the related art, and aims to provide a method capable of improving the equipment maintenance efficiency.

The following describes in detail the technical solutions provided by the embodiments of the present application with reference to the accompanying drawings.

As shown in fig. 1, an overhaul system provided by an embodiment of the present application may include: service device 500 and terminal 600. The service device 500 may be a server or other device having data processing capabilities. The terminal 600 may be an AR device, such as AR glasses; other devices with input/output functions and display functions, such as a mobile phone or a tablet computer, are also possible. The terminal 600 may be provided with an image capturing element (e.g., a camera) and a voice capturing element (e.g., a microphone). The service apparatus 500 may be connected with the terminal 600 through a network. The following description will take an overhaul facility as a server and a terminal as an AR facility as an example.

In the process of interaction between the server 500 and the AR device 600, the AR device 600 can acquire an overhaul operation image of an overhaul worker during overhaul, and then send the overhaul operation image to the server 500; after receiving the overhaul operation image, the server 500 judges whether the overhaul operation corresponding to the overhaul operation image is correct; if the maintenance operation is wrong, the server 500 acquires first prompt information and then sends the first prompt information to the AR device 600; and after the AR equipment receives the first prompt information, displaying the first prompt information. When the overhauling personnel sees the first prompt information displayed by the AR device 600, the overhauling operation is known to be wrong in time, and correction is performed in time. Thus, the equipment maintenance efficiency can be improved.

It should be understood that the terminal (for example, AR device) provided in the embodiment of the present application may also be a terminal having multiple functions such as a storage function and a data processing function. At this time, the overhaul system provided by the embodiment of the application may only include the terminal and not include the server.

Fig. 2 is a flowchart of an overhaul method provided in an embodiment of the present application. The service method may be performed by a server. As shown in fig. 2, the overhaul method provided by the embodiment of the application may include:

And 210, acquiring an overhaul operation image.

Step 220, based on the overhaul operation image, judging whether the overhaul operation corresponding to the overhaul operation image is correct.

And 230, if the overhaul operation is wrong, acquiring first prompt information, and enabling a terminal to display the first prompt information.

It should be noted that the method shown in fig. 2 may be performed by a server, or may be performed by other maintenance devices. When the overhaul system provided by the embodiment of the present application does not include a server, the method shown in fig. 2 may also be performed by the AR device.

When the terminal is AR glasses, after an maintainer wears the AR glasses, the AR glasses can display the first prompt information in front of eyes of the maintainer; when the terminal is a device with a display screen, such as a mobile phone or a tablet computer, the terminal can display the first display information on the display screen of the terminal.

And the overhaul operation image can be sent to a server after being acquired by the AR equipment. The AR equipment can be provided with a camera and a microphone, the AR equipment can acquire maintenance operation images of maintenance personnel in the maintenance process through the camera, and can acquire voice instructions of the maintenance personnel through the microphone. The AR device may acquire the inspection operation image in real time, may acquire the inspection operation image at regular time or at predetermined time intervals, and may acquire the inspection operation image after acquiring a voice instruction indicating that the inspection operation image is acquired.

Thus, when an overhaul worker overhauls the equipment to be overhauled, an overhaul operation image of the overhaul worker during overhaul operation can be obtained; judging whether the overhaul operation corresponding to the overhaul operation image is correct or not based on the overhaul operation image; if the overhaul operation is wrong, acquiring first prompt information, and enabling the AR equipment to display the first prompt information. Therefore, when an operator operates the error, the operator can timely find the error and prompt the error, so that the error of subsequent operation is avoided, and the equipment maintenance efficiency is improved.

In order to make the equipment maintenance efficiency higher, the AR equipment can acquire the maintenance operation image in real time, and once the maintenance operation image is found to correspond to the maintenance operation image, the maintenance personnel are immediately prompted through the first prompt information.

The first prompt may include text information for prompting an operation error, e.g. "operation error" or "operation error-! Please re-operate-! "etc. Meanwhile, the microphone on the AR equipment can play the first prompt information in a voice mode.

Optionally, in step 220, the determining, based on the overhaul operation image, whether the overhaul operation corresponding to the overhaul operation image is correct or not may include the following steps:

Extracting features of the overhaul operation image to obtain a first feature vector;

calculating the similarity of the first feature vector and each feature vector in a predetermined feature vector combination;

judging whether the overhaul operation corresponding to the overhaul operation image is correct or not based on the similarity of the first eigenvector and each eigenvector in the eigenvector combination;

the feature vector combination comprises a second feature vector and a third feature vector, wherein the second feature vector is a feature vector of a correct operation image corresponding to the overhaul operation, and the third feature vector is a feature vector of an incorrect operation image corresponding to the overhaul operation.

It will be appreciated that the predetermined feature vector combination may include feature vectors obtained by feature extraction from a repair operation image in a large number of previous repair videos. After the feature vector is obtained, the inspection operation image can be judged to be correct or incorrect manually, and then the feature vector is respectively put into a correct operation feature vector library and an incorrect operation feature vector library. At this time, the feature vector combination includes the feature vector in the correct operation feature vector library (i.e., the second feature vector) and the feature vector in the incorrect operation feature vector library (i.e., the third feature vector).

Optionally, the extracting the features of the overhaul operation image to obtain a first feature vector may include:

and carrying out feature extraction on the overhaul operation image by utilizing a pre-constructed convolutional neural network model to obtain a first feature vector.

It will be appreciated that the convolutional neural network may convolve the image and then represent the image as a length of eigenvector.

Meanwhile, in calculating the similarity of the first feature vector and each feature vector in the predetermined feature vector combination, a cosine similarity algorithm may be used for calculation.

Cosine similarity, also known as cosine similarity, is measured by measuring the cosine value of the angle between two vectors. The cosine value of the angle of 0 degree is 1, and the cosine value of any other angle is not more than 1; and its minimum value is-1. The cosine value of the angle between the two vectors thus determines whether the two vectors point approximately in the same direction. When the two vectors have the same direction, the cosine similarity value is 1; when the included angle of the two vectors is 90 degrees, the cosine similarity value is 0; when the two vectors point in diametrically opposite directions, the cosine similarity has a value of-1. This results in dependence on the length of the vector, only on the pointing direction of the vector. Cosine similarity is usually used for positive space and therefore gives values between-1 and 1.

Given two eigenvectors A and B, the remaining chord similarity formula is as follows:

ai. Bi represents the components of vectors A and B, respectively, the closer the cosine value is to 1, the closer the angle is to 0 degrees, and the more similar the vectors A and B are.

As described above, the closer the cosine similarity is to 1, the closer the included angle is to 0 degrees, the more similar the vectors a and B are, and the cosine value of any angle is not greater than 1. Thus, the greater the cosine similarity of the two vectors, the more similar the two vectors are.

After obtaining the similarity between the first feature vector and each feature vector in the predetermined feature vector combination, the determining, based on the similarity between the first feature vector and each feature vector in the feature vector combination, whether the overhaul operation corresponding to the overhaul operation image is correct may include:

selecting a preset number of feature vectors in the feature vector combination as a target feature vector set according to the descending order of the similarity between each feature vector in the feature vector combination and the first feature vector;

if the number of the second target feature vectors in the target feature vector set is greater than the number of the third target feature vectors, the maintenance operation corresponding to the maintenance operation image is correct;

If the number of the second target feature vectors in the target feature vector set is smaller than the number of the third target feature vectors, the maintenance operation corresponding to the maintenance operation image is wrong;

wherein the set of target feature vectors comprises a second target feature vector and/or a third target feature vector; the second target feature vector is a feature vector of the target feature vector set for a correct operation image corresponding to the overhaul operation, and the third target feature vector is a feature vector of the target feature vector set for an incorrect operation image corresponding to the overhaul operation.

The second target feature vector is a feature vector of the target feature vector set for the correct operation image corresponding to the overhaul operation, which can be understood as a feature vector from a correct operation feature vector library; the third target feature vector is a feature vector of the target feature vector set for the error operation image corresponding to the overhaul operation, which may be understood as a feature vector from an error operation feature vector library.

For easy understanding, the above method for determining whether the inspection operation corresponding to the inspection operation image is correct based on the similarity of each feature vector in the first feature vector and the feature vector combination is illustrated herein:

For example, nine feature vectors in the feature vector combination are selected as a target feature vector set in descending order of the magnitudes of the similarities of each feature vector in the feature vector combination and the first feature vector. The set of target feature vectors may include both feature vectors from the library of correctly operating feature vectors (i.e., the second target feature vector) and feature vectors from the library of incorrectly operating feature vectors (i.e., the third target feature vector), or may include only feature vectors from the library of correctly operating feature vectors (i.e., the second target feature vector) or feature vectors from the library of incorrectly operating feature vectors (i.e., the third target feature vector).

If there are 3 feature vectors from the correct operation feature vector library (i.e., the second target feature vector) and 6 feature vectors from the incorrect operation feature vector library (i.e., the third target feature vector) in the 9 feature vectors in the target feature vector set. Since the number of the second target feature vectors (i.e., 3) is smaller than the number of the third target feature vectors (i.e., 6), the overhaul operation is wrong.

Optionally, the first prompt information includes a target correct operation image, and the acquiring the overhaul operation image includes:

Receiving a maintenance operation image from the AR equipment;

if the maintenance operation is wrong, acquiring first prompt information, and enabling the AR equipment to display the first prompt information comprises:

if the maintenance operation is wrong, a third target feature vector with the highest similarity with the first feature vector in the target feature vector set is obtained and is used as a target wrong operation feature vector;

acquiring a target correct operation image associated with the target incorrect operation feature vector based on the target incorrect operation feature vector;

and sending the target correct operation image to the AR equipment so that the AR equipment displays the target correct operation image.

It will be appreciated that each feature vector in the library of incorrect operational feature vectors is associated with a correct service image, i.e. the target correct operation image. The target correct operation image may be stored in a local database in advance. Thus, a target correct operation image associated with the target incorrect operation feature vector can be acquired based on the target incorrect operation feature vector.

As described above, after the maintenance operation error is determined, a feature vector having the highest similarity with the first feature vector is selected from 6 feature vectors (i.e., third target feature vectors) from the error operation feature vector library in the target feature vector set as a target error operation feature vector. Then, a target correct operation image associated with the target incorrect operation feature vector is acquired, and the target correct operation image is sent to the AR device, so that the AR device displays the target correct operation image.

Optionally, before the receiving the overhaul operation image from the AR device, the overhaul method provided by the embodiment of the present application may further include:

receiving a voice instruction from the AR equipment, wherein the voice instruction indicates to start an overhaul flow;

responding to the voice instruction for indicating to start the overhaul flow, acquiring information to be confirmed, and sending the information to be confirmed to the AR equipment, so that the AR equipment displays the information to be confirmed;

receiving a confirmation voice instruction from the AR equipment, wherein the confirmation voice instruction indicates to confirm the information to be confirmed;

responding to the confirmation voice instruction, acquiring second prompt information and sending the second prompt information to the AR equipment, so that the AR equipment displays the second prompt information;

the to-be-confirmed information comprises a tool list and a safety attention point which are needed by overhaul, and the second prompt information comprises at least one of text prompt information, voice prompt information and animation prompt information which are related to overhaul.

The AR equipment can acquire a voice instruction of an maintainer through a microphone arranged on the AR equipment, and then the voice instruction is sent to a server; the voice command may be a voice command indicating to start the overhaul flow, or a voice command indicating to confirm the information to be confirmed, or a voice command indicating to perform other operations.

It can be understood that after the maintenance process is started, maintenance personnel needs to confirm the tool list and the safety attention point required by maintenance before maintenance. Therefore, the tool list required for overhaul can remind an overhaul worker to prepare before overhaul, and the safety attention points can remind the overhaul worker to pay attention to safety in the overhaul process.

The text prompt information is a detailed description of each overhaul step in the overhaul process, the voice prompt information is voice corresponding to the text prompt information, and the animation prompt information is a disassembly and assembly animation display of key parts of the equipment.

Meanwhile, in the overhaul process, the server can also acquire a three-dimensional model corresponding to the equipment to be overhauled, the three-dimensional model can comprise a three-dimensional model of the whole equipment and a three-dimensional model of a core component of the equipment, and the three-dimensional model can be stored in a local database of the server in advance. And then the three-dimensional model is sent to the AR equipment, and the AR equipment can display the three-dimensional model superimposed on the corresponding position of the equipment to be overhauled by utilizing a motion tracking technology. The three-dimensional model associated with the equipment to be overhauled can be the same as the equipment name, the equipment model and the equipment type of the equipment to be overhauled.

Optionally, the local database can also store demonstration videos of each step in the overhaul flow, and when required by an overhaul worker, the demonstration videos can be played through voice instructions. Namely, when the AR equipment acquires a voice instruction for indicating to play the demonstration video, the voice instruction for indicating to play the demonstration video is sent to a server; after receiving the voice instruction for indicating to play the demonstration video, the server acquires the demonstration video from a local database and sends the demonstration video to the AR equipment; and after the AR equipment receives the demonstration video, displaying the demonstration video.

And the text prompt information, the voice prompt information and the animation prompt information can be selectively opened or closed through voice instructions.

The camera on the AR equipment can capture video besides acquiring images. The AR equipment can record the whole overhaul process to obtain overhaul operation videos.

Optionally, after the overhaul process is finished, the overhaul method provided by the embodiment of the application may further include:

acquiring an overhaul recording table, and transmitting the overhaul recording table to the AR equipment so that the AR equipment displays the overhaul recording table;

Receiving a voice recording command related to the overhaul recording table from AR equipment, wherein the voice recording command comprises a command corresponding to information to be recorded;

acquiring the information to be recorded based on the voice recording instruction;

and inputting the information to be input into the overhaul recording table, wherein the information to be input comprises at least one of overhaul personnel name, overhaul content and overhaul completion time.

Alternatively, during service, critical component inspection data may need to be recorded while certain relatively important service steps are being performed. At this time, the server may acquire the key component detection record table after the relatively important maintenance is completed, and send the key component detection record table to the AR device. After the AR equipment receives the key component detection record list, displaying the key component detection record list; then acquiring a voice recording order related to the key component detection record table, wherein the voice recording order comprises detection data of a key component, and sending the voice recording order to a server; after receiving the voice recording command, the server records the detection data of the key component in the voice recording command into the key component detection record table.

And finally, storing the overhaul recording table and the overhaul operation video in a local database correspondingly, wherein the name of the overhaul operation video file can be information recorded in the overhaul recording table, such as at least one of overhaul personnel name, overhaul content and overhaul completion time, and the key component detection recording table can also be stored in the local database.

Optionally, when the server is connected to the network, the key component detection record table, the overhaul record table and the overhaul operation video may also be sent from the local database to a background data center for data archiving.

Optionally, before the maintenance flow starts, the maintenance method provided by the embodiment of the application may further include a preparation step before maintenance:

receiving a voice instruction indicating that overhaul is ready and an image of equipment to be overhauled from AR equipment;

selecting a to-be-overhauled equipment template library based on the voice instruction, and matching the image of the to-be-overhauled equipment with the image in the to-be-overhauled equipment template library;

if the matching is successful, equipment information is acquired according to a template image which is successfully matched with the image of the equipment to be overhauled, and the equipment information is sent to the AR equipment, wherein the equipment information comprises: at least one of a device name, a device model number, a device type, a device delivery date, and a device maintenance period;

If the matching fails, acquiring third prompt information and sending the third prompt information to the AR equipment, wherein the third prompt information comprises voice information or text information for prompting to re-match.

It will be appreciated that the device information may be pre-stored in a local database of the server. The selecting the equipment template library to be overhauled based on the voice command can be understood as selecting an overhauling object based on the voice command or selecting a function to be overhauled based on the voice command.

The equipment to be overhauled provided by the embodiment of the application can be various mechanical equipment or a certain part of various mechanical equipment; for example, the equipment to be overhauled can be fracturing equipment or a plunger pump in the fracturing equipment. The overhaul method provided by the embodiment of the application can overhaul a certain device, can overhaul a certain component of a certain device and can overhaul a certain assembly of a certain component of a certain device. The equipment template library to be overhauled can comprise templates of equipment, templates of all parts of the equipment and all assemblies of all parts of the equipment.

When the equipment to be overhauled is a plunger pump of the fracturing equipment, the template library of the equipment to be overhauled can comprise a plunger pump hydraulic end image, a plunger pump power end image and a fracturing truck image. At this time, the selecting of the equipment template library to be overhauled based on the voice command can be understood as selecting any one of overhauling functions of hydraulic end overhauling, plunger pump power end overhauling and plunger pump assembly disassembly based on the voice command.

When the AR equipment acquires the image of the equipment to be overhauled, a frame prompt line corresponding to the equipment to be overhauled is displayed on the AR equipment, and an overhauler needs to enable the frame prompt line to be aligned with the outline of the equipment to be overhauled. When the matching is failed, the AR equipment reminds an maintainer of aligning the frame prompting line with the outline of the equipment to be overhauled in a text or voice mode, and then the matching is performed again.

Optionally, after the image of the equipment to be overhauled is successfully matched with the template image of the equipment to be overhauled, the overhauling record can be checked through a voice instruction. That is, the overhaul method provided by the embodiment of the application may further include:

receiving a voice instruction from the AR equipment, wherein the voice instruction is used for indicating to view maintenance records;

Acquiring an overhaul record of the equipment to be overhauled;

and sending the maintenance record to the AR equipment, so that the AR equipment displays the maintenance record.

Thus, the maintenance personnel can check the maintenance record in a voice instruction mode. After the maintenance record is checked, the maintenance personnel can switch the display picture of the AR equipment back to the equipment information interface in a voice instruction mode.

Fig. 3 is a flow chart of another maintenance method provided by an embodiment of the present application. This overhaul method may be performed by the AR device. As shown in fig. 3, the overhaul method provided by the embodiment of the application may include:

in step 310, the AR device obtains a service operation image.

The ar device transmits the service operation image to the server in step 320.

In step 330, the ar device receives the first prompt sent by the server.

And step 340, the AR equipment displays the first prompt information.

The first prompt information may include a target correct operation image, that is, a repair operation corresponds to the correct operation image.

Before the ar device acquires the overhaul operation image in step 310, the overhaul method provided by the embodiment of the present application may further include:

the AR equipment acquires a voice instruction for indicating to start the overhaul process and sends the voice instruction for indicating to start the overhaul process to the server;

The AR equipment receives information to be confirmed sent by a server and displays the information to be confirmed;

the AR equipment acquires a voice instruction for indicating to confirm the information to be confirmed and sends the voice instruction for indicating to confirm the information to be confirmed to a server;

the AR equipment receives second prompt information sent by the server and displays the second prompt information;

the AR equipment acquires a voice instruction indicating to be overhauled and an image of equipment to be overhauled;

the voice instruction indicating the equipment to be overhauled and the image of the equipment to be overhauled are sent to a server by the AR equipment;

the AR device receives device information sent by a server, wherein the device information comprises: at least one of a device name, a device model number, a device type, a device delivery date, and a device maintenance period; displaying the equipment information;

Or, the AR equipment receives third prompt information sent by the server, wherein the third prompt information comprises voice information or text information for prompting to match again; and prompting according to the third prompting information.

The prompting according to the third prompting information comprises prompting to match again in a text mode or prompting to match again in a voice mode.

Optionally, in the overhauling process, the overhauling method provided by the embodiment of the application further comprises the following steps:

the AR equipment acquires a voice instruction for indicating to view the overhaul record;

the AR equipment receives an overhaul record sent by a server;

and the AR equipment displays the overhaul record.

Optionally, after the overhaul process is finished, the overhaul method provided by the embodiment of the application further may include:

the AR equipment receives an overhaul recording table sent by a server and displays the overhaul recording table;

the AR equipment acquires a voice recording instruction related to the overhaul recording table, wherein the voice recording instruction comprises an instruction corresponding to information to be recorded; and sending the voice input command related to the overhaul recording table to a server.

According to the overhaul method provided by the embodiment of the application, the voice instruction of an overhaul worker, the image of the equipment to be overhauled and the overhaul operation image can be obtained through the AR equipment, and various prompt information can be displayed to the overhaul worker for watching through the AR equipment. So, in the maintenance process, the prompt message can help the maintenance personnel to maintain to improve maintenance efficiency.

Fig. 4 is a flow chart of yet another service method provided by an embodiment of the present application. As shown in fig. 4, the method for overhauling provided by the embodiment of the application may include:

in step 401, the ar device acquires a voice command indicating to be overhauled and an image of the device to be overhauled, and sends the voice command indicating to be overhauled and the image of the device to be overhauled to a server.

Step 402, after receiving the voice command indicating that the equipment is ready to be overhauled and the image of the equipment to be overhauled, selecting a template library of the equipment to be overhauled based on the voice command indicating that the equipment is ready to be overhauled, and matching the image of the equipment to be overhauled with the image in the template library of the equipment to be overhauled.

Step 403, if the matching is successful, the server acquires equipment information according to the template image successfully matched with the image of the equipment to be overhauled and sends the equipment information to the AR equipment; and the AR equipment displays the equipment information after receiving the equipment information.

Step 404, if the matching fails, the server acquires a third prompting message for prompting to re-match and sends the third prompting message to the AR equipment; and after the AR equipment receives the third prompt information, prompting according to the third prompt information.

Step 405, the ar device obtains a voice command indicating to view the overhaul record and sends the voice command to the server; after receiving a voice instruction for indicating to check the overhaul record, the server acquires the overhaul record of the equipment to be overhauled and sends the overhaul record to the AR equipment; after the AR equipment receives the maintenance record, the maintenance record is displayed.

Step 406, the ar device obtains a voice command indicating to start the overhaul process and sends the voice command to the server; after receiving the voice instruction for indicating to start the overhaul flow, the server acquires information to be confirmed and sends the information to the AR equipment; and after the AR equipment receives the information to be confirmed, the information to be confirmed is displayed.

Step 407, the ar device obtains a confirmation voice command indicating to confirm the information to be confirmed and sends the confirmation voice command to the server; after receiving the confirmation voice instruction, the server acquires second prompt information and sends the second prompt information to the AR equipment; and after the AR equipment receives the second prompt information, the second prompt information is displayed.

Step 408, the AR equipment acquires an overhaul operation image and sends the overhaul operation image to the server; after the server receives the overhaul operation image, judging whether overhaul operation corresponding to the overhaul operation image is correct or not based on the overhaul operation image.

Step 409, if the overhaul operation is wrong, the server acquires first prompt information and sends the first prompt information to the AR device; and after the AR equipment receives the first prompt information, the first prompt information is displayed.

Step 410, the ar device obtains a voice command indicating that the overhaul is finished and sends the voice command to the server; after receiving the voice instruction for indicating the maintenance end, the server acquires a maintenance record table and sends the maintenance record table to the AR equipment; and after the AR equipment receives the maintenance record table, displaying the maintenance record table.

Step 411, the ar device obtains a voice input command related to the maintenance record table and sends the voice input command to the server; after receiving the voice recording instruction, the server records the information to be recorded in the voice recording instruction into the overhaul recording table.

In step 412, the ar device acquires the overhaul operation video and sends the overhaul operation video to the server, and the server stores the overhaul recording table and the overhaul operation video in a local database correspondingly.

It should be noted that, in the overhaul method provided by the embodiment of the present application, the interaction process between the AR device and the server may be implemented through a wireless network, a mobile network, or a wired network.

According to the overhaul method provided by the embodiment of the application, when an overhaul worker overhauls equipment to be overhauled, an overhaul operation image from the AR equipment can be received; judging whether the overhaul operation corresponding to the overhaul operation image is correct or not based on the overhaul operation image; if the overhaul operation is wrong, acquiring first prompt information and sending the first prompt information to the AR equipment, so that the AR equipment displays the first prompt information. Therefore, when an operator operates the error, the operator can timely find the error and prompt the error, so that the error of subsequent operation is avoided, and the equipment maintenance efficiency is improved.

Fig. 5 is a block diagram of an overhaul equipment according to an embodiment of the present application. As shown in fig. 5, an embodiment of the present application provides an inspection apparatus 500. The service apparatus 500 includes an acquisition module 510, a determination module 520, and a control module 530.

In the embodiment of the present application, the service device 500 may be a server or a terminal (for example, an AR device).

The acquiring module 510 is configured to acquire an overhaul operation image.

The determining module 520 is configured to determine whether the maintenance operation corresponding to the maintenance operation image is correct based on the maintenance operation image.

The obtaining module 510 is further configured to obtain a first prompt message if the overhaul operation is wrong;

the control module 530 is configured to cause the terminal to display the first prompt information.

Optionally, the judging module 520 may further include a feature extracting unit, a calculating unit, and a judging unit; the feature extraction unit is used for extracting features of the overhaul operation image to obtain a first feature vector; the computing unit is used for computing the similarity of each feature vector in the combination of the first feature vector and a predetermined feature vector; the judging unit is used for judging whether the overhaul operation corresponding to the overhaul operation image is correct or not based on the similarity of the first eigenvector and each eigenvector in the eigenvector combination; the feature vector combination comprises a second feature vector and a third feature vector, wherein the second feature vector is a feature vector of a correct operation image corresponding to the overhaul operation, and the third feature vector is a feature vector of an incorrect operation image corresponding to the overhaul operation.

Optionally, the feature extraction unit is specifically configured to perform feature extraction on the overhaul operation image by using a convolutional neural network model that is built in advance, so as to obtain a first feature vector; the judging unit is specifically configured to select a preset number of feature vectors in the feature vector combination as a target feature vector set according to a descending order of the similarity between each feature vector in the feature vector combination and the first feature vector; if the number of the second target feature vectors in the target feature vector set is greater than the number of the third target feature vectors, the maintenance operation corresponding to the maintenance operation image is correct; if the number of the second target feature vectors in the target feature vector set is smaller than the number of the third target feature vectors, the maintenance operation corresponding to the maintenance operation image is wrong; wherein the set of target feature vectors comprises a second target feature vector and/or a third target feature vector; the second target feature vector is a feature vector of the target feature vector set for a correct operation image corresponding to the overhaul operation, and the third target feature vector is a feature vector of the target feature vector set for an incorrect operation image corresponding to the overhaul operation.

Optionally, the first prompt information includes a target correct operation image; the obtaining module 510 is specifically configured to receive an overhaul operation image from the terminal when obtaining the overhaul operation image, and is further configured to obtain, if the overhaul operation is wrong, a third target feature vector with the highest similarity with the first feature vector in the target feature vector set as a target incorrect operation feature vector; acquiring a target correct operation image associated with the target incorrect operation feature vector based on the target incorrect operation feature vector; the control module 530 is specifically configured to send the target correct operation image to a terminal, so that the terminal displays the target correct operation image.

Optionally, the obtaining module 510 is further configured to receive a voice command from the terminal, where the voice command indicates to start the overhaul process; and responding to the voice instruction for indicating to start the overhaul flow, and acquiring information to be confirmed. The control module 530 is further configured to send the information to be confirmed to the terminal, so that the terminal displays the information to be confirmed. After the terminal displays the information to be confirmed, the obtaining module 510 is further configured to receive a voice command from the terminal indicating to confirm the information to be confirmed, and obtain a second prompt message in response to the voice command indicating to confirm the information to be confirmed. The control module 530 is configured to send the second prompt information to the terminal, so that the terminal displays the second prompt information; the to-be-confirmed information comprises a tool list and a safety attention point which are needed by overhaul, and the second prompt information comprises at least one of text prompt information, voice prompt information and animation prompt information which are related to overhaul.

Optionally, after the overhaul process is finished, the obtaining module 510 is further configured to obtain an overhaul record table; the control module 530 is further configured to send the service record table to the terminal, so that the terminal displays the service record table; the obtaining module 510 is further configured to obtain a voice entry command related to the maintenance record table, where the voice entry command includes information to be entered; based on the voice input instruction, inputting the information to be input into the overhaul recording table, wherein the information to be input comprises at least one of overhaul personnel name, overhaul content and overhaul completion time.

It should be appreciated that the method for overhauling a server described above may be applied to a server provided in an embodiment of the present application, so the content of the server may be referred to in the description of the method section above.

Fig. 6 is a block diagram illustrating an AR device according to an embodiment of the present application. As shown in fig. 6, an embodiment of the present application provides an AR device 600. The AR device 600 includes an acquisition module 610, a transmission module 620, a reception module 630, and a display module 640.

The acquiring module 610 is configured to acquire an overhaul operation image.

The sending module 620 is configured to send the overhaul operation image to the server.

The receiving module 630 is configured to receive the first prompt message sent by the server.

The display module 640 is configured to display the first prompt message.

Optionally, before the acquiring module 610 acquires the overhaul operation image, the acquiring module 610 is further configured to acquire a voice instruction indicating to start an overhaul process; the sending module 620 is configured to send the voice command to the server, where the voice command indicates to start the overhaul process; the receiving module 630 is configured to receive information to be confirmed sent by the server; the display module 640 is configured to display the information to be confirmed. The obtaining module 610 is further configured to obtain a voice command indicating to confirm the information to be confirmed; the sending module 620 is further configured to send the voice command indicating to confirm the information to be confirmed to the server; the receiving module 630 is further configured to receive a second prompt message sent by the server; the display module 640 is further configured to display the second prompt message; the to-be-confirmed information comprises a tool list and a safety attention point which are needed by overhaul, and the second prompt information comprises at least one of text prompt information, voice prompt information and animation prompt information which are related to overhaul.

It should be appreciated that the above-described method for overhauling an AR device may be applied to an AR device provided in an embodiment of the present application, so the content of the AR device may be referred to the description of the above method section.

Fig. 7 is a block diagram of an overhaul system according to an embodiment of the present application. As shown in fig. 7, an embodiment of the present application provides an inspection system 700; the service system 700 may include only the terminal 600, or may include the service apparatus 500 and the terminal 600; when the service system 700 includes the service device 500 and the terminal 600, the service device 500 may be a server and the terminal 600 may be an AR device.

It should be appreciated that the above-described maintenance method may be applied to the maintenance system provided in the embodiment of the present application, and thus, reference may be made to the description of the above method section for the content of the maintenance system.

Embodiments of the present application also provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements any of the methods of servicing a server as described above.

The embodiment of the present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements any one of the methods of overhauling an AR device as described above.

The embodiment of the application also provides a network device, which comprises: a memory, a processor, and a computer program stored on the memory and executable on the processor, which when executed by the processor, implements any of the methods of servicing a server as described above.

The embodiment of the application also provides a network device, which comprises: a memory, a processor, and a computer program stored on the memory and executable on the processor, which when executed by the processor, implements any of the methods of overhauling an AR device as described above.

From the description of the embodiments above, those skilled in the art will appreciate that embodiments of the application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.

The present invention is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

In one typical configuration, a computing device includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.

The memory may include volatile memory in a computer-readable medium, random Access Memory (RAM) and/or nonvolatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of computer-readable media.

Computer readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of storage media for a computer include, but are not limited to, phase change memory (PRAM), static Random Access Memory (SRAM), dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), read Only Memory (ROM), electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices, or any other non-transmission medium, which can be used to store information that can be accessed by a computing device. Computer-readable media, as defined herein, does not include transitory computer-readable media (transmission media), such as modulated data signals and carrier waves.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element.

The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those having ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are to be protected by the present application.

Claims

1. A method of service, the method comprising:

acquiring an overhaul operation image;

if the overhaul operation is wrong, acquiring first prompt information, and enabling a terminal to display the first prompt information;

based on the overhaul operation image, judging whether the overhaul operation corresponding to the overhaul operation image is correct or not includes:

the feature vector combination comprises a second feature vector and a third feature vector, wherein the second feature vector is a feature vector of a correct operation image corresponding to the overhaul operation, and the third feature vector is a feature vector of an error operation image corresponding to the overhaul operation;

the step of extracting the characteristics of the overhaul operation image to obtain a first characteristic vector comprises the following steps:

Performing feature extraction on the overhaul operation image by using a pre-constructed convolutional neural network model to obtain the first feature vector;

based on the similarity of each feature vector in the first feature vector and feature vector combination, judging whether the overhaul operation corresponding to the overhaul operation image is correct or not includes:

2. The service method according to claim 1, wherein the first prompt information includes a target correct operation image, and the acquiring the service operation image includes:

receiving an overhaul operation image from the terminal;

if the maintenance operation is wrong, acquiring first prompt information, and enabling a terminal to display the first prompt information comprises:

and sending the target correct operation image to the terminal, so that the terminal displays the target correct operation image.

3. The service method according to claim 2, wherein before the receiving the service operation image from the terminal, the service method further comprises:

receiving a voice instruction from the terminal, wherein the voice instruction indicates to start an overhaul flow;

responding to the voice instruction for indicating to start the overhaul flow, acquiring information to be confirmed, and sending the information to be confirmed to the terminal, so that the terminal displays the information to be confirmed;

Receiving a voice instruction from the terminal, wherein the voice instruction indicates to confirm the information to be confirmed;

responding to the voice instruction for indicating and confirming the information to be confirmed, acquiring second prompt information and sending the second prompt information to the terminal, so that the terminal displays the second prompt information;

4. A service method according to claim 3, wherein after the service flow is completed, the service method further comprises:

acquiring an overhaul record table, and sending the overhaul record table to the terminal so that the terminal displays the overhaul record table;

acquiring a voice recording instruction related to the overhaul recording table, wherein the voice recording instruction comprises information to be recorded;

based on the voice input instruction, inputting the information to be input into the overhaul recording table, wherein the information to be input comprises at least one of overhaul personnel name, overhaul content and overhaul completion time.

5. A method of service, the method comprising:

the augmented reality equipment acquires an overhaul operation image;

the augmented reality device sends the overhaul operation image to a server;

the augmented reality device receives first prompt information sent by the server, and the first prompt information is sent to the augmented reality device when the server judges whether the overhaul operation error corresponding to the overhaul operation image is correct or not based on the overhaul operation image and the judgment result is wrong;

the augmented reality device displays the first prompt information;

6. The overhaul method of claim 5, further comprising, prior to the augmented reality device acquiring an overhaul operation image:

the augmented reality equipment acquires a voice instruction for indicating to start the overhaul process and sends the voice instruction for indicating to start the overhaul process to the server;

the augmented reality equipment receives information to be confirmed sent by the server and displays the information to be confirmed;

the augmented reality equipment acquires a voice instruction for indicating to confirm the information to be confirmed and sends the voice instruction for indicating to confirm the information to be confirmed to the server;

the augmented reality equipment receives second prompt information sent by the server and displays the second prompt information;

7. An inspection apparatus, the inspection apparatus comprising:

the acquisition module is used for acquiring an overhaul operation image;

the control module is used for enabling the terminal to display the first prompt information;

8. An augmented reality device, the augmented reality device comprising:

the acquisition module is used for acquiring an overhaul operation image;

the receiving module is used for receiving first prompt information sent by the server, and the first prompt information is sent to the augmented reality equipment when the server judges whether the overhaul operation error corresponding to the overhaul operation image is correct or not based on the overhaul operation image and the judgment result is wrong;

the display module is used for displaying the first prompt information;