TWM597913U - Artificial intelligence operation assistive system - Google Patents

Abstract

An intelligent operation assistance system includes a portable device and an artificial intelligence operation assistance platform. The portable device includes a camera module, a wireless communication module, a processor and a display module. The camera module obtains the environment image with the target object. The wireless communication module sends environmental images and receives operation signals. The processor controls the wireless communication module to send environmental images, and generates an operation layer according to the operation signal. The display module presents the operation layer. Artificial intelligence operation auxiliary platform communication connection wireless communication module. The artificial intelligence operation assistance platform receives environmental images and executes the identification process to identify the target object, and generates and sends operation signals based on the target object and the object database.

Description

Smart operation assistance system

This creation involves image recognition, especially a smart operation assistance system based on Mixed Reality (MR) technology and high-speed communication network.

In petrochemical or chemical industry plants, there are complex pipelines, a wide variety of equipment and various instruments. In order to maintain the safety of various assets and equipment in the plant, it is necessary to deploy enough personnel to conduct inspections to facilitate daily routine The equipment index data monitoring, and at the same time to eliminate emergency events.

However, for new patrol personnel, because they are not immediately familiar with the operation methods and troubleshooting mechanisms of all equipment, operating errors often occur. Even if the inspectors use the walkie-talkie to talk to the central control center to get instructions on the operation, it is still possible that the operator may make mistakes in the operation due to the lack of clear meaning of the two parties, and cause the fault to be unable to be eliminated or the equipment to fail. This may result in unpredictable losses for the factory.

In view of this, the proposed creation of a kind of wisdom auxiliary operating system, in the bottom of the fifth generation of mobile communications technology ^{(5 th Generation mobile networks, 5G} ) dedicated spectrum of corporate network deployment made of high-speed communication environment to achieve the petrochemical industry plant area Auxiliary guidance for equipment operation can effectively enable new inspectors to operate correctly when facing the complex equipment and instruments in the petrochemical plant, reducing the possibility of human error.

An intelligent operation assistance system described according to an embodiment of the present creation includes a portable device and an artificial intelligence operation assistance platform. The portable device includes a camera module, a wireless communication module, a processor and a display module. The camera module is used to obtain environmental images. There are target objects in the environmental image. no The line communication module is used to send environmental images and receive operation signals. The processor is electrically connected to the wireless communication module and the camera module. The processor controls the wireless communication module to send environmental images, and generates an operation layer according to the operation signal. The display module is used to present the operation layer. The artificial intelligence operation auxiliary platform communicates with the wireless communication module. The artificial intelligence operation auxiliary platform is used to receive environmental images and execute a recognition process to identify the target object. The artificial intelligence operation auxiliary platform is used to generate and generate data based on the target object and the object database. Send operation signal.

The above description of the content of the disclosure and the description of the following implementation manners are used to demonstrate and explain the spirit and principle of this creation, and provide a further explanation of the scope of patent application of this creation.

100: Smart operation assistance system

3: portable device

32: camera module

34: wireless communication module

36: processor

38: display module

5: Artificial intelligence operation auxiliary platform

N: High-speed mobile communication network

AP: wireless access point

S1~S5: steps

A: Operation starting position

B: Operation end position

71: pipeline

73: Meter

75: Ball raft

77: handle

L: Operation layer

Fig. 1 is a block diagram of a smart operation assistance system according to an embodiment of the present creation.

FIG. 2 is a schematic diagram showing the appearance of a portable device according to an embodiment of the present invention.

FIG. 3 is a flowchart of a method for assisting smart operation according to an embodiment of the present creation.

Fig. 4 is a schematic diagram of operating layers drawn according to an embodiment of the present creation.

The detailed features and advantages of this creation will be described in detail below in the implementation mode. The content is sufficient to enable anyone familiar with the relevant art to understand the technical content of this creation and implement it accordingly, and in accordance with the content disclosed in this specification, the scope of patent application and drawings , Anyone who is familiar with relevant skills can easily understand the purpose and advantages of this creation. The following examples further illustrate the viewpoint of this creation in detail, but do not limit the scope of this creation by any viewpoint.

Please refer to FIG. 1, which shows a block diagram of a smart operation assistance system 100 according to an embodiment of the present creation. The smart operation assistance system 100 is suitable for petrochemical plants where a high-speed mobile communication network N has been deployed. In one embodiment, the high speed mobile communication network N by the radio access point AP and a plurality of base stations are composed of the wireless access point AP through the fifth-generation mobile communication system technologies ^{(5 th Generation mobile networks, 5G} ) dedicated spectrum of Enterprise private networks are connected to each other. As shown in FIG. 1, the intelligent operation assistance system 100 includes a portable device 3 and an artificial intelligence operation assistance platform 5. The portable device 3 is connected to the artificial intelligence operation auxiliary platform 5 through one of the aforementioned wireless access points AP.

As shown in FIG. 1, the portable device 3 used for transmitting the environmental images visually seen by the inspectors includes a camera module 32, a wireless communication module 34, a processor 36 and a display module 38. The processor 36 is electrically connected to the camera module 32, the wireless communication module 34 and the display module 38. The artificial intelligence operation auxiliary platform 5 is communicatively connected to the wireless communication module 34 to receive environmental images. In one embodiment, the artificial intelligence operation assistance platform 5 is a server or a cluster of servers, and is communicatively connected to a high-speed mobile communication network N implemented by 5G. In this way, the artificial intelligence operation assistance platform 5 can receive the environmental images returned by each portable device 3 with a small delay.

Please refer to FIG. 2, which shows a schematic diagram of the appearance of the portable device 3 described in an embodiment of the present creation. In one embodiment, the portable device 3 is a smart helmet worn by inspectors. Since the camera module 32 is arranged close to the eye position of the inspector wearing the portable device 3 in the form of a smart helmet, the inspector can easily collect the factory buildings he sees through the camera module 32 Images of various equipment inside. In other embodiments, the appearance of the portable device 3 can also be smart glasses, a smart phone with a camera module 32, a tablet computer with a camera module 32, or a personal digital assistant with a camera module 32. (Personal Digital Assistant, PDA) etc. This creation does not limit the appearance or hardware type of the portable device 3.

In the embodiment shown in FIG. 2, the display module 38 is disposed at the position of the lens. The display module 38 is a small transparent screen. Therefore, the inspectors can pass through the display module 38 to see the equipment inside the factory. At the same time, the inspector can also see the display layer presented on the small screen in the form of the lens. Therefore, in this embodiment, the display module 38 can provide mixed reality (Miexed Reality, MR) or augmented reality. The superimposed images presented in the form of Augemented Reality (AR) are for inspection by inspectors. In another embodiment, the display module 38 is, for example, a general screen, such as the display panel of a tablet computer, and the user can see the camera on the display module 38 The actual image captured by the image module 32, or the three-dimensional object image sent by the artificial intelligence operation auxiliary platform 5 is seen. This creation does not limit the hardware type of the display module 38.

Please refer to FIG. 3, which shows a flowchart of a smart operation assisting method described in an embodiment of the present creation. The aforementioned smart operation assistance method is applicable to the aforementioned smart operation assistance system 100. The following describes the composition and implementation of various components in the smart operation assistance system 100 in conjunction with the flow of the smart operation assistance method shown in FIG. 3.

Please refer to step S1, the camera module 32 of the portable device 3 obtains and sends an environment image. There are target objects in the environmental image. The target object generally refers to equipment or instruments that can be operated by inspectors. For example, an inspector wearing a portable device 3 as shown in FIG. 2 arrives at a designated coordinate point and starts to inspect or operate petrochemical pipelines, equipment or instruments around the coordinate point. First, the camera module 32 of the portable device 3 obtains the environment image with the target object, and then, through the aforementioned high-speed mobile communication network N realized by the 5G enterprise private network, the processor 36 of the portable device 3 controls the portable The wireless communication module 34 of the mobile device 3 sends the environment image with the target object to the artificial intelligence operation assistance platform 5.

Please refer to step S2, the artificial intelligence operation assistance platform 5 recognizes the target object in the environmental image. In one embodiment, the artificial intelligence operation assistance platform 5 can execute an image recognition process and load an object database to identify the target object in the environmental image. The object database includes the image information of all equipment in the petrochemical industry plant, for example, in the form of computer-aided design (CAD) files. Therefore, no matter what perspective the inspector views the target object, the artificial intelligence operation assistance platform 5 can recognize the type of the target object in the environmental image.

In another embodiment of step S2, when the target object has a readable value (for example, a meter on a pipeline), the identification procedure executed by the artificial intelligence operation auxiliary platform 5 further includes confirming the readable value. For example, the inspector confirms the readable value by himself, and then inputs the readable value through the input element of the portable device 3 and sends it back to the artificial intelligence operation auxiliary platform 5. Alternatively, the inspector can also read this value through other instruments and then return it to the artificial intelligence operation Auxiliary platform 5. Or, the meter belongs to the Intelligent Internet of Things (AIoT) type, which automatically returns the readable value to the artificial intelligence operation auxiliary platform 5 through the high-speed communication network N every fixed cycle. Alternatively, the artificial intelligence operation assistance platform 5 directly recognizes the readable value from the environmental image captured by the camera module 32. In summary, this creation does not limit the way that the smart operation assistance platform 5 can confirm the readable value.

In another embodiment of step S2, before the artificial intelligence operation assistance platform 5 executes the recognition process to recognize the target object, it further includes recognizing the identifiable objects around the target object. For example, the artificial intelligence operation assistance platform 5 loads the 3D model database, and then executes the identification process based on the environment image and the 3D model database to find all the identifiable objects in the environment image. According to the distribution of these identifiable objects, The artificial intelligence operation auxiliary platform 5 compares the object distribution information stored in the three-dimensional model database to confirm the coordinate point of the current location of the inspector. If the artificial intelligence operation assistance platform 5 determines that there is the target object around the coordinate point where the inspector is located, the artificial intelligence operation assistance platform 5 can send a notification signal to instruct the inspector to look toward the location of the target object. Regarding the aforementioned three-dimensional model database, it includes one or more Computer Aided Design (CAD) files and multiple coordinate points. For example, the CAD file of the petrochemical industry plant can be used to create this 3D model database. In other embodiments, laser scanning or detailed construction drawings of a petrochemical plant can be used to reversely build a 3D model database of pipelines and other equipment. For example, a three-dimensional scanning instrument is used to obtain the spatial information of the pipeline or equipment surface, and the point cloud recognition software is used to convert it into a three-dimensional model with physical meaning. For another example, find out the coordinates of the start and end points or reference points of the pipeline according to the detailed construction drawings, and then semi-automatically build a 3D model database by the CAD software system. As mentioned above, after the inspectors are confirmed and their line of sight is facing the target object, the artificial intelligence operation assistance platform 5 executes the recognition process to recognize the target object according to the environment image and location information. Because the pipelines, equipment, and meters in the petrochemical plant are basically fixed, all equipment belonging to the location information can be screened out in advance through the above mechanism, so the time for the identification process to identify the target object can be reduced.

Please refer to step S3, the artificial intelligence operation auxiliary platform 5 according to the target object and The object database generates and sends operation signals. In detail, after confirming the target object in step S2, the artificial intelligence operation assistance platform 5 further outputs the operation signal applicable to the target object according to the target object and the object database. In one embodiment, the object database not only stores the images of all target objects, but also stores the target object state judgment model applicable to each target object. In practice, the artificial intelligence operation assistance platform 5 pre-collects a number of state parameters associated with the target object, such as the factory specifications of the target object, the numerical items that can be adjusted, and the display of the target object at different points in time. The status (such as temperature, pressure or meter readings, etc.). The artificial intelligence operation assistance platform 5 uses the collected state parameters associated with the target object as an output layer, and substitutes it into a neural network algorithm to generate a target object state judgment model, the output of which is the aforementioned operation parameter. The neural network-like algorithms are, for example, MultiLayer Perceptron (MLP), Long Short-Term Memory (LSTM), and Convolutional Neural Network (CNN).

In another embodiment, the artificial intelligence operation assistance platform 5 sets operation parameters according to readable values. For example, when the temperature of the device exceeds a first threshold, the operating valve associated with the temperature control on the device is rotated to a specified angle to adjust the temperature, or when the pressure gauge of the device is lower than a first threshold. At the second threshold, start the boost switch on the device. It should be noted that the artificial intelligence operation assistance platform 5 created by this invention can dynamically adjust the configuration values of the operating parameters according to the current state of the equipment, so that the operating signals are suitable for the current situation of the inspectors.

Please refer to step S4 and step S5. The portable device 3 receives and generates an operation layer according to the operation signal, and the display module 38 of the portable device 3 presents the operation layer. In detail, the processor 36 of the portable device 3 generates an operation layer according to the operation signal and the operation parameters therein. The operation layer is, for example, text messages or graphic messages. In other embodiments, the processor 36 further generates an operating voice according to the operating signal, and plays the voice through the speaker on the portable device 3.

In one embodiment, the display module 38 is a non-transparent display panel, which is used to present a superimposed image of the operation layer and the environment image. The environment image described here may be taken by the camera module 32. The captured real scene image, the operation layer is overlaid on the real scene image to form a superimposed image, so the effect of augmented reality (AR) can be achieved. The environment can also be a three-dimensional object image of the target object sent by the artificial intelligence operation assistance platform 5, and the operation layer is overlaid on the virtual image to form a superimposed image, so that virtual reality (VR) or mixed reality (MR) can be achieved. )Effect. In another embodiment, the display module 38 is a display panel with transparency, which is used for presenting a screen of the operation diagram stacked on the physical target object. In the above embodiment, the portable device 3 further includes an Inertial Measurement Unit (IMU), which is used to sense the line-of-sight angle of the inspector to the target object, and feed this angle back to the processor 36 . The processor 36 is further used to adjust the presentation type of the operation layer according to the viewing angle, so that the virtual image and the target object presented by the display module 38 can be presented in a better form.

Please refer to FIG. 4, which illustrates a schematic diagram of the inspector viewing the operation layer L of an embodiment of the present invention through the display module 38. As shown in Figure 4, the inspector can see the physical target object (including the pipeline 71, the meter 73, the ball valve 75 and the handle 77) through the display module 38 of the transparent lens, and can see the display module 38 to mix at the same time. Mixed Reality presents the operation layer L above the pipeline 71, the meter 73, the ball valve 75 and the handle 77 (the display module 38 presents this operation layer in a translucent form, for example), and then follows the instructions of the operation signal to move the handle 77 is rotated from the operation start position A to the operation end position B.

In summary, the intelligent operation assistance system proposed in this creation can improve the accuracy of equipment operation by inspectors and improve industrial safety. And according to the current status of the equipment, the operation instructions are dynamically adjusted, and the inspection information is collected and feedback, and the assets and equipment in the petrochemical plant are monitored and managed.

Although this creation is disclosed above with the aforementioned embodiments, it is not intended to limit this creation. Without departing from the spirit and scope of this creation, all changes and modifications made are within the scope of patent protection of this creation. For the scope of protection defined by this creation, please refer to the attached patent scope.

100: Smart operation assistance system

3: portable device

32: camera module

34: wireless communication module

36: processor

38: display module

5: Artificial intelligence operation auxiliary platform

N: High-speed mobile communication network

AP: wireless access point

Claims (5)

A smart operation assistance system includes: a portable device, including: a camera module for obtaining an environment image, the environment image has a target object; a wireless communication module for sending the environment image, And receives an operation signal; a processor electrically connected to the wireless communication module and the camera module, the processor controls the wireless communication module to send the environment image, and generates an operation layer according to the operation signal; and The display module is electrically connected to the processor, and the display module is used to present the operation layer; and an artificial intelligence operation auxiliary platform is communicatively connected to the wireless communication module, and the artificial intelligence operation auxiliary platform is used to receive the environmental image An identification process is executed to identify the target object, and the artificial intelligence operation assistance platform is used to generate and send the operation signal according to the target object and an object database. The operation of the auxiliary system according to the wisdom of a request for a high-speed network environment, the network environment includes a plurality of high-speed wireless access point, wireless access point of the plurality of fifth-generation mobile communication technology applications (5 ^th Generation Mobile networks, 5G) dedicated spectrum enterprise private networks are connected to each other, and the wireless communication module is connected to the artificial intelligence operation auxiliary platform through one of the wireless access points. The intelligent operation assistance system according to claim 1, wherein the target object has a readable value, the operation signal further includes an operation parameter, and the identification process is further used to identify the readable value, and the artificial intelligence operation The auxiliary platform is further used to set the operating parameter according to the readable value. The intelligent operation assistance system according to claim 3, wherein the artificial intelligence operation assistance platform is further used to collect a plurality of state parameters associated with the target object, and generate a target object based on the state parameters and a type of neural network algorithm The state judgment model, and the output of the target object state judgment model is the operating parameter. The intelligent operation assistance system according to claim 1, wherein the display module is further used to present a superimposed image of the operation layer and the environment image.

Images