CN110796738B - Three-dimensional visualization method and device for state tracking of inspection equipment - Google Patents
Three-dimensional visualization method and device for state tracking of inspection equipment Download PDFInfo
- Publication number
- CN110796738B CN110796738B CN201910898913.2A CN201910898913A CN110796738B CN 110796738 B CN110796738 B CN 110796738B CN 201910898913 A CN201910898913 A CN 201910898913A CN 110796738 B CN110796738 B CN 110796738B
- Authority
- CN
- China
- Prior art keywords
- target
- dimensional visualization
- inspection
- picture
- area
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
- G06T19/003—Navigation within 3D models or images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/695—Control of camera direction for changing a field of view, e.g. pan, tilt or based on tracking of objects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Abstract
The disclosure provides a three-dimensional visualization method and device for tracking the state of inspection equipment, electronic equipment and a computer readable storage medium, and relates to the technical field of inspection. The method comprises the following steps: controlling the inspection equipment to acquire a target picture of a target object, wherein the target picture comprises an abnormal area; in a three-dimensional visualization environment, matching the position and the orientation of a virtual agent of the inspection equipment to the inspection equipment based on state information of the inspection equipment; determining a target area pointed by the virtual agent in the three-dimensional visual environment based on the position and the orientation of the virtual agent; displaying the target picture near the target area in the three-dimensional visualization environment. The method and the device help improve inspection efficiency and inspection precision and reduce inspection cost.
Description
Technical Field
The disclosure relates to the technical field of inspection, in particular to a three-dimensional visualization method for state tracking of inspection equipment, a three-dimensional visualization device for state tracking of inspection equipment, electronic equipment and a computer readable storage medium.
Background
Three-dimensional digital visualization technology is increasingly applied in the fields of industrial manufacturing, architectural design, analog simulation and the like. The current three-dimensional digital visualization software mainly loads and displays a three-dimensional model of a target scene, and can perform certain interactive operation and view related data of the model.
With the increasingly strict requirements on industrial production safety, more timely and accurate inspection of production environments and production equipment are required. The appearance of automatic inspection equipment (robot, unmanned aerial vehicle) helps people more efficient completion to patrol and examine the work, has reduced the manpower requirement. With more and more inspection devices, inspection data are more and more abundant, and how to present inspection results in front of engineering personnel effectively in time so as to help engineering personnel know current production conditions and accurate management factories is a necessary work.
It should be noted that the information disclosed in the above background section is only for enhancing understanding of the background of the present disclosure and thus may include information that does not constitute prior art known to those of ordinary skill in the art.
Disclosure of Invention
The invention aims to provide a three-dimensional visualization method, a three-dimensional visualization device, an electronic device and a computer readable storage medium for tracking the state of inspection equipment, so that inspection results are timely reflected in a three-dimensional digital visualization environment.
Other features and advantages of the present disclosure will be apparent from the following detailed description, or may be learned in part by the practice of the disclosure.
According to one aspect of the present disclosure, there is provided a three-dimensional visualization method for status tracking of a patrol equipment, including:
controlling the inspection equipment to acquire a target picture of a target object, wherein the target picture comprises an abnormal area;
matching the position and orientation of a virtual agent of the patrol equipment to the patrol equipment based on the state information of the patrol equipment in a three-dimensional visualization environment;
determining a target area pointed by the virtual agent in the three-dimensional visualization environment based on the position and the orientation of the virtual agent;
displaying the target picture near the target area in the three-dimensional visualization environment.
In an exemplary embodiment of the disclosure, the controlling the inspection apparatus to acquire the target frame of the target object includes:
controlling a camera of the inspection equipment to acquire a first picture of a target object;
when an abnormal area exists in the first picture, adjusting the angle of the camera towards the target object so that the abnormal area is positioned at the center of the picture shot by the camera;
and controlling the camera to acquire a target picture of the target object.
In an exemplary embodiment of the present disclosure, the abnormal region being located at a center position of the camera photographing screen includes:
the center of the rectangular area where the abnormal area is located at the center of the picture shot by the camera.
In an exemplary embodiment of the present disclosure, the abnormal region being located at a center position of the camera photographing screen includes:
when the number of the abnormal areas is more than two, the common center of the centers of the rectangular areas where all the abnormal areas are located is located at the center position of the camera shooting picture.
In an exemplary embodiment of the present disclosure, the abnormal region being located at a center position of the camera photographing screen includes:
in the coordinate system where the camera shooting picture is located, the center coordinates of the rectangular area where each abnormal area is located are respectively:
A 0 (x 0 ，y 0 )，A 1 (x 1 ，y 1 )，A 2 (x 2 ，y 2 )……A n-1 (x n-1 ，y n-1 ) Wherein the number of the abnormal areas is N;
the coordinates of the common center of the centers of the rectangular areas where all the abnormal areas are located are:
C 0 ((x 0 +x 1 +x 2 +…+x n-1 )/N,(y 0 +y 1 +y 2 +…+y n-1 )/N))；
the center coordinate of the picture shot by the camera is C 1 Adjustment ofThe camera is oriented at an angle to the target object such that C 0 Equal to C 1 。
In one exemplary embodiment of the present disclosure, the status information of the inspection device includes a position of the inspection device, an orientation of a camera of the inspection device.
In one exemplary embodiment of the present disclosure, the determining the target area to which the virtual agent points in the three-dimensional visualization environment includes:
in the three-dimensional visualization environment, the virtual agent sends out an extension line forwards in the direction of the virtual agent, the extension line intersects with the surface of the virtual agent of the target object to form an intersection point, and the target area is an area containing the intersection point.
According to a second aspect of the present disclosure, there is provided a three-dimensional visualization apparatus for inspection equipment status tracking, comprising:
the target picture acquisition module is used for controlling the inspection equipment to acquire a target picture of a target object, wherein the target picture comprises an abnormal area;
the inspection state synchronization module is used for matching the position and the orientation of the virtual agent of the inspection equipment to the inspection equipment based on the state information of the inspection equipment in a three-dimensional visual environment;
the target area determining module is used for determining a target area pointed by the virtual agent in the three-dimensional visualization environment based on the position and the orientation of the virtual agent;
and the target picture display module is used for displaying the target picture near the target area in the three-dimensional visual environment.
According to a third aspect of the present disclosure, there is provided an electronic device comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the three-dimensional visualization method of inspection equipment status tracking of any of the above via execution of the executable instructions.
According to a fourth aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the three-dimensional visualization method of inspection device state tracking of any one of the above.
In the three-dimensional visualization method, the three-dimensional visualization device, the electronic device and the computer readable storage medium for tracking the state of the inspection device, which are provided by the embodiment of the invention, the inspection device can be controlled to acquire a target picture of a target object, wherein the target picture comprises an abnormal area, the orientation and the direction of a virtual agent of the inspection device are matched to the inspection device in a three-dimensional visualization environment, and the target picture is displayed near the target area pointed by the virtual agent. According to the method and the device, on one hand, the target picture containing the abnormal region of the inspection target object can be automatically acquired, so that an accurate inspection result is obtained, inspection precision is improved, and manual operation is saved; on the other hand, the inspection result can be matched to a corresponding target area in the three-dimensional visual environment and displayed, so that engineering personnel can conveniently and accurately position the inspection result and check the inspection result, the inspection efficiency is improved, and the inspection cost is reduced.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the disclosure and together with the description, serve to explain the principles of the disclosure. It will be apparent to those of ordinary skill in the art that the drawings in the following description are merely examples of the disclosure and that other drawings may be derived from them without undue effort.
FIG. 1 illustrates a schematic diagram of an exemplary system architecture to which a method and apparatus for three-dimensional visualization of inspection device status tracking of embodiments of the present disclosure may be applied;
FIG. 2 illustrates a schematic diagram of a computer system suitable for use in implementing embodiments of the present disclosure;
FIG. 3 schematically illustrates a flow chart of a three-dimensional visualization method of inspection device status tracking, according to one embodiment of the disclosure;
FIG. 4 schematically illustrates a three-dimensional visual simulated environment schematic of inspection device state tracking in an exemplary embodiment of the present disclosure;
fig. 5 and 6 schematically illustrate process diagrams for adjusting a camera of a patrol equipment to acquire a target screen in an exemplary embodiment of the present disclosure;
FIG. 7 schematically illustrates a diagram of determining a target area to which a virtual agent is directed in a three-dimensional visualization environment in an exemplary embodiment of the present disclosure;
fig. 8 schematically illustrates a block diagram of a three-dimensional visualization apparatus for inspection equipment status tracking in an exemplary embodiment of the present disclosure.
Detailed Description
Example embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments may be embodied in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of embodiments of the present disclosure. One skilled in the relevant art will recognize, however, that the aspects of the disclosure may be practiced without one or more of the specific details, or with other methods, components, devices, steps, etc. In other instances, well-known technical solutions have not been shown or described in detail to avoid obscuring aspects of the present disclosure.
Furthermore, the drawings are merely schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and thus a repetitive description thereof will be omitted. Some of the block diagrams shown in the figures are functional entities and do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software or in one or more hardware modules or integrated circuits or in different networks and/or processor devices and/or microcontroller devices.
Fig. 1 shows a schematic diagram of an exemplary system architecture to which a method and apparatus for three-dimensional visualization of inspection device status tracking of embodiments of the present disclosure may be applied.
As shown in fig. 1, the system architecture 100 may include one or more of the terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others. The terminal devices 101, 102, 103 may be various electronic devices with display screens including, but not limited to, desktop computers, portable computers, smart phones, tablet computers, and the like. It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation. For example, the server 105 may be a server cluster formed by a plurality of servers.
The three-dimensional visualization method for tracking the state of the inspection equipment provided by the embodiments of the present disclosure is generally executed by the server 105, and accordingly, the three-dimensional visualization device for tracking the state of the inspection equipment is generally disposed in the server 105. However, it is easily understood by those skilled in the art that the three-dimensional visualization method for tracking the state of the inspection device provided in the embodiment of the present disclosure may also be performed by the terminal devices 101, 102, 103, and accordingly, the three-dimensional visualization apparatus for tracking the state of the inspection device may also be provided in the terminal devices 101, 102, 103, which is not particularly limited in the present exemplary embodiment.
Fig. 2 shows a schematic diagram of a computer system suitable for use in implementing embodiments of the present disclosure.
It should be noted that the computer system 200 of the electronic device shown in fig. 2 is only an example, and should not impose any limitation on the functions and the application scope of the embodiments of the present disclosure.
As shown in fig. 2, the computer system 200 includes a Central Processing Unit (CPU) 201, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 202 or a program loaded from a storage section 208 into a Random Access Memory (RAM) 203. In the RAM 203, various programs and data required for the system operation are also stored. The CPU 201, ROM 202, and RAM 203 are connected to each other through a bus 204. An input/output (I/O) interface 205 is also connected to bus 204.
The following components are connected to the I/O interface 205: an input section 206 including a keyboard, a mouse, and the like; an output portion 207 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker, and the like; a storage section 208 including a hard disk or the like; and a communication section 209 including a network interface card such as a LAN card, a modem, and the like. The communication section 209 performs communication processing via a network such as the internet. The drive 210 is also connected to the I/O interface 205 as needed. A removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed on the drive 210 as needed, so that a computer program read out therefrom is installed into the storage section 208 as needed.
In particular, according to embodiments of the present disclosure, the processes described below with reference to flowcharts may be implemented as computer software programs. For example, embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method shown in the flowcharts. In such an embodiment, the computer program may be downloaded and installed from a network via the communication portion 209, and/or installed from the removable medium 211. The computer program, when executed by a Central Processing Unit (CPU) 201, performs the various functions defined in the methods and apparatus of the present application.
It should be noted that the computer readable medium shown in the present disclosure may be a computer readable signal medium or a computer readable storage medium, or any combination of the two. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, however, the computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, with the computer-readable program code embodied therein. Such a propagated data signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination of the foregoing. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, fiber optic cable, RF, etc., or any suitable combination of the foregoing.
The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.
The units involved in the embodiments of the present disclosure may be implemented by means of software, or may be implemented by means of hardware, and the described units may also be provided in a processor. Wherein the names of the units do not constitute a limitation of the units themselves in some cases.
As another aspect, the present application also provides a computer-readable storage medium that may be included in the electronic device described in the above embodiments; or may exist alone without being incorporated into the electronic device. The computer-readable storage medium carries one or more programs which, when executed by the electronic device, cause the electronic device to implement the methods described in the embodiments below. For example, the electronic device may implement the steps shown in fig. 3-7, etc.
The following describes the technical scheme of the embodiments of the present disclosure in detail:
the embodiment firstly provides a three-dimensional visualization method for tracking the state of the inspection equipment. The three-dimensional visualization method for tracking the state of the inspection device may be applied to the server 105, or may be applied to one or more of the terminal devices 101, 102, 103, which is not particularly limited in this exemplary embodiment.
Fig. 3 schematically illustrates a flow chart of a three-dimensional visualization method of inspection device status tracking, according to one embodiment of the disclosure.
Referring to fig. 3, the three-dimensional visualization method for tracking the state of the inspection equipment may include the following steps:
s310, controlling the inspection equipment to acquire a target picture of a target object, wherein the target picture comprises an abnormal area;
s320, matching the position and the orientation of a virtual agent of the inspection equipment to the inspection equipment based on the state information of the inspection equipment in a three-dimensional visual environment;
s330, determining a target area pointed by the virtual agent in the three-dimensional visualization environment based on the position and the orientation of the virtual agent;
and S340, displaying the target picture near the target area in the three-dimensional visual environment.
In the three-dimensional visualization method, the three-dimensional visualization device, the electronic device and the computer readable storage medium for tracking the state of the inspection device, which are provided by the embodiment of the invention, the inspection device can be controlled to acquire a target picture of a target object, wherein the target picture comprises an abnormal area, the orientation and the direction of a virtual agent of the inspection device are matched to the inspection device in a three-dimensional visualization environment, and the target picture is displayed near the target area pointed by the virtual agent.
According to the method and the device, on one hand, the target picture containing the abnormal region of the inspection target object can be automatically acquired, so that an accurate inspection result is obtained, inspection precision is improved, and manual operation is saved; on the other hand, the inspection result can be matched to a corresponding target area in the three-dimensional visual environment and displayed, so that engineering personnel can conveniently and accurately position the inspection result and check the inspection result, the inspection efficiency is improved, and the inspection cost is reduced.
Typically, the inspection device may effect the movement of the location automatically or under personnel control. The inspection device may include an unmanned aerial vehicle, a ground-operated robot, or the like. The inspection device may be equipped with a positioning system so that the system monitors the position, movement status, etc. of the inspection device at any time. The inspection device may also be equipped with various sensors, such as vision sensors, thermal sensors, smoke sensors, etc., to perform the function of data acquisition.
In the present exemplary embodiment, in order to obtain a target screen of inspection, the inspection apparatus may be equipped with a camera for photographing an inspection target screen, and the camera may be rotated with respect to the inspection apparatus to achieve a change in photographing angle of view.
In addition, the inspection device may be configured with a calculation unit to analyze and identify the collected data (pictures, temperatures, etc.), thereby giving inspection results. The data collected by the inspection can be uploaded to a server, and the server can complete the task of analysis and identification and give out the inspection result.
In this example embodiment, the three-dimensional visualization system may track the status of the inspection apparatus in real time and feedback control the inspection apparatus. Wherein, the inspection equipment state may include: the system state, motion state, azimuth and other self state information of the inspection equipment can also comprise the defect state of the detected object in the inspection environment and the like.
FIG. 4 schematically illustrates a three-dimensional visual simulated environment schematic of inspection device state tracking in an exemplary embodiment of the present disclosure; fig. 5 and 6 schematically illustrate process diagrams for adjusting a camera of a patrol equipment to acquire a target screen in an exemplary embodiment of the present disclosure; fig. 7 schematically illustrates a schematic diagram of determining a target area to which a virtual agent points in a three-dimensional visualization environment in an exemplary embodiment of the present disclosure.
Next, the shelf scheduling method of the present exemplary embodiment will be described in more detail with reference to fig. 3 to 7.
In step S310, the inspection apparatus is controlled to acquire a target screen of a target object, the target screen including an abnormal region therein.
First, a three-dimensional visual simulation environment to which the method in the present exemplary embodiment is applied will be described. Referring to fig. 4, 401 is an operation command window of three-dimensional visualization software, 402 is a three-dimensional simulation area, 403, 404, 405 are virtual objects in a simulation environment, which represent objects in a real environment, and have various attribute data preset for the real environment objects. 406 is a virtual proxy for the inspection device in a three-dimensional visual simulation environment. 408 is a target picture acquired by the inspection device, and the target picture includes an abnormal area.
It should be added that, in the three-dimensional visualization system, the system can track the state of the inspection equipment in real time, and control the position of the virtual agent 406 according to the position information in the state of the equipment, and specifically, the position and the orientation of the virtual agent 406 can be included. The direction of the virtual agent 406 may be the direction of the image shot by the camera.
In this example embodiment, controlling the inspection apparatus to acquire the target screen of the target object may include:
controlling a camera of the inspection equipment to acquire a first picture of a target object;
when an abnormal area exists in the first picture, adjusting the angle of the camera towards the target object so that the abnormal area is positioned at the center of the picture shot by the camera;
and controlling the camera to acquire a target picture of the target object.
The target object may be various objects or personnel in the inspection environment, such as factory buildings, equipment, vehicles, articles, staff and external suspicious personnel. The first frame may be a frame of the inspection device capturing a picture of a target area of a preset target object when the inspection task is performed, or may be a frame of the inspection device capturing a picture of a non-target area of the target object when the inspection task is performed, or capturing a picture of a non-target object when the suspicious situation is accidentally found, which is not particularly limited in this disclosure with respect to the first frame.
Further, after the camera shoots the first picture, if the abnormal area is found in the first picture through preliminary identification analysis, the angle of the camera towards the target object can be adjusted so that the abnormal area is positioned at the center of the camera shooting picture, and the camera is controlled to acquire the target picture of the target object.
Referring to fig. 5 and 6, a specific method for adjusting an angle of a camera towards a target object to obtain a target image may include:
the camera photographs the first screen 508, finds that two abnormal areas exist through preliminary recognition analysis of the first screen 508, and frames each abnormal area, such as area a and area B, with a rectangular frame that just wraps each abnormal area.
The center coordinates of the region a and the region B are located at the coordinate positions on the coordinate system where the screen is located, and may be set to a (x 0 ，y 0 ) And B (x) 1 ，y 1 ) The common center coordinate of the two regions may be C 0 ：((x 0 +x 1 )/2,(y 0 +y 1 )/2)). Further, the center coordinate of the photographed picture may be set to C 1 。
It is possible to determine the common center coordinate C of the two abnormal regions 0 With the center coordinate C of the picture 1 And judging whether the abnormal areas are positioned at the center of the picture shot by the camera or not.
In one case, if C 0 With the center coordinate C of the picture 1 And if the images are equal, determining that the abnormal area is positioned at the center of the image shot by the camera, and taking the current image as the target image.
In another case, if C 0 With the center coordinate C of the picture 1 If the images are not equal, the shooting angle of the camera can be continuously adjusted, and the coordinates of the corresponding abnormal areas in the picture also correspondingly change, namely A (x 0 ，y 0 ) And B (x) 1 ，y 1 ) Where the coordinates of (C) change 0 Will also change until C 0 And C 1 Equal, i.e. C 0 ＝C 1 Even if all the abnormal areas are positioned at the center of the shooting picture of the camera, the shooting picture of the target object is obtained by controlling the camera, namely the target picture.
Further, in the process of adjusting the camera angle, along with the movement of the picture, the camera can acquire the current first picture according to a certain frequency, and the system can perform real-time identification analysis on the current first picture so as to judge whether an abnormal area exists in the picture. And when the abnormal area exists, calculating a common center coordinate of the center coordinates of the rectangular areas where all the abnormal areas are located, and judging whether the common center coordinate is equal to the center coordinate of the shot picture.
Furthermore, in some embodiments, as the frame moves, new outliers may appear in the frame and the outliers may leave the frame. The process of correcting the center of the abnormal region will be described below by taking the generation of a new abnormal region as an example:
for the first screen 508 in fig. 5, when a new abnormal region C is found to enter the screen at the time of camera angle adjustment, the coordinates of the region C may be set to C (x 2 ，y 2 ) The common center coordinate of the three regions may be C 0 ：((x 0 +x 1 +x 2 )/3,(y 0 +y 1 +y 2 ) (3)) continuing to adjust the camera angle until C 0 And C 1 Equal to each other, thereby obtaining a picture as in fig. 6, and a picture 608 is a desired target picture.
Further, when N abnormal areas occur in the screen, the central coordinates of the rectangular area where each abnormal area is located may be respectively: a is that 0 (x 0 ，y0)，A 1 (x 1 ，y1)，A 2 (x 2 ，y 2 )……A n-1 (x n-1 ，y n-1 ) The coordinate of the common center of the centers of the rectangular areas where all the abnormal areas are located may be C 0 ((x 0 +x 1 +x 2 +…+x n-1 )/N,(y 0 +y 1 +y 2 +…+y n-1 ) N), the camera is adjusted to angle towards the target object, C 0 Equal to C 1 。
In step S320, in a three-dimensional visualization environment, the position and orientation of the virtual agent of the patrol equipment are matched to the patrol equipment based on the status information of the patrol equipment.
In this step, in order to match the obtained target frame to a corresponding position in the three-dimensional visual environment, it is necessary to first determine the orientation and direction of the virtual agent of the inspection apparatus in the three-dimensional visual environment, and then determine the position of the target frame.
The state information of the inspection equipment can include the position of the inspection equipment, the orientation of a camera of the inspection equipment, the system state and the motion state of the inspection equipment, the defect state of a detected target in the inspection environment and the like.
After determining the status information of the inspection device currently acquiring the target picture, the azimuth and the orientation of the virtual agent 406 of the inspection device may be adjusted based on the status information. In particular, the location of the patrol device, the orientation of the patrol device camera, along with other information of the patrol device, may be transferred to a virtual proxy 406 in a three-dimensional visual simulated environment, where the virtual proxy 406 adjusts its location and orientation to match the patrol device. The method for matching the azimuth information of the virtual agent and the real inspection device can refer to the matching process of the virtual agent and the automobile in the application of the navigation map, and the disclosure is not limited in particular.
In step S330, determining a target area to which the virtual agent points in the three-dimensional visualization environment based on the position and orientation of the virtual agent;
after determining the position and orientation of the virtual agent 406 of the inspection device, in order to determine the position where the target frames need to be matched in the three-dimensional visualization environment, that is, to determine which area of which object is in the three-dimensional visualization environment is the area of the target frame photographed by the camera, the following method may be further adopted in this exemplary embodiment:
in the three-dimensional visualization environment, referring to fig. 7, the virtual agent 406 sends out an extension line 407 forward in the direction of the virtual agent, and the extension line 407 intersects with the surface of the virtual agent 404 of the target object to form an intersection T, so that the target area may be an area including the intersection T.
Specifically, in a three-dimensional visual simulation environment, virtual ray 407 may perform collision detection with an object in the simulation environment. First, a rectangular parallelepiped bounding box may be provided for each object in a virtual environment, such as 403, 404, 405, etc. shown in fig. 4, 7. The virtual ray 407 collides with the bounding box of the object in the simulation environment for the first time in the emitting direction, and the formed intersection point T can be confirmed as the target point. The region containing the intersection point T shown may be identified as the target region to which the virtual agent 406 is directed in the three-dimensional visualization environment. Wherein the range size of the target area can be preset.
In step S340, the target screen is displayed in the vicinity of the target area in the three-dimensional visualization environment.
Further, as shown in fig. 4 and 7, the target screen 408 may be displayed near a target area in the simulation environment. The target screen may be the target screen as obtained in step S310. In addition, for convenient viewing, the display position of the target picture can be as close to the target area as possible in the three-dimensional visualization environment on the premise of not shielding the target point or the target area. The display position of the target picture can be finely adjusted along with the change of the visual angle of the display picture of the current virtual environment so as to adapt to the observation habit of human eyes.
It should be added that the target picture can be displayed in a marking frame, and the marking frame can be rectangular or other shapes. The identification frame may display a target image obtained by the real inspection device, and may also display other types of image images after performing image processing on the target image.
In summary, according to the three-dimensional visualization method for tracking the state of the inspection equipment provided by the present disclosure, the inspection equipment can be controlled to accurately acquire the target picture including the abnormal region, and the target picture is displayed at the corresponding position in the three-dimensional visualization environment. Through the visual display mode, a user can clearly and real-timely see the actual detection result of the inspection equipment in the three-dimensional visual environment, the inspection efficiency and the inspection precision are improved, the inspection cost is reduced, and the production safety is improved.
The disclosure also provides a three-dimensional visualization device for tracking the state of the inspection equipment. As shown in fig. 8, the three-dimensional visualization apparatus 800 for status tracking of the inspection device may include:
a target picture obtaining module 810, configured to control the inspection device to obtain a target picture of a target object, where the target picture includes an abnormal area;
a patrol status synchronization module 820 configured to match, in a three-dimensional visualization environment, a position and an orientation of a virtual agent of the patrol equipment to the patrol equipment based on status information of the patrol equipment;
a target area determining module 830, configured to determine, based on the position and orientation of the virtual agent, a target area to which the virtual agent points in the three-dimensional visualization environment;
a target screen display module 840 for displaying the target screen in the vicinity of the target area in the three-dimensional visualization environment.
It should be noted that, the specific details of each module in the three-dimensional visualization device for tracking the state of the inspection equipment are described in detail in the corresponding three-dimensional visualization method for tracking the state of the inspection equipment, so that the details are not repeated here.
It should be noted that although in the above detailed description several modules or units of a device for action execution are mentioned, such a division is not mandatory. Indeed, the features and functionality of two or more modules or units described above may be embodied in one module or unit in accordance with embodiments of the present disclosure. Conversely, the features and functions of one module or unit described above may be further divided into a plurality of modules or units to be embodied.
Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any adaptations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice within the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.
It is to be understood that the present disclosure is not limited to the precise arrangements and instrumentalities shown in the drawings, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims (9)
1. The three-dimensional visualization method for tracking the state of the inspection equipment is characterized by comprising the following steps of:
controlling the inspection equipment to acquire a target picture of a target object, wherein the target picture comprises an abnormal area;
in a three-dimensional visualization environment, determining the position and the orientation of a virtual agent of the inspection equipment based on the state information of the inspection equipment;
determining a target area pointed by the virtual agent in the three-dimensional visualization environment based on the position and the orientation of the virtual agent;
displaying the target screen in the vicinity of the target area in the three-dimensional visualization environment;
wherein said determining a target area to which said virtual agent is directed in said three-dimensional visualization environment comprises: in the three-dimensional visualization environment, the virtual agent sends out an extension line forwards in the direction of the virtual agent, the extension line intersects with the surface of the virtual agent of the target object to form an intersection point, and the target area is an area containing the intersection point.
2. The method for three-dimensional visualization of status tracking of a patrol apparatus according to claim 1, wherein said controlling the patrol apparatus to acquire a target picture of a target object comprises:
controlling a camera of the inspection equipment to acquire a first picture of a target object;
when an abnormal area exists in the first picture, adjusting the angle of the camera towards the target object so that the abnormal area is positioned at the center of the picture shot by the camera;
and controlling the camera to acquire a target picture of the target object.
3. The three-dimensional visualization method for tracking the status of a patrol equipment according to claim 2, wherein the positioning of the abnormal area at the center of the picture shot by the camera comprises:
the center of the rectangular area where the abnormal area is located at the center of the picture shot by the camera.
4. The three-dimensional visualization method for tracking the status of a patrol equipment according to claim 2, wherein the positioning of the abnormal area at the center of the picture shot by the camera comprises:
when the number of the abnormal areas is more than two, the common center of the centers of the rectangular areas where all the abnormal areas are located is located at the center position of the camera shooting picture.
5. The three-dimensional visualization method for tracking a status of a patrol equipment according to claim 3 or 4, wherein the positioning of the abnormal area at the center of the camera shot comprises:
in the coordinate system where the camera shooting picture is located, the center coordinates of the rectangular area where each abnormal area is located are respectively:
A 0 (x 0 ，y 0 )，A 1 (x 1 ，y 1 )，A 2 (x 2 ，y 2 )……A n-1 (x n-1 ，y n-1 ) Wherein the number of the abnormal areas is N;
the coordinates of the common center of the centers of the rectangular areas where all the abnormal areas are located are:
C 0 ((x 0 +x 1 +x 2 +…+x n-1 )/N,(y 0 +y 1 +y 2 +…+y n-1 )/N))；
the center coordinate of the picture shot by the camera is C 1 Adjusting the angle of the camera towards the target object to enable C 0 Equal to C 1 。
6. The method of three-dimensional visualization of inspection equipment status tracking of claim 5, wherein the inspection equipment status information includes a location of the inspection equipment, an orientation of a camera of the inspection equipment.
7. A three-dimensional visualization device for tracking the status of inspection equipment, comprising:
the target picture acquisition module is used for controlling the inspection equipment to acquire a target picture of a target object, wherein the target picture comprises an abnormal area;
the inspection state synchronization module is used for determining the position and the orientation of the virtual agent of the inspection equipment based on the state information of the inspection equipment in a three-dimensional visual environment;
the target area determining module is used for determining a target area pointed by the virtual agent in the three-dimensional visualization environment based on the position and the orientation of the virtual agent;
a target screen display module for displaying the target screen in the vicinity of the target area in the three-dimensional visualization environment;
wherein said determining a target area to which said virtual agent is directed in said three-dimensional visualization environment comprises: in the three-dimensional visualization environment, the virtual agent sends out an extension line forwards in the direction of the virtual agent, the extension line intersects with the surface of the virtual agent of the target object to form an intersection point, and the target area is an area containing the intersection point.
8. An electronic device, comprising:
a processor; and
a memory for storing executable instructions of the processor;
wherein the processor is configured to perform the three-dimensional visualization method of inspection device status tracking of any of claims 1-6 via execution of the executable instructions.
9. A computer readable storage medium having stored thereon a computer program, wherein the computer program when executed by a processor implements the three-dimensional visualization method of inspection device status tracking of any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910898913.2A CN110796738B (en) | 2019-09-23 | 2019-09-23 | Three-dimensional visualization method and device for state tracking of inspection equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910898913.2A CN110796738B (en) | 2019-09-23 | 2019-09-23 | Three-dimensional visualization method and device for state tracking of inspection equipment |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110796738A CN110796738A (en) | 2020-02-14 |
| CN110796738B true CN110796738B (en) | 2023-04-25 |
Family
ID=69439832
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910898913.2A Active CN110796738B (en) | 2019-09-23 | 2019-09-23 | Three-dimensional visualization method and device for state tracking of inspection equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110796738B (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111447416B (en) * | 2020-04-10 | 2022-06-21 | 北京旷视机器人技术有限公司 | Display method and device of monitoring video and electronic system |
| CN112669487A (en) * | 2020-12-21 | 2021-04-16 | 北京佳讯飞鸿电气股份有限公司 | Target tracking method and inspection robot |
| CN112866579B (en) * | 2021-02-08 | 2022-07-01 | 上海巡智科技有限公司 | Data acquisition method and device and readable storage medium |
| CN113310407A (en) * | 2021-07-02 | 2021-08-27 | 四川华能宝兴河水电有限责任公司 | Dynamic monitoring method for flood discharge gate of high dam reservoir hydropower station |
Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001054099A (en) * | 1999-08-10 | 2001-02-23 | Secom Co Ltd | Image monitoring system |
| JP2005323229A (en) * | 2004-05-11 | 2005-11-17 | Yamatake Corp | Patrol method and patrol system |
| JP2006338187A (en) * | 2005-05-31 | 2006-12-14 | Secom Co Ltd | Monitoring device |
| CN105259899A (en) * | 2015-12-01 | 2016-01-20 | 国网重庆市电力公司电力科学研究院 | Control system for transformer substation patrol robot |
| CN106909169A (en) * | 2017-03-30 | 2017-06-30 | 广东容祺智能科技有限公司 | A kind of full automatic power patrol UAV system |
| JP2018088065A (en) * | 2016-11-28 | 2018-06-07 | 株式会社Ｎｔｔファシリティーズ | Information visualization system, information visualization method, and program |
| CN109141647A (en) * | 2018-08-01 | 2019-01-04 | 北京百度网讯科技有限公司 | Computer room temperature monitoring method and device |
| CN109190921A (en) * | 2018-08-10 | 2019-01-11 | 广东电网有限责任公司 | A kind of monitoring information treating method and apparatus |
| CN110139037A (en) * | 2019-05-20 | 2019-08-16 | 深圳铂石空间科技有限公司 | Object monitoring method and device, storage medium and electronic equipment |
| CN110188643A (en) * | 2019-05-21 | 2019-08-30 | 北京市商汤科技开发有限公司 | A kind of information display method and device, storage medium |
| CN110223270A (en) * | 2019-04-26 | 2019-09-10 | 五邑大学 | A method of it is positioned using GEOGRAPHICAL INDICATION photo and three-dimensional visualization |
| CN110223413A (en) * | 2019-06-12 | 2019-09-10 | 深圳铂石空间科技有限公司 | Intelligent polling method, device, computer storage medium and electronic equipment |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160328887A1 (en) * | 2015-05-04 | 2016-11-10 | The Trustees Of Columbia University In The City Of New York | Systems and methods for providing assistance for manipulating objects using virtual proxies and virtual replicas |
| US9892558B2 (en) * | 2016-02-19 | 2018-02-13 | The Boeing Company | Methods for localization using geotagged photographs and three-dimensional visualization |
| US10261747B2 (en) * | 2016-09-09 | 2019-04-16 | The Boeing Company | Synchronized side-by-side display of live video and corresponding virtual environment images |
-
2019
- 2019-09-23 CN CN201910898913.2A patent/CN110796738B/en active Active
Patent Citations (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2001054099A (en) * | 1999-08-10 | 2001-02-23 | Secom Co Ltd | Image monitoring system |
| JP2005323229A (en) * | 2004-05-11 | 2005-11-17 | Yamatake Corp | Patrol method and patrol system |
| JP2006338187A (en) * | 2005-05-31 | 2006-12-14 | Secom Co Ltd | Monitoring device |
| CN105259899A (en) * | 2015-12-01 | 2016-01-20 | 国网重庆市电力公司电力科学研究院 | Control system for transformer substation patrol robot |
| JP2018088065A (en) * | 2016-11-28 | 2018-06-07 | 株式会社Ｎｔｔファシリティーズ | Information visualization system, information visualization method, and program |
| CN106909169A (en) * | 2017-03-30 | 2017-06-30 | 广东容祺智能科技有限公司 | A kind of full automatic power patrol UAV system |
| CN109141647A (en) * | 2018-08-01 | 2019-01-04 | 北京百度网讯科技有限公司 | Computer room temperature monitoring method and device |
| CN109190921A (en) * | 2018-08-10 | 2019-01-11 | 广东电网有限责任公司 | A kind of monitoring information treating method and apparatus |
| CN110223270A (en) * | 2019-04-26 | 2019-09-10 | 五邑大学 | A method of it is positioned using GEOGRAPHICAL INDICATION photo and three-dimensional visualization |
| CN110139037A (en) * | 2019-05-20 | 2019-08-16 | 深圳铂石空间科技有限公司 | Object monitoring method and device, storage medium and electronic equipment |
| CN110188643A (en) * | 2019-05-21 | 2019-08-30 | 北京市商汤科技开发有限公司 | A kind of information display method and device, storage medium |
| CN110223413A (en) * | 2019-06-12 | 2019-09-10 | 深圳铂石空间科技有限公司 | Intelligent polling method, device, computer storage medium and electronic equipment |
Non-Patent Citations (3)
| Title |
|---|
| Application and realization of power line panorama based on the geographic information;Song haihua 等;《Power System Technology》;第32卷(第7期);全文 * |
| 三维可视化技术在水电站运维管理中的应用;李谧 等;《电子技术与软件工程》;第125卷(第3期);全文 * |
| 机动卫勤分队三维可视化战备物资管理系统的设计;郭栋等;《医疗卫生装备》(第09期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110796738A (en) | 2020-02-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110796738B (en) | Three-dimensional visualization method and device for state tracking of inspection equipment | |
| CN110162083B (en) | Method and system for executing three-dimensional structure inspection and maintenance tasks by using unmanned aerial vehicle | |
| WO2022120567A1 (en) | Automatic calibration system based on visual guidance | |
| CN112652016B (en) | Point cloud prediction model generation method, pose estimation method and pose estimation device | |
| CN108363995B (en) | Method and apparatus for generating data | |
| CN109032348B (en) | Intelligent manufacturing method and equipment based on augmented reality | |
| US10438412B2 (en) | Techniques to facilitate accurate real and virtual object positioning in displayed scenes | |
| US11713970B2 (en) | Positioning method, electronic device and computer readable storage medium | |
| CN111766951B (en) | Image display method and apparatus, computer system, and computer-readable storage medium | |
| CN112549034B (en) | Robot task deployment method, system, equipment and storage medium | |
| CN110555838A (en) | Image-based part fault detection method and device | |
| Ziaei et al. | Real-time markerless Augmented Reality for Remote Handling system in bad viewing conditions | |
| CN112288825A (en) | Camera calibration method and device, electronic equipment, storage medium and road side equipment | |
| CN113272871A (en) | Camera calibration method and system | |
| CN107316272A (en) | Method and its equipment for image procossing | |
| CN113870439A (en) | Method, apparatus, device and storage medium for processing image | |
| CN111753739A (en) | Object detection method, device, equipment and storage medium | |
| Ren et al. | Camera calibration with pose guidance | |
| CN110458108A (en) | Method for real-time monitoring, system, terminal device and storage medium hand-manipulated | |
| CN115719436A (en) | Model training method, target detection method, device, equipment and storage medium | |
| CN115797406A (en) | Out-of-range warning method, device, equipment and storage medium | |
| CN113808186B (en) | Training data generation method and device and electronic equipment | |
| CN115294207A (en) | Fusion scheduling system and method for smart campus monitoring video and three-dimensional GIS model | |
| CN116136408A (en) | Indoor navigation method, server, device and terminal | |
| Huang et al. | Design and application of intelligent patrol system based on virtual reality |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |