WO2023131124A1

WO2023131124A1 - Virtual interaction method, apparatus and system for work machine and work environment

Info

Publication number: WO2023131124A1
Application number: PCT/CN2023/070137
Authority: WO
Inventors: 胡立辛
Original assignee: 上海三一重机股份有限公司
Priority date: 2022-01-04
Filing date: 2023-01-03
Publication date: 2023-07-13
Also published as: CN114327076A

Abstract

Provided in the present application are a virtual interaction method, apparatus and system for a work machine and a work environment. The method comprises: acquiring an open degree proportion signal of a control assembly; determining motion state data of a controlled work machine; respectively determining, according to the motion state data and pre-constructed virtual environment information, the state of interaction between the controlled work machine and each spatial element in the virtual environment information, and generating scenario update information according to the state of interaction; and finally, intercepting corresponding scenario image information from the scenario update information, according to an actual angle-of-view range of a user, and outputting same. Therefore, an interaction process between a work machine and a work environment in a virtual scenario is realized. The process can be realized by means of a virtual environment, thereby overcoming the problem of the interaction process being labour-intensive and time-consuming due to the limitations of an actual application environment in a scenario in which the work machine is operated and controlled to interact with an application environment thereof.

Description

Virtual interaction method, device and system for operating machinery and operating environment

Cross References to Related Applications

This application claims the priority of the Chinese patent application with the application number 202210002264.5 filed on January 04, 2022, and the title of the invention is "virtual interaction method, device and system for operating machinery and operating environment", which is incorporated by reference in its entirety This article.

technical field

The present application relates to the field of virtual reality technology, in particular to a method, device and system for virtual interaction between an operating machine and an operating environment.

Background technique

Considering that the application environment of the operating machine is mainly a complex construction operation scene, in the scene involving the interaction between the operating machine and the application environment, the interaction process is often time-consuming and labor-intensive due to the limitations of the application environment, and it is difficult to meet the actual needs.

For example, in the control performance test scene of operating machinery, it generally needs to be completed under various working conditions. Considering the actual test site is limited, one site can only test one working condition environment. When testing multiple working conditions environments, it is necessary to Transferring the operating machinery to be tested to different test sites leads to time-consuming and labor-intensive testing and low testing efficiency.

Contents of the invention

The present application provides a method, device and system for virtual interaction between an operating machine and an operating environment, which are used to solve the problem of time-consuming interactions in the interaction process often caused by the limitations of the application environment in the prior art involving the interaction between the operating machine and its application environment. Time-consuming and labor-intensive, it is difficult to meet the defects of actual needs.

In a first aspect, the present application provides a virtual interaction method between an operating machine and an operating environment, the method comprising:

Obtain the opening proportional signal of the control component;

Determine the motion state data of the controlled working machine according to the opening ratio signal;

According to the motion state data and the pre-constructed virtual environment information, respectively determine the interaction state between the controlled working machine and each space element in the virtual environment information, and generate scene update information according to the interaction state;

The user's actual viewing angle range is acquired, and corresponding scene image information is intercepted from the scene update information and output according to the user's actual viewing angle range.

According to a virtual interaction method between an operating machine and an operating environment provided in the present application, the distance between the controlled operating machine and each spatial element in the virtual environment information is respectively determined according to the motion state data and the pre-constructed virtual environment information. interaction state, and generate scene update information according to the interaction state, including:

According to the motion state data and the pre-constructed virtual environment information, when it is determined that the controlled working machine is in contact with at least one spatial element in the virtual environment information, determine the interaction type of the spatial element in contact;

Determine scene change information according to the interaction category and the motion state data of the controlled working machine;

The virtual environment information is updated in real time according to the scene change information to generate scene update information.

According to a virtual interaction method between an operating machine and an operating environment provided by the present application, the interaction categories of the space elements include interactable categories and non-interactive categories.

According to a virtual interaction method between an operating machine and an operating environment provided in the present application, the scene change information is determined according to the interaction category and the motion state data of the controlled operating machine, including:

When the interaction category of the space element in contact is interactive, determine the deformation state of the space element according to the motion state data of the controlled working machine, and compare the deformation state and the The motion state data of the controlled operating machine is used as scene change information;

When the interaction type of the space element in contact is non-interactive, the motion state data of the controlled working machine at the current moment is used as the scene change information.

According to a virtual interaction method between an operating machine and an operating environment provided by the present application, the actual viewing angle range of the user is obtained, including:

Obtain the rotation center coordinates and rotation angle of the controlled operating machine;

Determine the front head-up direction vector of the cab of the controlled working machine according to the coordinates of the center of rotation and the angle of rotation;

According to the front head-up direction vector and the preset user viewing angle range, the user's actual viewing angle range is determined.

In the second aspect, the present application also provides a virtual interaction device between the working machine and the working environment, the device comprising:

An acquisition module, configured to acquire an opening proportional signal of the manipulation component;

The first processing module is configured to determine the motion state data of the controlled working machine according to the opening ratio signal;

The second processing module is configured to respectively determine the interaction state between the controlled working machine and each spatial element in the virtual environment information according to the motion state data and the pre-built virtual environment information, and generate scene update information;

The third processing module is configured to acquire the actual viewing angle range of the user, and intercept and output corresponding scene image information from the scene update information according to the actual viewing angle range of the user.

In a third aspect, the present application also provides a virtual interaction system between an operating machine and an operating environment, the system comprising: a manipulation component, a data processing device, and at least one imaging device, and the manipulation component and the at least one imaging device are both compatible with the connection to the above-mentioned data processing equipment;

The control component is used for the user to initiate a control action on the controlled working machine;

The data processing device is used to obtain the opening proportional signal of the control component; determine the motion state data of the controlled working machine according to the opening proportional signal; according to the motion state data and pre-built virtual environment information, Respectively determine the interaction state between the controlled working machine and each spatial element in the virtual environment information, and generate scene update information according to the interaction state; obtain the user's actual viewing angle range, and based on the user's actual viewing angle range, from Intercepting and outputting corresponding scene image information from the scene update information;

The imaging device is used to display the scene image information.

According to a virtual interaction system between an operating machine and an operating environment provided by the present application, the imaging device is a display screen or a head-mounted imaging device.

According to a virtual interaction system between an operating machine and an operating environment provided in the present application, when the controlled operating machine is an excavator, the control assembly includes left and right handles and left and right pedals.

In a fourth aspect, the present application also provides a test system for operating machine handling performance, which uses any one of the above-mentioned virtual interaction methods between the working machine and the working environment.

The method, device and system for virtual interaction between the working machine and the working environment provided by this application determine the motion state data of the controlled working machine by obtaining the opening ratio signal of the control component, and according to the motion state data and the pre-built virtual environment information, Determine the interaction state of the controlled operating machine and each space element in the virtual environment information, and generate scene update information according to the interaction state, and finally intercept and output the corresponding scene image information from the scene update information according to the user's actual viewing angle range, to realize The interactive process between the operating machine and the operating environment in the virtual scene is realized. Since this process can be realized through the virtual environment, it overcomes the time-consuming interaction process due to the limitations of the actual application environment in the scene involving the interaction between the operating machine and its application environment. energy-consuming problem.

Description of drawings

In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are the For some embodiments of the present invention, those of ordinary skill in the art can also obtain other drawings based on these drawings on the premise of not paying creative efforts.

Fig. 1 is a schematic flow chart of a virtual interaction method between an operating machine and an operating environment provided by the present application;

Fig. 2 is a schematic structural diagram of a virtual interaction device between an operating machine and an operating environment provided by the present application;

Fig. 3 is a schematic structural diagram of the virtual interaction system between the operating machine and the operating environment provided by the present application;

Fig. 4 shows a schematic structural diagram of the virtual interaction system between the working machine and the working environment when the controlled object is an excavator;

Fig. 5 shows a schematic diagram of the data processing flow in the computing host;

FIG. 6 is a schematic structural diagram of an electronic device provided by the present application.

Detailed ways

In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the accompanying drawings in this application. Obviously, the described embodiments are part of the embodiments of this application , but not all examples. Based on the embodiments in this application, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of this application.

The method, device and system for virtual interaction between the working machine and the working environment provided by the embodiments of the present application are described below with reference to FIGS. 1 to 5 .

Fig. 1 shows the virtual interaction method between the working machine and the working environment provided by the embodiment of the present application, the method includes:

Step 110: Obtain an opening proportional signal of the control component.

The control components mentioned here refer to the components that can control the movement and construction actions of the controlled working machine. For example, if the controlled working machine is an excavator, the control components can be two control handles on the left and right and two pedals on the left and right. The board can simulate the left and right joysticks and left and right walking pedals of a real excavator, so that the user can trigger the corresponding control commands by operating the control components.

Specifically, for an excavator, the forward and backward movements of the left-hand handle correspond to the extension and recovery control commands of the arm respectively, and the left and right movements of the left-hand handle respectively correspond to the left and right rotation control commands of the upper body; the forward and backward movements of the right handle correspond to the lowering and Lifting control command, the left and right movements of the right handle correspond to the excavation and unloading control commands of the bucket respectively; Track forward and reverse control commands on the right.

When the manipulation component is operated by the user, the corresponding opening proportional signal can be obtained, so as to sense the user's manipulation action on the manipulation component in the form of an electric signal.

Step 120: Determine the motion state data of the controlled working machine according to the opening ratio signal.

In an exemplary embodiment, the conversion from the opening proportional signal to the motion state data of the controlled working machine can be realized by constructing a system simulation model.

Specifically, the system simulation model is a one-dimensional physical parameterized model, which specifically includes the basic electric control logic model, the hydraulic system model, the operating machine boarding and the dynamic model of the working device, and the electric control logic model outputs various The electronic control valve signal is sent to the hydraulic system model, and the hydraulic system model outputs the real-time pressure of each cylinder to the dynamic model of the working machine and the working device. By sensing the travel, rotation of the working machine and the motion state of the working device on it, the output is controlled The motion state data of the work machine, the motion state data includes the space coordinates, speed, acceleration and other information corresponding to the body of the work machine and the working devices on it.

It can be understood that the working device mentioned in this embodiment refers to the components that the working machine needs to use during walking and construction operations. For example, when the working machine is an excavator, the working device can refer to the arm of the excavator, the bucket A system consisting of rods, buckets and hydraulic cylinders.

Step 130: According to the motion state data and the pre-constructed virtual environment information, respectively determine the interaction state between the controlled working machine and each spatial element in the virtual environment information, and generate scene update information according to the interaction state.

Specifically, according to the motion state data and the pre-constructed virtual environment information, respectively determine the interaction state of the controlled operation machine and each space element in the virtual environment information, and generate scene update information according to the interaction state, which may include:

Step 1: According to the motion state data and the pre-constructed virtual environment information, when it is determined that the controlled operation machine is in contact with at least one spatial element in the virtual environment information, determine the interaction category of the spatial element that has been in contact;

Step 2: Determine the scene change information according to the interaction category of the contacted spatial elements and the motion state data of the controlled operating machine;

Step 3: Update the virtual environment information in real time according to the scene change information to generate scene update information.

In this embodiment, the interaction categories of the spatial elements include interactive objects and non-interactive objects. Correspondingly, the spatial elements are divided into interactive objects and non-interactive objects.

It can be understood that non-interactive objects include spatial elements such as terrain, roads, other machinery, obstacles, and people. Interactable objects include construction objects corresponding to operating machines such as soil and stone.

In an exemplary embodiment, judging whether the controlled operating machine is in contact with the spatial element in the virtual environment information may be based on the known external normal vector defined by each point in the contour information of the spatial element, when the controlled operating machine When the distance between the coordinates of each point corresponding to the contour of the working device and some points on the contour of the spatial element is less than the preset threshold, calculate the vector formed by the line connecting the contour point of the spatial element to the contour point of the nearest working device at each time step, and the distance between the contour point of the spatial element The dot product operation is performed on the outer normal vector, and if the operation result is less than or equal to 0, it is determined that the controlled operation machine is in contact with a certain spatial element in the virtual environment information.

In an exemplary embodiment, the process of determining the scene change information according to the interaction type of the spatial element in contact and the motion state data of the controlled working machine may include:

On the one hand, when the interaction category of the space element in contact is interactive, the deformation state of the space element is determined according to the motion state data of the controlled operating machine, and the deformation state and the motion state of the controlled operating machine after construction work Data as scene change information;

On the other hand, when the interaction category of the contacted spatial element is non-interactive, the motion state data of the controlled working machine at the current moment is used as the scene change information.

In this embodiment, in order to construct virtual scene information, a space and obstacle environment model can be built in advance. The model is a space data model, which records all the space coordinates corresponding to the outline or boundary of each element in the space. The motion state data of the machine is the input. By comparing the relative position, speed and acceleration of each element in the space with the operating machine in real time, it is judged whether each element is in contact with the operating machine.

After determining that a certain element is in contact with the operating machine, it is further determined whether the element belongs to an interactive object or a non-interactive object. For non-interactive objects, the model can output information such as the position and posture of the operating machine under environmental constraints, that is, the contact state The motion status data of the controlled working machine.

For interactive objects, an interactive model of construction objects can be built in advance, which is a spatial data model and a material property model of construction objects (such as soil and stones). The construction can be simulated through a simulation algorithm based on discrete element method or material point method The deformation and residual shape of the object under the action of the controlled operation machine, so as to output the coordinates of each point of the outer contour of the construction object after construction.

Specifically, when the controlled operation machine is in contact with the interactive object, the model can simulate the construction action process, specifically according to the position, speed and other information of the construction component, and based on the material properties of the construction object, it can calculate in real time what the construction component is under the interactive condition. The resistance received and the deformation state of the construction object, and then the deformation state of the construction object and the motion state data of the controlled operating machine after the construction operation are fed back to the space and obstacle environment model to realize the acquisition of scene change information during the construction process. Finally, the The scene change information is recombined with other immutable environment elements in the virtual environment information to generate updated 3D virtual environment information, that is, scene update information.

Therefore, this embodiment combines the application system simulation model, space and obstacle environment model, and construction object interaction model, from handle and pedal signals to high-precision simulation of soil and attachments in the excavation process, to achieve the effect of virtual reality.

Step 140: Obtain the user's actual viewing angle range, intercept and output corresponding scene image information from the scene update information according to the user's actual viewing angle range.

In an exemplary embodiment, the process of obtaining the user's actual viewing angle range may include:

First, obtain the coordinates of the center of rotation and the angle of rotation of the controlled operating machine;

Then, according to the slewing center coordinates and slewing angle, determine the front head-up direction vector of the cab of the controlled working machine;

Finally, the user's actual viewing angle range is determined according to the forward head-up direction vector and the preset user's viewing angle range.

It can be understood that the above-mentioned space and obstacle environment model includes the coordinates of each point on the ground in the movable space of the working machine, and the data is stored in the form of point cloud data with a certain resolution. In this embodiment, an excavator is taken as an example to illustrate the acquisition process of the actual viewing angle range, as follows:

When the excavator is moving in any direction, the ground in the driving direction must ensure that the z-axis coordinates of the bottom of the excavator’s crawler (set as a plane) are greater than or equal to the z-axis coordinates of the corresponding point on the ground projected by it, otherwise it will occur At the same time, it is necessary to ensure that the z-axis coordinates of at least three points are equal to the corresponding coordinates on the ground.

At this time, the normal vector n and the traveling direction vector s of the entire excavator can be determined by the plane formed by the bottom of the crawler, and the coordinates of the excavator's turning center o and the turning angle θ are known parameters, which can be obtained by the DH homogeneous matrix transformation method Find the head-up direction vector b in front of the excavator cab at each time point, which can be calculated by the travel direction vector s passing through the excavator’s rotation center o and the axis rotation angle θ in the direction n, assuming that the user’s viewing angle range is head-up The up-down angle of the direction is ±α ₁ , and the left-right angle is ±α ₂ , and then the user's viewing angle and range can be obtained, so as to determine the user's actual viewing angle range.

According to the real-time actual viewing angle range of the user, the virtual environment area that the user can see under the current viewing angle can be obtained from the scene update information. The coordinates (should be the coordinates of the eyes of the cab personnel) are extended along the line connecting the above viewing angle range, and the intersection formed with the outline of the virtual environment in the scene update information is the virtual environment area that the user can see.

The virtual environment area can be fed back to the user through data such as images and depth information. During the feedback process, it can be displayed to the user through the imaging device, and the user can also continue or change the input of the handle and pedal according to the visual feedback seen by his operation. Realize the function of real-time virtual human-computer interaction, and then realize the virtual interaction between the working machine and the working environment.

Of course, in the actual application process, the imaging device can not only be one or more display screens, but also a wearable imaging device that can follow the movement of the human head in real time, such as head-mounted glasses. The real-time viewing angle movement of the human body is combined with the actual viewing angle range of the user obtained above. Specifically, the viewing angle transformation (including viewing angle translation and rotation) captured by the head-mounted imaging device can be described by the D-H homogeneous matrix transformation method, namely The head-up direction vector b is transformed to obtain a more accurate actual viewing angle range, and the final real-time image can be output to the user by the head-mounted glasses.

It can be seen that the virtual interaction method between the working machine and the working environment provided by this embodiment can construct a corresponding virtual scene for the construction scene, which can meet the specific needs of the construction scene. At the same time, the mechanism model of the actual construction operation process and the virtual reality The combination of digital realization technology makes the interactive process of construction and walking of the controlled operation machine closer to the real scene, and better meets the actual application requirements.

The virtual interaction device between the working machine and the working environment provided by this application is described below. The virtual interaction device between the working machine and the working environment described below and the virtual interaction method between the working machine and the working environment described above can be referred to in correspondence.

Fig. 2 shows the virtual interaction device between the working machine and the working environment provided by the embodiment of the present application, the device includes:

An acquisition module 210, configured to acquire an opening ratio signal of the manipulation component;

The first processing module 220 is configured to determine the motion state data of the controlled working machine according to the opening ratio signal;

The second processing module 230 is configured to respectively determine the interaction state between the controlled working machine and each spatial element in the virtual environment information according to the motion state data and the pre-constructed virtual environment information, and generate scene update information according to the interaction state;

The third processing module 240 is configured to acquire the user's actual viewing angle range, and intercept and output corresponding scene image information from the scene update information according to the user's actual viewing angle range.

In an exemplary embodiment, the above-mentioned second processing module 230 is specifically configured to: judge whether the controlled working machine is in contact with each spatial element in the virtual environment information according to the motion state data and the pre-constructed virtual environment information; When the controlled operation machine is in contact with at least one spatial element in the virtual environment information, determine the interaction category of the contacted spatial element; determine the scene change information according to the interaction category of the contacted spatial element and the motion state data of the controlled operation machine; The scene change information updates the virtual environment information in real time to generate scene update information.

Specifically, the interaction categories of the spatial elements in this embodiment may include interactive categories and non-interactive categories.

The above-mentioned second processing module 230 determines the function of the scene change information according to the interaction type of the contacted spatial element and the motion state data of the controlled operating machine, which can be specifically realized in the following manner:

When the interaction category of the contacted spatial element is interactive, the deformation state of the spatial element is determined according to the motion state data of the controlled operating machine, and the deformation state and the motion state data of the controlled operating machine after construction work are used as the scene change information;

When the interaction category of the contacted spatial elements is non-interactive, the motion state data of the controlled working machine at the current moment is used as the scene change information.

The above-mentioned function of the third processing module 240 to obtain the user's actual viewing angle range can be specifically implemented in the following manner:

According to the coordinates of the center of rotation and the angle of rotation, determine the front head-up direction vector of the cab of the controlled operating machine;

FIG. 3 shows a virtual interaction system between an operating machine and an operating environment provided by an embodiment of the present application. The system includes: a manipulation component 310, a data processing device 320, and at least one imaging device 330. Both the manipulation component 310 and the at least one imaging device 330 are Connect with data processing equipment 320;

The manipulation component 310 is used for the user to initiate a manipulation action on the controlled working machine;

The data processing device 320 is used to obtain the opening proportional signal of the control component; determine the motion state data of the controlled operating machine according to the opening proportional signal; determine the controlled operating machine and the The interactive state of each spatial element in the virtual environment information, and generate scene update information according to the interactive state; obtain the user's actual viewing angle range, intercept and output the corresponding scene image information from the scene update information according to the user's actual viewing angle range;

The imaging device 330 is used to display scene image information.

It should be noted that the imaging device 330 in this embodiment may be a display screen or a head-mounted imaging device.

In an exemplary embodiment, when the controlled working machine is an excavator, the control assembly includes left and right handles and left and right pedals.

Fig. 4 shows the structural framework of the virtual interaction system between the working machine and the working environment when the controlled object is an excavator. The system includes three display screens 410, two left and right pedals 420, a left handle 430, Right handle 440 and computing host 450;

The user operates the left and right pedals 420, the left handle 430, and the right handle 440, and the corresponding handle and pedal opening ratio signals are transmitted to the computer through a certain transmission method (either wired transmission or wireless transmission). The host computer 450 obtains the scene image information through a series of data processing procedures through the calculation host computer 450, and the scene image information can be displayed to the user in real time through the display screen 410. In the virtual scene, the real-time interaction between the excavator and the virtual operating environment is realized by means of human-computer interaction.

Referring to FIG. 4 , the system is also provided with a seat 460 on which the user can manipulate the above-mentioned control components.

For the data processing flow in the computing host 450, please refer to accompanying drawing 5, the processing flow is specifically as follows:

Step 510: the user operates the handle and the pedal, and then outputs the handle signal and the pedal signal;

Step 520: Input the handle signal and pedal signal into the system simulation model, that is, the physical parameterized model, and output the motion state data of the excavator after being processed by the system simulation model;

Step 530: The motion state data of the excavator is further input into the space and obstacle environment model. The model is obtained through virtual mapping or real-scene scanning transformation. After the model is processed, and the head-mounted device is used to follow the change of the perspective of the human body, the excavation can be output. The angle of view and position change corresponding to the rotation and walking of the machine;

Step 540: At the same time, through the interaction between the construction object interaction model and the space and obstacle environment model, the soil virtual environment changes caused by excavation can be obtained;

Step 550: Integrate the data obtained in step 530 and step 540 to obtain the interactive virtual environment and real-time perspective;

Step 560: Present the interactive virtual environment to the user on the imaging device, so as to facilitate the user to perform the next operation.

In addition, the present application also provides a control performance test system of an operating machine, which uses the above-mentioned virtual interaction method between the operating machine and the operating environment. The test system can use the virtual environment to test the control performance of operating machinery, which is more convenient and efficient than the existing test methods based on actual scenarios.

FIG. 6 illustrates a schematic diagram of the physical structure of an electronic device. As shown in FIG. 6, the electronic device may include: a processor (processor) 610, a communication interface (Communications Interface) 620, a memory (memory) 630 and a communication bus 640, Wherein, the processor 610 , the communication interface 620 , and the memory 630 communicate with each other through the communication bus 640 . The processor 610 can call the logic instructions in the memory 630 to execute the virtual interaction method between the working machine and the working environment. The method includes: acquiring the opening proportional signal of the control component; determining the movement of the controlled working machine according to the opening proportional signal State data; according to the motion state data and the pre-constructed virtual environment information, respectively determine the interaction state of the controlled operation machine and each space element in the virtual environment information, and generate scene update information according to the interaction state; obtain the user's actual viewing angle range, according to The user's actual viewing angle range is intercepted from the scene update information and the corresponding scene image information is output.

In addition, the logic instructions in the above-mentioned memory 630 may be implemented in the form of software functional units and when sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc., which can store program codes. .

On the other hand, the present application also provides a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, and when the program instructions are executed by a computer When executing, the computer can execute the virtual interaction method between the working machine and the working environment provided by the above methods, the method includes: obtaining the opening proportional signal of the control component; according to the opening proportional signal, determining the motion state data of the controlled working machine ;According to the motion state data and the pre-constructed virtual environment information, respectively determine the interaction state of the controlled operation machine and each space element in the virtual environment information, and generate scene update information according to the interaction state; obtain the user's actual viewing angle range, according to the user's The actual viewing angle range is intercepted from the scene update information and the corresponding scene image information is output.

In yet another aspect, the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it is implemented to perform the above-mentioned methods for virtual interaction between an operating machine and an operating environment. , the method includes: obtaining the opening proportional signal of the control component; determining the motion state data of the controlled operating machine according to the opening proportional signal; respectively determining the controlled operating machine and the virtual The interactive state of each spatial element in the environmental information, and generate scene update information according to the interactive state; obtain the user's actual viewing angle range, intercept and output the corresponding scene image information from the scene update information according to the user's actual viewing angle range.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without any creative efforts.

Through the above description of the implementations, those skilled in the art can clearly understand that each implementation can be implemented by means of software plus a necessary general-purpose hardware platform, and of course also by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in computer-readable storage media, such as ROM/RAM, magnetic discs, optical discs, etc., including several instructions to make a computer device (which may be a personal computer, server, or network device, etc.) execute the methods described in various embodiments or some parts of the embodiments.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limiting them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

A virtual interaction method between an operating machine and an operating environment, comprising:

Obtain the opening proportional signal of the control component;

Determine the motion state data of the controlled working machine according to the opening ratio signal;

According to the motion state data and the pre-constructed virtual environment information, respectively determine the interaction state between the controlled working machine and each space element in the virtual environment information, and generate scene update information according to the interaction state;

The user's actual viewing angle range is acquired, and corresponding scene image information is intercepted from the scene update information and output according to the user's actual viewing angle range.
The virtual interaction method between an operating machine and an operating environment according to claim 1, wherein, according to the motion state data and the pre-built virtual environment information, respectively determine the controlled operating machine and the virtual environment information Interaction state of each spatial element, and generate scene update information according to the interaction state, including:

According to the motion state data and the pre-constructed virtual environment information, when it is determined that the controlled working machine is in contact with at least one spatial element in the virtual environment information, determine the interaction type of the spatial element in contact;

Determine scene change information according to the interaction category and the motion state data of the controlled working machine;

The virtual environment information is updated in real time according to the scene change information to generate scene update information.
The method for virtual interaction between a working machine and a working environment according to claim 2, wherein the interaction categories of the spatial elements include interactive categories and non-interactive categories.
A method for virtual interaction between an operating machine and an operating environment according to claim 3, wherein determining scene change information according to the interaction category and the motion state data of the controlled operating machine includes:

When the interaction category of the space element in contact is interactive, determine the deformation state of the space element according to the motion state data of the controlled working machine, and compare the deformation state and the The motion state data of the controlled operating machine is used as scene change information;

When the interaction category of the space element in contact is non-interactive, the motion state data of the controlled working machine at the current moment is used as the scene change information.
A virtual interaction method between an operating machine and an operating environment according to claim 1, wherein obtaining the user's actual viewing angle range includes:

Obtain the rotation center coordinates and rotation angle of the controlled operating machine;

Determine the front head-up direction vector of the cab of the controlled working machine according to the coordinates of the center of rotation and the angle of rotation;

According to the front head-up direction vector and the preset user viewing angle range, the user's actual viewing angle range is determined.
A virtual interaction device for an operating machine and an operating environment, comprising:

An acquisition module, configured to acquire an opening proportional signal of the manipulation component;

The first processing module is configured to determine the motion state data of the controlled working machine according to the opening ratio signal;

The second processing module is configured to respectively determine the interaction state between the controlled working machine and each spatial element in the virtual environment information according to the motion state data and the pre-built virtual environment information, and generate Scene update information;

The third processing module is configured to acquire the actual viewing angle range of the user, and intercept and output corresponding scene image information from the scene update information according to the actual viewing angle range of the user.
A virtual interaction system between an operating machine and an operating environment, comprising: a manipulation component, a data processing device, and at least one imaging device, and the manipulation component and the at least one imaging device are both connected to the data processing device;

The control component is used for the user to initiate a control action on the controlled working machine;

The data processing device is used to obtain the opening proportional signal of the control component; determine the motion state data of the controlled working machine according to the opening proportional signal; according to the motion state data and pre-built virtual environment information, Respectively determine the interaction state between the controlled working machine and each spatial element in the virtual environment information, and generate scene update information according to the interaction state; obtain the user's actual viewing angle range, and based on the user's actual viewing angle range, from Intercepting corresponding scene image information from the scene update information;

The imaging device is used to display the scene image information.
The virtual interaction system between an operating machine and an operating environment according to claim 7, wherein the imaging device is a display screen or a head-mounted imaging device.
The virtual interaction system between an operating machine and an operating environment according to claim 7, wherein when the controlled operating machine is an excavator, the control components include left and right handles and left and right pedals.
A control performance test system for an operating machine, which uses the virtual interaction method between an operating machine and an operating environment as claimed in any one of claims 1 to 5.