WO2023195489A1

WO2023195489A1 - Construction machine operating device and construction machine operating system using said operating device

Info

Publication number: WO2023195489A1
Application number: PCT/JP2023/014104
Authority: WO
Inventors: 純平佐藤; 聡志猪瀬
Original assignee: 日立建機株式会社
Priority date: 2022-04-08
Filing date: 2023-04-05
Publication date: 2023-10-12

Abstract

Provided is a device for operating a construction machine, with which the construction machine can be operated in a simple manner. This invention provides a device for operating a construction machine operated by an operator, the operating device characterized by having an input device that receives input of operation information from the operator, and a control device that outputs an action command to a drive device of the construction machine on the basis of the inputted operation information, a gesture sensing area being set in the input device, the input device detecting movements of the operator's fingers in the gesture sensing area, and the control device outputting the action command to the drive device so that the construction machine moves in accordance with the action of the fingers.

Description

Construction machinery operating device and construction machinery operating system using the operating device

The present invention relates to a construction machine operating device and a construction machine operating system using the operating device.

BACKGROUND ART Conventionally, a technique has been known in which an operation target is operated by the movement of a fingertip on a touch panel (for example, Patent Document 1). Patent Document 1 discloses a technology for executing or changing a taught robot motion by detecting a user motion (for example, a finger movement such as a long press, flick, or swipe) of a worker on a touch panel. ing. When a user motion is identified, the robot device operates according to operation commands (for example, deceleration, acceleration, stop, restart, start position control, start force control, start error reset, etc.) stored in advance in association with the user motion. is executed or changed.

However, in the technology disclosed in Patent Document 1, there is no relationship between the user motion of the worker and the corresponding motion of the robot device. Therefore, it is necessary for the operator to memorize the user motions that correspond to the movements of the robot device in advance, or check the correspondence between the movements and the user motions on the monitor, and then input the appropriate user motions. be.

JP 2021-142608 Publication

By the way, construction machines such as hydraulic excavators often require complex and continuous operations, such as raising an arm while traveling, and the operations are complicated. In recent years, the functions of construction machinery have diversified, and their operation tends to become more complex.

When attempting to operate a construction machine using the technology disclosed in Patent Document 1, it is necessary to prepare a large number of user motions corresponding to complicated operations of the construction machine. In this case, the operator needs to memorize a large number of user motions or check on a monitor a large number of correspondences between the motions of the construction machine and the user motions. This imposes a burden on the operator, and it is required to operate the construction machine more easily.

The present invention has been made in view of the above-mentioned problems, and its purpose is to provide a construction machine operating device that allows easy operation of the construction machine, and a construction machine operating system using the operating device. It is.

In order to achieve the above object, an operating device for a construction machine according to the present invention is an operating device for a construction machine operated by an operator, and includes an input device into which operation information from the operator is input; a control device that outputs an operation command to a drive device of the construction machine based on input operation information, a gesture sensing area is set in the input device, and a gesture sensing area is set in the input device; The control device detects the movement of the operator's fingers in the sensing area, and outputs the operation command to the drive device so that the construction machine moves in accordance with the movement of the fingers. do.

According to the operating device for a construction machine according to the present invention, the construction machine can be easily operated.

FIG. 2 is a conceptual diagram showing an example of operation of a construction machine using the operating device according to the present embodiment. FIG. 1 is a conceptual diagram showing an example of an operating device according to the present embodiment. FIG. 2 is a conceptual diagram showing an example of an input device of the operating device according to the present embodiment. FIG. 2 is a functional block diagram showing an example of the operating device according to the present embodiment. 5 is a flowchart illustrating an example of operating a construction machine using the operating device according to the present embodiment. FIG. 7 is a conceptual diagram showing another example of the operating device according to the present embodiment. FIG. 3 is a conceptual diagram illustrating an example of the relationship between finger movements and construction machine movements in the operating device according to the present embodiment. FIG. 3 is a conceptual diagram illustrating an example of the relationship between finger movements and construction machine movements in the operating device according to the present embodiment. FIG. 3 is a conceptual diagram illustrating an example of the relationship between finger movements and construction machine movements in the operating device according to the present embodiment. FIG. 3 is a conceptual diagram illustrating an example of the relationship between finger movements and construction machine movements in the operating device according to the present embodiment. FIG. 3 is a conceptual diagram illustrating an example of the relationship between finger movements and construction machine movements in the operating device according to the present embodiment. FIG. 3 is a conceptual diagram illustrating an example of the relationship between finger movements and construction machine movements in the operating device according to the present embodiment. FIG. 3 is a conceptual diagram illustrating an example of the relationship between finger movements and construction machine movements in the operating device according to the present embodiment. FIG. 3 is a conceptual diagram showing an example of an operation for stopping the operation of a construction machine in the operating device according to the present embodiment. FIG. 3 is a conceptual diagram illustrating an example of the relationship between finger movements and construction machine movements in the operating device according to the present embodiment. FIG. 3 is a conceptual diagram illustrating an example of the relationship between finger movements and construction machine movements in the operating device according to the present embodiment. FIG. 3 is a conceptual diagram illustrating an example of the relationship between finger movements and construction machine movements in the operating device according to the present embodiment. FIG. 7 is a conceptual diagram showing a modification of the operating device according to the present embodiment. FIG. 7 is a conceptual diagram illustrating an example of an operation of an operating device according to a modification. FIG. 1 is a conceptual diagram showing an example of an operation system according to the present embodiment. FIG. 1 is a functional block diagram showing an example of an operation system according to the present embodiment. FIG. 6 is a conceptual diagram showing another example of the operating system according to the present embodiment. 5 is a flowchart illustrating an example of operating a construction machine using the operating system according to the present embodiment. FIG. 3 is a conceptual diagram illustrating an example of a detection method in the gesture sensing unit of the operating device according to the present embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Construction machinery operating device>
FIG. 1 is a conceptual diagram showing an example of operation of a hydraulic excavator (construction machine) 100 using an operating device 150 according to the present embodiment. As shown in FIG. 1, the operating device 150 according to the present embodiment operates the hydraulic excavator 100 by operating an input device 200 instead of operating a conventional operating lever provided in the operator's cab. It is. In the driver's cab, only the operating device 150 may be provided, or both the operating device 150 and the operating lever may be provided. Note that the object to be operated by the operating device 150 is not limited to the hydraulic excavator 100. For example, the operating device 150 can operate general construction machinery different from the hydraulic excavator 100, such as an electrically driven excavator, a dump truck, and a wheel loader.

FIG. 2 is a conceptual diagram showing an example of the operating device 150 according to the present embodiment. As shown in FIG. 2, the operating device 150 according to the present embodiment is provided in the driver's cab of the hydraulic excavator 100, and is operated by a driver (hereinafter also referred to as an operator) seated in a driver's seat. The operating device 150 of this embodiment is installed on a monitor stand in the driver's cab of the hydraulic excavator 100.

FIG. 3 is a conceptual diagram showing an example of the input device 200 of the operating device 150 according to the present embodiment. FIG. 4 is a functional block diagram showing an example of the operating device 150 according to this embodiment. As shown in FIG. 4, the operating device 150 has an input device 200 into which operating information from an operator is input, and a drive device (not shown) of the hydraulic excavator 100 based on the input operating information. , and an operation determination device 210 (control device) that outputs an operation command.

The input device 200 will be explained. The input device 200 of this embodiment is a so-called touch panel type input device that has a touch pad function to detect the touch of an operator's finger and a display function to display predetermined information. As shown in FIG. 4, the input device 200 includes a gesture sensing section 201, an auxiliary command input section 202, a mode selection section 204, and a status display section 205 as internal functions. A gesture sensing area 301, an auxiliary command input area 302, a mode selection area 304, and a status display area 305 are displayed on the display of the input device 200.

The gesture sensing unit 201 detects the movements of the operator's fingers. Movements of the operator's fingers are input to the gesture sensing unit 201 from the gesture sensing area 301. The gesture sensing unit 201 detects the movement of the operator's hand when the finger comes into contact with the gesture sensing area 301 of the display. Here, the concept of finger movement includes a trajectory (also referred to as a gesture) of movement of the finger on the surface of the gesture sensing area 301 and the speed of the movement.

The gesture sensing unit 201 detects the number and movement of the operator's fingers that are in contact with the gesture sensing area 301, as well as the movement of the palm of the hand. For example, the gesture sensing unit 201 may touch the gesture sensing area 301 with three fingers, the thumb, index finger, and middle finger, and pull them toward you, or gesture sense the five fingers, the thumb, index finger, middle finger, ring finger, and little finger. It is possible to detect movements such as touching the area 301 and pushing the palm out to the back, or pulling the palm toward the user while the palm is in contact with the gesture sensing area 301.

The operating device 150 operates predetermined operating parts of the hydraulic excavator 100 according to the detection results of the gesture sensing unit 201 (raising and lowering the boom, raising and lowering the arm, rotating the bucket, moving the traveling body forward and backward, etc.). Hereinafter, the boom, arm, and bucket of the hydraulic excavator 100 will also be collectively referred to as a working machine.

As shown in FIG. 4, the auxiliary command input section 202 includes a safety device input section 202a, a continuous operation input section 202b, and an individual operation input section 202c. An auxiliary command that assists the type of operation of the hydraulic excavator 100 is input to the auxiliary command input section 202 . The auxiliary command is input to the auxiliary command input section 202 from the auxiliary command input area 302 displayed on the display of the input device 200.

The safety device input section 202a corresponds to the "safety device" of the hydraulic excavator 100, and is used to avoid unintended operation of the hydraulic excavator 100 and to easily stop the operation of the hydraulic excavator 100. The safety device input signal is input from the safety device input area 302a to the safety device input section 202a. For example, by touching the gesture sensing area 301 with one hand of the operator while touching the safety device input area 302a, a predetermined operation part of the hydraulic excavator 100 can be controlled in a predetermined operation type. becomes possible to operate.

On the other hand, the operator can stop the operation of the hydraulic excavator 100 by removing his hand from the safety device input area 302a. In other words, if the operator's hand does not touch the safety device input area 302a, the hydraulic excavator 100 will not operate even if the operator's finger accidentally moves on the gesture sensing area 301. . Therefore, it is possible to prevent the hydraulic excavator 100 from moving due to an erroneous operation by the operator. Further, the operator can stop the operation of the hydraulic excavator 100 simply by removing his/her hand from the safety device input area 302a.

A command for determining continuous motion of a predetermined type of motion is input to the continuous motion input section 202b. A continuous motion command is input from the continuous motion input area 302b to the continuous motion input section 202b. When the operator's hand comes into contact with the continuous motion input area 302b, the operating device 150 outputs a motion command to repeat (continue) a predetermined type of motion to the drive device of a predetermined motion portion of the hydraulic excavator 100. .

For example, when moving the hydraulic excavator 100 forward, the operator makes a movement of pushing his hand out on the gesture sensing area 301. However, the gesture sensing area 301 is limited. Therefore, when the operator's fingers reach the end of the gesture sensing area 301, the operator's fingers cannot be pushed further back, and the hydraulic excavator 100 stops moving forward. On the other hand, if the operator makes a movement to push his/her finger to the back on the gesture sensing area 301 while the operator's hand is in contact with the continuous motion input area 302b, even if the operator's finger is Even when the end of the region 301 is reached, the hydraulic excavator 100 can continue moving forward.

A command for determining an independent action in which only the specified action type is performed independently is input to the independent action input section 202c. A solo action command is input from the solo action input area 302c to the solo action input section 202c. When the operator's hand comes into contact with the independent motion input area 302c, the operating device 150 issues a motion command to the drive device of a predetermined motion part of the hydraulic excavator 100 to perform only that motion in a predetermined motion type. Output. "Swing" displayed in the independent operation input area 302c means a swing of the upper revolving structure. "Travel" means the traveling motion of the lower traveling body of the hydraulic excavator 100, and "Rotate" means the rotation when a special attachment that involves rotational motion is attached.

For example, when the operator performs a movement of pulling the palm toward the user on the gesture sensing area 301, the hydraulic excavator 100 moves backward. On the other hand, when the operator pulls three fingers, the thumb, index finger, and middle finger, toward the user on the gesture sensing area 301, the boom is raised. In this way, for one movement of moving a finger forward on the gesture sensing area 301, the movement of the hydraulic excavator 100 is changed depending on whether the object in contact with the gesture sensing area 301 is a palm or a finger. Different types. However, when the operator's hand is in contact with the display area indicating "Travel" in the independent motion input area 302c and moves the finger forward on the gesture sensing area 301, the operator can touch the gesture sensing area 301. Hydraulic excavator 100 can be moved backwards regardless of whether the object is a palm or a finger.

The mode selection unit 204 is a means for selecting an operation mode, such as energy saving operation, for example. A mode selection command is input to the mode selection unit 204 from the mode selection area 304. In FIG. 3, a state is shown in which "ECO" indicating energy-saving operation is selected in the mode selection area 304. The status display area 305 is an area for displaying information regarding the operating status of the hydraulic excavator 100 during operation. The status display unit 205 notifies the operator that the hydraulic excavator 100 is ready for operation by displaying text information in the status display area 305, emitting sound or vibration, or the like. In FIG. 3, as an example, "ENG ON" is displayed in the status display area 305, indicating that the engine of the hydraulic excavator 100 is running.

When the gesture sensing unit 201 detects the movement of the operator's fingers and a predetermined auxiliary command is input to the auxiliary command input unit 202, the detection result and a signal corresponding to the input result are transmitted to the operation determination device 210. be done.

Next, the operation determination device 210 will be explained. The operation determination device 210 is configured by, for example, a microcomputer having a CPU, memory, and an input/output interface. As shown in FIG. 4, the operation determination device 210 has, as its internal functions, a gesture determination section 210a, an emergency motion control section 991, a specific machine determination section 211, a specific attachment determination section 211-1, and an operation command matching section. section 212, an operation command recording section 213, a special operation command recording section 213-1, a specific machine operation pattern inspection section 214, a physical information recording section 215, and a working machine information recording section 216.

The operation determination device 210 determines the operation of the hydraulic excavator 100 based on the finger movements detected by the gesture sensing unit 201. Information (operation command) for controlling the operation of the hydraulic excavator 100 is transmitted to the electronic controller of the hydraulic excavator 100 by wired or wireless communication. The electronic controller operates each operating part of the hydraulic excavator 100 (boom, arm, bucket, traveling body, etc.) in a predetermined type of operation and with a predetermined amount of operation.

The operation determination device 210 controls the actuator (drive device) of the hydraulic excavator 100 via the electronic controller so that the hydraulic excavator 100 moves according to the finger movements of the operator input to the input device 200. The operation determination device 210 detects finger movements reminiscent of the movements of the hydraulic excavator 100, determines the operation of the hydraulic excavator 100 according to the finger movements, and issues an operation command to the electronic controller to perform the operation. Send. The electronic controller controls the actuator based on the received operation command. Therefore, the driver of the hydraulic excavator 100 can operate the hydraulic excavator 100 intuitively.

The gesture determination unit 210a determines what type of finger movement detected by the gesture sensing unit 201. Specifically, the gesture determination unit 210a determines whether it is a finger or a palm that is in contact with the gesture sensing area 301, and if it is a finger, the position of the finger (thumb, index finger, middle finger, ring finger, little finger). It is determined where the position is and what progress the hand and fingers have made on the gesture sensing area 301. Note that the characteristics (size, etc.) of the fingers used for control differ depending on the operator. Therefore, as an example, the characteristics of the hand and fingers are registered in advance in the physical information recording section 215, and the gesture determination section 210a determines the type of the above-mentioned movement of the hand and fingers.

The emergency operation control unit 991 performs control related to emergency operations such as emergency stop during operation of the hydraulic excavator 100. When the hydraulic excavator 100 is brought to an emergency stop, the emergency operation control unit 991 controls the drive device of the hydraulic excavator 100 so that the hydraulic excavator 100 stops as smoothly as possible.

The operation command recording unit 213 records a basic pattern of finger movements reminiscent of the movement of the hydraulic excavator 100, and information on the type of movement and the operating part of the hydraulic excavator 100, which are associated with the basic pattern. The basic pattern of finger movements is set to intuitively represent the movement of the hydraulic excavator 100. As an example, a basic finger pattern for moving the hydraulic excavator 100 forward may be a pattern in which the finger slides forward on the gesture sensing area 301. For example, when a special device such as a cutting machine (cutter) or a demolition attachment (hydraulic breaker) is attached as an attachment to a working machine, the special operation command recording unit 213-1 records the operation type and this operation. The basic patterns of hand and finger movements associated with the types are recorded.

The motion command matching section 212 matches the finger movement detected by the gesture sensing section 201 (that is, the type of hand motion determined by the gesture determining section 210a) with the basic pattern recorded in the motion command recording section 213. Then, the operation type and operation part of the corresponding hydraulic excavator 100 are specified. The specific machine operation pattern inspection unit 214 generates an operation command according to the operation type and operation part specified by the operation command comparison unit 212, and transmits the operation command to the electronic controller. The electronic controller operates the drive device of the hydraulic excavator 100 based on the operation command.

Further, the specific machine operation pattern inspection unit 214 generates an auxiliary operation command different from the above-mentioned operation command according to the auxiliary command input to the auxiliary command input unit 202, and sends the auxiliary operation command to the electronic controller. do. The electronic controller operates the drive device of the hydraulic excavator 100 based on the auxiliary operation command. This allows the operator to operate the hydraulic excavator 100 according to the movement of the operator's fingers, such as when the operator slides his or her fingers forward on the gesture sensing area 301, the hydraulic excavator 100 moves forward in synchronization. can do. By operating the gesture sensing unit 201 and the auxiliary command input unit 202, the operator can perform operations with intuitive, easy-to-understand, and simple movements reminiscent of the movements of the hydraulic excavator 100.

In the work equipment information recording section 216, information on the model of the work equipment (ie, boom, arm, bucket) corresponding to the model of the hydraulic excavator 100 is recorded in advance. The information on the model of the work machine includes, for example, information on the movable range (rotation angle, etc.) for each model of the work machine. Further, the work equipment information recording unit 216 records information on the operating speed of the work equipment relative to the speed of finger movement for each model of the hydraulic excavator 100. For example, in the case of a hydraulic excavator 100 with a wide movable range of the work machine, the operating speed of the work machine with respect to finger movements is higher than that of a hydraulic excavator 100 with a narrow movable range of the work machine. It has been recorded that it speeds up

The specific machine determination unit 211 determines which type of work machine is attached, based on information such as the model of the hydraulic excavator 100. Then, the specific machine operation pattern inspection unit 214 identifies the model (and movable range) of each work machine by referring to the work machine information recording unit 216, and compares the work of the hydraulic excavator 100 with respect to the speed of finger movement. The operating speed of the machine is determined, and an operating command is generated according to the determined operating speed. Specifically, in the case of a type of hydraulic excavator 100 in which the working machine has a wide movable range, the specific machine operation pattern inspection unit 214 increases the operating speed of the working machine relative to the speed of finger movement. Further, in the case of the hydraulic excavator 100 of a type in which the working machine has a narrow movable range, the specific machine operation pattern inspection unit 214 slows down the operating speed of the working machine relative to the speed of finger movement.

For example, in the case of the hydraulic excavator 100 whose arm has a narrow movable range, when a finger is moved on the gesture sensing area 301, the arm may quickly reach the limit position of its movable range. In this case, it becomes difficult to make fine adjustments to the arm. In this regard, according to the operating device 150 according to the present embodiment, in the case of the hydraulic excavator 100 whose working equipment has a narrow movable range, the operation speed of the working equipment is slowed down relative to the speed of hand and finger movements. It becomes easier to make fine adjustments to the machine.

Furthermore, in the case of the hydraulic excavator 100 where the arm has a wide movable range, when a finger is moved on the gesture sensing area 301, it may take some time for the arm to reach the limit position of the movable range. In this case, it becomes difficult to move the arm quickly. In this respect, according to the operating device 150 according to the present embodiment, in the case of a hydraulic excavator 100 having a wide movable range of the working machine, the operating speed of the working machine is increased relative to the speed of finger movement. The machine can be quickly moved to the limit of its range of motion.

The specific attachment determination unit 211-1 determines whether or not a special machine such as a cutting machine (cutter) or a demolition attachment (hydraulic breaker) is attached as an attachment to the working machine. According to the determination result of the specific attachment determination unit 211-1, the motion command matching unit 212 records the detected finger movement in the basic pattern recorded in the motion command recording unit 213 and in the special motion command recording unit 213-1. Matches one of the specified basic patterns.

The physical information recording unit 215 records physical information regarding the fingers of each operator. Examples of the physical information include loss of a fingertip, paralysis of a fingertip, and the like. In this case, the motion command recording unit 213 changes the basic pattern of the operator's finger movements according to the physical information. Then, the operation command recording unit 213 records the changed pattern of hand and finger movements, and the operation type and operation part of the hydraulic excavator 100 that are associated with the changed pattern. The motion command matching unit 212 matches the detected finger movement with the above-mentioned changed pattern. Thereby, even if the operator has a defect in a fingertip, the hydraulic excavator 100 can be easily operated.

FIG. 5 is a flowchart illustrating the operation of the hydraulic excavator 100 using the operating device 150 according to the present embodiment. When operating the hydraulic excavator 100, first, the gesture sensing unit 201 detects (acquires) the movement (gesture) of the operator's fingers (S1). Next, the gesture determination unit 210a determines the type of finger movement (gesture) detected by the gesture sensing unit 201 (S2). Next, the operation command verification unit 212 refers to the operation command recording unit 213 and the special operation command recording unit 213-1, and determines whether the movement of the hand or finger corresponds to a “normal machine (for example, a hydraulic excavator 100)” or The "type of operation" is determined depending on whether the machine corresponds to a "special machine (for example, a demolition machine in which the attachment of the hydraulic excavator 100 is replaced with a hydraulic breaker)" (S3). This is because the attachments attached to special machines are different from those of normal machines, and therefore require corresponding operation controls and signals to execute them. In this embodiment, a case will be described in which the movement of the fingers corresponds to a "normal machine (for example, a hydraulic excavator 100)". Next, the motion command matching section 212 matches the finger movement detected by the gesture sensing section 201 with the basic pattern of hand and finger movement recorded in the motion command recording section 213, and calculates the hydraulic pressure associated with the movement. The operation type and operation part of the shovel 100 are specified (S4). Next, the operation determination device 210 determines, for example, the communication state with the electronic controller of the hydraulic excavator 100 (S5). Finally, the operation determination device 210 outputs an operation command to the actuator of the hydraulic excavator 100 via the electronic controller, and thereby outputs an operation command to each operating part of the hydraulic excavator 100 (boom, arm, bucket, traveling body, etc.). ) moves in a predetermined type of motion and with a predetermined amount of motion in synchronization with the movement of the fingers (S6).

FIG. 6 is a conceptual diagram showing another example of the operating device 150 according to the present embodiment. As shown in FIG. 6, the input device 200 of the operating device 150 may be provided on each driver's seat (eg, left and right armrest portions) of the hydraulic excavator 100. For example, if you perform a predetermined gesture with one hand and input an auxiliary command with the other hand, but the hydraulic excavator 100 does not perform the intended operation, you may be so distracted by the operation at hand that your field of vision is restricted. This raises concerns about drivers not paying attention to the road ahead when driving.

Therefore, as shown in FIG. 6, the gesture sensing area 301 may be provided on one armrest portion (for example, the right armrest), and the auxiliary command input area 302 may be provided on the other armrest portion (for example, the left armrest). Note that the auxiliary command input area 302 is not necessarily limited to a touch panel type, and may have a physical shape such as a protruding auxiliary operation button. With this configuration, it is possible to achieve intuitive and smooth operation at hand while ensuring a wide field of view ahead.

Next, the relationship between the basic pattern of finger movements and the movement of the hydraulic excavator 100 will be explained while presenting various implementation examples. 7 to 13 and 15 to 17 are conceptual diagrams showing an example of the relationship between the movement of fingers and the movement of the hydraulic excavator 100 in the operating device 150 according to the present embodiment. FIG. 14 is a conceptual diagram showing an example of an operation for stopping the operation of a construction machine in the operating device according to the present embodiment.

In FIG. 7, one hand is in contact with the safety device input area 302a of the auxiliary command input area 302. Further, on the gesture sensing area 301, a movement of pulling three fingers (for example, the thumb, index finger, and middle finger) of the other hand toward the user (a movement similar to lifting a boom with the hand) is shown. Control is performed to raise the boom of the hydraulic excavator 100 by such finger movements and finger positions (pulling the thumb, index finger, and middle finger toward you).

In FIG. 8, one hand is in contact with the safety device input area 302a of the auxiliary command input area 302. Further, on the gesture sensing area 301, a movement of pushing three fingers (for example, the thumb, index finger, and middle finger) of the other hand toward the back (a movement similar to lowering a boom with the hand) is shown. The boom of the hydraulic excavator 100 is controlled to be lowered by such finger movements and finger positions (the thumb, index finger, and middle finger are pushed out to the back). In this way, by sliding the thumb, index finger, and middle finger back and forth on the gesture sensing area 301, the boom can be moved up and down.

In FIG. 9, one hand is in contact with the safety device input area 302a of the auxiliary command input area 302. Further, on the gesture sensing area 301, a movement of pulling the three fingers (for example, the little finger, ring finger, and middle finger) of the other hand toward the user (a movement similar to hugging the arm with the hand) is shown. By such finger movements and finger positions (pulling the little finger, ring finger, and middle finger toward you), the arm of the hydraulic excavator 100 is controlled to be lowered (embraced). Although not shown, the arm is raised (pushed out) by pushing three fingers (little finger, ring finger, middle finger) to the back (movement of pushing out the arm with the hand) on the gesture sensing area 301. , is controlled.

In FIG. 10, one hand is in contact with the safety device input area 302a of the auxiliary command input area 302. Further, on the gesture sensing area 301, a movement of opening five fingers of the other hand outward is shown. Control is performed such that earth and sand are dumped by the bucket of the hydraulic excavator 100 based on the movement of the fingers and the position of the fingers (spreading the five fingers). Furthermore, although not shown in the drawings, the bucket of the hydraulic excavator 100 scoops up dirt, etc. (tilt) by performing a movement of squeezing five fingers (a hand movement similar to scooping up dirt) on the gesture sensing area 301. , is controlled.

In FIG. 11, one hand is in contact with the safety device input area 302a of the auxiliary command input area 302. Furthermore, on the gesture sensing area 301, a movement of finely shaking the five fingers of the other hand (hand movement like shaking off dirt) is shown. This causes the bucket of the hydraulic excavator 100 to be controlled to vibrate in order to remove the remaining soil attached to the bucket.

In FIG. 12, one hand is in contact with the safety device input area 302a of the auxiliary command input area 302. Also, while touching the five fingers of the other hand on the gesture sensing unit 201, a movement of rotating the wrist to the left or right on the gesture sensing unit 201 (a hand movement such as rotating the upper rotating body) is performed. )It is shown. As a result, the upper rotating body of the hydraulic excavator 100 is controlled to rotate according to the direction of rotation of the wrist.

In FIG. 13, a "compound operation" in which each operation is performed simultaneously is shown using the boom and arm of the hydraulic excavator 100 as an example. Specifically, one hand is in contact with the safety device input area 302a of the auxiliary command input area 302. Also, a movement of pulling the entire hand toward you on the gesture sensing area 301 while touching the five fingers of the other hand on the gesture sensing area 301 (a movement similar to lifting the boom and arm by hand) is shown. ing. In this way, the boom and arm are simultaneously raised by a gesture of pulling five fingers toward the user on the gesture sensing area 301.

Furthermore, although not shown, control is performed to lower the boom and the arm at the same time by pushing five fingers toward the back on the gesture sensing unit 201 (a hand movement that lowers the boom and arm). .

Additionally, if you pull your five fingers toward you on the gesture sensing area 301 (a movement similar to lifting the boom and arm with your hand) and narrow your fingertips (a movement of your hand similar to scooping up dirt), the boom and Control is such that when the arm is raised, the bucket is simultaneously tilted.

Also, while pushing five fingers to the back on the gesture sensing area 301 (hand movement like lowering a boom and arm), open the fingertips outward (hand movement like releasing dirt), and move the fingertips left and right. By making small shakes (hand movements like shaking off dirt), the boom and arm are lowered, the bucket is simultaneously dumped, and the bucket is vibrated.

FIG. 14 shows a method for stopping the operation, taking as an example the raising and lowering of the boom of the hydraulic excavator 100. In one example (operation stopping method 1), one hand is in contact with the gesture sensing area 301 and the other hand is away from the safety device input area 302a. In this way, the operation of the hydraulic excavator 100 stops when the operator's hand leaves the safety device input area 302a.

In another example (operation stopping method 2), it is shown that the operation of the hydraulic excavator 100 is stopped by removing one hand from the gesture sensing area 301. In this case, even if the other hand is in contact with the safety device input area 302a, the operation of the hydraulic excavator 100 is stopped. That is, the operation of the hydraulic excavator 100 stops when the finger leaves either the safety device input area 302a or the gesture sensing area 301.

FIG. 15 shows an example of a traveling operation of the hydraulic excavator 100. Specifically, one hand is in contact with the safety device input area 302a. Further, a movement of bringing the fist (palm) of the other hand into contact with the gesture sensing area 301 and pulling the entire hand forward (a movement similar to pulling the hydraulic excavator 100 backward with the hand) is shown. As a result, the hydraulic excavator 100 is controlled to move backward.

FIG. 16 shows that the hydraulic excavator 100 is controlled by selecting the independent operation input area 302c of the auxiliary command input area 302. In FIG. 16, one hand is in contact with the display area indicating "Travel" in the independent action input area 302c. Also shown is a movement in which five fingers of the other hand are brought into contact with the gesture sensing area 301 and pulled forward (a movement in which the entire fingers are used to pull the hydraulic excavator 100 backward). As a result, the hydraulic excavator 100 is controlled to move backward.

The operator of the hydraulic excavator 100 may wish to perform the same operation continuously. FIG. 17 shows that the hydraulic excavator 100 is controlled by selecting the continuous operation input region 302b. Specifically, the middle finger of one hand (or fingertip) is brought into contact with the continuous motion input region 302b, and the index finger is brought into contact with the region indicating "Travel" in the single motion input region 302c. Also shown is a movement in which five fingers of the other hand are brought into contact with the gesture sensing area 301 and pushed out to the back (a movement in which the entire fingers push the hydraulic excavator 100 forward). As a result, the hydraulic excavator 100 is controlled to continue moving forward.

In this way, for example, the operator can intuitively operate the hydraulic excavator 100 by pushing his/her hand out to the back (front) when moving the hydraulic excavator 100 forward, and by pulling it toward the front (back) when moving the hydraulic excavator 100 backward. can be done. Therefore, the operator can use simple, intuitive and easy-to-understand operations reminiscent of the movements of the hydraulic excavator 100 to perform complex and continuous movements unique to the hydraulic excavator 100 (for example, moving forward while moving dirt, sand, etc. into the bucket). loading, etc.).

Next, a modification of the operating device 150 according to the present embodiment will be described. FIG. 18 is a conceptual diagram showing a modification of the operating device 150 according to the present embodiment. FIG. 19 is a conceptual diagram showing an operation example of the operating device 150 according to the modification.

In the above embodiment, an example was shown in which the operating device 150 is mounted in the driver's cab of the hydraulic excavator 100, and the operator rides in the driver's cab and operates the operating device 150. However, the operating device 150 may be a detachable wireless communication terminal, and may be operated at a distance where the hydraulic excavator 100 can be visually seen, for example, via a wireless communication network. In this case, as shown in FIG. 19, the operating device 150 may be a shoulder-type terminal of the operator. For example, the operation determining device 210 of the operating device 150 wirelessly communicates with the electronic Communicate with the controller.

Note that the operating device 150 according to the modification may have the operation determining device 210 mounted on the hydraulic excavator 100, and the input device 200 may be detachable. In this case, the input device 200 communicates with the operation determination device 210 in a wireless manner.

<Construction machinery operation system>
Next, the operation system 160 of the hydraulic excavator 100 according to this embodiment will be explained. The operating system 160 is a system that operates the hydraulic excavator 100 using the above-described operating device 150. The operation system 160 is provided with an operation verification device (server device) 220, which will be described later, in addition to the operation device 150 described above, and an emergency operation control unit 991 provided in the operation determination device 210 is installed in the operation verification device 220. They differ in terms of the points provided. Hereinafter, components having the same or similar functions as the operating device 150 described above will be given the same reference numerals as those of the operating device 150, and the explanation thereof will be omitted, and different parts will be explained.

FIG. 20 is a conceptual diagram showing an example of the operation system 160 according to the present embodiment. FIG. 21 is a functional block diagram showing an example of the operating system 160 according to this embodiment. The operation system 160 according to this embodiment realizes remote operation of the hydraulic excavator 100 from outside the site where the hydraulic excavator 100 is located (for example, in a foreign country), for example, via a wireless communication network.

As shown in FIG. 21, the operating system 160 includes an operation verification device (server device) 220 in addition to the above-described operating device 150. The operation verification device 220 is capable of wireless communication with the operation device 150, specifically, the operation determination device 210 of the operation device 150, and receives operation commands according to the operation type and operation part specified by the operation command matching unit 212. do. Then, the operation verification device 220 determines whether or not to operate the hydraulic excavator 100 with the specified operation type. For example, when the operation verification device 220 determines that the hydraulic excavator 100 should be operated in the specified operation type, the operation verification device 220 transmits information for controlling the operation of the hydraulic excavator 100 to the hydraulic excavator 100 via wireless communication. Send to electronic controller. That is, the operation verification device 220 controls the actuator (drive device) of the hydraulic excavator 100 via the electronic controller.

The operation verification device 220 includes an operation determination section 220a, an emergency operation control section 991, a driver condition monitoring section 992, an abnormal operation/abnormal physical condition detection section 994, a trajectory information recording section 221, and a configuration section 222. , an operation performance information recording section 993, a qualification information recording section 225, a driving qualification authentication section 223, and a communication state determination section 224.

The operation determination unit 220a determines whether the hydraulic excavator 100 is in operation based on the detected finger movements. As described above, as an example, when the gesture sensing unit 201 detects a movement of pulling three fingers (thumb, index finger, and middle finger) toward the user, the boom of the hydraulic excavator 100 should rise. However, the hydraulic excavator 100 may be overturned for some reason, and in this case, it is preferable that the operation of the hydraulic excavator 100 (that is, raising the boom in an overturned state) be stopped. At this time, the emergency operation control unit 991 stops the operation of the hydraulic excavator 100. Thereby, the hydraulic excavator 100 can be operated more safely.

The qualification information recording unit 225 records qualification information indicating that the operator is authorized to operate the hydraulic excavator 100. For example, the operator qualification authentication unit 223 recognizes the operator of the hydraulic excavator 100 by inputting a password assigned to each operator, and allows the operator to operate the hydraulic excavator 100 based on the qualification information. Authenticate that the person is authorized to perform the operation. If the operator is not authorized to operate the hydraulic excavator 100, the emergency operation control unit 991 stops the operation of the hydraulic excavator 100. This prevents the hydraulic excavator 100 from being operated by an operator who is not qualified to operate the hydraulic excavator 100, and allows the hydraulic excavator 100 to be operated more safely.

The trajectory information recording unit 221 records the trajectory of finger movements for each operator. When the operator is authenticated by the operator qualification authentication unit 223 as a person authorized to operate the hydraulic excavator 100, the configuration unit 222 determines the trajectory of the operator's fingers recorded in the trajectory information recording unit 221. is applied to a hydraulic excavator different from the hydraulic excavator 100. This allows the finger movements of one operator to be applied to various hydraulic excavators. Therefore, for example, even if one hydraulic excavator 100 is being used by another operator, another operator can operate the other hydraulic excavator, and work efficiency at the work site can be improved.

It is not desirable from a safety standpoint for the driver to change while operating the hydraulic excavator 100. Therefore, the driver condition monitoring unit 992 constantly monitors changes in finger movements detected by the gesture sensing unit 201 of the input device 200, such as finger sizes and fingertip movements that differ slightly from operator to operator. As an example, when a child who is not qualified to drive attempts to perform a driving operation, the driver condition monitoring unit 992 detects an abnormality based on the difference between the hand and finger movements of an adult who is qualified to drive. In this case, the emergency operation control unit 991 does not transmit a signal related to operation control to the hydraulic excavator 100 and stops the operation of the hydraulic excavator 100. As a result, after the operator is authenticated by the operator's qualification authentication unit 223 as a person authorized to operate the hydraulic excavator 100, the operator is transferred from the operator who has the operator's qualification to another person who does not have the operator's qualification. Therefore, it is possible to avoid operating the hydraulic excavator 100 instead.

The abnormal movement/abnormal physical condition detection unit 994 detects modulation regarding the operator's own body. For example, if the operator becomes dizzy while operating the hydraulic excavator 100, loses consciousness, or is driving under the influence of alcohol, the gesture sensing unit 201 detects the movement of the fingers over a certain period of time. Unnatural changes are observed in the basic pattern of finger movements recorded in the motion command recording unit 213, such as not being detected, or sudden movements being detected more rapidly than usual. Therefore, the abnormal movement/abnormal physical condition detection unit 994 detects the modulation of the operator's body based on changes in the movements of the operator's fingers. In this case, the emergency operation control unit 991 does not transmit a signal related to operation control to the hydraulic excavator 100 and stops the operation of the hydraulic excavator 100. As a result, even if the operator is authenticated by the operator qualification authentication unit 223 as a person authorized to operate the hydraulic excavator 100, subsequent modulations related to the operator's own body are detected and the hydraulic excavator The safety of 100 operations can be improved.

If an accident occurs while operating the hydraulic excavator 100, the operation information recording unit 993 records the situation as a history. Thereby, even when hydraulic excavators 100 of the same model are operated in an environment similar to the environment in which the accident occurred, information regarding the accident can be shared between the hydraulic excavators. Therefore, accidents can be prevented.

The communication state determination unit 224 determines whether the communication state between the operation verification device 220 and the electronic controller of the hydraulic excavator 100 is normal.

FIG. 22 is a conceptual diagram showing another example of the operating system 160 according to this embodiment. The functions of the input device 200 of the operating device 150 do not necessarily have to be integrated into one housing. As shown in FIG. 22, the gesture sensing unit 201 is provided in two

operating devices

150a and 150b, one operating device 150a is provided with an auxiliary command input area, and the other operating device 150b is provided with a mode selection area and a status display. A region may be provided. For example, when remotely controlling the hydraulic excavator 100, it may be necessary to control the operation of the hydraulic excavator 100 while ensuring visibility from the vehicle body. In this case, by providing a plurality of

operating devices

150a and 150b and also providing a plurality of monitors for visibility from the vehicle body (see FIG. 20), flexible operations such as operation with both hands can be performed while ensuring forward visibility. becomes possible.

FIG. 23 is a flowchart illustrating the operation of a construction machine using the operation system according to this embodiment. In FIG. 23, an authentication process S4a is added between S4 and S5 to determine the presence or absence of driving qualifications. In S4a, the driving qualification authentication unit 223 authenticates that the operator is authorized to operate the hydraulic excavator 100 based on the qualification information recorded in the qualification information recording unit 225. In addition, in S4a, the operator status monitoring unit 992 determines that even if the operator is authenticated by the operating qualification authentication unit 223 as a person who is authorized to operate the hydraulic excavator 100, the operator status monitoring unit 992 determines that the operator Based on this, it is determined that the operator has been replaced by another person who is not qualified to drive, based on the difference between the hand and finger movements of the person who is qualified to drive. Further, in S4a, even if the operator is authenticated by the operator qualification authentication unit 223 as a person authorized to operate the hydraulic excavator 100, the abnormal operation/abnormal physical condition detection unit 994 detects that the operator Based on changes in the movements of the operator's fingers, the operator's physical condition (such as alcohol consumption) is determined. After that, in S5, the communication state determination unit 224 determines whether the communication state between the operation verification device 220 and the electronic controller of the hydraulic excavator 100 is normal. If the communication is normal, S6 is executed.

Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the operating device 150 and the operating system 160 according to the above embodiments, and is within the concept of the present invention and the scope of the claims. This includes all aspects that may occur. Moreover, each structure may be selectively combined as appropriate so as to achieve at least some of the problems and effects described above. For example, the shape, material, arrangement, size, etc. of each component in the above embodiments may be changed as appropriate depending on the specific aspect of the present invention.

Here, FIG. 24 is a conceptual diagram illustrating a detection method in the gesture sensing section of the operating device according to the present embodiment. In the embodiment described above, a case has been described in which the gesture sensing unit 201 detects the movement of a finger that is in contact with the gesture sensing area 301. However, the gesture sensing area and the gesture sensing unit may use any detection method such as an optical type, an electrostatic type, or a pressure-sensitive type as long as the state and movement of the fingers can be detected. That is, as long as the movement of the fingers can be determined, a fingertip sensor method (gesture sensing area 301a in the middle part of FIG. 24) or an optical camera method (gesture sensing area 301b in the lower part of FIG. 24) may be used. . For example, by providing the

gesture sensing areas

301a and 301b with a simple configuration such as a fingertip sensor type or an optical camera type, it is possible to have the effect of being lighter than the so-called touch panel type gesture sensing area 301 and requiring less time and effort to prepare. be.

The input device 200 may be a commercially available smartphone or tablet. In this case, using a general-purpose product eliminates the need for a special operating device. Furthermore, to improve operability, multiple smartphones or tablet computers may be used. When using wireless communication, WiFi, local wireless, or business wireless may be used, and a public communication line or a dedicated line may be used.

100 Hydraulic excavator (construction machinery)
150 Operation device 160 Operation system 200 Input device 210 Operation determination device (control device)
220 Operation verification device (server device)

Claims

An operating device for construction machinery operated by an operator,
an input device into which operation information from the operator is input;
a control device that outputs an operation command to a drive device of the construction machine based on the input operation information,
A gesture sensing area is set in the input device,
The input device detects the movement of the operator's fingers in the gesture sensing area,
The control device outputs the operation command to the drive device so that the construction machine moves according to the movement of the hand.
An operating device for construction machinery characterized by:
The input device detects a trajectory of the finger movement of the operator,
The control device includes:
recording a basic pattern of a trajectory of hand and finger movements reminiscent of the movement of the construction machine, and information on the movement part and type of movement of the construction machine that is associated with the basic pattern;
Compare the trajectory of the detected finger movement with the basic pattern,
identifying a movement part and movement type of the construction machine corresponding to the trajectory;
generating the movement command according to the identified movement part and movement type;
transmitting the operation command to the drive device;
The operating device for construction machinery according to claim 1, characterized in that:
The input device includes, as auxiliary commands to assist the operation type, a command for determining an operation in a specified operation type and a stop of the operation, a command for determining continuous operation in the specified operation type, and a command for determining a continuous operation in the specified operation type; At least one of the commands for determining an independent action that performs only the action in the specified action type is input,
The control device includes:
Generating an auxiliary operation command different from the operation command in response to the auxiliary command,
transmitting the auxiliary operation command to the drive device;
The operating device for construction machinery according to claim 2, characterized in that:
The control device includes:
Recording physical information regarding the hands and fingers of each operator,
changing the basic pattern of the trajectory of the finger movement according to the physical information;
recording a changed pattern of the changed trajectory, and a movement part and movement type of the construction machine that are associated with the changed pattern;
comparing the detected hand and finger movement trajectory with the changed pattern;
The operating device for construction machinery according to claim 2, characterized in that:
The operating device for a construction machine according to claim 2, wherein the input device notifies the operator that the construction machine is ready for operation.
The input device detects the speed of finger movement of the operator,
The control device includes:
determining the movable range of a working machine installed in the construction machine as the type of the construction machine;
In accordance with the result of the determination, determining the operating speed of the working machine of the construction machine with respect to the detected speed of finger movement;
generating the operation command according to the determined operation speed;
The operating device for construction machinery according to claim 2, characterized in that:
The control device includes:
In the case of a construction machine in which the working machine has a wide range of motion, generating a motion command that increases the operating speed of the working machine relative to the speed of the hand and finger movement;
In the case of a construction machine in which the working machine has a narrow range of movement, generating a motion command that slows down the operating speed of the working machine relative to the speed of movement of the fingers;
7. The construction machine operating device according to claim 6.
An operating system for a construction machine using the operating device according to any one of claims 2 to 7,
a server device capable of wireless communication with the operating device;
The server device determines whether or not to operate the construction machine according to the operation type specified by the operating device.
A construction machine operation system characterized by:
The server device includes:
determining whether the construction machine is operating based on the detected trajectory of the movement of the hand and fingers;
If it is determined that the construction machine is not operating in a motion corresponding to the trajectory, controlling the drive device so that the construction machine stops;
9. The construction machine operating system according to claim 8.
The server device includes:
Recording qualification information indicating that the operator is a person authorized to operate the construction machine, and based on the qualification information, confirming that the operator is a person authorized to operate the construction machine. authenticate,
controlling the drive device so that the construction machine stops if the operator is not authorized to operate the construction machine;
9. The construction machine operating system according to claim 8.
The server device includes:
recording a basic pattern of the trajectory of the finger movement for each operator;
If the operator is a person authorized to operate the construction machine, applying the basic pattern of the hand and finger trajectory of the operator to a construction machine different from the construction machine;
The construction machine operating system according to claim 10.