JP2021042569A - Information processing device, information processing system, information processing method, computer program, and construction machine - Google Patents

Abstract

To provide an information processing device for a work machine that can estimate the moving speed of a work unit with a simple configuration.SOLUTION: An information processing device 10 includes a storage unit 30, an image information acquisition unit 12, and a speed estimation unit 20e. The storage unit 30 stores the correspondence information generated by associating data Gs of the reference image of a work unit 40 of a work machine 100 with posture data Ks of the work unit 40 of the work machine 100. The image information acquisition unit 12 acquires image data Gj of the work unit 40 of the work machine 100 for comparison with the data Gs of the reference image. The speed estimation unit 20e generates speed information Ve regarding the speed of the work unit 40 of the work machine 100 based on the image data Gj acquired by the image information acquisition unit 12 and the correspondence information.SELECTED DRAWING: Figure 2

Description

The present invention relates to information processing devices, information processing systems, information processing methods, computer programs and construction machines.

Construction machines equipped with cameras that image working equipment are known. For example, Patent Document 1 describes a construction machine including a camera installed on a swivel body to image a working device, an angle detecting unit for detecting a relative angle, and a posture specifying unit for specifying a posture. .. This construction machine detects the relative angle between the links from the edge of each link extracted based on the image of the camera, and identifies the posture of the working device with respect to the swivel body based on the relative angle.

JP-A-2017-053627

The present inventors have obtained the following recognition about a construction machine provided with a boom arm and an attachment driven by power such as flood control.

A certain construction machine uses power to drive an arm mechanism including a boom and an arm, and moves an attachment such as a bucket attached to the arm mechanism to perform a predetermined construction. For example, when constructing the ground, it is desirable to perform accurate construction in a short time and efficiently according to the design drawing. However, when moving the attachment, if the moving speed is too fast, the overshoot becomes large and the positioning accuracy deteriorates. On the contrary, if the moving speed is too slow, the moving time becomes long and the work efficiency decreases. Therefore, from the viewpoint of improving the movement accuracy of the attachment, it is necessary to detect the movement speed of the attachment and perform precise airframe control.

In order to detect the moving speed, it is conceivable to provide a position sensor in each part of the arm mechanism and the attachment, and estimate the speed from the change in the detection result per time. However, in this configuration, the configuration is complicated because the sensor and its wiring are provided. In addition, the cost of the sensor and wiring is extra, which is disadvantageous in terms of cost. From the viewpoint of detecting the moving speed, it cannot be said that the construction machine described in Patent Document 1 sufficiently copes with this problem. Such a problem may occur not only in the above-mentioned construction machines but also in other types of work machines.

The present invention has been made in view of these problems, and one object of the present invention is to provide an information processing device for a work machine capable of estimating the moving speed of a work unit with a simple configuration.

In order to solve the above-mentioned problems, the information processing apparatus according to an aspect of the present invention provides correspondence information generated by associating the data of the reference image of the working part of the working machine with the data of the posture of the working part of the working machine. Work based on the storage unit that is stored, the acquisition unit that acquires the image data of the work unit of the work machine for comparison with the data of the reference image, and the image data acquired by the acquisition unit and the corresponding information. It is provided with a speed estimation unit that generates speed information regarding the speed of the working unit of the machine.

It should be noted that any combination of the above, and those in which the components and expressions of the present invention are mutually replaced between methods, devices, programs, temporary or non-temporary storage media on which programs are recorded, systems, and the like are also used. It is effective as an aspect of the present invention.

According to the present invention, it is possible to provide an information processing device for a work machine capable of estimating the moving speed of a work unit with a simple configuration.

It is a side view which shows typically the work machine provided with the information processing apparatus of 1st Embodiment. It is a block diagram which shows schematic the information processing apparatus of FIG. It is explanatory drawing explaining the posture of the work part of the work machine of FIG. It is a figure which shows the learning data of the information processing apparatus of FIG. It is explanatory drawing explaining the posture estimation process of the information processing apparatus of FIG. It is explanatory drawing explaining the speed estimation processing and acceleration estimation processing of the information processing apparatus of FIG. It is a figure which shows an example of attitude information, velocity information, and acceleration information of the information processing apparatus of FIG. It is a block diagram which shows schematic the abnormality detection part of the information processing apparatus of FIG. It is a block diagram which shows schematic the appearance detection part of the information processing apparatus of FIG. It is a flowchart which shows the operation of the information processing apparatus of FIG. It is a flowchart which shows the operation of the information processing apparatus of FIG. It is a flowchart which shows the operation of the information processing apparatus of FIG. It is a flowchart which shows the operation of the information processing apparatus of FIG. It is a block diagram which shows schematic the work machine which used the information processing system of 5th Embodiment.

Hereinafter, the present invention will be described with reference to each drawing based on a preferred embodiment. In the embodiments and modifications, the same or equivalent components and members are designated by the same reference numerals, and redundant description will be omitted as appropriate. In addition, the dimensions of the members in each drawing are shown enlarged or reduced as appropriate for easy understanding. In addition, some of the members that are not important for explaining the embodiment in each drawing are omitted and displayed.

Also, terms including ordinal numbers such as 1st and 2nd are used to describe various components, but this term is used only for the purpose of distinguishing one component from other components. The components are not limited by.

[First Embodiment]
The configuration of the information processing apparatus 10 of the work machine according to the first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a side view schematically showing a work machine 100 provided with the information processing device 10 of the first embodiment. FIG. 2 is a block diagram schematically showing the information processing apparatus 10.

The information processing device 10 includes an image information acquisition unit 12, an environment information acquisition unit 14, a posture information acquisition unit 16, a control unit 20, and a storage unit 30. The information processing device 10 has a machine learning time and a normal operation time (hereinafter, referred to as “non-learning operation time”) other than the machine learning time. At the time of machine learning, the information processing device 10 can generate a posture estimation model based on the learning image information and the learning posture information of each part of the work unit 40 of the work machine 100. This attitude estimation model exemplifies correspondence information. The information processing device 10 can estimate the posture of the working unit 40 based on real-time image information and a posture estimation model during a non-learning operation. The work machine 100 can control the operation of the work unit 40 based on the posture of the work unit 40 estimated by the information processing device 10.

The image information acquisition unit 12 acquires image data of the work unit 40. The environmental information acquisition unit 14 acquires information on the surrounding environment of the work machine 100. The posture information acquisition unit 16 acquires data related to the posture of the work unit 40 (hereinafter, referred to as “posture data”). The control unit 20 executes various data processing related to the generation of the posture estimation model and the posture estimation of the work unit 40. The storage unit 30 stores data that is referenced or updated by the control unit 20. First, the configuration of the work machine 100 will be described, and other configurations will be described later.

The work machine 100 of the present embodiment is a construction machine that moves a bucket 46 to perform work, and functions as a so-called power excavator. The work machine 100 has a lower traveling portion 36, an upper vehicle body portion 34, an arm mechanism 48, and a bucket 46. In the present embodiment, the arm mechanism 48 and the bucket 46 form a working unit 40. The lower traveling portion 36 is configured to be able to travel in a predetermined direction by an endless track or the like. The upper vehicle body portion 34 is mounted on the lower traveling portion 36. The upper vehicle body portion 34 and the working portion 40 are configured to be rotatable around the turning shaft La with respect to the lower traveling portion 36 by the turning drive unit 60. The swivel drive unit 60 can be composed of, for example, a swivel motor (not shown) and a swivel gear (not shown). A cockpit 38 is provided in the upper vehicle body portion 34.

The control room 38 is provided with an operation unit 54 that controls the work unit 40. When an operation is input from the operation unit 54, the plurality of hydraulic valves 58 open and close according to the operation. The hydraulic oil supplied from the hydraulic pump (not shown) is sent to the plurality of hydraulic cylinders 56 according to the opening and closing of the hydraulic valve 58. The hydraulic cylinder 56 includes hydraulic cylinders 56a, 56b, and 56c that are sequentially arranged from the base end side to the tip end side of the arm mechanism 48. The hydraulic cylinders 56a, 56b, and 56c expand and contract according to the amount of hydraulic oil delivered. The cockpit 38 is provided with a display unit 38d for displaying information from an abnormality detection unit and an appearance detection unit, which will be described later.

Further, the work machine 100 of the present embodiment has a work machine control unit 62 that controls the operation of the work unit 40 based on a command from a higher-level control system. The work machine control unit 62 will be described later.

The base end portion of the arm mechanism 48 is provided on the right side of the cockpit 38 in the upper vehicle body portion 34 as an example. The arm mechanism 48 includes, for example, a boom 42 and an arm 44 extending forward from the upper body portion 34. A bucket 46 is attached to the tip end side of the arm mechanism 48. In this way, the work machine 100 can drive the bucket 46 to perform the desired work by changing the posture of the work unit 40 according to the operation of the operator. Further, the work machine 100 can move the bucket 46 three-dimensionally by turning the upper vehicle body portion 34 and the work portion 40.

FIG. 3 is an explanatory diagram illustrating the posture of the working unit 40. The expansion and contraction lengths L1, L2, and L3 of the hydraulic cylinders 56a, 56b, and 56c can be changed according to the flood pressure. The boom 42 is configured such that the tip portion rotates up and down around the base end portion on the upper vehicle body portion 34 side due to the expansion and contraction of the hydraulic cylinder 56a. The arm 44 is configured such that the tip end portion rotates back and forth around the base end portion on the boom 42 side due to the expansion and contraction of the hydraulic cylinder 56b. The bucket 46 is configured such that the tip portion rotates back and forth or up and down around the base end portion on the arm 44 side due to the expansion and contraction of the hydraulic cylinder 56c.

The working unit 40 changes the flexion angles θ1, θ2, and θ3 of the joints connecting the boom 42, the arm 44, and the bucket 46 by changing the expansion and contraction lengths L1, L2, and L3 of the hydraulic cylinders 56a, 56b, and 56c. be able to. As an example, the angle θ1 is the angle of the boom 42 with respect to the horizontal plane, the angle θ2 is the bending angle of the joint connecting the boom 42 and the arm 44, and the angle θ3 is the bending angle of the joint connecting the arm 44 and the bucket 46. The bending angle of.

The posture of the working unit 40 can be defined by the positions and relative angles of the boom 42, the arm 44 and the bucket 46. Assuming that the shapes of the boom 42, the arm 44 and the bucket 46 are constant, the posture of the working portion 40 is the size of each part of the boom 42, the arm 44 and the bucket 46 and the expansion and contraction lengths L1, L2 and L3 of the hydraulic cylinders 56a, 56b and 56c. Alternatively, it can be specified by geometric calculation based on the bending angles θ1, θ2, and θ3.

Return to FIG. Each block shown in FIG. 2 can be realized by elements such as a computer processor, CPU, and memory, electronic circuits, and mechanical devices in terms of hardware, and can be realized by a computer program or the like in terms of software. Then, I draw a functional block realized by their cooperation. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by combining hardware and software.

The image information acquisition unit 12 will be described. The image information acquisition unit 12 of the present embodiment has an image sensor that captures an image of the work unit 40. At the time of machine learning, which will be described later, the image information acquisition unit 12 provides the control unit 20 with the image capture result obtained by imaging the work unit 40 as image data. Hereinafter, the image data acquired during machine learning is referred to as "reference image data Gs". The image information acquisition unit 12 provides the control unit 20 with the image capture result of the image of the work unit 40 as image data even during the non-learning operation. Hereinafter, the image data acquired during the non-learning operation is simply referred to as "image data Gj". The image data Gj may be real-time image data.

The image information acquisition unit 12 is configured to rotate integrally with the work unit 40. Specifically, the image information acquisition unit 12 is arranged so that the work unit 40 can be imaged on the roof of the cockpit 38. When the working unit 40 turns, the image information acquisition unit 12 moves integrally with the working unit 40, so that the relative positional relationship with the working unit 40 does not change even if the working unit 40 turns.

The posture information acquisition unit 16 will be described. The posture information acquisition unit 16 of the present embodiment includes stroke sensors 16a, 16b, 16c that acquire the expansion / contraction lengths L1, L2, and L3 of the hydraulic cylinders 56a, 56b, and 56c. The posture information acquisition unit 16 is attached during machine learning and is removed during non-learning operation. The posture information acquisition unit 16 provides the control unit 20 with data of the expansion / contraction lengths L1, L2, and L3 during machine learning.

The environmental information acquisition unit 14 will be described. When acquiring image data, if the surrounding environment such as the weather is different, the brightness and color temperature of the image will also be different, which will increase the error in posture estimation. Therefore, in the present embodiment, the environment information acquisition unit 14 acquires information on the surrounding environment, and corrects the image data according to the acquisition result. The environmental information acquisition unit 14 of the present embodiment includes an illuminance sensor that acquires the ambient brightness and a color temperature sensor that acquires the color temperature. The environmental information acquisition unit 14 provides the acquisition result as the environmental information Mp to the control unit 20. In this example, the environmental information acquisition unit 14 is arranged on the roof of the cockpit 38.

The storage unit 30 will be described. The storage unit 30 includes a model storage unit 32. The model storage unit 32 stores a posture estimation model that estimates the posture of the working unit 40, which is a model generated by known machine learning based on the data Gs of the reference image and the posture data. The posture estimation model can be said to be a function in which the input and output data formats are predetermined. Image data Gj is input to the posture estimation model of the embodiment. Further, the posture estimation model of the embodiment outputs information on the estimated posture corresponding to the image data. The method for generating the posture estimation model will be described later.

The control unit 20 will be described. The control unit 20 includes a model generation unit 22, an image information reception unit 20a, a posture information reception unit 20b, a posture estimation unit 20c, a position estimation unit 20d, a speed estimation unit 20e, an acceleration estimation unit 20f, and an individual. It includes an information holding unit 20h, an environmental information receiving unit 20g, an image information correction unit 20j, a background information removing unit 20k, and a color information removing unit 20m. Further, the control unit 20 includes an abnormality detection unit 24 and an appearance detection unit 26. An application program in which a plurality of modules corresponding to the plurality of functional blocks are mounted may be installed in a storage (for example, a storage unit 30) of the information processing apparatus 10. The processor (for example, CPU) of the information processing device 10 may exert the function of each functional block by reading the application program into the main memory and executing the program.

First, the model generation unit 22, the image information reception unit 20a, the posture information reception unit 20b, and the posture estimation unit 20c will be described. The image information receiving unit 20a receives input of the imaging result of the working unit 40 from the image information acquisition unit 12. In particular, at the time of machine learning, the image information receiving unit 20a receives the data Gs of the reference image for learning from the image information acquisition unit 12. Further, during the non-learning operation, the image information receiving unit 20a receives the image data Gj from the image information acquisition unit 12.

The posture information reception unit 20b receives input of posture information of the work unit 40 from the posture information acquisition unit 16. Specifically, the posture information receiving unit 20b receives the data of the expansion / contraction lengths L1, L2, and L3 of the stroke sensors 16a, 16b, and 16c during machine learning. When the data of the expansion and contraction lengths L1, L2, and L3 are collectively referred to as the posture data Ks.

FIG. 4 is a diagram showing learning data of the information processing device 10. In this figure, in order to facilitate understanding, the data Gs of the reference image from the image information acquisition unit 12 is replaced with the figure on the plane together with the posture data Ks. At the time of machine learning, the control unit 20 associates the received reference image data Gs with the posture data Ks and stores them in the storage unit 30. At the time of machine learning, the control unit 20 widely changes the posture of the work unit 40 within the movable range, and each time the change is made, the reference image data Gs and the posture data Ks are associated with each other and stored in the storage unit 30. The reference image data Gs and the posture data Ks stored in the storage unit 30 are referred to as learning data Sd. It is desirable that the learning data Sd covers the postures that the working unit 40 can take. Therefore, as shown in FIG. 4, the learning data Sd includes a large amount of reference image data Gs and posture data Ks associated with each other.

The model generation unit 22 generates a posture estimation model by using the data Gs of the reference images and the posture data Ks of the learning data Sd associated with each other as teacher data. The model generation unit 22 of the present embodiment generates a posture estimation model by machine learning (teachered learning) using the reference image data Gs and the posture data Ks as teacher data. The model generation unit 22 may generate a posture estimation model using a known machine learning method such as a support vector machine, a neural network (including deep learning), or a random forest. The model generation unit 22 stores the generated posture estimation model in the model storage unit 32.

The posture estimation unit 20c estimates the posture of the working unit 40 based on the image data Gj and the stored information of the storage unit 30 during the non-learning operation. As an example, the posture estimation unit 20c compares the image data Gj with the reference image data Gs of the training data Sd, and obtains the posture data Ks associated with the reference image data Gs having the highest degree of similarity. It may be an estimation result. In this case, since the data Gs of a large number of reference images are referred to, it may take time to obtain the result. Therefore, the posture estimation unit 20c of the present embodiment derives the estimated posture from the image data Gj by using the posture estimation model generated by the model generation unit 22.

FIG. 5 is an explanatory diagram illustrating a posture estimation process in the posture estimation model generated by the model generation unit 22. When the image data Gj is input, this posture estimation model outputs the estimated posture information Ke corresponding to the image data. The estimated posture information Ke of the present embodiment includes the flexion angles θ1, θ2, and θ3 of the joints of the boom 42, the arm 44, and the bucket 46 of the working unit 40. The posture estimation unit 20c transmits the estimated posture information Ke to the work machine control unit 62 of the work machine 100.

Next, the position estimation unit 20d, the velocity estimation unit 20e, and the acceleration estimation unit 20f will be described. FIG. 6 is an explanatory diagram illustrating a speed estimation process in the speed estimation unit 20e. In FIG. 6, the posture of the working unit 40 at a certain timing T (n) is shown by a solid line, and the posture of the working unit 40 at the timing T (n-1) past the timing T (n) is shown by a broken line. FIG. 7 shows an example of position information Pe, velocity information Ve, and acceleration information Ae at each timing T. In this figure, the position information Pe at the timing T (n) is indicated by Pe (n), the velocity information Ve is indicated by Ve (n), and the acceleration information Ae is indicated by Ae (n). The position information Pe, the velocity information Ve, and the acceleration information Ae will be described later.

The position estimation unit 20d estimates the position of a predetermined portion of the work unit 40 based on the estimated posture information Ke. The predetermined portion is not limited, but in this example, the predetermined portion is the bucket 46. The position of the bucket 46 can be specified by calculation based on the shape information of the boom 42, the arm 44 and the bucket 46 and the estimated posture information Ke (bending angles θ1, θ2, θ3). The shape information of the boom 42, the arm 44, and the bucket 46 can be stored in the storage unit 30 in advance.

Information about the position estimated by the position estimation unit 20d is called position information Pe. The position estimation unit 20d transmits the position information Pe to the work machine control unit 62 and stores it in the storage unit 30 in time series. As an example, the position information Pe may include data of expansion / contraction lengths L1, L2, and L3. The position information Pe may be two-dimensional or three-dimensional information.

The speed estimation unit 20e generates speed information Ve based on the data Gj of a plurality of images acquired at a plurality of timings having a time difference and the time difference. In particular, the speed estimation unit 20e estimates the speed of a predetermined portion (for example, the bucket 46) of the work unit 40 based on the position information Pe, and generates speed information Ve related to the speed. The velocity of the bucket 46 can be obtained by differentiating the change in the position of the bucket 46 with respect to time. For example, with respect to the position information Pe stored in the storage unit 30 in time series, the difference between the plurality of position information Pes generated and stored at a plurality of different timings is divided by the time difference dT of the timings of the bucket 46. The speed can be estimated. This time difference dT may be the time difference for each frame of the image data Gj (one frame of the still image that is the basis of the image).

As an example, the timing T (n) may be the timing of the current frame, and the timing T (n-1) may be the timing one or more frames before the current frame. In this case, the time difference dT is the reciprocal of the frame rate (the number of frames per second (fps)). Information about the speed estimated by the speed estimation unit 20e is called speed information Ve. In this example, the velocity information Ve (n) at the timing T (n) is obtained by the equation (1).
Ve (n) = (Pe (n) -Pe (n-1)) / dT ... (1)
That is, the velocity information Ve is obtained by dividing the difference between the two position information Pes of each of the two frames by the time difference of the frames.

The speed estimation unit 20e transmits the speed information Ve to the work machine control unit 62 and stores it in the storage unit 30 in time series. The velocity information Ve may be two-dimensional or three-dimensional information.

The acceleration estimation unit 20f generates acceleration information Ae regarding the acceleration of the working unit 40 based on the plurality of velocity information Ve and the time difference. In particular, the acceleration estimation unit 20f estimates the acceleration of a predetermined portion (for example, the bucket 46) of the working unit 40 based on the velocity information Ve, and generates acceleration information Ae related to the acceleration. The acceleration of the bucket 46 can be obtained by differentiating the velocity change of the bucket 46 with respect to time. For example, with respect to the speed information Ve stored in the storage unit 30 in time series, the acceleration of the bucket 46 is obtained by dividing the difference between a plurality of speed information Ve generated and stored at different timings by the time difference dT of the specific timing. Can be estimated. This time difference dT may be the time difference of each frame of the image data Gj. Information about the acceleration estimated by the acceleration estimation unit 20f is called acceleration information Ae. In this example, Ae (n) at the timing T (n) is calculated by the equation (2).
Ae (n) = (Ve (n) -Ve (n-1)) / dT ... (2)
That is, the acceleration information Ae is obtained by dividing the difference between the two velocity information Ves of each of the two frames by the time difference of the frames.

The acceleration estimation unit 20f transmits the acceleration information Ae to the work machine control unit 62 and stores it in the storage unit 30 in time series. The acceleration information Ae may be two-dimensional or three-dimensional information. Hereinafter, the position information Pe, the speed information Ve, and the acceleration information Ae are collectively referred to as feedback information.

Returning to FIG. 2, the work machine control unit 62 will be described. The work machine control unit 62 controls the operation of the work machine 100 based on the command information from the upper control system. The command information of the present embodiment relates to the target position information Ps regarding the position of the predetermined portion (bucket 46), the target speed information Vs regarding the speed of the predetermined portion (bucket 46), and the acceleration of the predetermined portion (bucket 46). It includes the target acceleration information As. Hereinafter, the target position information Ps, the target speed information Vs, and the target acceleration information As are collectively referred to as target information.

As shown in FIG. 2, the work machine control unit 62 has an operation control unit 62a. The operation control unit 62a gives the control signal Sj to the flood control valve 58 and the control signal Sk to the swivel drive unit 60 by a predetermined control algorithm based on the target information and the feedback information. The operation control unit 62a controls the opening and closing of the hydraulic valve 58 by the control signal Sj, expands and contracts the hydraulic cylinders 56a, 56b, and 56c, and controls the operation of the boom 42, the arm 44, and the bucket 46. The motion control unit 62a can control a predetermined portion (bucket 46) to a desired position by feeding back the position information Pe. Further, the operation control unit 62a can control the turning drive unit 60 by the control signal Sk.

The motion control unit 62a can control the speed of a predetermined portion (bucket 46) to a desired level by feeding back the speed information Ve. Further, by feeding back the differential element of the position information Pe, overshoot and hunting of the position control can be suppressed. Further, the motion control unit 62a can control the acceleration of a predetermined portion (bucket 46) to a desired level by feeding back the acceleration information Ae. Further, by feeding back the differential element of the speed information Ve, overshoot and hunting of speed control can be suppressed. As described above, according to the present embodiment, highly responsive and highly accurate control can be realized.

Next, the abnormality detection unit 24 will be described. The abnormality detection unit 24 detects a malfunction state or a failure state of the work unit 40 based on the estimated posture information Ke, and outputs the detection result to the host control system. FIG. 8 is a block diagram schematically showing the abnormality detection unit 24. As shown in this figure, the abnormality detection unit 24 of the present embodiment includes a prediction posture calculation unit 24b, an abnormality determination unit 24c, and a state output unit 24d. The predicted posture calculation unit 24b calculates information on the posture of the work unit 40 (hereinafter, referred to as “predicted posture information Ka”) that is expected as a result of the work unit 40 being controlled. That is, the expected posture calculation unit 24b determines the original posture of the work unit 40 as a result of the control.

In the present embodiment, the predicted posture calculation unit 24b simulates the opening / closing operation of the hydraulic valve 58 and the expansion / contraction lengths L1, L2, and L3 of the hydraulic cylinders 56a, 56b, and 56c based on the control signal Sj. The expected posture information Ka is determined based on L2 and L3. Further, as another example, the hydraulic pressure supplied to the hydraulic cylinders 56a, 56b, 56c is detected by the hydraulic sensor, and the predicted posture calculation unit 24b detects the expansion and contraction lengths L1, L2, and L3 based on the detection result of the hydraulic sensor. May be simulated. Further, the expansion / contraction lengths L1, L2, and L3 may be simulated based on the detection result of the oil pressure sensor and the control signal Sj.

The abnormality determination unit 24c determines the operation abnormality of the work unit 40 based on the estimated posture information Ke derived from the image data Gj and the control signal Sj. In particular, the abnormality determination unit 24c determines a malfunction state or a failure state of the work unit 40 according to the difference dK between the estimated posture information Ke and the expected posture information Ka. The abnormality determination unit 24c may divide the difference dK by a plurality of threshold values and output the division as the determination result J24. For example, the abnormality determination unit 24c uses a first threshold value and a second threshold value larger than the first threshold value. The abnormality determination unit 24c determines that the difference dK is less than the first threshold value as a normal state, and if the difference dK is greater than or equal to the first threshold value and less than the second threshold value, determines that the state is defective, and the difference dK is greater than or equal to the second threshold value. In that case, it may be determined as a failure state. In this specification, a state in which there is a high possibility of failure and the operation should be stopped immediately is defined as a failure state, and a state in which a failure may occur and should be inspected at an early stage is defined as a failure state.

The state output unit 24d presents the determination result J24 of the abnormality determination unit 24c to the host control system or a predetermined device. In this example, the state output unit 24d causes the display unit 38d provided in the cockpit 38 to display the determination result J24 of the abnormality determination unit 24c. The state output unit 24d may display the determination result of the abnormality determination unit 24c on a mobile terminal (not shown) possessed by the operator or the administrator via a communication means such as a network.

Next, the appearance detection unit 26 will be described. The appearance detection unit 26 displays the appearance shape that should exist, such as one of the constituent members such as the hydraulic cylinders 56a, 56b, and 56c being removed from the image data Gj acquired by the image information acquisition unit 12. If not, it is determined that the working unit 40 is externally damaged. The appearance detection unit 26 detects the presence or absence of an abnormality in the appearance of the work unit 40 based on the estimated posture information Ke, and outputs the detection result to the host control system. FIG. 9 is a block diagram schematically showing the appearance detection unit 26. As shown in this figure, the appearance detection unit 26 of the present embodiment includes a reference image generation unit 26b, an appearance determination unit 26c, and a state output unit 26d.

The reference image generation unit 26b generates data related to the image of the work unit 40 in a state considered to be normal in the posture (hereinafter, referred to as “reference image data Gh”) based on the estimated posture information Ke. The reference image data Gh may be data related to the image of the working unit 40 in the normal initial state. The image data for each part of the working unit 40 is stored in the storage unit 30 in advance, and the reference image generation unit 26b synthesizes the image data of each part stored based on the estimated posture information Ke, and the data of the reference image. Gh can be generated.

The appearance determination unit 26c determines the appearance abnormality of the work unit 40 based on the image data Gj and the reference image data Gh. In particular, the appearance determination unit 26c compares the image data Gj with the reference image data Gh generated by the reference image generation unit 26b, and according to the degree of difference (hereinafter, referred to as “difference degree Sh”). It is determined whether or not there is an abnormality in the appearance of the working unit 40. In this example, when the degree of similarity between the two images is high, the degree of difference Sh is low, and when the degree of similarity between the two images is low, the degree of difference Sh is high.

The appearance determination unit 26c may divide the difference degree Sh by a plurality of threshold values and output the division as the determination result J26. For example, the appearance determination unit 26c can use a first threshold value and a second threshold value larger than the first threshold value. The appearance determination unit 26c determines that there is no appearance abnormality when the difference degree Sh is less than the first threshold value, and determines that there is a partial appearance abnormality when the difference degree Sh is equal to or more than the first threshold value and is less than the second threshold value. When the degree Sh is equal to or higher than the second threshold value, it may be determined that there is an appearance abnormality.

The state output unit 26d presents the determination result J26 of the appearance determination unit 26c to the host control system or a predetermined device. In this example, the state output unit 26d causes the display unit 38d provided in the cockpit 38 to display the determination result of the appearance determination unit 26c. The state output unit 26d may display the determination result of the appearance determination unit 26c on a mobile terminal (not shown) possessed by the operator or the administrator via a communication means such as a network.

Returning to FIG. 2, the image information correction unit 20j, the environmental information reception unit 20g, the individual information holding unit 20h, the background information removal unit 20k, and the color information removal unit 20m will be described.

The image information correction unit 20j of the present embodiment corrects the image data Gj according to the information on the surrounding environment when acquiring the image data Gj, the age of the work machine, or the individual difference of the work machine. Further, the environmental information receiving unit 20g and the individual information holding unit 20h provide correction information to the image information correction unit 20j.

The environmental information reception unit 20g accepts the input of the acquisition result from the environmental information acquisition unit 14. In particular, the environmental information reception unit 20g receives the environmental information Mp from the environmental information acquisition unit 14. The image information correction unit 20j corrects the brightness and color temperature of the image data Gj based on the environmental information Mp. The image information correction unit 20j corrects the brightness of the image data Gj so as to be the same as the brightness of the reference image data Gs. The image information correction unit 20j corrects the color temperature of the image data Gj so as to be the same as the color temperature of the reference image data Gs. With this configuration, the estimation error due to the surrounding environment can be reduced.

The appearance of the work unit 40 varies from individual to individual work machine 100. This individual difference may be an error factor in posture estimation. Therefore, the individual information holding unit 20h of the present embodiment holds the individual information Me of each individual. The individual information Me includes information on individual differences in appearance due to age of the work machine 100, scratches on the work unit 40, deposits, deformation, and the like. The image information correction unit 20j corrects the image data Gj according to the individual information Me. With this configuration, the estimation error due to individual differences can be reduced.

The image data Gj includes a background image that differs depending on the site where the work machine 100 operates. Therefore, the background image included in the image data Gj may be an error factor in the posture estimation. Therefore, the background information removing unit 20k of the present embodiment removes information about the background image from the image data Gj. With this configuration, the estimation error caused by the background image can be reduced.

If the reference image data Gs and the image data Gj are stored and processed as full-color image data, the amount of data to be handled becomes large, which is disadvantageous in terms of processing speed and storage capacity. Therefore, the color information removing unit 20m of the present embodiment removes the color information from the reference image data Gs and the image data Gj to obtain grayscale image data. With this configuration, the amount of data to be handled is reduced, which is advantageous in terms of processing speed and storage capacity.

The operation of the information processing apparatus 10 configured as described above will be described. FIG. 10 is a flowchart showing the operation of the information processing device 10. This figure shows an operation S70 that generates a posture estimation model by machine learning during machine learning. The operation S70 is started at the timing when the posture information acquisition unit 16 is attached to the work machine 100 in advance and the instruction for model creation is input by the administrator.

When the model generation timing is reached (Y in step S71), the control unit 20 of the information processing device 10 receives the reference image data Gs and the attitude data Ks from the image information acquisition unit 12 and the attitude information acquisition unit 16 (Y). Step S72). In this step, the control unit 20 widely changes the posture of the work unit 40 within the movable range, and each time the change is made, the control unit 20 receives the reference image data Gs and the posture data Ks and stores them in the storage unit 30.

The background information removing unit 20k removes information about the background image from the data Gs of the reference image (step S73). The color information removing unit 20m removes color information from the data Gs of the reference image (step S74). The removal of the background image and the removal of the color information may be executed for the received reference image data Gs each time, or may be executed for the reference image data Gs stored in the storage unit 30. ..

The model generation unit 22 generates a posture estimation model by machine learning based on the data Gs of the reference image from which the background image and the color information have been removed and the posture data Ks, and stores the posture estimation model in the model storage unit 32 (step S75). .. When the posture estimation model is stored, the operation S70 ends. After the end of the operation S70, the posture information acquisition unit 16 may be removed from the work machine 100.

If the model generation timing is not reached (N in step S71), S72 to S75 are skipped. This operation S70 is merely an example, and the order of the steps may be changed, or some steps may be added / deleted / changed.

FIG. 11 is also a flowchart showing the operation of the information processing device 10. This figure shows an operation S80 that estimates the posture of the working unit 40 based on the image data Gj using the posture estimation model. The operation S80 is started at the timing when the posture estimation instruction is input by the administrator during the non-learning operation.

When the posture estimation timing is reached (Y in step S81), the control unit 20 of the information processing device 10 receives the image data Gj from the image information acquisition unit 12 (step S82). In this step, the received image data Gj is stored in the storage unit 30.

The background information removing unit 20k removes information about the background image from the image data Gj (step S83). The color information removing unit 20m removes color information from the image data Gj (step S84).

The image information correction unit 20j corrects the image data Gj according to the individual information Me held in the individual information holding unit 20h (step S85). The removal of the background image, the removal of the color information, and the correction of the image are executed for the image data Gj stored in the storage unit 30.

The posture estimation unit 20c estimates the posture of the working unit 40 based on the posture estimation model based on the image data Gj in which the background image is removed, the color information is removed, and the image is corrected (step S86). In this step, the estimated posture information Ke is output from the posture estimation model.

The speed estimation unit 20e transmits the estimated attitude information Ke output from the attitude estimation model to the outside of the information processing device 10 (step S87). For example, the speed estimation unit 20e transmits the estimated posture information Ke to the work machine control unit 62. When the estimated posture information Ke is transmitted, the operation S80 ends. The operation S80 is repeatedly executed until there is no instruction for posture estimation.

If the timing of posture estimation is not reached (N in step S81), S82 to S87 are skipped. This operation S80 is merely an example, and the order of the steps may be changed, or some steps may be added / deleted / changed.

FIG. 12 is also a flowchart showing the operation of the information processing device 10. This figure shows an operation S90 in which the position, speed, and acceleration of the work unit 40 are specified based on the estimated posture information Ke, and the identification result is transmitted to the work machine control unit 62. The operation S90 is started at the timing when the administrator inputs an instruction to detect the speed and the acceleration during the non-learning operation.

When the timing for detecting the velocity and the acceleration is reached (Y in step S91), the position estimation unit 20d generates the position information Pe of the predetermined part (bucket 46) of the work unit 40 based on the estimated posture information Ke. It is transmitted to the work machine control unit 62 and stored in the storage unit 30 (step S92).

When the position information Pe is stored, the speed estimation unit 20e generates the speed information Ve of the work unit 40 based on the position information Pe, transmits it to the work machine control unit 62, and stores it in the storage unit 30 (step). S93).

When the speed information Ve is stored, the acceleration estimation unit 20f generates the acceleration information Ae of the work unit 40 based on the speed information Ve, transmits it to the work machine control unit 62, and stores it in the storage unit 30 (step). S94). When the acceleration information Ae is stored, the operation S90 ends. The operation S90 is repeatedly executed until there are no instructions for detecting the speed and acceleration.

If the timing for detecting the velocity and acceleration is not reached (N in step S91), S92 to S94 are skipped. This operation S90 is merely an example, and the order of the steps may be changed, or some steps may be added / deleted / changed.

FIG. 13 is also a flowchart showing the operation of the information processing device 10. In this figure, the presence or absence of a malfunction state, a failure state, or an abnormality in the appearance of the work unit 40 is detected based on the estimated posture information Ke, and the detection result is displayed on the display unit 38d provided in the cockpit 38. The operation S100 to be output to is shown. The operation S100 is started at the timing when the state detection instruction is input by the administrator during the non-learning operation.

When the timing of state detection is reached (Y in step S101), the predicted posture calculation unit 24b acquires the control signal Sj from the operation control unit 62a of the work machine control unit 62 and determines the predicted posture information Ka (step S102). ..

When the predicted posture information Ka is determined, the abnormality determination unit 24c determines a malfunction state or a failure state of the working unit 40 according to the difference dK between the estimated posture information Ke and the predicted posture information Ka (step S103).

When the state of the working unit 40 is determined, the state output unit 24d causes the display unit 38d to display the determination result of the abnormality determination unit 24c (step S104).

When the above-mentioned determination result is displayed, the reference image generation unit 26b generates the data Gh of the reference image in the posture based on the posture data Ks (step S105).

After the reference image data Gh is generated, the appearance determination unit 26c determines whether or not there is an abnormality in the appearance of the work unit 40 according to the degree of difference Sh between the reference image data Gh and the image data Gj (step). S106).

When the appearance state of the work unit 40 is determined, the state output unit 26d causes the display unit 38d to display the determination result of the appearance determination unit 26c (step S107). When the above determination result is displayed, the operation S100 ends. The operation S100 is repeatedly executed until there is no instruction for state detection.

If the timing of state detection is not reached (N in step S101), S102 to S107 are skipped. This operation S100 is merely an example, and the order of the steps may be changed, or some steps may be added / deleted / changed.

The features of the information processing apparatus 10 of the present embodiment configured as described above will be described. The information processing device 10 has a storage unit 30 that stores correspondence information generated by associating the reference image data Gs of the work unit 40 with the attitude data Ks of the work unit 40, and the reference image data Gs. The image information acquisition unit 12 that acquires the image data Gj of the work unit 40 for comparison with, and the speed estimation unit 20e that generates the speed information Ve related to the speed of the work unit 40 based on the image data Gj and the corresponding information. And. According to this configuration, the speed information Ve related to the speed of the working unit 40 can be generated from the image data Gj acquired by the image information acquisition unit 12, the reference image data Gs, and the posture data Ks. The correspondence information may be the above-mentioned posture estimation model.

The image information acquisition unit 12 may be configured to rotate integrally with the work unit 40. In this case, since the positional relationship between the image information acquisition unit 12 and the work unit 40 is constant, the speed information Ve of the work unit 40 can be easily generated.

The information processing device 10 may have a posture estimation unit 20c that estimates the posture of the work unit 40 based on the image data Gj and the corresponding information. In this case, the posture of the working unit 40 can be easily estimated from the image data Gj.

The storage unit 30 may store the posture estimation model generated by machine learning based on the reference image data Gs and the posture data Ks. In this case, the attitude estimation model generated by machine learning can be used.

The speed estimation unit 20e may generate speed information Ve based on the data Gj of a plurality of images acquired by the image information acquisition unit 12 at a plurality of timings having a time difference and the time difference. In this case, the speed information Ve can be generated by a simple arithmetic process.

The information processing device 10 may include an acceleration estimation unit 20f that generates acceleration information Ae regarding the acceleration of the work unit 40 based on a plurality of speed information Ves generated by the speed estimation unit 20e and a time difference. In this case, the acceleration information Ae can be generated by a simple arithmetic process.

The information processing device 10 determines an abnormality determination unit 24c that determines an operation abnormality of the work unit 40 based on the image data Gj acquired by the image information acquisition unit 12 and a control signal for controlling the operation of the work unit 40. May be provided. In this case, the operation abnormality can be detected by the image data Gj and the control signal without using a special sensor.

The information processing device 10 determines the appearance abnormality of the work unit 40 based on the image data Gj acquired by the image information acquisition unit 12 and the reference image data Gh of the work unit 40 in a state considered to be normal. The determination unit 26c may be further provided. In this case, the appearance abnormality can be detected from the image data Gj and the normal image data without using a special sensor.

The information processing device 10 has an image information correction unit 20j that corrects the image data Gj according to the information on the surrounding environment when acquiring the image data Gj, the age of the work machine 100, or the individual difference of the work machine 100. You may. In this case, the image data Gj can be corrected to improve the posture estimation accuracy.

The information processing device 10 may have a color information removing unit 20m that compresses or removes the color information of the data Gs of the reference image. In this case, it is advantageous in terms of the storage capacity and processing speed of the storage unit 30.

The information processing device 10 may have a background information removing unit 20k that removes the background image of the data Gs of the reference image. In this case, it is possible to suppress a decrease in estimation accuracy due to the background image.

Next, the second to fifth embodiments of the present invention will be described. In the drawings and description of the second to fifth embodiments, the same or equivalent components and members as those of the first embodiment are designated by the same reference numerals. The description that overlaps with the first embodiment will be omitted as appropriate, and the configuration different from that of the first embodiment will be mainly described.

[Second Embodiment]
A second embodiment of the present invention is an information processing method for a work machine. In this information processing method, the steps (S72 to S75) of storing the correspondence information generated by associating the data Gs of the reference image of the work unit 40 of the work machine 100 with the attitude data Ks of the work unit 40 in the storage unit 30. ), The step (S82) of acquiring the image data Gj of the working unit 40 for comparison with the reference image data Gs, and the data Gj of a plurality of images acquired at a plurality of timings having a time difference and the time difference. It includes a step (S92) of generating speed information Ve based on the corresponding information.

The step of generating the speed information Ve may include a step of referring to the posture estimation model generated by machine learning based on the reference image data Gs and the posture data Ks. That is, the correspondence information may be a posture estimation model. According to the configuration of the second embodiment, the same function and effect as those of the first embodiment are obtained.

[Third Embodiment]
A third embodiment of the present invention is a construction machine 1000. The construction machine 1000 refers to a storage unit 30 that stores correspondence information generated by associating the work unit 40 with the data Gs of the reference image of the work unit 40 and the attitude data Ks of the work unit 40. The image information acquisition unit 12 that acquires the image data Gj of the work unit 40 for comparison with the image data Gs, and the speed information Ve related to the speed of the work unit 40 is generated based on the image data Gj and the corresponding information. It is provided with a speed estimation unit 20e. The correspondence information may be the above-mentioned posture estimation model.

The construction machine 1000 may be, for example, a machine that moves a bucket 46 attached to the arm mechanism 48 to perform construction work. Various attachments such as a fork, a hammer, and a crusher may be attached to the arm mechanism 48 of the construction machine 1000 instead of the bucket. According to the configuration of the third embodiment, the same function and effect as those of the first embodiment are obtained.

[Fourth Embodiment]
A fourth embodiment of the present invention is a computer program P100. The computer program P100 has a function of storing the correspondence information generated by associating the reference image data Gs of the work unit 40 with the posture data Ks of the work unit 40 in the storage unit 30, and the data Gs of the reference image. The speed information Ve of the working unit 40 based on the function of acquiring the data Gj of the image of the working unit 40 for comparison, the data Gj of a plurality of images acquired at a plurality of timings having a time difference, the time difference, and the corresponding information. The function to generate is realized in the computer. The correspondence information may be the above-mentioned posture estimation model.

In the computer program P100, these functions may be installed in the storage (for example, the storage unit 30) of the information processing apparatus 10 as an application program in which a plurality of modules corresponding to the functional blocks of the control unit 20 are mounted. The computer program P100 may be read into the main memory of the processor (for example, CPU) of the information processing device 10 and executed. According to the configuration of the fourth embodiment, the same effects as those of the first embodiment are obtained.

[Fifth Embodiment]
A fifth embodiment of the present invention is an information processing system 1. FIG. 14 is a block diagram schematically showing a work machine 100 using the information processing system 1 of the present embodiment, and corresponds to FIG. The information processing system 1 is similar to the first embodiment in that the speed information Ve regarding the speed of the work unit 40 is estimated based on the data Gj of the image of the work unit 40 of the work machine 100. The information processing system 1 is different from the information processing device 10 of the first embodiment in that a part of its configuration is provided in the network server 120. In this example, the estimation unit 120j is provided on the network server 120.

The information processing system 1 of the present embodiment includes an information processing device 110 and a network server 120. The information processing device 110 and the network server 120 communicate with each other via a communication network NW using wired communication or wireless communication. As a communication network NW, a general-purpose network such as the Internet or a dedicated network can be adopted. As the network server 120, a cloud server, which is also called a cloud computing system, can be adopted.

The information processing device 110 has an acquisition unit 12 and a control unit 20. The control unit 20 includes an image information receiving unit 20a, an image information transmitting unit 20p, and an estimation result receiving unit 20q. The acquisition unit 12 acquires the data Gj of the image of the work unit 40 of the work machine 100. The image information transmission unit 20p transmits the image data Gj acquired by the acquisition unit 12 to the network server 120 via the communication network NW.

The network server 120 has a storage unit 120m, an estimation unit 120j, a reception unit 120q, and a transmission unit 120p. The estimation unit 120j includes a posture estimation unit 120c, a position estimation unit 120d, a velocity estimation unit 120e, and an acceleration estimation unit 120f. The storage unit 120m stores the correspondence information Ci generated by associating the data Gs of the reference image of the work unit 40 with the posture data Ks of the work unit 40 of the work machine 100. The correspondence information Ci in this example is a posture estimation model generated by machine learning based on the reference image data Gs and the posture data Ks. The posture estimation model is generated in advance based on the data of the reference work machine and stored in the model storage unit 120n.

The receiving unit 120q receives the image data Gj. The posture estimation unit 120c operates in the same manner as the posture estimation unit 20c to generate the estimated posture information Ke. The position estimation unit 120d operates in the same manner as the position estimation unit 20d to generate position information Pe. The speed estimation unit 120e operates in the same manner as the speed estimation unit 20e to generate speed information Ve. In this example, the speed estimation unit 120e generates speed information Ve regarding the speed of the work unit 40 based on the correspondence information Ci (posture estimation model) and the data Gj of the received image. The acceleration estimation unit 120f operates in the same manner as the acceleration estimation unit 20f to generate acceleration information Ae. The transmission unit 120p transmits the estimated posture information Ke, the position information Pe, the velocity information Ve, and the acceleration information Ae to the information processing device 110 via the communication network NW.

The estimation result receiving unit 20q of the information processing device 110 receives the estimated posture information Ke, the position information Pe, the speed information Ve, and the acceleration information Ae from the network server 120. The estimation result receiving unit 20q transmits the estimated posture information Ke, the position information Pe, the velocity information Ve, and the acceleration information Ae to the work machine control unit 62. The work machine control unit 62 operates in the same manner as in the first embodiment to control the operation of the work machine 100.

The information processing system 1 configured as described above operates in the same manner as the information processing apparatus 10 of the first embodiment, and exhibits the same actions and effects. In addition, since the estimation unit 120j is provided in the network server 120, the estimation accuracy can be improved by using a more advanced algorithm. In addition, one network server can support a plurality of work machines.

The examples of the embodiments of the present invention have been described in detail above. All of the above-described embodiments are merely specific examples for carrying out the present invention. The content of the embodiment does not limit the technical scope of the present invention, and many design changes such as modification, addition, and deletion of components are made within the scope of the invention defined in the claims. It is possible. In the above-described embodiment, the contents that can be changed in such a design are described with notations such as "in the embodiment" and "in the embodiment", but the contents are designed without such notations. It's not that changes aren't tolerated.

[Modification example]
Hereinafter, a modified example will be described. In the drawings and description of the modified examples, the same or equivalent components and members as those in the embodiment are designated by the same reference numerals. The description that overlaps with the embodiment will be omitted as appropriate, and the configuration different from that of the first embodiment will be mainly described.

In the first embodiment, an example is shown in which the velocity information Ve and the acceleration information Ae are information related to the velocity and acceleration of the bucket 46, but the present invention is not limited thereto. The speed information Ve and the acceleration information Ae may be information on the speed and acceleration of a portion of the working unit 40 other than the bucket 46, such as the arm 44 and the boom 42. Further, the velocity information Ve and the acceleration information Ae may include information on the velocity and acceleration of two or more parts or parts.

In the first embodiment, an example in which the velocity information Ve and the acceleration information Ae are fed back to control the velocity and the acceleration is shown, but the present invention is not limited to this. For example, the information processing device 10 may control an avoidance operation in which the working unit 40 avoids contact with surrounding people or objects according to at least one of position information Pe, velocity information Ve, and acceleration information Ae. ..

In the first embodiment, the posture estimation model is generated by each work machine 100 and stored in the model storage unit 32 of the work machine 100, but the present invention is not limited thereto. The posture estimation model may be generated by a reference work machine and stored in advance in the model storage unit 32 of each work machine 100. Further, the posture estimation model may be updated at an appropriate timing.

In the description of the first embodiment, an example in which the image information acquisition unit 12 is composed of one image sensor has been shown, but the present invention is not limited to this. The image information acquisition unit 12 may be composed of a plurality of image sensors. For example, the image information acquisition unit 12 may include a so-called stereo camera.

In the description of the first embodiment, an example in which the image information acquisition unit 12 is provided on the roof of the cockpit 38 is shown, but the present invention is not limited to this. For example, the image information acquisition unit 12 may be arranged on the side surface of the cockpit 38 or on the cover of the upper vehicle body unit 34. Further, the image information acquisition unit 12 may be arranged on the work unit 40.

In the description of the first embodiment, an example is shown in which the work machine 100 is a construction machine for performing construction work by moving the bucket 46, but the present invention is not limited to this, and is applicable to work machines other than the construction machine. it can.

In the first embodiment, an example is shown in which the color information removing unit 20m completely removes the color information from the reference image data Gs and the image data Gj, but the present invention is not limited thereto. The color information removing unit 20m may compress the color information of the reference image data Gs and the image data Gj by color reduction or the like.

In the description of the first embodiment, an example in which the arm mechanism 48 is provided on the right side of the cockpit 38 has been shown, but the present invention is not limited thereto. For example, the arm mechanism may be provided on the left side of the cockpit or in front of the cockpit.

In the description of the first embodiment, an example in which the work machine 100 is operated by the operator from the cockpit 38 is shown, but the present invention is not limited thereto. For example, the work machine may be autopilot or remote controlled.

In the description of the fifth embodiment, the control unit 20 includes an individual information holding unit 20h, an environmental information receiving unit 20g, an image information correction unit 20j, a background information removing unit 20k, a color information removing unit 20m, an abnormality detecting unit 24, and an appearance detection unit. Although an example not including the part 26 is shown, the present invention is not limited thereto. For example, the control unit 20 of the fifth embodiment includes an individual information holding unit 20h, an environmental information receiving unit 20g, an image information correction unit 20j, a background information removing unit 20k, a color information removing unit 20m, an abnormality detecting unit 24, or an appearance detecting unit. It may contain one or more of 26.

The above-mentioned modification has the same action / effect as that of the first embodiment.

Any combination of the above-described embodiments and modifications is also useful as an embodiment of the present invention. The new embodiment produced by the combination has the effects of the combined embodiment and the modified example.

10 Information processing equipment, 12 Image information acquisition unit, 14 Environmental information acquisition unit, 16 Attitude information acquisition unit, 20 Control unit, 20e Speed estimation unit, 20f Acceleration estimation unit, 22 Model generation unit, 24c Abnormality judgment unit, 26c Appearance judgment Department, 30 storage unit, 32 model storage unit, 40 work unit, 42 boom, 44 arm, 46 bucket, 62 work machine control unit, 100 work machine, 1000 construction machine.

Claims (14)

A storage unit that stores correspondence information generated by associating the data of the reference image of the work unit of the work machine with the posture data of the work unit of the work machine.
An acquisition unit that acquires image data of the work unit of the work machine for comparison with the data of the reference image, and an acquisition unit.
An information processing device including a speed estimation unit that generates speed information regarding the speed of the work unit of the work machine based on the correspondence information and the data of the image. The information processing device according to claim 1, wherein the acquisition unit is configured to rotate integrally with the work unit of the work machine. The information processing apparatus according to claim 1 or 2, further comprising an estimation unit that estimates the posture of the work unit of the work machine based on the image data and the corresponding information. The information processing device according to any one of claims 1 to 3, wherein the storage unit stores a posture estimation model generated by machine learning based on the reference image data and the posture data. The information processing according to any one of claims 1 to 4, wherein the speed estimation unit generates the speed information based on the data of a plurality of images acquired by the acquisition unit at a plurality of timings having a time difference and the time difference. apparatus. The information processing apparatus according to claim 5, further comprising an acceleration estimation unit that generates acceleration information regarding acceleration of the work unit of the work machine based on a plurality of speed information generated by the speed estimation unit and the time difference. A claim further comprising an abnormality determination unit for determining an operation abnormality of the work unit of the work machine based on the image data acquired by the acquisition unit and a control signal for controlling the operation of the work unit of the work machine. Item 4. The information processing apparatus according to any one of Items 1 to 6. A claim further comprising an appearance determination unit for determining an appearance abnormality of the work unit of the work machine based on the image data acquired by the acquisition unit and the image data of the work unit of the work machine in a normal state. The information processing apparatus according to any one of 1 to 7. A storage unit that stores a posture estimation model generated by machine learning based on the reference image data of the work unit of the work machine and the posture data of the work unit of the work machine.
An acquisition unit that is configured to rotate integrally with the work unit of the work machine and acquires image data of the work unit of the work machine for comparison with the data of the reference image.
A posture estimation unit that estimates the posture of the work unit of the work machine by referring to the posture estimation model based on the image data acquired by the acquisition unit.
A position estimation unit that estimates the position of a predetermined part of the work unit of the work machine based on the estimated posture information regarding the posture of the work unit of the work machine estimated by the posture estimation unit.
An information processing device including a speed estimation unit that generates speed information regarding the speed of the work unit of the work machine based on the position information regarding the position of the predetermined portion estimated by the position estimation unit. An information processing device that acquires and transmits image data of the work part of a work machine,
The above is based on the correspondence information generated in advance by associating the image data received from the information processing apparatus with the reference image data of the work unit of the work machine and the attitude data of the work unit of the work machine. An information processing system including a network server that estimates speed information related to the speed of a working unit of a work machine and transmits the speed information to the information processing apparatus. A step of storing correspondence information generated by associating the reference image data of the work part of the work machine with the posture data of the work part of the work machine, and
A step of acquiring image data of a working part of the work machine for comparison with the data of the reference image, and
An information processing method including a step of generating speed information based on data of a plurality of images acquired at a plurality of timings having a time difference, the time difference, and the corresponding information. The information processing method according to claim 11, wherein the step of generating the speed information includes a step of referring to a posture estimation model generated by machine learning based on the data of the reference image and the data of the posture. A function to store the correspondence information generated by associating the reference image data of the work part of the work machine with the posture data of the work part of the work machine, and
A function of acquiring image data of a working part of the work machine for comparison with the data of the reference image, and
A computer program for realizing a function of generating speed information of a work unit of a work machine based on data of a plurality of images acquired at a plurality of timings having a time difference, the time difference, and the corresponding information. The work part of the work machine and
A storage unit that stores correspondence information generated by associating the data of the reference image of the work unit of the work machine with the posture data of the work unit of the work machine.
An acquisition unit that acquires image data of the work unit of the work machine for comparison with the data of the reference image, and an acquisition unit.
A construction machine including a speed estimation unit that generates speed information regarding the speed of the work unit of the work machine based on the image data acquired by the acquisition unit and the corresponding information.

Images