JP2022025195A - Arm position detection system for construction machine - Google Patents

Arm position detection system for construction machine Download PDF

Info

Publication number
JP2022025195A
JP2022025195A JP2020127865A JP2020127865A JP2022025195A JP 2022025195 A JP2022025195 A JP 2022025195A JP 2020127865 A JP2020127865 A JP 2020127865A JP 2020127865 A JP2020127865 A JP 2020127865A JP 2022025195 A JP2022025195 A JP 2022025195A
Authority
JP
Japan
Prior art keywords
coordinates
construction machine
joint
arm
detection system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
JP2020127865A
Other languages
Japanese (ja)
Other versions
JP7458262B2 (en
Inventor
弘隆 熊倉
Hirotaka Kumakura
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
IHI Aerospace Co Ltd
Original Assignee
IHI Aerospace Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by IHI Aerospace Co Ltd filed Critical IHI Aerospace Co Ltd
Priority to JP2020127865A priority Critical patent/JP7458262B2/en
Publication of JP2022025195A publication Critical patent/JP2022025195A/en
Application granted granted Critical
Publication of JP7458262B2 publication Critical patent/JP7458262B2/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Landscapes

  • Component Parts Of Construction Machinery (AREA)

Abstract

To provide an arm position detection system for construction machines that can reduce the number of angle sensors.SOLUTION: An arm position detection system of the present invention comprises a camera fixed to a construction machine and a computing device. The computing device has a machine-learned AI that extracts a position of a second joint and a position of a third joint of the construction machine from the video taken with the camera toward the working direction of the construction machine, and outputs the video coordinates of the position of the second joint and the position of the third joint, a coordinate transformation unit that transforms the video coordinates into a two-dimensional real space coordinate system by means of a predetermined projection vector and outputs transformed coordinates, and a filtering unit that corrects the transformed coordinates to true coordinates that can actually be taken from the pre-stored dimensional information of the construction machine and outputs them. The number of angle sensors, which may cause failures, is reduced, and the work interruption time due to angle sensor failures can be reduced.SELECTED DRAWING: Figure 3

Description

本発明は、建設機械のアーム位置検出システムに係り、更に詳細には、アーム位置の検出に使用する角度センサの数を削減できる建設機械のアーム位置検出システムに関する。 The present invention relates to an arm position detection system for a construction machine, and more particularly to an arm position detection system for a construction machine capable of reducing the number of angle sensors used to detect the arm position.

オペレータが建設機械に乗って操縦することが困難である場合、例えば、危険が高い所や過酷な気象条件下での工事などでは、工事現場から離れた場所でオペレータが建設機械から送信される映像や情報をモニタで見ながら建設機械を遠隔操縦することが行われている。 When it is difficult for the operator to get on and maneuver the construction machine, for example, in a high-risk place or construction under harsh weather conditions, the image transmitted by the operator from the construction machine at a place away from the construction site. It is practiced to remotely control construction machinery while viewing information and information on a monitor.

建設機械の遠隔操縦者は、体感により建設機械のアーム位置を知ることができないが、施工精度を向上させるために、遠隔操縦者は正確なアーム位置を知る必要がある。 The remote controller of the construction machine cannot know the arm position of the construction machine by feeling, but the remote controller needs to know the accurate arm position in order to improve the construction accuracy.

特許文献１の特開２００６－２１４２４６号公報には、作業装置のブーム、アーム及びバケットの関節部それぞれに角度センサを設けてアーム位置を検出し、検出したアーム位置を遠隔操縦者に伝えることが開示されている。 In Japanese Patent Application Laid-Open No. 2006-214246 of Patent Document 1, an angle sensor is provided at each of the joints of the boom, arm and bucket of the working device to detect the arm position, and the detected arm position is notified to the remote controller. It has been disclosed.

特開２００６－２１４２４６号公報Japanese Unexamined Patent Publication No. 2006-214246

しかしながら、建設機械は、その作業中に大きな振動や衝撃が角度センサに伝わり易く、設置する角度センサの数が多いと故障の確率が高くなるという建設機械特有の問題があり、角度センサの故障により作業が中断され、また、角度センサは高価であるため無人化建設機械のコストが上昇してしまう。 However, construction machinery has a problem peculiar to construction machinery that large vibrations and shocks are easily transmitted to the angle sensor during the work, and the probability of failure increases when the number of angle sensors installed is large. Work is interrupted and the cost of unmanned construction machinery increases due to the high cost of angle sensors.

本発明は、このような従来技術の有する課題に鑑みてなされたものであり、その目的とするところは、故障の原因となる角度センサの数を削減できる建設機械のアーム位置検出システムを提供することにある。 The present invention has been made in view of the problems of the prior art, and an object of the present invention is to provide an arm position detection system for a construction machine capable of reducing the number of angle sensors that cause a failure. There is something in it.

本発明者は、上記目的を達成すべく鋭意検討を重ねた結果、建設機械に固定されたカメラが撮影した建設機械の作業方向の映像情報からアーム位置を検出することにより、上記目的が達成できることを見出し、本発明を完成するに至った。 As a result of diligent studies to achieve the above object, the present inventor can achieve the above object by detecting the arm position from the image information of the working direction of the construction machine taken by the camera fixed to the construction machine. And completed the present invention.

上記課題は、本発明の下記（１）～（５）により解決される。
（１）建設機械に固定されたカメラと、演算装置とを備え、
上記演算装置が、
上記カメラが建設機械の作業方向を撮影した映像から、上記建設機械の第２関節部の位置と第３関節部の位置を抽出し、上記第２関節部の位置と上記第３関節部の位置の映像座標を出力する機械学習済ＡＩと、
上記映像座標を、予め設定された射影ベクトルにより２次元の実空間座標系に変換し変換座標を出力する座標変換部と、
上記変換座標を、予め記憶した建設機械の寸法情報から実際に取り得る真座標に修正するフィルタリング部と、を有することを特徴とする建設機械のアーム位置検出システム。
（２）上記フィルタリング部が、実際に取り得る真座標のうち、上記変換座標との空間距離が最も短い真座標に修正することを特徴とする上記第（１）項に記載の建設機械のアーム位置検出システム。
（３）上記フィルタリング部が、先の真座標と、該先の真座標を得た時点からの経過時間とから、該経過時間内に移動可能な範囲内の真座標に修正することを特徴とする上記第（１）項又は上記第（２）項に記載の建設機械のアーム位置検出システム。
（４）上記フィルタリング部が、上記第２関節部と第３関節部の可動域内の真座標に修正することを特徴とする上記第（１）項～上記第（３）項のいずれか１つの項に記載の建設機械のアーム位置検出システム。
（５）さらに、上記建設機械の第３関節部に角度センサを備え、
該第３関節部に設けられたアタッチメントの位置を検出することを特徴とする上記第（１）項～上記第（４）項のいずれか１つの項に記載の建設機械のアーム位置検出システム。 The above problems are solved by the following (1) to (5) of the present invention.
(1) Equipped with a camera fixed to the construction machine and an arithmetic unit,
The above arithmetic unit
The position of the second joint and the position of the third joint of the construction machine are extracted from the image taken by the camera of the working direction of the construction machine, and the position of the second joint and the position of the third joint are extracted. Machine-learned AI that outputs the video coordinates of
A coordinate conversion unit that converts the above video coordinates into a two-dimensional real space coordinate system using a preset projection vector and outputs the converted coordinates.
An arm position detection system for a construction machine, which comprises a filtering unit that corrects the converted coordinates to true coordinates that can be actually obtained from the dimensional information of the construction machine stored in advance.
(2) The arm of the construction machine according to the above item (1), wherein the filtering unit corrects the true coordinates to the true coordinates having the shortest spatial distance from the converted coordinates among the true coordinates that can be actually obtained. Position detection system.
(3) The feature is that the filtering unit corrects the true coordinates of the destination and the elapsed time from the time when the true coordinates of the destination are obtained to the true coordinates within the movable range within the elapsed time. The arm position detection system for construction machinery according to the above item (1) or the above item (2).
(4) One of the above items (1) to (3), wherein the filtering unit corrects the coordinates to the true coordinates within the movable range of the second joint part and the third joint part. Arm position detection system for construction machinery as described in section.
(5) Further, an angle sensor is provided at the third joint of the construction machine.
The arm position detection system for a construction machine according to any one of the above items (1) to (4), which detects the position of the attachment provided on the third joint portion.

本発明によれば、建設機械に固定されたカメラが撮影した建設機械のアーム映像から、アーム位置を検出することとしたため、故障の原因となる角度センサの数が削減され、角度センサの故障による作業中断時間を削減できる建設機械のアーム位置検出システムを提供することができる。 According to the present invention, since the arm position is detected from the arm image of the construction machine taken by the camera fixed to the construction machine, the number of angle sensors that cause a failure is reduced, and the failure of the angle sensor causes the failure. It is possible to provide an arm position detection system for construction machinery that can reduce work interruption time.

本発明の建設機械のアーム位置検出システムを設けた建設機械の概略図である。It is a schematic diagram of the construction machine provided with the arm position detection system of the construction machine of this invention. カメラ映像（左図）から機械学習済ＡＩがブーム部分とアーム部分を抽出した図（右図）である。It is the figure (right figure) that the machine-learned AI extracted the boom part and the arm part from the camera image (left figure). フィルタリングにより変換座標を真座標に修正する概念を説明する図である。It is a figure explaining the concept which corrects a transformation coordinate to a true coordinate by filtering. 検出したアーム位置を出力する状態の一例を示す図である。It is a figure which shows an example of the state which outputs the detected arm position. 演算装置の処理を示す図である。It is a figure which shows the processing of the arithmetic unit.

本発明の建設機械のアーム位置検出システムを、建設機械がバックホウである場合を例に説明する。 The arm position detection system of the construction machine of the present invention will be described by exemplifying the case where the construction machine is a backhoe.

上記建設機械のアーム位置検出システムは、建設機械に固定されたカメラと、演算装置とを備える。
そして、上記演算装置が、上記カメラが建設機械の作業方向を撮影した映像から上記アームの第２関節部の位置と第３関節部の位置を抽出し、第２関節部の位置と第３関節部の位置の映像座標を出力する機械学習済ＡＩと、
上記映像座標を、予め設定された射影ベクトルにより２次元の実空間座標系に変換し変換座標を出力する座標変換部と、上記変換座標を、予め記憶した建設機械の寸法情報から修正し、真座標を出力するフィルタリング部と、を有する。 The arm position detection system of the construction machine includes a camera fixed to the construction machine and an arithmetic unit.
Then, the arithmetic unit extracts the position of the second joint portion and the position of the third joint portion of the arm from the image taken by the camera in the working direction of the construction machine, and the position of the second joint portion and the third joint portion. Machine-learned AI that outputs the video coordinates of the position of the part,
The coordinate conversion unit that converts the above video coordinates into a two-dimensional real space coordinate system using a preset projection vector and outputs the conversion coordinates, and the above conversion coordinates are corrected from the dimensional information of the construction machine stored in advance, and are true. It has a filtering unit that outputs coordinates.

バックホウ１００は、図１に示すように、上部旋回体１と下部走行体２とを備える。上部旋回体１は下部走行体２に対して旋回自在に連結され、下部走行体２に対して方向を変えることができる。下部走行体２はクローラを回転させて地表面を走行する。 As shown in FIG. 1, the backhoe 100 includes an upper swing body 1 and a lower traveling body 2. The upper swivel body 1 is rotatably connected to the lower traveling body 2 and can change its direction with respect to the lower traveling body 2. The lower traveling body 2 rotates the crawler and travels on the ground surface.

上記上部旋回体１は、操縦席を備えるキャビンまたはキャノピーを有する。
また、上記上部旋回体１には、第１関節部１０を介して上下に搖動可能なブーム１１が設けられ、該ブーム１１の先端には第２関節部２０を介して上下に搖動可能なアーム２１が連結される。
さらに、上記アーム２１の先端には、アタッチメント３１として第３関節部３０を介して上下に搖動可能なバケットが連結される。 The upper swivel body 1 has a cabin or a canopy provided with a cockpit.
Further, the upper swing body 1 is provided with a boom 11 that can swing up and down via the first joint portion 10, and an arm that can swing up and down at the tip of the boom 11 via the second joint portion 20. 21 are connected.
Further, a bucket that can swing up and down is connected to the tip of the arm 21 via a third joint portion 30 as an attachment 31.

バックホウは、上記ブーム、アーム及びバケットを協働して動かすことで地面を掘削する。なお、上記アームにはバケット以外の他のアタッチメントが取り付けられてもよい。 The backhoe excavates the ground by moving the boom, arm and bucket together. An attachment other than the bucket may be attached to the arm.

上記カメラ５０は、上部旋回体１に設けられ、図１に示すように、建設機械の作業方向に向け、少なくともブーム１１とアーム２１とを繋ぐ第２関節部２０、及びアーム２１とバケット３１とを繋ぐ第３関節部３０を撮影できる位置に固定される。なお、第１関節部１０は撮影できなくても構わない。 The camera 50 is provided on the upper swing body 1, and as shown in FIG. 1, has at least a second joint portion 20 connecting the boom 11 and the arm 21 and an arm 21 and a bucket 31 in the working direction of the construction machine. The third joint portion 30 is fixed at a position where it can be photographed. It is not necessary that the first joint portion 10 can be photographed.

上記演算装置は、機械学習済ＡＩと座標変換部とフィルタリング部とを有する。図５に演算装置における処理の流れを示す。 The arithmetic unit has a machine-learned AI, a coordinate conversion unit, and a filtering unit. FIG. 5 shows the flow of processing in the arithmetic unit.

（機械学習済ＡＩ）
本発明における機械学習済ＡＩは、画像解析で一般的なＳｅｍａｎｔｉｃ Ｓｅｇｍｅｎｔａｔｉｏｎを用い、対象物を塗り潰すことで求める。具体的にはブーム及びアームが写り、第２関節部と第３関節部とがポインティングされた画像と、該画像中のブーム部分及びアーム部分がそれぞれ単一色で塗り潰され、第２関節部と第３関節部とがポインティングされた塗り潰し画像と、の組み合わせを多数作成し、ブーム及びアームの画像と、それに対応するブーム及びアームの塗り潰し画像から、ブームとアームとの特徴を学習させ、上記塗り潰し画像中の第２関節部と第３関節部との位置を認識する塗り絵検出型のＡＩである。 (Machine-learned AI)
The machine-learned AI in the present invention is obtained by painting an object using a Semantic Segmentation, which is common in image analysis. Specifically, the image in which the boom and the arm are shown and the second joint and the third joint are pointed, and the boom and the arm in the image are painted in a single color, respectively, and the second joint and the third joint are shown. A large number of combinations of a filled image in which the three joints are pointed are created, and the characteristics of the boom and the arm are learned from the image of the boom and the arm and the corresponding filled image of the boom and the arm, and the above filled image is obtained. It is a painting detection type AI that recognizes the positions of the second joint and the third joint inside.

上記ブーム及びアームの特徴を学習した機械学習済ＡＩは、図２に示すように、上記カメラが撮影した映像中のブーム部分とアーム部分とを抽出して、映像中の位置を示す第２関節部の映像座標（ｘ２”,ｙ２”）と第３関節部の映像座標（ｘ３”,ｙ３”）を出力する。関節部の座標を求める具体的方法としては、例えば塗り潰したブーム領域の左端部又は下端部から決められた距離を（ｘ２”,ｙ２”）、アーム領域の下端部から決められた距離を（ｘ３”,ｙ３”）とし、画面上の座標を決定する。 As shown in FIG. 2, the machine-learned AI that has learned the characteristics of the boom and the arm extracts the boom part and the arm part in the image taken by the camera, and shows the position in the image of the second joint. The video coordinates of the part (x2 ", y2") and the video coordinates of the third joint part (x3 ", y3") are output. As a specific method for obtaining the coordinates of the joint portion, for example, the determined distance from the left end portion or the lower end portion of the filled boom region (x2 ", y2") and the determined distance from the lower end portion of the arm region (x3). ", Y3") and determine the coordinates on the screen.

（座標変換部）
上記座標変換部は、上記映像座標を予め設定された射影ベクトルにより２次元の実空間座標系に変換する。 (Coordinate conversion part)
The coordinate conversion unit converts the video coordinates into a two-dimensional real-space coordinate system using a preset projection vector.

上記機械学習済ＡＩが出力した上記映像座標は、カメラが設置された位置から見た関節部位置を示す座標であり、実空間における位置を示す座標ではないため、上記映像座標を実空間座標系に変換する必要がある。 The video coordinates output by the machine-learned AI are coordinates indicating the joint position as seen from the position where the camera is installed, not the coordinates indicating the position in the real space. Therefore, the video coordinates are used as the real space coordinate system. Need to be converted to.

ここで、実空間座標系が２次元であるのは、建設機械のブーム、アーム及びバケットの関節部は同一平面上に存在し、その平面内で協働して作業を行うため、関節部位置は２次元の座標系で特定することができ、３次元の座標系は必要ないためである。 Here, the reason why the real space coordinate system is two-dimensional is that the boom, arm, and bucket joints of the construction machine are on the same plane, and the joints are located because they work together in that plane. This is because can be specified by a two-dimensional coordinate system, and a three-dimensional coordinate system is not required.

上記射影ベクトルは、当該建設機械における第２関節部及び第３関節部の映像座標と、それに対応する実空間の座標との関係から予め設定される。 The projection vector is preset from the relationship between the image coordinates of the second joint portion and the third joint portion in the construction machine and the corresponding coordinates in the real space.

具体的には、上記映像座標、すなわち建設機械に固定されたカメラ位置から見た関節部の方向と距離とを示すベクトルを、ブーム、アーム及びバケットが存在する平面上に射影して２次元の実空間座標系に変換し、第２関節部の変換座標（ｘ２’,ｙ２’）と第３関節部の変換座標（ｘ３’,ｙ３’）を出力する。
なお、第１関節部が映像中に写っていなくても、第１関節部の角度は第２関節部の方向から求めることができる。 Specifically, the above image coordinates, that is, a vector indicating the direction and distance of the joint as seen from the camera position fixed to the construction machine, are projected onto a plane in which the boom, arm, and bucket are present, and are two-dimensional. It is converted into a real space coordinate system, and the converted coordinates of the second joint portion (x2', y2') and the converted coordinates of the third joint portion (x3', y3') are output.
Even if the first joint portion is not shown in the image, the angle of the first joint portion can be obtained from the direction of the second joint portion.

（フィルタリング部）
フィルタリング部は、上記変換座標を建設機械が実際に取り得る真座標に修正する。 (Filtering section)
The filtering unit corrects the converted coordinates to the true coordinates that the construction machine can actually take.

機械学習済ＡＩは、映像の明るさや映像中の影や反射等によって、ブーム部分やアーム部分の抽出精度が低下するため、機械学習済ＡＩが出力する第２関節部や第３関節部の映像座標には誤差が生じる。また、機械学習済ＡＩは、映像中のブーム部分とアーム部分とを抽出を行い、塗り潰し画像中の第２関節部と第３関節部との位置を認識するだけであり、映像中のブームとアームとが物理的に取り得る位置にあるか否かを考慮しないため、映像座標に誤差が生じ、該映像座標を変換した変換座標も誤差を含んでいる。 Machine-learned AI reduces the extraction accuracy of the boom part and arm part due to the brightness of the image, shadows and reflections in the image, etc., so the image of the second joint part and the third joint part output by the machine-learned AI There will be an error in the coordinates. In addition, the machine-learned AI only extracts the boom part and the arm part in the image and recognizes the positions of the second joint part and the third joint part in the filled image, and the boom in the image. Since it is not considered whether or not the arm is in a position that can be physically taken, an error occurs in the video coordinates, and the converted coordinates obtained by converting the video coordinates also include the error.

フィルタリング部は、上記映像座標を建設機械の寸法情報、例えば、ブームの長さ及びアームの長さから第２関節部と第３関節部とが物理的に取り得る真座標の組み合わせに修正する。 The filtering unit corrects the video coordinates to a combination of true coordinates that can be physically obtained by the second joint portion and the third joint portion from the dimensional information of the construction machine, for example, the length of the boom and the length of the arm.

図３に示すように、第２関節部は第一関節部からブームの長さだけ離れた円弧上に位置するはずであり、第３関節部は上記円弧上の第２関節部位置からアームの長さだけ離れた円弧上に位置するはずである。 As shown in FIG. 3, the second joint should be located on an arc separated by the length of the boom from the first joint, and the third joint is on the arm from the position of the second joint on the arc. It should be located on an arc that is length apart.

本発明においては、第２関節部の変換座標と第３関節部位置の変換座標との組み合わせに最も近い、第２関節部の真座標と第３関節部真座標の組み合わせに修正した。 In the present invention, the combination of the true coordinates of the second joint and the true coordinates of the third joint, which is the closest to the combination of the converted coordinates of the second joint and the converted coordinates of the position of the third joint, is modified.

上記最も近い真座標の組み合わせに修正する方法としては、例えば、第２関節部の真座標（ｘ２,ｙ２）と第２関節部の変換座標（ｘ２’,ｙ２’）との空間距離ｄ２と、第３関節部の真座標（ｘ３,ｙ３）と第３関節部の変換座標（ｘ３’,ｙ３’）との空間距離ｄ３との和（ｄ２＋ｄ３）や、これらの二乗和（ｄ２ ２＋ｄ３ ２）が最も小さくなる真座標に修正する方法等が挙げられる。 As a method of modifying the combination of the closest true coordinates, for example, the spatial distance d 2 between the true coordinates (x2, y2) of the second joint and the converted coordinates (x2', y2') of the second joint is used. , The sum of the spatial distance d 3 between the true coordinates (x3, y3) of the third joint and the converted coordinates (x3', y3') of the third joint, and the sum of their squares (d 2 + d 3 ). There is a method of correcting to the true coordinates where d 2 2 + d 3 2 ) is the smallest.

また、フィルタリング部は、変換座標と空間距離が最も近い真座標が、先の真座標と該先の真座標を得た時点から経過した時間内に移動可能な範囲内の真座標であるか否か、第２関節部や第３関節部の可動域内の真座標であるか否か、等のフィルタをかけることで、アームの位置検出精度が向上する。 Further, in the filtering unit, whether or not the true coordinates having the closest spatial distance to the converted coordinates are the true coordinates within the range that can be moved within the time elapsed from the time when the previous true coordinates and the previous true coordinates are obtained. Or, by applying a filter such as whether or not the coordinates are true coordinates within the movable range of the second joint portion and the third joint portion, the position detection accuracy of the arm is improved.

さらに、複数のカメラを設け、異なる角度から撮影したそれぞれのカメラ映像から複数の変換座標を得て、該複数の変換座標間の中心座標を変換座標としてバラツキを軽減することや、上記複数の変換座標のうち最も外れたバラツキが大きい変換座標を除いた中心座標を元に上記フィルタをかけることで、アームの位置検出精度が向上する。 Further, a plurality of cameras are provided, a plurality of conversion coordinates are obtained from each camera image taken from different angles, and the center coordinates between the plurality of conversion coordinates are used as the conversion coordinates to reduce the variation, and the above-mentioned plurality of conversions are performed. By applying the above filter based on the center coordinates excluding the converted coordinates with the largest deviation among the coordinates, the position detection accuracy of the arm is improved.

フィルタリング部は、予め、第２関節部と第３関節部の真座標の組み合わせをデータベースに記憶しておいてもよく、予め記憶した建設機械の寸法情報から、実際に取り得る第２関節部と第３関節部の真座標の組み合わせを計算してもよい。 The filtering unit may store in advance the combination of the true coordinates of the second joint part and the third joint part in the database, and can actually obtain the second joint part from the dimensional information of the construction machine stored in advance. The combination of the true coordinates of the third joint may be calculated.

本発明の建設機械のアーム位置検出システムは、建設機械の第３関節部に装着したアタッチメントの先端がカメラ映像に映り、作業部分を見渡すことができる場合は、第２関節部や第３関節部の真座標と同様にして、アタッチメントの先端の真座標を検出する。 In the arm position detection system of the construction machine of the present invention, when the tip of the attachment attached to the third joint part of the construction machine is reflected in the camera image and the working part can be seen, the second joint part or the third joint part can be seen. The true coordinates of the tip of the attachment are detected in the same way as the true coordinates of.

建設機械がバックホウのように、第３関節部に装着したアタッチメントがバケットである場合等、アタッチメントが地中に潜り、その先端がカメラに映らない範囲で作業を行う場合は、アタッチメントを装着した関節部、例えば、アームとアタッチメントとを繋ぐ第３関節部に角度センサを設ける。 When the attachment attached to the third joint is a bucket, such as when the construction machine is a backhoe, when the attachment is submerged in the ground and the tip of the attachment is not visible in the camera, the joint with the attachment is attached. An angle sensor is provided in a portion, for example, a third joint portion connecting the arm and the attachment.

アタッチメントが装着された関節部に角度センサを設けることで、アタッチメントの先端位置が見えない場合であっても、図４に示すように、上記第２関節部と第３関節部の真座標と併せて、ブーム、アーム、及びアタッチメントの位置の他、該アタッチメントの角度を遠隔操縦者に伝えることができる。 By providing an angle sensor at the joint to which the attachment is attached, even if the tip position of the attachment cannot be seen, as shown in FIG. 4, the true coordinates of the second joint and the third joint are combined. In addition to the positions of the boom, arm, and attachment, the angle of the attachment can be communicated to the remote controller.

そして、例えば、第２関節部と第３関節部の真座標と図４に示すような画像とを遠隔操縦画面に映し、遠隔操縦者はカメラ画像と共にそれらの情報を見ながら操縦することができる。 Then, for example, the true coordinates of the second joint portion and the third joint portion and the image as shown in FIG. 4 are projected on the remote control screen, and the remote control operator can operate while viewing the information together with the camera image. ..

本発明の建設機械のアーム位置検出システムを、建設機械がバックホウである場合を例に説明したが、作業部分を見渡せる建設機械であれば適用でき、例えば、ブルドーザ等にも適用が可能である。 The arm position detection system of the construction machine of the present invention has been described by taking the case where the construction machine is a backhoe as an example, but it can be applied to any construction machine that can overlook the working part, and can be applied to, for example, a bulldozer.

１ 上部旋回体
２ 下部走行体
１０ 第１関節部
１１ ブーム
２０ 第２関節部
２１ アーム
３０ 第３関節部
３１ アタッチメント（バケット）
５０ カメラ
１００ バックホウ 1 Upper swing body 2 Lower running body 10 1st joint part 11 Boom 20 2nd joint part 21 Arm 30 3rd joint part 31 Attachment (bucket)
50 camera 100 backhoe

Claims (5)

建設機械に固定されたカメラと、演算装置とを備え、
上記演算装置が、
上記カメラが建設機械の作業方向を撮影した映像から、上記建設機械の第２関節部の位置と第３関節部の位置を抽出し、上記第２関節部の位置と上記第３関節部の位置の映像座標を出力する機械学習済ＡＩと、
上記映像座標を、予め設定された射影ベクトルにより２次元の実空間座標系に変換し変換座標を出力する座標変換部と、
上記変換座標を、予め記憶した建設機械の寸法情報から実際に取り得る真座標に修正するフィルタリング部と、を有することを特徴とする建設機械のアーム位置検出システム。 Equipped with a camera fixed to the construction machine and an arithmetic unit,
The above arithmetic unit
The position of the second joint and the position of the third joint of the construction machine are extracted from the image taken by the camera of the working direction of the construction machine, and the position of the second joint and the position of the third joint are extracted. Machine-learned AI that outputs the video coordinates of
A coordinate conversion unit that converts the above video coordinates into a two-dimensional real space coordinate system using a preset projection vector and outputs the converted coordinates.
An arm position detection system for a construction machine, which comprises a filtering unit that corrects the converted coordinates to true coordinates that can be actually obtained from the dimensional information of the construction machine stored in advance. 上記フィルタリング部が、実際に取り得る真座標のうち、上記変換座標との空間距離が最も短い真座標に修正することを特徴とする請求項１に記載の建設機械のアーム位置検出システム。 The arm position detection system for a construction machine according to claim 1, wherein the filtering unit corrects the true coordinates that can actually be obtained to the true coordinates having the shortest spatial distance from the converted coordinates. 上記フィルタリング部が、先の真座標と、該先の真座標を得た時点からの経過時間とから、該経過時間内に移動可能な範囲内の真座標に修正することを特徴とする請求項１又は２に記載の建設機械のアーム位置検出システム。 The claim is characterized in that the filtering unit corrects the true coordinates of the destination and the elapsed time from the time when the true coordinates of the destination are obtained to the true coordinates within a movable range within the elapsed time. The arm position detection system for construction machinery according to 1 or 2. 上記フィルタリング部が、上記第２関節部と第３関節部の可動域内の真座標に修正することを特徴とする請求項１～３のいずれか１つの項に記載の建設機械のアーム位置検出システム。 The arm position detection system for a construction machine according to any one of claims 1 to 3, wherein the filtering unit corrects the coordinates to the true coordinates within the movable range of the second joint portion and the third joint portion. .. さらに、上記建設機械の第３関節部に角度センサを備え、
該第３関節部に設けられたアタッチメントの位置を検出することを特徴とする請求項１～４のいずれか１つの項に記載の建設機械のアーム位置検出システム。 Furthermore, an angle sensor is provided at the third joint of the construction machine.
The arm position detection system for a construction machine according to any one of claims 1 to 4, wherein the position of an attachment provided on the third joint portion is detected.
JP2020127865A 2020-07-29 2020-07-29 Arm position detection system for construction machinery Active JP7458262B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2020127865A JP7458262B2 (en) 2020-07-29 2020-07-29 Arm position detection system for construction machinery

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2020127865A JP7458262B2 (en) 2020-07-29 2020-07-29 Arm position detection system for construction machinery

Publications (2)

Publication Number Publication Date
JP2022025195A true JP2022025195A (en) 2022-02-10
JP7458262B2 JP7458262B2 (en) 2024-03-29

Family

ID=80264538

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2020127865A Active JP7458262B2 (en) 2020-07-29 2020-07-29 Arm position detection system for construction machinery

Country Status (1)

Country Link
JP (1) JP7458262B2 (en)

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4362452B2 (en) 2005-02-07 2009-11-11 青木あすなろ建設株式会社 Work equipment construction support system
JP2016061054A (en) 2014-09-17 2016-04-25 住友重機械工業株式会社 Shovel, and supporting device for the shovel
JP6491576B2 (en) 2015-09-07 2019-03-27 Ｋｙｂ株式会社 Angle detector
JP6559529B2 (en) 2015-10-06 2019-08-14 日立建機株式会社 Construction machinery
WO2018203091A1 (en) 2017-05-05 2018-11-08 J.C. Bamford Excavators Ltd Working machine
JP7177608B2 (en) 2018-06-11 2022-11-24 株式会社小松製作所 Systems including working machines, computer-implemented methods, methods of producing trained localization models, and training data

Also Published As

Publication number Publication date
JP7458262B2 (en) 2024-03-29

Similar Documents

Publication Publication Date Title
US10781572B2 (en) Working machine
KR101843382B1 (en) Periphery monitoring device for crawler-type working machine
WO2018043301A1 (en) Work machine graphics display system
WO2015182455A1 (en) Work state detection system of work vehicle, and work vehicle
WO2015198410A1 (en) Outside recognition device
KR102606049B1 (en) construction machinery
JP4740890B2 (en) Construction machine and backward movement guidance method for construction machine
US11732439B2 (en) Remote control system and main control device
JP7023813B2 (en) Work machine
JP7155516B2 (en) construction machinery
KR20220102804A (en) System and method of controlling construction machinery
US20230137344A1 (en) Work machine
CN115362297A (en) Periphery detection device for construction machine
JP2022025195A (en) Arm position detection system for construction machine
US20160150189A1 (en) Image processing system and method
KR20230017724A (en) System and method of controlling construction machinery
JP6901710B1 (en) Safety devices and work machines
KR102505529B1 (en) work machine
KR102011386B1 (en) an excavator working radius representation method
US20220372732A1 (en) Periphery monitoring device for working machine
US20240209595A1 (en) Object visualization in construction heavy equipment
WO2023100533A1 (en) Image display system, remote operation assistance system, and image display method
JP6954145B2 (en) Tip attachment dimensional measuring device
US20230370568A1 (en) Control device for work machine
JP6567736B2 (en) Shovel, shovel display method, and shovel display device

Legal Events

Date Code Title Description
A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20200904

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20230622

RD02 Notification of acceptance of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: A7422

Effective date: 20230622

RD07 Notification of extinguishment of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: A7427

Effective date: 20231010

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A821

Effective date: 20231010

RD04 Notification of resignation of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: A7424

Effective date: 20231101

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20240123

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20240214

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20240229

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20240313

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20240318

R150 Certificate of patent or registration of utility model

Ref document number: 7458262

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150