CN110956230A

CN110956230A - Reinforcing steel bar contact identification using artificial intelligent vision

Info

Publication number: CN110956230A
Application number: CN201910095956.7A
Authority: CN
Inventors: 樊家伦; 罗恩妮; 黄敬修; 梁颂恒
Original assignee: Logistics and Supply Chain Multitech R&D Centre Ltd
Current assignee: Logistics and Supply Chain Multitech R&D Centre Ltd
Priority date: 2018-09-27
Filing date: 2019-01-31
Publication date: 2020-04-03

Abstract

An apparatus or system for rebar (rebar) joint identification using artificial intelligence vision, comprising a rebar tying tool connected to a micromanipulator; and a macro manipulator for moving the position of the micro manipulator; wherein the macro manipulator is adapted to move the micro manipulator on a first plane; the micromanipulator is adapted to move the rebar tying tool in a second plane parallel to the first plane and in a first axis perpendicular to the second plane.

Description

Reinforcing steel bar contact identification using artificial intelligent vision

Technical Field

The present invention relates to construction equipment, and more particularly, to a reinforcing bar (reinforging bar/rebar) joint recognition apparatus or system for using artificial intelligent vision.

Background

Rebar tying is traditionally labor intensive, dangerous, and a cause of delays in construction processes. Particularly for large projects such as forming concrete roads, floors or floors, which involve joining intersecting sections of rebar covering an area of countless square kilometers. For example, concrete roads are typically between several meters and several tens of meters wide and over hundreds of kilometers long. These concrete road, slab or floor surfaces typically require multiple layers of rebar in a grid pattern, where the rebar intersects each other along the length and width of the surface. Each crossing point of the reinforcing bars must be fixed to secure its position and prevent the reinforcing bars from moving during the concrete construction.

The crossover point may be secured with a binding wire around the crossover point. Construction workers need to manually tie the rebar intersections together with a tying wire. Portable power strapping tools are commonly used to increase the efficiency of this task. However, the construction worker still needs to walk along the length and width of the rebar grid, identify each intersection, bend over at each intersection one at a time, and actuate a tool gun to apply a strap or clip at each intersection. Tying the rebar along a wide surface is a repetitive, time consuming, and physically demanding task. Furthermore, it is not safe to walk on the reinforcing steel grid at the construction site. Having many workers work and walk on the rebar grid only significantly increases the risk of injury.

Disclosure of Invention

The invention provides a device or a system for identifying a steel bar joint by using artificial intelligent vision.

In one aspect of the present invention, there is provided a reinforcing bar joint recognition apparatus using artificial intelligence vision, comprising:

a rebar tying tool attached to the micromanipulator; and

a macro manipulator for moving a position of the micro manipulator;

wherein the macro manipulator is adapted to move the micro manipulator on a first plane; the micromanipulator is adapted to move the rebar tying tool in a second plane parallel to the first plane and in a first axis perpendicular to the second plane.

Preferably, the macro-manipulator is adapted to move along a larger amplitude than the micro-manipulator.

Preferably, the macro manipulator comprises a gantry axis servo for moving the micro manipulator along a second axis on a first plane.

Preferably, the macro manipulator comprises a carriage axis servo for moving the micro manipulator along a third axis on the first plane such that the second axis is orthogonal to the third axis.

Preferably, the macro manipulator is adapted to move with an amplitude of meters.

Preferably, the micromanipulator includes a longitudinal axis servo for moving the rebar tying tool along a fourth axis on the second plane.

Preferably, the micromanipulator includes a transverse axis servo for moving the rebar tying tool along a fifth axis on the second plane such that the second axis is orthogonal to the fourth axis.

Preferably, the micromanipulator is adapted to move in millimeters.

Preferably, the apparatus further comprises: a system processor coupled to one or more controllers for controlling the macro manipulator and the micro manipulator.

Preferably, the system processor includes a local memory, a storage device, and a communicator.

Preferably, the system processor is adapted to connect to one or more controllers through a communicator using wireless signals.

Preferably, a controller is adapted to control an actuating servomotor for actuating the rebar tying tool.

Preferably, the apparatus further comprises one or more sensors, including an image sensor or an infrared sensor for capturing an image of the workpiece.

Preferably, the sensor is disposed on the rebar tying tool.

Preferably, the system processor is adapted to send the image to a server.

Preferably, the server is adapted to identify the work location on the workpiece by a pattern recognition algorithm.

Preferably, the pattern recognition algorithm comprises an artificial intelligence algorithm.

Preferably, the server is adapted to calculate relative coordinates of the work location.

Preferably, the system processor is adapted to move the rebar tying tool to a work position based on the relative coordinates.

Preferably, the reinforcing bar binding tool is adapted to bind the work pieces with steel wires or welding.

Advantageously, the present invention provides an improved solution for robotically automating the rebar tying process. For example, a new and novel rebar junction identification device or system using artificial intelligence vision may overcome or ameliorate at least one of the disadvantages of the prior art, or provide a useful alternative.

Other objects and advantages will become apparent when considering the following specification and the accompanying drawings.

Drawings

The features and advantages of the present invention will become apparent from the following description of embodiments thereof, given by way of example only, with reference to the accompanying drawings, in which:

FIG. 1 shows a schematic diagram of a reinforcement bar joint identification system using artificial intelligence vision according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a reinforcement bar joint identification apparatus using artificial intelligence vision according to an embodiment of the present invention;

fig. 3 illustrates the rebar tying tool of fig. 2;

fig. 4 illustrates a process of reinforcement joint identification using artificial intelligence vision of an embodiment of the present invention.

Detailed Description

The inventors have devised, through their own studies, experiments and experiments, a general operation that can facilitate a reinforcing bar binding process using a robot tool.

In one exemplary embodiment, a device may be used to bind a rectangular array of longitudinal and transverse reinforcement of paving material. The apparatus may be comprised of a self-propelled frame assembly. The self-propelled frame assembly includes means for lifting each individual one of the longitudinally aligned rebars and for periodically strapping the transverse rebars with the lifted transverse rebars in a plurality of strapping bands disposed at selected intersections of the longitudinal and transverse rebars. The self-propelled frame further comprises a transverse alignment device for supporting a plurality of longitudinal rebars and inserting a transverse rebar thereunder to receive a tying band; thereafter, a carriage assembly with a pair of spaced apart lashing assemblies movably supported on the frame is intermittently driven on the frame to effect lashing on every other longitudinal rebar. The next transverse reinforcement is similarly tied with a bracket that moves back through the frame in the opposite direction.

In alternative embodiments, an automated rebar tying machine may be used that includes an automobile, a tie-line opening, a high-definition camera, a tying machine, a spool slot, a processor, a robotic arm, an alarm, a controller, a driver, a counter, and an internal power source. The vehicle includes an alarm, a controller, a driver, a counter, and an internal power source. The wire opening is mounted at one end of the strapping machine. The high-definition camera is installed on the binding line opening. The strapping machine includes a handler, a spool slot, and is connected at one end to a robotic arm. The other end of the arm is connected to the car. The processor respectively controls the high-definition camera, the alarm, the controller, the driver, the counter and the built-in power supply.

In yet another example embodiment, an autonomous assembly may be used that includes a rack subassembly, a carriage subassembly movably mounted on the rack subassembly, a tool actuation subassembly mounted on the carriage subassembly, and an autonomous control system. The autonomous control system includes a perception subsystem, a motion planning subsystem, and a motion control subsystem. The frame subassembly includes a bridge member for traversing a selected portion of the work site and a frame drive system for effecting travel of the frame subassembly along a first path that generally travels longitudinally along the length of the selected portion of the work site. The carriage subassembly includes a carriage and a carriage drive system for effecting travel of the carriage along a second path, the second path generally transverse to the bridge member. The tool actuation subassembly includes a motion actuator operably connected to an end effector of the motion actuator, and an actuator drive system for effecting linear travel of the end effector along a third path, the third path being generally perpendicular to the second path.

However, these rebar tying machines or systems fail to provide an effective and efficient method of identifying rebar intersections. Accordingly, a skilled builder is still required to control the reinforcing bar binding machine. Construction workers with such skills are a scarce resource and tend to increase the cost of construction projects. Accordingly, there is a need for an improved method and system for performing repetitive physical demanding tasks, such as rebar tying.

Fig. 1 shows a schematic diagram of a rebar junction identification device or system using artificial intelligence vision. The system 1 of fig. 1 comprises a server 2 associated with a rebar tying device 3 as shown in fig. 2. The rebar tying device 3 includes a system processor 4 having local memory, data storage, and a communication module or for sending and receiving data and signals from the server. The system processor 4 is connected to one or more controllers 5. In one embodiment, one or more controllers 5 are connected directly to the system processor 4 through a system bus, while other controllers 5 are connected to the system processor 4 through wireless signals.

The controller 5 is adapted to send and receive remote data and signals from the system processor 4. The controller 5 may control one or

more servomotors

7, 8 and/or actuators 9. A controller 5 is connected to a frame axis servomotor 7, and the frame axis servomotor 7 controls the rebar tool 18 shown in fig. 3 to traverse the longitudinal axis (y-axis) of the work piece or rebar grid. Another controller 5 is connected to a carriage axis servomotor 8, and the carriage axis servomotor 8 controls the rebar tool 18 to pass through the transverse (x-axis) axis of the work piece or rebar grid. And another controller 5 connected to a tool actuator 9, the tool actuator 9 being capable of effecting movement of the rebar tying tool 18 in a longitudinal (y-axis), a transverse (x-axis) and a vertical (z-axis) axis. The x, y, z axes are orthogonal. The tool actuator 9 is also adapted to actuate a rebar tying tool 18 for tying down rebar junctions. In a preferred embodiment, the controller 5 of the tool actuator 9 is further adapted to control one or more sensors 6, such as infrared sensors, image sensors, etc., attached to the rebar tying tool 18 or the tool actuator 9.

In one embodiment, the gantry axis servomotor 7 and the carriage axis servomotor 8 are macro manipulators, which can move very fast over long distances (in meters). Since the gantry axis servomotor 7 and the carriage axis servomotor 8 have to be moved away from the other parts of the system, each of them will have a separate controller 5. In one embodiment, the tool actuator 9 comprises at least four servo motors; one on the longitudinal axis; one on the horizontal axis; one on the vertical axis; one for actuating the rebar tying tool. A single controller 5 may control four servo motors. In one embodiment, the four servo motors are connected in a series configuration. In one embodiment, the servo motors on the tool actuators 9 are micromanipulators, which can be moved with great precision (in millimeters) over small distances.

In operation, the system processor 4 moves the rebar tool around the workpiece and takes a plurality of images with the sensor 6. The system processor 4 then sends the images to the server 2 for image recognition using artificial intelligence algorithms to identify the work site or unbundled rebar junctions and calculate the relative coordinates of the work site. The server 2 then sends the relative coordinates of the unbundled rebar junctions to the system processor 4. Then, the system processor 4 moves the bar binding tool to the corresponding unbounded bar junction and actuates the bar binding tool to bind the bar junction.

It is contemplated that rebar tying tool 18 can tie rebar with steel wire or by welding.

Reference is now made to fig. 2 and 3. The apparatus 10 of fig. 2 includes a frame axle rail 12 that produces movement along a longitudinal path of the work site. Carriage axle track 14 moves in a lateral direction, generally transverse to the longitudinal path; and tool actuator 16 has rebar tying tool 18 attached thereto. The tool actuator 16 effects movement of the rebar tying tool 18 in a vertical direction as shown in fig. 2.

The typical size of the rail grid is about 4.9m x 3m and the typical diameter of the rebar is 10-12 mm. In one embodiment of the present invention, a micro-macro manipulator with AI vision is provided. The macro manipulator includes a large 2-axis gantry rail. The vision sensor or camera is adapted to identify the rebar junction or intersection location, for example, the depth camera may sense a bar object having a diameter of 10mm 30cm above the workpiece grid. The micromanipulator with the automated tendon layer then precisely positions the joints. In one embodiment, one or more piezoelectric motors or voice coil motors.

The carriage shaft rail 12 allows the carriage shaft rail 14 to move laterally along the longitudinal axis of the workpiece or rebar mesh. In one embodiment, the frame shaft rail comprises a conventional steel rail. In an alternative embodiment, the frame axle track may be constructed with wheels.

Carriage axle track 14 includes a servo motor or actuator for controlling movement of the carriage axle track along carriage axle track 12. A servo motor or actuator is typically connected to one or more wheels to facilitate lateral movement. In one embodiment of the present invention, the servo motors or actuators of carrier shaft rail 14 are connected to brakes for securing the carrier shaft rail in a precise position on rack shaft rail 12.

Carriage axle track 14 is adapted to allow movement of tool actuator 16 over a transverse span of carriage axle track 14 that is substantially perpendicular to rack axle track 12. In one embodiment, the servo motor or actuator of the carriage shaft track 14 is connected to the microcontroller 5, and the microcontroller 5 is communicable with the system processor 4 of the apparatus 10. In one embodiment, the system processor is adapted to communicate with the microcontroller 5 by wireless means, such as radio, WiFi, bluetooth, etc.

Carriage shaft track 14 is adapted to allow tool actuator 16 to traverse its entire length laterally. In one embodiment, the tool actuators 16 include a first set of servo motors or actuators that move along the carriage shaft rail 14. Optionally, carriage shaft track 14 includes a servo motor or actuator to propel tool actuator 16 over its length. In one embodiment, the servo motor or actuator of carriage shaft track 14 is coupled to a microcontroller that is communicatively coupled to the system processor of device 10. The system processor is adapted to communicate with the microcontroller 5 by wireless means, such as radio, Wi-Fi, bluetooth, etc.

The gantry axle track 12 and the carriage axle track 14 include one or more macro manipulators. In one embodiment, one or more voice coil motors are used to drive the macro manipulator.

In a preferred embodiment as shown in fig. 2, apparatus 10 includes one or more tool actuators 16 on carriage axle track 14. Each tool actuator 16 is connected to a rebar tying tool. Each tool actuator has a first set of servo motors or actuators or micromanipulators to move the rebar tying tool 18 along the longitudinal, lateral, and vertical axes with less displacement than the macro-manipulator. Each tool actuator has a second set of servo motors or actuators to actuate the rebar tying tool 18 so that the rebar tying tool can tie joints of rebar. Each tool actuator 16 has a tool actuator microcontroller for controlling the first and second sets of servo motors or actuators and is in communication with the system processor.

In one embodiment, each tool actuator microcontroller is associated with a separate system processor or separate system core so that each tool actuator 16 can simultaneously perform independent tasks through parallel processing.

In one embodiment, one or more sensors are attached to rebar tying tool 18. The sensor may be an infrared sensor, an RGB sensor, a laser sensor, a high definition camera, or a combination thereof. In one application of an embodiment of the present invention, rebar tying tool 18 has an infrared light source, an infrared sensor, and an RGB sensor to produce a three-dimensional depth color image. In another embodiment, the rebar tying tool 18 has two cameras to produce a stereoscopic image. In general, using only RGB sensors may make joint identification at multiple layers difficult. The combination of depth and RGB sensors may improve the reliability of images captured in the background of a multi-layer workpiece.

In the preferred embodiment, rebar junction detection and positioning is accomplished by a so-called hand-eye depth or stereo vision system, in which a sensor or camera is mounted on the rebar tying tool 18. This solution is particularly suitable for use in situations where it is not possible to secure the mounting equipment (e.g. camera, arm) above the rebar grid. Alternatively, the sensor is attached to the tool actuator 16.

The depth or stereo vision system consists of two sensors rigidly mounted to the rebar tying tool 18. Hand-eye depth or stereo vision systems provide many benefits because the sensors can be manipulated to obtain an unobstructed view of the rebar. Their small size also allows better access to a wider variety of rebar tying locations. In general, it is difficult to capture the young rebar using a depth sensor, but the eye-of-hand configuration allows the camera to move closer to the rebar to capture a better depth image.

Depth or stereo vision systems are implemented using multiple sensors or cameras. Although depth or stereo vision may be achieved using a single movement sensor or camera, or a single sensor or camera at two adjacent tool actuators 16, dual sensors or cameras allow for efficient capture of rebar junction images, as adjacent tool actuators 16 need not be moved multiple times. Furthermore, the front position of the dual sensors or cameras makes it easier to control the distance between the two sensors or cameras to produce a more accurate depth stereo image.

In one embodiment, the system processor is adapted to forward the depth or stereo image to the server 2 for joint identification using artificial intelligence vision. The server 2 may be a single system or a cloud system located remotely from the work site.

Fig. 4 provides a schematic diagram of a process 30 for an apparatus or system for rebar junction identification using artificial intelligence vision. In operation, the device 10 must calibrate its position and hand-eye coordination, as shown by calibration step 32. The calibration step 32 may be performed only once at device start-up.

After the calibration step 32, the apparatus 10 may move the rebar tying tool 18 and capture a plurality of images in an image capture step 34. The system process of device 10 may store the previous location so that device 10 may recover the contact identification from the previous location.

In one embodiment, the apparatus 10 must be calibrated in a calibration step 32 prior to performing the rebar tying task. Calibration requires the use of a vision system to establish the relationship between the sensors, the rebar tying tool 18, and the work piece. The first step is to capture a set of images of a calibration object of known dimensions to provide a correspondence between pixels and real world or relative coordinates. The process of sensor calibration may involve processing multiple images at each rebar tying tool 18 and may be said to be the most important step in the overall calibration of the vision system.

In one embodiment, sensor calibration is achieved by capturing multiple images of a calibration pattern. The calibration pattern provides a set of feature points with known 3D real world or relative coordinates. By obtaining the correspondence between the feature points in the 2D image plane and the 3D real world from a plurality of images, the sensor parameters can be determined. Since sensor calibration of depth or stereo systems is time consuming and highly repetitive. From this point of view, it makes sense to automate the camera calibration process.

In one embodiment, an auto-calibration algorithm is used to determine intrinsic and extrinsic parameters of the camera. Based on data obtained from the initial sensor calibration, estimates of hand-eye and tool-world transformations are determined. Finally, the actual hand-eye and tool world transformations and the calibration of the apparatus 10 are simultaneously addressed using a non-linear optimization method.

Once the depth or stereo image is captured in the image capture step 34, the depth or stereo image is subjected to one or more image filters for an image pre-processing step 36. The image filter will remove noise from the image for pattern recognition. In general, the performance of a depth or RGB camera may degrade in a semi-outdoor environment. In one embodiment, the image pre-processing step 36 may reduce noise due to outdoor environments.

Referring to the image pre-processing step 36, outdoor image degradation due to severe weather conditions is considered to be a major problem in most vision-based applications.

In one embodiment, the image pre-processing step 36 utilizes image enhancement methods, such as model-based and non-model-based methods. Non-model based methods use information in the image for processing and model based methods use additional information about the imaging device and the environment for processing.

In another embodiment, a polarizing filter or sensor is used to reduce glare, reduce surface reflections, and increase the clarity of structures, defects, and shapes.

After the image preprocessing step 36, the image is subjected to an artificial intelligence visual recognition step 38 to identify the unbundled rebar junctions. In one embodiment of the present invention, the apparatus 10 uses a neural network pattern recognition algorithm to identify the unbundled rebar junctions. This process is typically performed in a remote server or cloud with significantly higher processing power. Neural networks have been pre-trained to optimize the recognition process.

Once the unbundled rebar junction is determined, the system processor will send an instruction to the device 10 to move the tool actuator to the position of the rebar junction and actuate the rebar tying tool 18 to tie the rebar junction. In this case, the apparatus 10 only requires automatic rebar tying of at least 50% or more of the internal joints.

It is to be understood that variations may be made in the core teachings of the invention as will be appreciated by those skilled in the art.

Although the present invention has been described with reference to specific examples, it will be appreciated by those skilled in the art that the present invention may be embodied in many other forms to conform to the broad principles and spirit of the invention described herein.

The invention and the described embodiments include in particular the best mode known to the applicant for carrying out the invention. The invention and the described preferred embodiments comprise in particular at least one industrially applicable feature.

Claims

1. A reinforcement contact identification apparatus using artificial intelligence vision, comprising:

a rebar tying tool attached to the micromanipulator; and

a macro manipulator for moving a position of the micro manipulator;

2. The apparatus of claim 1, wherein the macro-manipulator is adapted to move along a larger amplitude than the micro-manipulator.

3. The apparatus of claim 2, wherein the macro-manipulator comprises a gantry axis servo for moving the micro-manipulator along a second axis on the first plane.

4. The apparatus of claim 3, wherein the macro manipulator comprises a carriage axis servo for moving the micro manipulator along a third axis on the first plane such that the second axis is orthogonal to the third axis.

5. The apparatus of claim 4, wherein the macro manipulator is adapted to move in meters.

6. The apparatus of claim 1, wherein the micromanipulator includes a longitudinal axis servo for moving the rebar tying tool along a fourth axis on the second plane.

7. The apparatus of claim 6, wherein the micromanipulator includes a lateral axis servo for moving the rebar tying tool along a fifth axis on the second plane such that the second axis is orthogonal to the fourth axis.

8. The apparatus of claim 7, wherein the micromanipulator is adapted to move in millimeters.

9. The apparatus of claim 1, further comprising: a system processor coupled to one or more controllers for controlling the macro manipulator and the micro manipulator.

10. The apparatus of claim 9, wherein the system processor comprises a local memory, a storage device, and a communicator.

11. The apparatus of claim 10, wherein the system processor is adapted to connect to one or more of the controllers through the communicator using wireless signals.

12. The apparatus of claim 9, wherein one of the controllers is adapted to control an actuating servomotor for actuating the rebar tying tool.

13. The apparatus of claim 10, further comprising one or more sensors including an image sensor or an infrared sensor for capturing an image of the workpiece.

14. The apparatus of claim 13, wherein the sensor is located on the rebar tying tool.

15. The apparatus of claim 14, wherein the system processor is adapted to send the image to a server.

16. The apparatus of claim 15, wherein the server is adapted to identify the work location on the workpiece by a pattern recognition algorithm.

17. The apparatus of claim 16, wherein the pattern-aware algorithm comprises an artificial intelligence algorithm.

18. The apparatus of claim 17, wherein the server is adapted to calculate relative coordinates of the work location.

19. The apparatus of claim 18, wherein the system processor is adapted to move the rebar tying tool to a work position based on the relative coordinates.

20. The apparatus of claim 19, wherein the rebar tying tool is adapted to tie the work pieces with wires or welds.