WO2022246477A1

WO2022246477A1 - A trailer hitch assist system and method

Info

Publication number: WO2022246477A1
Application number: PCT/US2022/072504
Authority: WO
Inventors: Joseph Burtch; Nizar Ahamed; Sina DAVANI
Original assignee: Continental Autonomous Mobility US, LLC
Priority date: 2021-05-21
Filing date: 2022-05-23
Publication date: 2022-11-24

Abstract

A trailer hitch assist method and system includes receiving images from a camera positioned at the rear of the tow vehicle. Instructions are sent to present the images on a display. Based upon the images, a bounding box overlay is determined indicative of an acceptable area in the images for a representation of a trailer coupler for hitching thereto. Instructions are sent to display the overlay on the display with the images. A user selection of a trailer coupler representation within the bounding box overlay is received. A second bounding box overlay surrounding the trailer coupler representation is determined and instructions sent to depict on the display the second bounding box overlay. A trailer hitching operation is initiated, including sending to a vehicle steering system instructions causing the tow vehicle to be autonomously steered in a rearward direction towards the trailer associated with the selected trailer coupler representation.

Description

A TRAILER HITCH ASSIST SYSTEM AND METHOD

TECHNICAL FIELD

[0001] This disclosure relates to a method and system for performing a trailer hitch assist operation.

BACKGROUND

[0002] Trailers are usually unpowered vehicles that are pulled by a powered tow vehicle. A trailer may be a utility trailer, a popup camper, a travel trailer, livestock trailer, flatbed trailer, enclosed car hauler, and boat trailer, among others. The tow vehicle may be a car, a crossover, a truck, a van, a sports-utility -vehicle (SUV), a recreational vehicle (RV), or any other vehicle configured to attach to the trailer and pull the trailer. The trailer may be attached to a powered vehicle using a trailer hitch. A receiver hitch mounts on the tow vehicle and connects to the trailer hitch to form a connection. The trailer hitch may be a ball and socket, a fifth wheel and gooseneck, or a trailer jack. Other attachment mechanisms may also be used.

[0003] Recent advancements in camera sensor technology have led to improved vehicle trailer assist functions. As such, it is desirable to provide an vehicle reverse system that is capable of maneuvering a tow vehicle towards the trailer for hitching the vehicle to the trailer.

SUMMARY

[0004] Example embodiments of the present disclosure are directed to a method for maneuvering a tow vehicle towards a trailer positioned behind the tow vehicle. The method includes receiving, at data processing hardware, one or more images from a camera positioned on a back portion of the tow vehicle and in communication with the data processing hardware, and sending instructions to present on a user display the one or more images. Based upon the received one or more images, a bounding box overlay is determined indicative of an acceptable area in the one or more images for a representation of a trailer coupler for hitching thereto. Instructions are sent to display the overlay on the user display with the one or more images. A user selection of a trailer coupler representation within the bounding box overlay is received at the data processing hardware. Responsive to receiving the user selection of the trailer coupler representation, a second bounding box overlay surrounding the trailer coupler representation is determined and instructions are sent to depict on the user display the second bounding box overlay. A trailer hitching operation is initiated following the second bounding box determination. During the trailer hitching operation, instructions are sent from the data processing hardware to a steering system of the tow vehicle that is in communication with the data processing hardware. The instructions, when executed, cause the tow vehicle to be autonomously steered throughout the trailer hitching operation in a rearward direction towards the trailer associated with the selected trailer coupler representation. [0005] During the trailer hitching operation, images are received at the data processing hardware. A speed of the tow vehicle is determined. A vehicle speed indicator overlay is determined based upon the determined tow vehicle speed.

Instructions are sent to the user display for presenting on the user display with the plurality of images the speed indicator overlay.

[0006] In one implementation, the vehicle speed indicator is a bar graph including a maximum speed value. A visual appearance of the bar graph changes as the determined tow vehicle speed approaches the maximum speed value.

[0007] The method may further include, during the trailer hitching operation, receiving, at the data processing hardware, a plurality of images from the camera, and initially tracking the representation of the trailer coupler in the plurality of images. Following the tracking, the method may include detecting the trailer coupler representation until the trailer coupler representation is determined to be aligned with a tow ball representation corresponding to a tow ball of the tow vehicle.

[0008] The method may further include receiving, at the data processing hardware, a user selection of a representation of a tow ball of the tow vehicle in the one or more images. Responsive to receiving the user selection of the tow ball representation, a marker overlay is determined on or surrounding the tow ball representation and instructions are sent for depicting on the user display the marker overlay surrounding the tow ball representation. [0009] The method may further include, during the trailer hitching operation, receiving, at the data processing hardware, a plurality of images from the camera, determining at least one of a maximum velocity or maximum acceleration of the tow vehicle, detecting at least one of a speed of the tow vehicle or an acceleration of the tow vehicle, and sending, from the data processing hardware to a drive system of the tow vehicle instructions for limiting at least one of the tow vehicle speed or the tow vehicle acceleration to be less than the determined maximum velocity or maximum acceleration, respectively.

[0010] In one implementation, the method includes determining, by the data processing hardware, a distance between the tow vehicle tow ball and the trailer coupler based in part upon the received images, wherein sending instructions to depict on the user display the second bounding box overlay includes instructions to visually change the second bounding box overlay on the user display when the determined distance is less than a predetermined distance threshold. [0011] A trailer hitch assist system includes data processing hardware and non- transitory memory having program code instructions stored therein which, when executed by the data processing hardware, causes the data processing hardware to perform one or more of the method operations described above.

DESCRIPTION OF DRAWINGS [0012] FIG. l is a schematic top view of an exemplary tow vehicle at a distance from trailers positioned behind the tow vehicle.

[0013] FIG. 2 is a schematic view of an exemplary tow vehicle of FIG. 1.

[0014] FIGS. 3-6 are rearward views presented on a tow vehicle user display associated with a trailer hitching operation, according to an example embodiment. [0015] FIGS. 7 and 8 are example flowcharts illustrating methods associated with one or more trailer hitching operation, according to an example embodiment.

[0016] Like reference symbols in the various drawings indicate like elements. DETAILED DESCRIPTION

[0017] A tow vehicle, such as, but not limited to a car, a crossover, a truck, a van, a sports-utility -vehicle (SUV), and a recreational vehicle (RV) may be configured to tow a trailer. The tow vehicle connects to the trailer by way of a trailer coupler. Manually backing up to a trailer may need a lot of effort from the driver. The driver must constantly monitor a rear vehicle camera and steer the vehicle to bring the tow ball of the vehicle underneath the coupler of a trailer. Therefore, it is desirable to have a tow vehicle that is capable of assisting in backing up the tow vehicle towards a driver specified position, for example a trailer, identified from an image of the rearward environment of the vehicle and displayed on a user interface, such as a user display.

[0018] Referring to FIGS. 1 and 2, the tow vehicle 100 may include a drive system 110 that maneuvers the tow vehicle 100 across a road surface based on drive commands having x, y, and z components, for example. As shown, the drive system 110 includes a front right wheel 112, 112a, a front left wheel 112, 112b, a rear right wheel 112, 112c, and a rear left wheel 112, 112d. The drive system 110 may include other wheel configurations as well. The drive system 110 may also include a brake system 114 that includes brakes associated with each wheel 112, 112a-d, and an acceleration system 116 that is configured to adjust a speed and direction of the tow vehicle 100. In addition, the drive system 110 may include a suspension system 118 that includes tires associates with each wheel 112, 112a-d, tire air, springs, shock absorbers, and linkages that connect the tow vehicle 100 to its wheels 112, 112a-d and allows relative motion between the tow vehicle 100 and the wheels 112, 112a-d. The suspension system 118 may be configured to adjust a height of the tow vehicle 100 allowing a tow vehicle hitch 120 (e.g., a tow vehicle tow ball 122) to align with a trailer hitch 210 (e.g., trailer hitch coupler 212 and a trailer hitch bar 214), which allows for connection between the tow vehicle 100 and the trailer 200.

[0019] The tow vehicle 100 may move across the road surface 10 by various combinations of movements relative to three mutually perpendicular axes defined by the tow vehicle 100: a transverse axis X, a fore-aft axis Y, and a central vertical axis Z. The transverse axis x extends between a right side and a left side of the tow vehicle 100. A forward drive direction along the fore-aft axis Y is designated as F, also referred to as a forward motion. In addition, an aft or rearward drive direction along the fore-aft direction Y is designated as R, also referred to as rearward motion. When the suspension system 118 adjusts the suspension of the tow vehicle 100, the tow vehicle 100 may tilt about the X axis and or Y axis, or move along the central vertical axis Z. [0020] The tow vehicle 100 may include a user interface 130. The user interface 130 may include the display 140, a knob 132, and a button 134, which are used as input mechanisms. In some examples, the display 140 may show the knob 132 and the button 134. While in other examples, the knob 132 and the button 134 are a knob button combination. In some examples, the user interface 130 receives one or more driver commands from the driver via one or more input mechanisms or a touch screen display 140 and/or displays one or more notifications to the driver. The user interface 130 is in communication with a vehicle controller 160, which is in turn in communication with a sensor system 150. In some examples, the display 140 displays an image 151 of an environment of the tow vehicle 100 leading to one or more commands being received by the user interface 130 (from the driver) that initiate execution of one or more behaviors. In some examples, the user display 140 displays an image 151 of the rearward environment of the tow vehicle 100. In this case, the driver may select a position within the image 151 that is indicative of the environment (i.e., the trailer 200) that the driver wants the vehicle to semi-autonomously maneuver towards. In some examples, the user display 140 displays one or more representations 146 of trailers 200 positioned behind the tow vehicle 100.

[0021] The tow vehicle 100 may include the sensor system 150 to provide reliable and robust driving. The sensor system 150 may include different types of sensors that may be used separately or with one another to create a perception of the environment of the tow vehicle 100 that is used for the tow vehicle 100 to drive and aid the driver in make intelligent decisions based on objects and obstacles detected by the sensor system 150.

[0022] The sensor system 150 may include the one or more cameras 152 supported by the tow vehicle 100. In some implementations, the tow vehicle 100 includes a rear camera 152 that is mounted to provide a view of a rear-driving path for the tow vehicle 100. The rear camera 152 may include a fisheye lens that includes an ultra wide-angle lens that produces strong visual distortion intended to create a wide panoramic or hemispherical image. Fisheye cameras capture images having an extremely wide angle of view. Moreover, images captured by the fisheye camera have a characteristic convex non-rectilinear appearance. Other types of cameras may also be used to capture images of the rear of the tow vehicle 100. As previously described, the images 151 captured by the rear camera 152 are displayed on the display 140 and show a rear environment of the tow vehicle 100 that may include one or more trailer representations 146, 146a-b of the one or more trailers 200, 200a-b positioned behind the tow vehicle 100.

[0023] In some examples, the sensor system 150 also includes one or more wheel encoders 154 associated with one or more wheels 112, 112a-d of the tow vehicle 100.

The wheel encoder 154 is an electro-mechanical device that converts an angular position or motion of the wheel to analog or digital output signals. Therefore, the wheel encoder 154 determines the speed and distance the wheel 112, 112a-d has traveled. The sensor system 150 may also include a steering wheel angle sensor 156 that measures a steering wheel data 157 such as position angle and rate of turn of the steering wheel.

[0024] The sensor system 150 may also include other sensors 158 that detect the vehicle motion, i.e., speed, angular speed, position, etc. The other sensors 158 may include an inertial measurement unit (EMU) configured to measure the vehicle’s linear acceleration (using one or more accelerometers) and rotational rate (using one or more gyroscopes). In some examples, the EMU also determines a heading reference of the tow vehicle 100. Therefore, the IMU determines the pitch, roll, and yaw of the tow vehicle 100. In some examples, the other sensors 158 may include radar sensors configured to determine a range, angle, or velocity of objects (e.g., the trailer 200). The radar sensors 158 transmit radio signals and measure the time it takes for a reflection of the transmitted signal to return. The radar sensors determine a distance to the object as one-half the round-trip time multiplied by the speed of the signal. The radar sensors may also determine a speed of the tow vehicle 100. The speed may be based on a distance from the radar sensors 158 to the object at a first time, and a distance from the radar sensors 158 to the object at a second time. Therefore, the radar sensors 158 may determine the speed based on a difference between the distances at the first and second time divided by the difference between the first and second time. The other sensors 158 may include, but are not limited to, sonar, LIDAR (Light Detection and Ranging, which can entail optical remote sensing that measures properties of scattered light to find range and/or other information of a distant target), LADAR (Laser Detection and Ranging), ultrasonic, etc. [0025] The vehicle controller 160 includes a computing device (or processor) 162 (e.g., central processing unit having one or more computing processors) in communication with non-transitory memory 164 (e.g., a hard disk, flash memory, random-access memory) capable of storing instructions executable on the computing processor(s) 162. In some examples, the vehicle controller 160 executes a trailer hitch assist system 170 and a drive assistance system 190.

[0026] The trailer hitch assist system 170, which may include one or more algorithms or modules for performing a trailer hitching operation, allows for a driver of a tow vehicle 100 who wants to tow a trailer 200 to position the tow ball 122 of the tow vehicle 100 at or under the coupler 212 of the trailer 200. The driver initially activates the trailer hitch assist system 170, such as through use of the user display 140 of the tow vehicle’s user interface 130. Referring to FIG. 3, the user display 140 presents a rearward view 310 from a rear camera 152 of the tow vehicle 100, including an overlay or bounding box 320 of an acceptable region for the location of a representation 212’ of the trailer coupler 212 of a trailer 200 in order for the trailer hitch assist system 170 to operate successfully. [0027] In an example embodiment, characteristics of the bounding box overlay 320 (e.g., the size, shape, and position) within the rear view is determined based upon limitations of the trailer hitch assist system 170 and the steering system 113 of the tow vehicle 100. In the example embodiment illustrated, the overlay 320 has a trapezoidal shape but it is understood that the overlay 320 may have other shapes. Specifically, the trapezoidal shaped overlay 320 includes two largely parallel line segments 321 and 322 with the latter being longer than the former. As illustrated, the line segment 321 is positioned closer to the representation of the tow ball 122 in the displayed image compared to the line segment 322 so that the bounding box 320 illustrates an expanding region, from the representation 122’ of the tow ball 122, from which the representation 212’ of a trailer coupler 212 may be selected. Based upon the above-mentioned limitations, the representation 212’ of the trailer coupler 212 is expected to be within the bounding box 320. [0028] The trailer hitch assist system 170 includes a trailer coupler identifier module 171 (FIG. 2) which allows for driver selection of the trailer coupler 212 of the trailer 200 that is desired to be hitched to the tow vehicle 100. This may be helpful if a plurality of trailer coupler representations are depicted in the rear view on the user display 140 that are located within the bounding box 320. In response to the driver touching the user display 140 near the representation 212’ of the trailer coupler 212 in the display, the trailer coupler identifier 171 causes the image displayed to “zoom in” or otherwise present a magnified view of the trailer coupler representation 212’ that is within the overlay 320. In one implementation, the amount of magnification is fixed and in another implementation, the magnification amount may vary based upon user input, such as user input via touching the touch screen user display 140. For example, the driver may set the magnification amount by touching the user display 140 with two fingers and then separating the fingers (which changes the image magnification) until the desired magnification is provided. With the representation 212’ of the trailer coupler 212 being within the magnified view on the user display 140 (see FIG. 4), the driver may touch the trailer coupler representation 212’ or near such representation to identify the trailer coupler 212 to hitch to during the trailer hitch assist operation. Upon the driver’s identification of the representation 212’ of the trailer coupler 212, a bounding box overlay 402 (see FIG. 5) may be included on the user display 140 surrounding the representation 212’ of the selected trailer coupler 212. In the example embodiment illustrated, the bounding box overlay 402 is noticeably smaller than the bounding box overlay 320 described above. As shown, the bounding box overlay 402 immediately surrounds the trailer coupler representation 212’ and is colored so that the driver is able to quickly identify the location of the trailer coupler representation 212’ in the displayed image.

[0029] In addition, the trailer assist system 170 includes a tow ball identifier module 173 (see FIG. 2) which provides for driver selection of the representation 122’ of the tow ball 122 of the tow vehicle 100. Specifically, the tow ball identifier 173 may perform the same as the trailer coupler identifier 171 but for the tow ball 122 instead of the trailer coupler 212, including the zoom/image magnification feature. The trailer hitch assist system 170 may have the driver select the image or representation 122’ of the tow ball 122 using the tow ball identifier 173 before the driver selects the image/representation 212’ of the trailer coupler 212 using the trailer coupler identifier 171. Upon identification of the tow ball representation 122’, a marker overlay (seen in FIGS. 3 and 5) may be provided on the tow ball representation 122’ in the view provided to the driver. The marker overlay illustrated is a cross of two intersecting line segments. It is understood that the marker overlay may have different symbols.

[0030] The trailer assist system 170 is configured to allow the driver to initiate the trailer hitch assist operation by providing a start button depiction 330 on the user display 140 (FIG. 3) that may be activated by the driver touching it on the touch screen user display. This activation is performed following the driver selection of the trailer coupler 212 and the vehicle tow ball 122. It is understood that other techniques for activating the trailer hitch assist operation may be utilized, such as automatic initiation of the trailer hitch assist operation following driver selection of the trailer coupler 212 and the vehicle tow ball 122.

[0031] According to an example embodiment, upon activation of the trailer hitch assist operation, the trailer hitch assist system 170 cooperates with the tow vehicle’s steering system to the steering of the tow vehicle 100 as the driver controls the vehicle speed (by controlling the brake system 114 and the acceleration system 116) in moving the tow vehicle 100 towards the driver-selected coupler 212 of the trailer 200.

[0032] During the trailer hitch assist operation, images 151 of the rearward view of the tow vehicle 100, provided by the rear camera 152, are presented on the user display 140. In addition, the bounding box overlay 402 surrounds the representation 212’ of the selected trailer coupler 212, and a vehicle speed overlay 404 is displayed in a part of the image 151 of the user display 140 that is not between the trailer coupler bounding box 402 and the representation 122’ of the vehicle tow ball 122. As shown in FIG. 5, the vehicle speed overlay 404 is positioned towards an outer region of the image 151. In the example embodiment illustrated, the vehicle speed overlay 404 is a bar graph in which a bar 406, corresponding tow vehicle speed, is shown within a bounding box 408. The speed of the tow vehicle 101 may be determined by the trailer hitch assist system 170 using wheel encoders 154. A line 410, corresponding to the maximum allowed speed during a trailer hitch assist operation, is disposed across the bounding box 408. The maximum allowed speed may be predetermined by the trailer hitch assist system 170. If tow vehicle speed, which is controlled by the driver depressing the accelerator pedal of the tow vehicle 100, approaches and surpasses the maximum allowed vehicle speed, the color of the vehicle speed bar 406 may change color. For example, the vehicle speed bar 406 may be green and gradually turn to yellow as tow vehicle speed approaches the maximum allowed speed, and gradually turn to red if the vehicle speed bar 406 exceeds the maximum allowed speed.

[0033] In addition, the trailer hitch assist system 170 may provide audio and/or haptic alerts to the driver as vehicle speed approaches and surpasses the maximum allowed vehicle speed. For instance, a series of beep sounds may be generated having a frequency that increases as the vehicle speed approaches the maximum allowable vehicle speed, and becomes a constant beep sound upon the vehicle speed surpassing the maximum allowed vehicle speed. It is understood that alternatively, other audio alerts may be used.

[0034] In one implementation, the maximum allowable vehicle speed may be fixed.

In another implementation, the maximum allowable vehicle speed may vary based upon, for example, the computed distance between the trailer coupler 212 and the vehicle tow ball 122, using images 151 or any other means like using radars in the distance computation.

[0035] The trailer hitch assist system 170 may further include a motion limiting function/module 175. The motion limiting function/module 175 may override driver control of the tow vehicle’s acceleration system 116 and brake system 114 if the motion of the tow vehicle 100 exceeds the maximum allowed vehicle speed, or a maximum allowed torque/acceleration. In one implementation, the motion limiting function 175 sets a maximum velocity and torque/acceleration of the vehicle so that neither vehicle velocity nor torque exceeds predetermined maximums.

[0036] During the trailer hitch assist operation, the trailer hitch assist system 170 uses two algorithms or approaches as the tow vehicle 100 moves from its original position until the tow ball 122 is positioned under and aligned with the trailer coupler 212. Initially, the trailer hitch assist system 170 uses a lower accuracy image tracker algorithm 172 to guide the tow vehicle 100 in part by controlling the vehicle steering system 113. As the tow vehicle 100 gets closer to the trailer 200, the trailer hitch assist system 170 seamlessly switches or transitions from use of the image tracker 172 to use of a higher accuracy image detection algorithm 174. The point at which the system switches to the image detection algorithm 174 may occur when the representation 212’ of the trailer coupler 212 is more directly visible and/or detectable in the received images 151.

[0037] According to an example embodiment, the image tracker 172 performs feature tracking. In particular, the image tracker 172 looks at a set of feature(s) (e.g., pixels in the area of the representation 212’ of the trailer coupler 212) in a first image 151 and looks for the same feature in the second (next) image 151, based upon the first image. The tracking proceeds in this image-by-image fashion. When the tow vehicle 100 reaches a point that the vehicle is sufficiently close, such that the system 170 determines that the image detector 174 may accurately detect the trailer coupler 212, the trailer hitch assist system 170 utilizes the image detector 174 to detect the trailer coupler 212.

[0038] In the image detector 174, a specific object (i.e., the representation 212’ of the trailer coupler 212) is continuously searched for in at least a portion of each image 151, and the detector provides the location (x, y coordinates) where the object is located in the images 151. In one implementation, the image detector 174 searches for the object in cropped images to reduce computational power. In another implementation, the image detector 174 searches for the object in each entire image 151. [0039] The system 170 may optionally continue using the image tracker 172 as a backup option for any frame where the image detector 174 fails to detect the trailer coupler representation 212’.

[0040] The image detector 174 may use artificial intelligence and/or classical computer vision in detecting representations of the trailer coupler 212 in received images 151. For example, the image detector 174 may use feature detection in detecting representations 212’ of the trailer coupler 212 in images 151 received from the rear camera. Alternatively, the image detector 174 uses a neural network for performing the image detecting.

[0041] When the trailer hitch assist system 170 is activated, the driver is presented with a back-up camera view having an overlay of the acceptable region for the trailer coupler 212 to be in order to operate. This area corresponds to the above-described bounding box 320 and is determined from the system’s image detection and vehicle steering limitations. See FIG. 3.

[0042] During the set-up of a trailer hitch assist operation, the driver touches the user display 140, which then automatically zooms in relative to the touched area on the screen for allowing the driver to then touch close to representation 212’ of the trailer coupler 212. The driver can then start the trailer hitch assist system 170, as described above. [0043] Following the trailer hitch assist operation set up, the system 170 controls the steering of the tow vehicle 100 while the driver controls the vehicle reverse speed via the brake and accelerator if needed (i.e. uphill). The vehicle speed indicator overlay 404 is shown on the back-up view to show the acceptable speed range for the driver to control. Also, the bounding box 402 showing the current tracking point can be shown to the driver to check accuracy.

[0044] While backing up, the system first uses the lower accuracy image tracker 172 to guide the steering of the vehicle tow ball 122 to the trailer coupler 212. This allows for maximum usable distance from the trailer 200. As the vehicle 100 gets closer to the trailer 200, the higher accuracy image detection algorithm 174 seamlessly takes over tracking when the representation 212’ of the trailer coupler 212 is more directly visible. [0045] The system alerts the driver when the vehicle tow ball 122 is lined up under the trailer coupler 212. The system may also use any means of alerting the user that the tow ball 122 is close to be lined up under the trailer coupler 212 and thus the user should be prepared to stop the vehicle 100. In one example embodiment, the color of the displayed bounding box 402 changes. In another example embodiment, a separate graphical overlay is displayed that represents the distance between the tow ball 122 and the coupler 212. The driver shifts to park which automatically actuates the electric parking brake to keep alignment for successful truck-to-trailer hitching. See FIG. 6. In another example embodiment, the driver may select a touch button on the user display to prompt or confirm completion of the trailer hitch assist function/operation. The trailer hitch assist system 170 will then automatically engage the park brake and automatically change the gearbox shifts to park.

[0046] In some example embodiments, the trailer hitch assist system 170 determines a destination (e.g., a selected trailer 200) for the tow vehicle 100 to drive towards, and generates instructions for controlling the vehicle steering system 113 accordingly. The trailer hitch assist system 170 receives images 151 from the camera 152 and/or sensor data 153 from the one or more sensors 154-158 and determines a location of the trailer 200, identified by the driver via the user interface 130, with respect to the tow vehicle 100, for example, a longitudinal distance DL_g, a lateral distance Du, a vertical distance Hca and/or an angle between the rear camera 152 and a front face of the trailer 200 or the trailer coupler 212. In addition, in some example embodiments, the trailer hitch assist system 170 also determines an angle (not shown) between the tow vehicle 100 and the trailer 200, i.e., an angle between the vehicle tow ball 122 and the trailer coupler 212 or the front face of the trailer 200. The longitudinal distance DLg, the lateral distance Du, and the vertical distance Hca may be used in the instructions for controlling the steering system 113 as the driver backs up the tow vehicle 100 towards the trailer 200.

[0047] During a trailer hitch assist operation, the vehicle controller 160 executes a drive assistance system 190, which in turn includes autonomous behaviors 192. The autonomous behaviors 192 receive the vehicle autonomous steering maneuvers and executes one or more steering behaviors 192c that send commands/instructions 191 to the drive system 110, causing the vehicle 100 to autonomously steer the tow vehicle 100 based on the vehicle autonomous steering maneuvers, which causes the vehicle 100 to be autonomously steered as the vehicle 100 moves, under control of the driver, towards the bounding box overlay 402.

[0048] Each behavior 192c causes the tow vehicle 100 to take an action, such as turning at a specific angle among others. The vehicle controller 160 may assist in maneuvering the tow vehicle 100 in any direction across the road surface by controlling the steering system 113 of the drive system 110, more specifically by issuing signals, commands, or instructions 191 to the steering system 113. The steering behavior 192c may be executed to change the direction of the tow vehicle 100 based on the vehicle maneuvers.

[0049] As previously discussed, the controller 160 executes the trailer hitch assist system 170 and the drive assistance system 190 in real time. Therefore, the described system provides an automated method that needs limited driver input. [0050] FIG. 7 provides an example arrangement of operations of a trailer hitch assist method for maneuvering a vehicle 100 (e.g., a tow vehicle) in a rearward direction R towards a trailer 200, using the trailer hitch assist system 170 described above. At block 702, the method includes receiving, at data processing hardware 162, one or more images 151 from the camera 152 positioned on a back portion of the tow vehicle 100 and in communication with the data processing hardware 162. At block 704, the data processing hardware 162 receives a user (i.e., driver) selection of a tow ball representation 122’ within the one or more images. This may include providing a magnified view of a tow ball representation 122’ for selection by the user, as described above. At block 706, the data processing hardware 162 receives a user selection of a trailer coupler representation 212’ within the one or more images. This may include providing a magnified view of a trailer coupler representation 212’ for selection by the user, as described above

[0051] At block 708, the data processing hardware receives a user request to begin the trailer hitching operation. This may include, for example, the user touching the start button representation 330 as described above. At block 710, in response to the user request to begin the trailer hitching operation, the data processing hardware sends to the steering system 113 of the tow vehicle 100 instructions for autonomously steering the tow vehicle 100 as the driver controls the motion of the vehicle 100, so that the tow vehicle is maneuvered towards the trailer 200.

[0052] FIG. 8 is a flowchart illustrating details of the trailer hitch assist operation. At block 802, the trailer hitch assist system 170 continuously displays on the user display 140 the vehicle speed overlay 404 with the images 151 of the rear view of the tow vehicle captured by the rear camera 152. At block 804, the image tracker 172 is used to initially track an image of the trailer coupler 212. As the tow vehicle 100 gets closer to the trailer 200, a determination is made at 806 that the representation 212’ of the trailer coupler 212 in the images can be accurately detected. In response to the determination, the image detector 174 is used in block 808 to detect the location of the representation 212’ of the trailer coupler 212 in the received images 151. Upon a determination at block 810 that the representation 212’ of the trailer coupler 212 in the images is aligned (i.e., positioned over) the representation 122’ of the tow ball 122, the driver is alerted in 812 as described above. Upon the driver stopping the tow vehicle 100, an electric brake is automatically applied by the tow vehicle 100 by the trailer hitch assist system 170 or an audio/visual instruction is sent to the driver to cause the brake operation to occur.

[0053] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. [0054] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor. [0055] Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Moreover, subject matter described in this specification can be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The terms “data processing apparatus”, “computing device” and “computing processor” encompass all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. A propagated signal is an artificially generated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal that is generated to encode information for transmission to suitable receiver apparatus. [0056] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multi-tasking and parallel processing may be advantageous. Moreover, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0057] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

WHAT IS CLAIMED IS:

1. A method of maneuvering a tow vehicle towards a trailer positioned behind the tow vehicle, the method comprising: receiving, at data processing hardware, one or more images from a camera positioned on a back portion of the tow vehicle and in communication with the data processing hardware; sending instructions to present on a user display the one or more images; based upon the received one or more images, determining a bounding box overlay indicative of an acceptable area in the one or more images for a representation of a trailer coupler for hitching thereto, and sending instructions to display the overlay on the user display with the one or more images; receiving, at the data processing hardware, a user selection of a trailer coupler representation within the bounding box overlay; responsive to receiving the user selection of the trailer coupler representation, determining a second bounding box overlay surrounding the trailer coupler representation and sending instructions to depict on the user display the second bounding box overlay; initiating a trailer hitching operation following determining the second bounding box; and during the trailer hitching operation, sending, from the data processing hardware to a steering system of the tow vehicle that is in communication with the data processing hardware, instructions causing the tow vehicle to be autonomously steered throughout the trailer hitching operation in a rearward direction towards the trailer associated with the selected trailer coupler representation.

2. The method of claim 1, further comprising, during the trailer hitching operation, receiving, at the data processing hardware, a plurality of images from the camera, determining a speed of the tow vehicle, determining a vehicle speed indicator overlay based upon the determined tow vehicle speed, and sending instructions to the user display for presenting on the user display with the plurality of images the speed indicator overlay.

3. The method of claim 2, wherein the vehicle speed indicator is a bar graph including a maximum speed value, wherein a visual appearance of the bar graph changes as the determined tow vehicle speed approaches the maximum speed value.

4. The method of claim 1, further comprising, during the trailer hitching operation, receiving, at the data processing hardware, a plurality of images from the camera, initially tracking the representation of the trailer coupler in the plurality of images and, following the tracking, detecting the trailer coupler representation until the trailer coupler representation is determined to be aligned with a tow ball representation corresponding to a tow ball of the tow vehicle.

5. The method of claim 1, further comprising: receiving, at the data processing hardware, a user selection of a representation of a tow ball of the tow vehicle in the one or more images, and responsive to receiving the user selection of the tow ball representation, determining a marker overlay on or surrounding the tow ball representation and sending instructions for depicting on the user display the marker overlay surrounding the tow ball representation.

6. The method of claim 1, further comprising, during the trailer hitching operation, receiving, at the data processing hardware, a plurality of images from the camera, determining at least one of a maximum velocity and maximum acceleration of the tow vehicle, detecting at least one of a speed of the tow vehicle or an acceleration of the tow vehicle, and sending, from the data processing hardware to a drive system of the tow vehicle instructions for limiting at least one of the tow vehicle speed or the tow vehicle acceleration to be less than the determined maximum velocity or maximum acceleration, respectively.

7. The method of claim 1, further comprising determining, by the data processing hardware, a distance between the tow vehicle tow ball and the trailer coupler based in part upon the received images, wherein sending instructions to depict on the user display the second bounding box overlay includes instructions to visually change the second bounding box overlay on the user display when the determined distance is less than a predetermined distance threshold.

8. A trailer hitch assist system for a tow vehicle, comprising: data processing hardware and non-transitory memory having program code instructions stored therein which, when executed by the data processing hardware, causes the data processing hardware to perform operations comprising receiving, at data processing hardware, one or more images from a camera positioned on a back portion of the tow vehicle and in communication with the data processing hardware; sending instructions to present on a user display the one or more images; based upon the received one or more images, determining a bounding box overlay indicative of an acceptable area in the one or more images for a representation of a trailer coupler for hitching thereto, and sending instructions to display the overlay on the user display with the one or more images; receiving, at the data processing hardware, a user selection of a trailer coupler representation within the bounding box overlay; responsive to receiving the user selection of the trailer coupler representation, determining a second bounding box overlay surrounding the trailer coupler representation and sending instructions to depict on the user display the second bounding box; initiating a trailer hitching operation following determining the second bounding box; and during the trailer hitching operation, sending, from the data processing hardware to a steering system of the tow vehicle that is in communication with the data processing hardware, instructions causing the tow vehicle to be autonomously steered throughout the trailer hitching operation in a rearward direction towards the trailer associated with the selected trailer coupler representation.

9. The system of claim 2, wherein the operations further comprise, during the trailer hitching operation, receiving a plurality of images from the camera, determining a speed of the tow vehicle, determining a vehicle speed indicator overlay based upon the determined tow vehicle speed, and sending instructions to the user display for presenting on the user display with the plurality of images the speed indicator overlay.

10. The system of claim 9, wherein the vehicle speed indicator is a bar graph including a maximum speed value, wherein a visual appearance of the bar graph changes as the determined tow vehicle speed approaches the maximum speed value.

11. The system of claim 8, wherein the operations further comprise, during the trailer hitching operation, receiving a plurality of images from the camera, initially tracking the representation of the trailer coupler in the plurality of images and, following the tracking, detecting the trailer coupler representation until the trailer coupler representation is determined to be aligned with a tow ball representation corresponding to a tow ball of the tow vehicle.

12. The system of claim 8, wherein the operations further comprise: receiving a user selection of a representation of a tow ball of the tow vehicle in the one or more images, and responsive to receiving the user selection of the tow ball representation, determining a marker overlay on or surrounding the tow ball representation and sending instructions for depicting on the user display the marker overlay surrounding the tow ball representation.

13. The system of claim 8, wherein the operations further comprise, during the trailer hitching operation, receiving a plurality of images from the camera, determining at least one of a maximum velocity and maximum acceleration of the tow vehicle, detecting at least one of a speed of the tow vehicle or an acceleration of the tow vehicle, and sending, from the data processing hardware to a drive system of the tow vehicle instructions for limiting at least one of the tow vehicle speed or the tow vehicle acceleration to be less than the determined maximum velocity or maximum acceleration, respectively.

14. The system of claim 8, further comprising determining, by the data processing hardware, a distance between the tow vehicle tow ball and the trailer coupler based in part upon the received images, wherein sending instructions to depict on the user display the second bounding box overlay includes instructions to visually change the second bounding box overlay on the user display when the determined distance is less than a predetermined distance threshold.