JP6762863B2 - Imaging equipment, image processing equipment, display systems, and vehicles - Google Patents

Description

The present disclosure relates to imaging devices, image processing devices, display systems, and vehicles.

Conventionally, a technique for displaying an image of an external region of a moving body such as a vehicle has been known. For example, Patent Document 1 discloses a technique for controlling power supply to a monitor that displays an image of a camera provided in a vehicle.

JP-A-2009-40113

Conventionally, there has been room for improvement in the technique of displaying an image of an external region of a moving body.

The present disclosure relates to an image processing device, an imaging device, and a display system that improve the convenience of a technique for displaying an image in an external region of a moving body.

The image pickup device according to the embodiment of the present disclosure includes an image pickup element that images the rear of the vehicle to generate a first image, and a guide that indicates a predicted course of the vehicle on a road surface portion included in a display area on the first image. It is equipped with a control unit that synthesizes images, and extends a predetermined distance from the road surface in the height direction between the detection target and the guide image in the first image, and the more the distance from the road surface in the height direction, the higher the transparency. The first recognition wall image showing the surface to be raised is combined.

The image processing device according to the embodiment of the present disclosure includes a communication unit that acquires a first image of the rear of the vehicle, and a guide that indicates a predicted course of the vehicle on a road surface portion included in a display area on the first image. It is equipped with a control unit that synthesizes images, and extends a predetermined distance from the road surface in the height direction between the detection target and the guide image in the first image, and the more the distance from the road surface in the height direction, the higher the transparency. The first recognition wall image showing the surface to be raised is combined.

The display system according to the embodiment of the present disclosure includes an image sensor that images the rear of the vehicle to generate a first image, and a guide that indicates a predicted course of the vehicle on a road surface portion included in a display area on the first image. A control unit for synthesizing images and a display device for displaying an image in a display area in which guide images are combined are provided, and a predetermined distance between a detection target and the guide image in the first image is determined in the height direction from the road surface. A first recognition wall image showing a surface whose transparency increases as the distance increases and the distance from the road surface increases in the height direction is synthesized.

The vehicle according to an embodiment of the present disclosure includes a display system. The display system includes an image sensor that images the rear of the vehicle to generate a first image, a control unit that synthesizes a guide image showing a predicted course of the vehicle on a road surface portion included in a display area on the first image, and a control unit. A display device for displaying an image in a display area in which a guide image is combined is provided, and a predetermined distance extends from the road surface in the height direction between the detection target and the guide image in the first image, and the height from the road surface. A first recognition wall image showing a surface whose transparency increases as the distance increases in the direction is combined.

According to the image processing device, the image pickup device, and the display system according to the embodiment of the present disclosure, the convenience of the technique for displaying the image of the external region of the moving body is improved.

It is a block diagram which shows the schematic structure of the display system which concerns on one Embodiment of this invention. It is the figure which looked at the vehicle equipped with the display system from the left side. It is a figure which shows the appearance of the display device of FIG. 1 schematically. It is a figure which shows the 1st example of the 1st video. It is a figure which shows the 1st example of the 2nd video corresponding to the display area of the 1st video of FIG. It is a figure which shows the example of the 3rd marker superimposed on the detection target. It is a figure which shows the 1st example of the 1st marker and the 2nd marker displayed around the detection target. It is a figure which shows the 2nd example of the 1st marker and the 2nd marker displayed around the detection target. It is a figure which shows the 3rd example of the 1st marker and the 2nd marker displayed around the detection target. It is a figure which shows the 2nd example of the 1st video. It is a figure which shows the 2nd example of the 2nd video corresponding to the display area of the 1st video of FIG. It is a figure which shows the 3rd example of the 1st video. It is a figure which shows the 3rd example of the 2nd video corresponding to the display area of the 1st video of FIG. It is a figure which shows the 4th example of the 1st video. It is a figure which shows the 4th example of the 2nd video corresponding to the display area of the 1st video of FIG. It is a figure which shows the 5th example of the 1st video. It is a figure which shows the 5th example of the 2nd video corresponding to the display area of the 1st video of FIG. It is a figure which shows the other example of the 2nd video corresponding to the display area of the 1st video of FIG. It is a figure which shows the 6th example of the 1st video. It is a figure which shows the 6th example of the 2nd video corresponding to the display area of the 1st video of FIG. It is a figure which shows another example of the 2nd video. It is a figure which shows the 1st modification of the 6th example of the 2nd video. It is a figure which shows the 2nd modification of the 6th example of the 2nd video. It is a figure which shows the 3rd modification of the 6th example of the 2nd video. It is a figure which shows the 7th example of the 1st video. It is a figure which shows the 7th example of the 2nd video corresponding to the display area of the 1st video of FIG. It is a figure which shows the 8th example of the 1st video. It is a figure which shows the positional relationship of a vehicle and an object to be detected. It is a figure which shows the 8th example of the 2nd video corresponding to the display area of the 1st video of FIG. It is a figure which shows the modification of the 8th example of the 2nd video. It is a figure which shows the example which changed the display range of FIG. 30 to wide view display.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Display system)
The display system 10 according to the embodiment of the present invention will be described with reference to FIG.

As shown in FIG. 1, the display system 10 includes an image pickup device 20, an image processing device 30, and a display device 40. Each component of the image pickup apparatus 20 and the display system 10 can transmit and receive information via, for example, a network 51. The network 51 may include, for example, wireless, wired, CAN (Controller Area Network), or the like.

In other embodiments, some or all of the components of the display system 10 may be integrally configured as one device. For example, an image processing device 30 may be incorporated in the image pickup device 20 or the display device 40.

As shown in FIG. 2, the image pickup device 20, the image processing device 30, and the display device 40 may be provided in the moving body 50. The "moving body" in the present disclosure may include, for example, a vehicle, a ship, an aircraft, and the like. Vehicles may include, for example, automobiles, industrial vehicles, railroad vehicles, living vehicles, fixed-wing aircraft traveling on runways, and the like. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, trolley buses and the like. Industrial vehicles may include, for example, industrial vehicles for agriculture and construction. Industrial vehicles may include, for example, forklifts and golf carts. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, lawnmowers and the like. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, crane trucks, dump trucks, road rollers and the like. The vehicle may include a vehicle that runs manually. The classification of vehicles is not limited to the above examples. For example, automobiles may include industrial vehicles that can travel on the road. The same vehicle may be included in multiple categories. Vessels may include, for example, marine jets, boats, tankers and the like. Aircraft may include, for example, fixed-wing aircraft, rotorcraft, and the like.

The imaging device 20 can image an external region of the moving body 50. The position of the image pickup apparatus 20 is arbitrary inside and outside the moving body 50. For example, as shown in FIG. 2, the image pickup apparatus 20 is located behind the moving body 50 capable of imaging an external region behind the moving body 50. The position of the image processing device 30 is arbitrary within the moving body 50. The display device 40 is visible to the subject 60. The position of the display device 40 is arbitrary on the moving body 50. For example, as shown in FIG. 2, the display device 40 is located in the dashboard of the moving body 50.

(Imaging device)
The image pickup apparatus 20 will be described in detail. For example, as shown in FIG. 1, the image pickup device 20 includes an image pickup optical system 21, an image pickup element 22, a communication unit 23, and a control unit 24.

The imaging optical system 21 forms a subject image. For example, the imaging optics 21 may include an aperture and one or more lenses.

The image sensor 22 has a plurality of pixels arranged in two dimensions. The image pickup device 22 may include, for example, a CCD (Charge Coupled Device) image pickup device or a CMOS (Complementary Metal Oxide Semiconductor) image pickup device. The image pickup device 22 can capture an image of a subject imaged by the image pickup optical system 21 and generate an image pickup image.

The communication unit 23 may include a communication interface capable of communicating with an external device. The communication unit 23 may be able to send and receive information via the network 51. The external device may include, for example, an image processing device 30. The "communication interface" in the present disclosure may include, for example, a physical connector and a wireless communication device. The physical connector may include an electric connector corresponding to transmission by an electric signal, an optical connector corresponding to transmission by an optical signal, and an electromagnetic connector corresponding to transmission by an electromagnetic wave. The electrical connector includes a connector compliant with IEC60603, a connector compliant with the USB standard, a connector corresponding to the RCA terminal, a connector corresponding to the S terminal specified in EIAJ CP-1211A, and a D terminal specified in EIAJ RC-5237. It may include a corresponding connector, a connector conforming to the HDMI® standard, and a connector corresponding to a coaxial cable including a BNC (British Naval Connector or Baby-series N Connector, etc.). Optical connectors may include various connectors according to IEC 61754. The wireless communication device may include a wireless communication device conforming to each standard including Bluetooth (registered trademark) and 802.11. The wireless communication device includes at least one antenna.

The control unit 24 includes one or more processors. The "processor" in the present disclosure may include a dedicated processor specialized for a specific process and a general-purpose processor that executes a specific function by loading a specific program. Dedicated processors may include DSPs (Digital Signal Processors) and application specific integrated circuits (ASICs). The processor may include a programmable logic device (PLD). The PLD may include an FPGA (Field-Programmable Gate Array). The control unit 24 may be either a SoC (System-on-a-Chip) in which one or a plurality of processors cooperate, or a SiP (System In a Package).

The control unit 24 controls the operation of the entire image pickup apparatus 20. The control unit 24 may cause the image pickup device 22 to generate an image pickup image at an arbitrary frame rate. The frame rate may substantially match, for example, a frame rate that can be displayed by the display device 40. The control unit 24 may execute a predetermined image processing on the generated captured image. The image processing may include, for example, exposure adjustment processing, white balance processing, distortion correction processing, and the like. The control unit 24 outputs the captured image to the image processing device 30 via the communication unit 23. For example, the control unit 24 may sequentially output captured images at the above-mentioned frame rate. Hereinafter, each captured image output at the above-mentioned frame rate is also simply referred to as a frame. The plurality of captured images output from the imaging device 20 are also referred to as a first image. For example, when the frame rate is 60 fps (Flame per Seconds), 60 captured images per second are output as the first image.

(Image processing device)
The image processing device 30 will be described in detail. The image processing device 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

The communication unit 31 may include a communication interface capable of communicating with various external devices. The external devices include, for example, an image pickup device 20, a display device 40, an ECU (Electronic Control Unit or Engine Control unit) provided in the moving body 50, a speed sensor, an acceleration sensor, a rotation angle sensor, a steering steering angle sensor, and an engine rotation speed sensor. , Accelerator sensor, brake sensor, illuminance sensor, raindrop sensor, mileage sensor, millimeter wave radar, obstacle detection device using ultrasonic sonar, ETC (Electronic Toll Collection System) receiver, GPS (Global Positioning System) device , Navigation devices, servers on the Internet, mobile phones, etc. may be included.

The communication unit 31 may include a communication interface for pedestrian-to-vehicle communication, road-to-vehicle communication, and vehicle-to-vehicle communication. The communication unit 31 may include a receiver corresponding to DSRC (Dedicated Short-Range Communication) and VICS (registered trademark) (Vehicle Information and Communication System) optical beacons provided in Japan. The communication unit 31 may include a receiver corresponding to the road traffic information providing system of another country.

The communication unit 31 may be able to acquire various information from an external device. For example, the communication unit 31 may be able to acquire mobile body information and environmental information.

The moving body information may include arbitrary information about the moving body 50. The moving body information includes, for example, the speed, acceleration, turning gravity, tilt, direction, and turning status of the moving body 50, steering wheel steering angle, cooling water temperature, remaining fuel, remaining battery, battery voltage, and the like. Engine speed, gear position, reverse signal presence / absence, accelerator operation presence / absence, accelerator opening, brake operation presence / absence, brake step, parking brake operation presence / absence, front / rear wheel or four wheel rotation speed difference, tire pressure, damper Amount of expansion and contraction, space position of driver's eyes, number of occupants and seat position, seatbelt wearing information, door opening / closing, window opening / closing, vehicle interior temperature, air conditioning operation presence / absence, air conditioning set temperature, air conditioning air flow , Outside air circulation setting, wiper operation status, driving mode, connection information with external devices, current time, average fuel consumption, instantaneous fuel consumption, lighting status of various lamps, position information of moving body 50, and destination of moving body 50 It may include route information to and from. The various lamps may include, for example, headlamps, fog lamps, back lamps, position lamps, and turn signals.

The environmental information may include arbitrary information regarding the external environment of the moving body 50. Environmental information includes, for example, the brightness around the moving body 50, weather, atmospheric pressure, outside air temperature, map information, traffic information, road construction information, temporary change in the speed limit of the driving path, and an object detected by another vehicle. , And the lighting state of the traffic light and the like may be included.

The storage unit 32 may include a temporary storage device and a secondary storage device. The storage unit 32 may be configured by using, for example, a semiconductor memory, a magnetic memory, an optical memory, or the like. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk, a magnetic tape, and the like. The optical memory may include, for example, a CD (Compact Disc), a DVD (Digital Versatile Disc), a BD (Blu-ray Disc) (registered trademark), and the like. The storage unit 32 stores various information and programs necessary for the operation of the image processing device 30.

The control unit 33 includes one or more processors. The control unit 33 controls the operation of the entire image processing device 30.

The control unit 33 may acquire the moving body information and the environmental information from the external device via the communication unit 31. The control unit 33 may determine the predicted course of the moving body 50 based on, for example, the moving body information. Hereinafter, the predicted course of the moving body 50 is also referred to as a first predicted course.

The control unit 33 may acquire the first image from the image pickup apparatus 20 via the communication unit 31. The first video includes a detection area and a display area.

The control unit 33 may detect at least a part of the detection target in the detection area on the acquired first image. The detection region on the first image may be at least a part of the captured image which is each frame of the first image. Each frame of the first image can be called a captured image. The detection area on the first image may be larger than the display area. The detection area on the first image may include a display area. The control unit 33 can detect the detection target inside the display area. The control unit 33 can detect the detection target outside the display area and inside the detection area. The area inside the detection area and the display area can be referred to as a first area. The area inside the detection area and outside the display area can be referred to as a second area.

The detection target may include a plurality of types of objects. Types of objects may include, for example, people, other moving objects, runways, lanes, white lines, gutters, sidewalks, pedestrian crossings, road signs, traffic signs, guardrails, walls, traffic lights, and the like. The types of detection targets that can be detected by the control unit 33 are not limited to these. At least a part of the detection target includes a part of the detection target that is not hidden behind the other object, for example, when a part of the detection target on the first image is hidden behind another object. It's fine. For example, the control unit 33 may be able to detect the upper body of the pedestrian when the lower body of the pedestrian is hidden behind an obstacle on the first image. Any object detection algorithm may be adopted for detecting at least a part of the detection target. For example, the control unit 33 may detect at least a part of the detection target by an algorithm such as pattern matching or feature point extraction using the captured image which is each frame of the first image.

When at least a part of the detection target is detected on the first video, the control unit 33 may determine the predicted course of the detection target based on the first video. Hereinafter, the predicted course to be detected is also referred to as a second predicted course. Any algorithm may be adopted for determining the second predicted course. For example, the control unit 33 may determine the second predicted course based on the change in the orientation and position of the detection target on the captured image which is each frame of the first image.

When at least a part of the detection target is detected on the first image, the control unit 33 may estimate the relative positional relationship between the moving body 50 and the detection target based on the first image. The relative positional relationship may include, for example, the distance between the moving body 50 and the detection target, and whether or not the first predicted course of the moving body 50 and the second predicted course of the detection target overlap. Any algorithm may be adopted for estimating the distance between the moving body 50 and the detection target. For example, the control unit 33 may estimate the distance between the moving body 50 and the detection target by the motion stereo method using the captured image which is each frame of the first video signal. In another embodiment, the control unit 33 may acquire information indicating the relative positional relationship between the moving body 50 and the detection target from the external device via the communication unit 31.

When the distance between the moving body 50 and the detection target decreases, the control unit 33 may determine which of the moving body 50 and the detection target contributes more to the decrease in the distance. Any algorithm may be employed to determine the contribution of the moving body 50 and the detection target to the reduction in the distance. In one example, the control unit 33 may detect the moving speed of the moving body 50 based on the moving body information. The control unit 33 may detect the moving speed of the detection target based on, for example, a change in the position of the detection target on the captured image which is each frame of the first image. The control unit 33 may determine that the moving body 50 or the detection target, whichever has the higher moving speed, contributes more to the decrease in the distance. In another example, when the moving speed of the moving body 50 is less than the reference value, the control unit 33 may determine that the detection target contributes greatly to the decrease in the distance. When the moving speed of the moving body 50 is equal to or higher than the reference value, the control unit 33 may determine that the moving body 50 contributes significantly to the decrease in the distance. The reference value may be arbitrarily set, but may be set to, for example, substantially zero. Details of the operation of the moving body 50 and the image processing device 30 according to the contribution of the detection target to the decrease in the distance will be described later.

When at least a part of the detection target is detected on the first image, the control unit 33 determines whether or not there is a possibility that the moving body 50 and the detection object come into contact with each other based on the first image. You can. Any algorithm may be adopted to determine the possibility of contact between the moving body 50 and the detection target. For example, the control unit 33 has at least one of a condition that the distance between the moving body 50 and the detection target is less than a predetermined threshold value and a condition that the decrease rate of the distance is equal to or more than a predetermined threshold value. If it is satisfied, it may be determined that the moving body 50 and the detection target may come into contact with each other. The details of the operation of the image processing device 30 according to the presence or absence of the possibility will be described later.

The control unit 33 may display the second image corresponding to the display area on the first image acquired from the image pickup apparatus 20 on the display device 40. Specifically, the control unit 33 may output the second image to the display device 40 via the communication unit 31. For example, when the control unit 33 detects the reverse movement of the moving body 50 based on the moving body information, the second image may be displayed on the display device 40. For example, the control unit 33 can detect reverse movement based on the shift position of the transmission gear. For example, the control unit 33 can detect the reverse movement based on the reverse signal output from the moving body at the time of the reverse movement. The second image may be, for example, an image obtained by cutting out a display area on the captured image which is each frame of the first image. The display area on the first image may be at least a part area on the captured image which is each frame of the first image. The display area may be smaller than the detection area. The display area may be included in the detection area. The position, shape, and size of the display area can be arbitrarily determined. The control unit 33 can change the position, shape, and size of the display area. By changing the position, shape, and size of the display area, for example, the display area and the detection area may substantially match.

The control unit 33 may combine various markers with the second image and display it on the display device 40. Synthesis includes overwriting and mixing. The marker may include, for example, one or more images. The control unit 33 may dynamically change the display mode of at least a part of the markers superimposed on the second image. The display mode may include, for example, the position, size, shape, color, and shade of at least a part of the marker on the second image. When displaying the marker corresponding to the detection target detected on the first image, the control unit 33 may determine the display mode of the marker according to the type of the detection target. Details of the operation of the image processing device 30 for displaying various markers on the display device 40 will be described later.

(Display device)
The display device 40 will be described in detail. The display device 40 may include, for example, a liquid crystal display, an organic EL (Electroluminescence) display, and the like. The display device 40 may display a second image input from the image processing device 30 via, for example, the network 51. The display device 40 may function as a touch screen capable of accepting operations by the user. The display device 40 may include switches and keys capable of accepting operations by the user. The switch may include a physical switch and an electronic switch. The key may include a physical key and an electronic key. When the display device 40 receives the user's operation on its own device, the display device 40 may transmit the user input based on the operation to the image processing device 30.

The display device 40 can be arranged at various locations on the moving body 50. 3A to 3E are diagrams showing an example of arrangement of the display device 40 when the moving body 50 is a vehicle. FIG. 3A shows an in-dashboard type display device 40a stored in the dashboard of the vehicle. FIG. 3B shows an on-dashboard type display device 40b arranged on the dashboard. In FIG. 3B, the display device 40b is incorporated in the vehicle 50. The display device 40b may be detachably mounted on the dashboard. FIG. 3C shows a display device 40c built in the room mirror and capable of displaying an image as needed. FIG. 4D shows a display device 40d built in the instruments panel. In FIG. 4D, the display device 40d is arranged adjacent to instruments such as a speedometer and a tachometer. In one embodiment, the entire instrument panel may be used as a display device 40d such as an LCD, and a second image may be displayed in the display device 40d together with images such as a speedometer and a tachometer. FIG. 4E shows a display device 40e using a portable information terminal, for example, a tablet terminal. A mobile phone display can be used as the display device 40e.

The second image and various markers displayed on the display device 40 by the image processing device 30 will be specifically described with reference to FIGS. 4 to 18. In the present disclosure, the terms "vertical" and "horizontal" for an image or image correspond to the image or the two-dimensional direction in the image. In the present disclosure, the terms "height direction", "horizontal direction", and "depth direction" for an image or image correspond to the three-dimensional direction of the image or the space in which the image is projected.

(First example (reference example))
FIG. 4 shows a first example of the detection region 61 in the first image acquired by the image processing device 30 from the image pickup device 20. In the example shown in FIG. 4, the detection region 61 is longer in the horizontal direction than in the vertical direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The control unit 33 may detect each of the pedestrian 63 and the vehicle 64 reflected inside the display area 62 on the first image as detection targets.

The control unit 33 determines whether or not one or more conditions are satisfied based on the relative positional relationship between the detection target detected inside the display area 62 on the first image and the moving body 50. judge. The one or more conditions may include, for example, the first condition that the detection target is located in the first predicted course 65 of the moving body 50. The one or more conditions may include, for example, a second condition in which at least a part of the first predicted course 65 of the moving body 50 and at least a part of the second predicted course to be detected overlap. When it is determined that the one or more conditions are satisfied, the control unit 33 may superimpose a predetermined marker corresponding to the detection target on the second image and display it on the display device 40. The predetermined marker may include a first marker, a second marker, and a third marker.

In the first example, the control unit 33 may determine that the one or more conditions are satisfied for the pedestrian 63. In such a case, the control unit 33 may display a marker corresponding to the pedestrian 63. The control unit 33 may determine that the one or more conditions are not satisfied for the vehicle 64. In such a case, the control unit 33 does not display the marker corresponding to the vehicle 64.

FIG. 5 shows an example of the second image corresponding to the display area 62 of the first image shown in FIG. When the aspect ratio of the display area 62 of the first image and the aspect ratio of the screen of the display device 40 are different, the control unit 33 cuts out the display area 62 of the first image and then the aspect ratio of the screen of the display device 40. The second image deformed according to the above may be output to the display device 40. As shown in FIG. 5, a pedestrian 63 and a vehicle 64 are shown on the second image.

For example, the control unit 33 may superimpose the guide line 66 indicating at least a part of the first predicted course 65 of the moving body 50 shown in FIG. 4 on the second image and display it on the display device 40. The control unit 33 may dynamically change the guide line 66 in response to a change in the steering angle of the steering wheel, for example.

The first image has a wider range than the display area 62. The control unit 33 may change the range of the display area 62. The control unit 33 may superimpose the icon image 67 on the second image and display it on the display device 40. For example, the outline 67a of the icon image 67 shown in FIG. 5 indicates the maximum range when the range of the display area 62 is changed. The white rectangle 67b of the icon image 67 indicates the display area 62. The icon image 67 shown in FIG. 5 shows the relative position and size of the display area 62 with respect to the maximum range of the display area 62.

FIG. 6 shows an example of a marker superimposed on the pedestrian 63 on the second image, for example. Hereinafter, the marker is also referred to as a third marker 68. The contour line 69 of the third marker 68 may substantially coincide with the contour line of the pedestrian 63 detected on the second image. The area 70 inside the contour line 69 of the third marker 68 may be filled with, for example, a color or pattern corresponding to the type of detection target “person”. When the pedestrian 63 is detected on the first image, the control unit 33 may superimpose the third marker 68 on the pedestrian 63 on the second image and display it on the display device 40. According to such a configuration, the subject 60 can easily see the pedestrian 63 on the second image. The control unit 33 may hide the third marker 68 after a predetermined time has elapsed after displaying the third marker 68.

FIG. 7 shows an example of two types of markers superimposed on the periphery of the pedestrian 63 on the second image, for example. Hereinafter, each of the two types of markers will also be referred to as a first marker 71 and a second marker 72. For example, after hiding the third marker 68, the control unit 33 may superimpose the first marker 71 and the second marker 72 on the second image and display them on the display device 40.

The control unit 33 may move the position of the first marker 71 following the pedestrian 63 on the second image. Since the first marker 71 follows the pedestrian 63, the subject 60 can easily catch the pedestrian 63. The first marker 71 is displayed around the pedestrian 63 away from the pedestrian 63. The subject 60 can easily grasp the behavior of the pedestrian 63 when the first marker 71 is displayed on the display device 40. The control unit 33 may change the superimposition position of the second marker 72 relative to the superimposition position of the first marker 71 on the second image. The control unit 33 may relatively move the second marker with reference to the position of the first marker 71.

For example, when the distance between the moving body 50 and the pedestrian 63 is decreasing, the control unit 33 can determine that the moving body 50 contributes greatly to the decrease in the distance. In such a case, the control unit 33 may move the second marker 72 toward the first marker 71. First, the control unit 33 displays the second marker 72 at a position away from the first marker 71. Next, the control unit 33 moves the second marker 72 toward the first marker 71 until the distance from the first marker 71 reaches a predetermined distance. Next, the control unit 33 erases the second marker 72. Next, the control unit 33 displays the second marker 72 at a position away from the first marker 71 again, and repeats the above process. In this example, since the second marker 72 is moving toward the object called the first marker 71, the subject 60 can understand that the second marker 72 is approaching the first marker 71. ..

For example, when the distance between the moving body 50 and the pedestrian 63 is decreasing, the control unit 33 can determine that the pedestrian 63 contributes greatly to the decrease in the distance. In such a case, the control unit 33 may move the second marker 72 away from the first marker 71. First, the control unit 33 displays the second marker 72 at a position close to the first marker 71. Next, the control unit 33 moves the second marker 72 away from the first marker 71 until the distance from the first marker 71 reaches a predetermined distance. Next, the control unit 33 erases the second marker 72. Next, the control unit 33 displays the second marker 72 at a position close to the first marker 71 again, and repeats the above process. In this example, since the second marker 72 is moving from the object called the first marker 71, the subject 60 can understand that the second marker 72 is separated toward the first marker 71.

The control unit 33 changes the moving direction of the second marker 72 with respect to the first marker 71 according to the amount of contribution of the moving body 50 and the pedestrian 63 to the decrease in the distance between the moving body 50 and the pedestrian 63. .. The subject 60 can identify, for example, whether the moving body 50 is approaching the pedestrian 63 or the pedestrian 63 is approaching the moving body 50 according to the moving direction of the second marker 72.

As shown in FIG. 8, for example, the control unit 33 may repeatedly enlarge or reduce the second marker 72 centering on the first marker 71 on the second image. As shown in FIG. 9, for example, the control unit 33 may superimpose the first marker 71 and the second marker 72 having the same shape as the contour line 69 of the pedestrian 63 on the second image. The control unit 33 may repeatedly enlarge or reduce the second marker 72 around the first marker 71. The control unit 33 switches the enlargement or reduction of the second marker 72 according to the amount of contribution of the moving body 50 and the pedestrian 63 to the decrease in the distance between the moving body 50 and the pedestrian 63.

When the distance between the detection target displaying the first marker 71 and the second marker 72 and the moving body 50 is less than a predetermined threshold value, the control unit 33 superimposes a new marker on the second image. You can. Hereinafter, the new marker is also referred to as a fourth marker. The fourth marker may include any image. For example, the fourth marker may include an image indicating an exclamation mark "!". According to this configuration, for example, when the pedestrian 63 displaying the first marker 71 and the second marker 72 and the moving body 50 come closer to each other by a certain degree or more, the fourth marker is superimposed and displayed on the second image. .. The subject 60 can identify, for example, that the pedestrian 63 is located in the vicinity of the moving body 50 by the fourth marker. In another embodiment, when the distance between the detection target displaying the first marker 71 and the second marker 72 and the moving body 50 is less than a predetermined threshold value, the control unit 33 performs the first marker. The display mode of the 71 and the second marker 72 may be changed. The control unit 33 may change the colors of the first marker 71 and the second marker 72, for example. Even with such a configuration, the subject 60 can identify, for example, that the pedestrian 63 is located in the vicinity of the moving body 50 by the change in the color of the marker.

The control unit 33 can detect two detection targets arranged in front and behind in the depth direction. The control unit 33 may display the first marker 71 and the second marker 72 on each of the two detection targets located in the front and rear positions. The control unit 33 may attach different first markers 71 and second markers 72 to the two detection targets. For example, the control unit 33 has a first marker 71 and a first marker 71 that are less noticeable than the first marker 71 and the second marker 72 attached to the second detection target located on the front side with respect to the first detection target located on the back side. A second marker 72 may be attached. For example, the control unit 33 has the first marker 71 and the second marker attached to the first detection target located on the back side of the first marker 71 and the second marker 72 attached to the second detection target located on the front side. 72 can be changed such as darkening the color, increasing the transmittance, and thinning the line.

(2nd to 4th examples (reference example))
FIG. 10 shows a second example of the detection region 61 in the first image acquired by the image processing device 30 from the image pickup device 20. In the example shown in FIG. 10, the detection region 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The control unit 33 separates the pedestrian 63a reflected inside the display area 62 on the first image and the pedestrians 63b and 63c appearing outside the display area 62 and inside the detection area 61, respectively. It may be detected as a detection target. The processing of the control unit 33 relating to the pedestrian 63a is the same as the processing relating to the pedestrian 63 shown in FIG. 4, for example.

When the detection position on the first image in which the detection target is detected is outside the display area 62 and inside the detection area 61, the control unit 33 superimposes and displays the marker corresponding to the detection target on the second image. It may be displayed on the device 40. Hereinafter, the marker is also referred to as a fifth marker. The control unit 33 may display the fifth marker when it is determined that the moving body 50 and the detection target may come into contact with each other. When the detection position on the first image in which the detection target is detected is on the right side of the display area 62, the control unit 33 superimposes the fifth marker on the right end on the second image on the display device 40. It may be displayed. When the detection position on the first image in which the detection target is detected is on the left side of the display area 62, the control unit 33 superimposes the fifth marker on the left end on the second image on the display device 40. It may be displayed.

In the second example, the detection position where the pedestrian 63b is detected is on the right side of the display area 62. The control unit 33 can determine that the moving body 50 and the pedestrian 63b may come into contact with each other. In such a case, the control unit 33 may superimpose the fifth marker corresponding to the pedestrian 63b on the right end on the second image and display it on the display device 40. Details of the fifth marker corresponding to the pedestrian 63b will be described later. The detection position where the pedestrian 63c is detected is on the left side of the display area 62. The control unit 33 can determine that there is no possibility that the moving body 50 and the pedestrian 63c come into contact with each other. In such a case, the control unit 33 does not have to display the fifth marker corresponding to the pedestrian 63b.

FIG. 11 shows an example of the second image corresponding to the display area 62 of the first image shown in FIG. As shown in FIG. 11, a pedestrian 63a is shown on the second image. Pedestrians 63b and 63c are not shown on the second image.

As shown in FIG. 11, the control unit 33 may superimpose, for example, the obstacle image 74 on the second image and display it on the display device 40. The obstacle image 57 shows the detection result of the obstacle detection device using the ultrasonic sonar or the like provided in the moving body 50. The obstacle image 74 may include image 74a, image 74b, and image 74c. Image 74a is an image of the moving body 50 viewed from above. Image 74b is an image showing that an obstacle has been detected behind the moving body 50. Image 74c is an image showing that an obstacle has been detected behind the moving body 50. The obstacle detection result by the obstacle detection device and the detection result of the detection target by the control unit 33 do not necessarily have to match. For example, in the example shown in FIG. 11, the obstacle image 74 shows that obstacles are detected at the right rear and the left rear of the moving body 50, respectively. On the other hand, the control unit 33 may determine that the pedestrian 63c existing on the left rear side of the moving body 50 is unlikely to come into contact with the moving body 50. In such a case, the control unit 33 does not have to display the fifth marker corresponding to the pedestrian 63c.

An example of the fifth marker 73 corresponding to the pedestrian 63b is shown in FIG. The fifth marker 73 may include an icon image 73a and a band image 73b. The icon image 73a may be an image corresponding to the "person" which is the type of the detection target. The subject 60 who visually recognizes the icon image 73a can recognize that a person exists to the right of the second image. The band image 73b is, for example, a band-shaped image extending in the vertical direction on the second image. The band image 73b may be filled with, for example, a color or pattern corresponding to the type of detection target "person". The control unit 33 may move the band image 73b within the rightmost region 73c on the second image. The control unit 33 may change the moving speed and width of the band image 73b.

The fifth marker 73 will be described in detail. The control unit 33 may determine the width of the band image 73b according to the distance between the moving body 50 and the pedestrian 63b. For example, the control unit 33 may increase the width of the band image 73b as the distance approaches. The subject 60 who visually recognizes the band image 73b can recognize the distance between the moving body 50 and the pedestrian 63b based on the width of the band image 73b.

The control unit 33 may determine that, for example, the contribution of the moving body 50 is large to the decrease in the distance between the moving body 50 and the pedestrian 63b. In such a case, the control unit 33 repeatedly moves the band image 73b in the first direction within the right end region 73c on the second image. The first direction may be, for example, a direction from the outside to the inside in the left-right direction on the second image. The control unit 33 may determine that, for example, the pedestrian 63b contributes significantly to the decrease in the distance between the moving body 50 and the pedestrian 63b. In such a case, the control unit 33 repeatedly moves the band image 73b in the second direction within the right end region 73c on the second image. The second direction may be, for example, a direction from the inside to the outside in the left-right direction on the second image. The subject 60 who visually recognizes the band image 73b can recognize whether the moving body 50 is approaching the pedestrian 63b or the pedestrian 63b is approaching the moving body 50 based on the moving direction of the band image 73b. Is.

The control unit 33 may determine the moving speed of the band image 73b according to the decreasing speed of the distance between the moving body 50 and the pedestrian 63b. For example, the faster the reduction speed of the distance, the faster the moving speed of the band image 73b may be. The subject 60 who visually recognizes the band image 73b can recognize the decreasing speed of the distance between the moving body 50 and the pedestrian 63b based on the moving speed of the band image 73b.

When the control unit 33 detects, for example, a user input corresponding to a predetermined user operation in the state where the fifth marker 73 is displayed, the detection position of the pedestrian 63b on the first image is included in the display area 62. The display area 62 may be changed as described above. For example, as shown in FIG. 12, the control unit 33 may extend the display area 62 on the first image in the left-right direction and move the display area 62 to the right side in the detection area 61. With this configuration, for example, as shown in FIG. 13, a pedestrian 63b is projected on the second image.

The predetermined user operation described above may include any user operation. For example, the predetermined user operation described above may include a first user operation for changing the steering angle of the moving body 50. The fifth marker 73 may function as a GUI (Graphic User Interface) that accepts a second user operation. Hereinafter, the GUI is also referred to as an interface image. In such a case, the predetermined user operation described above may include a second user operation.

The control unit 33 may automatically change the display area 62 so that the detection position of the pedestrian 63b on the first image is included inside the display area 62. In this case, the control unit 33 can maintain the automatic change of the display area 62 until the pedestrian 63b is no longer detected in the detection area 61 of the first image.

The control unit 33 may change the icon image 67 according to the change of the display area 62, for example, as shown in FIG.

The control unit 33 may change the display area 62 on the first image in response to, for example, a pinch-in operation and a pinch-out operation on the display device 40. For example, as shown in FIG. 14, the control unit 33 may substantially match the display area 62 with the detection area 61. In such a case, for example, as shown in FIG. 15, all the detection targets in the detection area 61 are displayed on the display device 40.

(Fifth example (reference example))
FIG. 16 shows a fifth example of the detection region 61 in the first image acquired by the image processing device 30 from the image pickup device 20. In the example shown in FIG. 16, the detection region 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The control unit 33 may detect the vehicle 64a reflected in the first predicted course 65 of the moving body 50, the vehicle 64b and the pedestrian 63d reflected outside the first predicted course 65, respectively, as detection targets. ..

In the fifth example, a case where the external region of the moving body 50 is dark, for example, at night or in a tunnel, will be described. When the external region of the moving body 50 is dark, the characteristic values of the first image and the second image may decrease. The characteristic value may include any parameter related to the visibility of the image. For example, the characteristic value may include, for example, at least one of the brightness value and the contrast ratio of the image. When the characteristic value of the second image is lowered, the visibility of the second image may be lowered.

The control unit 33 may execute the specific image processing on the region corresponding to the detection target on the second image. The specific image processing may include a first process of superimposing a marker corresponding to the detection target on the region. Hereinafter, the marker is also referred to as a sixth marker. The sixth marker may include, for example, an image that substantially matches the contour shape of the detection target on the second image. According to such a configuration, the sixth marker is superimposed and displayed on the detection target on the second video. Therefore, the target person 60 can easily recognize the detection target on the second video even when the characteristic value of the second video is low. The specific image processing may include a second process of changing the characteristic value of the region corresponding to the detection target on the second image. For example, the control unit 33 may change the characteristic value of the region so as to improve the visibility of the region on the second image. According to such a configuration, the visibility of the detection target on the second image is improved. Therefore, the target person 60 can easily recognize the detection target on the second video even when the characteristic value of the second video is low.

The control unit 33 may execute the above-mentioned specific image processing when one or more conditions are satisfied. One or more conditions may include a condition that the detection target is located within the first predicted course 65 of the moving body 50. One or more conditions may include a condition that the first predicted course 65 of the moving body 50 and the second predicted course to be detected overlap. One or more conditions may include a condition that the distance between the moving body 50 and the detection target is less than a predetermined threshold. One or more conditions may include a condition that the characteristic value of at least a part of the second image is less than a predetermined threshold value.

In the fifth example, the control unit 33 may determine that one or more of the above-mentioned conditions are satisfied for each of the vehicles 64a, 64b, and the pedestrian 63d. In such a case, as shown in FIG. 17, for example, the control unit 33 superimposes and displays the three sixth markers 75a, 75b, and 75c corresponding to the vehicles 64a, 64b, and the pedestrian 63d, respectively, on the second image. It may be displayed on the device 40. The control unit 33 may display the sixth marker superimposed on the detection target in a bright place.

The control unit 33 may change the shape of the guide wire 66. The control unit 33 may change the shape of the guide line 66 in the region where the guide line 66 and the detection target overlap. FIG. 18 is one of the shape examples of the guide wire 66. In the guide line 66 of FIG. 18, the guide line is not displayed in the area where the guide line 66 and the sixth marker 75a overlap. The shape example of the guide wire 66 is not limited to erasing, and other design changes are allowed. Design changes include color changes, transmittance changes, type changes to broken lines, line thickness changes, blinking, and the like. The control unit 33 may change the shape of the guide line 66 when the sixth marker is not displayed. The control unit 33 may change the shape of the guide line 66 when the first marker 71 and the second marker 72 are displayed on the detection target.

As described above, according to the display system 10 according to the first embodiment, various markers according to the detection target detected on the first image are superimposed on the second image and displayed on the display device 40. .. The target person 60 who visually recognizes the marker can recognize the relative positional relationship between the moving body 50 and the detection target at a glance. Therefore, the convenience of the technique for displaying the image of the external region of the moving body 50 is improved.

Hereinafter, with reference to FIGS. 19 to 31, examples of various images in which the image processing device 30 synthesizes the guide wall image 80 and other images in the display area on the first image and displays them on the display device 40. This will be described in detail.

(6th example)
FIG. 19 shows a sixth example of the detection region 61 in the first image acquired by the image processing device 30 from the image pickup device 20. The image pickup device 20 may be arranged so as to image the rear of the moving body 50 which is a vehicle. In the example shown in FIG. 19, the detection region 61 is longer in the horizontal direction than in the vertical direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The detection region 61 is not limited to a shape that is longer in the left-right direction than in the up-down direction, and can have various shapes such as a square, a vertically long rectangle, and a circle. The display area 62 can have various positions, sizes, and shapes. For example, the display area 62 is not always located in the center of the detection area 61 and may be closer to the left or right. The display area 62 is not limited to a part of the detection area 61, and the entire detection area 61 may be used as the display area 62. The control unit 33 may change the display area 62 on the first image in response to a pinch-in operation, a pinch-out operation, or the like on the display device 40.

The control unit 33 synthesizes the guide wall image 80 showing the first predicted course 65 of the moving body 50 with the display area on the first image acquired from the image pickup apparatus 20 to generate the second image. The control unit 33 may output an image in which the guide wall image 80 is superimposed on the second image and display it on the display device 40. The guide wall image 80 can show a predicted course when the moving body 50 moves backward. The guide wall image 80 is a marker that gives a three-dimensional impression to the subject 60. The guide wall image 80 can be considered as an image obtained by projecting a virtual guide wall for guiding the predicted course, which is arranged in the real space on which the first image is projected, onto the first image when the image pickup apparatus 20 captures the image. The guide wall image 80 can be visually recognized as being composed of a plurality of virtual translucent wall surfaces extending from the road surface to a predetermined height in the height direction in the field of view of the subject 60. It can be said that the guide wall image 80 is a three-dimensional display of the guide line displayed on the display device 40 when the vehicle moves backward by a wall-shaped display. The wall-shaped display includes a display in other aspects such as a surface, a film, and a plate. The three-dimensional display of the guide wall image 80 makes it easy for the subject 60 to grasp the position of the parking space on the road surface, the distance to an obstacle, or the like.

The control unit 33 can recognize the detection target in the display area 62 of the first image. The control unit 33 may detect a detection target that is inside the display area 62 on the first image and is reflected in the first predicted course 65. When the detection target is detected, the control unit 33 can synthesize a virtual plane as the first recognition wall image 83 on the moving body 50 side of the recognized detection target. When the detection target moves, the first recognition wall image 83 moves together with the detection target.

FIG. 20 shows an example of the second image corresponding to the display area 62 of the first image shown in FIG. The guide wall image 80 is arranged away from the lower end of the second image on the upper side. The guide wall image 80 may include two side wall images 81a, 81b extending in the height direction and the depth direction. The side wall image 81a is located on the left side in the horizontal direction, and the side wall image 81b is located on the right side in the horizontal direction. The region between the two side wall images 81a and 81b shows a first predicted course 65 through which the moving body 50, which is a vehicle, passes. The spacing between the side wall images 81a and 81b may be set to indicate the vehicle width of the moving body 50. In FIG. 20, the side wall images 81a and 81b are displayed linearly. The control unit 33 may acquire the moving body information from the communication unit 31 and bend the side wall images 81a and 81b according to the predicted course. In this case, the moving body information includes the steering angle of the steering wheel.

The guide wall image 80 may include a plurality of distance wall images 82a, 82b, 82c that extend in the height and horizontal directions and indicate the distance in the depth direction from the moving body 50. The distance wall images 82a, 82b, 82c may connect between the side wall images 81a and 81b. FIG. 20 may show three distance wall images 82a, 82b and 82c. The number of distance wall images is not limited to three, but may be two or four or more. The distance between the distance wall images 82a and 82b in the depth direction may be narrower than the distance between the distance wall images 82b and 82c in the depth direction. The length of the distance wall images 82a, 82b, 82c in the height direction may be longer or shorter than the length of the side wall images 81a, 81b in the height direction.

A frame may be displayed on the outer periphery of each side wall image 81a, 81b and the distance wall image 82a, 82b, 82c of the guide wall image 80. The guide wall image 80 may include at least one of an auxiliary line 84a extending in the depth direction, an auxiliary line 84b extending in the horizontal direction, and an auxiliary line 84c extending in the height direction. The guide wall image 80 may include all of the auxiliary line 84a extending in the depth direction, the auxiliary line 84b extending in the horizontal direction, and the auxiliary line 84c extending in the height direction.
The auxiliary line 84a extending in the depth direction can be displayed on the wall surface of the side wall images 81a and 81b. The auxiliary line 84b extending in the horizontal direction can be displayed on the wall surface of the distance wall images 82a, 82b, 82c. The auxiliary line 84c extending in the height direction can be displayed on the wall surfaces of the side wall images 81a, 81b and the distance wall images 82a, 82b, 82c. Each of the auxiliary lines 84a, 84b, 84c can be displayed in any number of 1 or more. When there are a plurality of auxiliary lines of the auxiliary lines 84a, 84b, 84c, the intervals between the auxiliary lines 84a, 84b, 84c can be equal. The distance between the auxiliary lines 84a, 84b, 84c may be different depending on conditions such as the height from the road surface and the distance from the moving body 50. The frame displayed on the outer periphery of each side wall image 81a, 81b and the distance wall image 82a, 82b, 82c can be regarded as a part of the plurality of auxiliary lines 84a, 84b, 84c. In FIG. 20 and the following figures, even if a plurality of auxiliary lines 84a, 84b, and 84c are present, only one of each is appropriately coded.

The distance wall image 82a is a first distance wall image closer to the moving body 50 than the other distance wall images 82b and 82c when mapped in the real space on which the first image is projected. The distance wall image 82a can be different in color from the other distance wall images 82b, 82c. For example, the distance wall images 82b and 82c can be displayed in translucent white, and the distance wall images 82a can be displayed in translucent red. The side wall images 81a and 81b can be displayed in translucent white. The above color setting is an example, and the colors of the side wall images 81a and 81b may be different from those of the distance wall images 82b and 82c.

The transmittance of the side wall images 81a and 81b may be changed according to the position in the depth direction. The transmittance of the side wall images 81a and 81b can be set to be lower toward the front side and higher toward the back side in the depth direction. The side wall images 81a and 81b may have different saturations or brightnesses depending on their positions in the depth direction. For example, the saturation can be increased toward the front side in the depth direction. The saturation of an image can also be rephrased as the color depth of the image. Continuous changes in transmittance, saturation, lightness, chromaticity, etc. can be said to have a gradation.

The transmittance of the distance wall images 82a, 82b, 82c may be changed according to the height from the road surface. The transmittance of the distance wall images 82a, 82b, and 82c can be set to increase as the distance from the road surface increases in the height direction. The distance wall images 82a, 82b, 82c may have different saturations or brightnesses depending on the height from the road surface. For example, the saturation can be reduced as the distance from the road surface increases.

The auxiliary lines 84b and 84c on the distance wall image 82a, which is the first distance wall image, may have the same color as the auxiliary lines 84b and 84c of the other distance wall images 82b and 82c. The auxiliary lines 84a, 84b, 84c can be of any color. For example, when the color of the wall surface of the guide wall image 80 other than the distance wall image 82a is white, the color of the auxiliary lines 84a, 84b, 84c may be white having higher brightness or lower transmittance than these wall surfaces.

The control unit 33 may detect the vehicle 64 displayed in the display area 62 of the first image as a detection target. In the example of FIG. 20, the vehicle 64 is located in the first predicted course 65. The detection target detected by the control unit 33 is not limited to the vehicle, but may be a building, a fence, an obstacle on the road, a person, an animal, or the like. When the detection target is detected, the control unit 33 can synthesize the first recognition wall image 83 on the moving body 50 side of the recognized detection target. When the detection target is detected, the control unit 33 may change the length of the guide wall image 80 in the height direction. When the detection target is detected, the control unit 33 can shorten the length of the guide wall image 80 in the height direction. As a result, the control unit 33 can easily see the display even when the first recognition wall image 83 is displayed, and can direct the attention of the target person 60 to the detection target.

The first recognition wall image 83 of the detection target located in the first prediction course 65 extends in the height direction and the horizontal direction. The first recognition wall image 83 may be displayed as an opaque or translucent surface of any color. In FIG. 20, the first recognition wall image 83 is displayed as a translucent surface having a rectangular shape on the moving body 50 side of the vehicle 64. The horizontal width of the first recognition wall image 83 may correspond to the horizontal width of the vehicle 64. The horizontal width of the first recognition wall image 83 can be substantially matched with the horizontal width of the vehicle 64. The height of the first recognition wall image 83 can be set to a predetermined value. The transmittance of the first recognition wall image 83 may be changed according to the height direction from the road surface. The transmittance of the first recognition wall image 83 can be set to increase as the distance from the road surface increases in the height direction. Further, the first recognition wall image 83 may have different saturation or brightness depending on the distance in the height direction from the road surface. For example, the farther away from the road surface in the height direction, the lower the saturation can be. The first recognition wall image 83 may include auxiliary lines extending in the horizontal and height directions.

The control unit 33 can move the auxiliary line extending in the horizontal direction of the first recognition wall image 83 in the height direction within the frame of the first recognition wall image 83. The auxiliary line can be displayed so as to appear from the lower end of the frame of the first recognition image 83 and disappear at the upper end. The control unit 33 moves both the relative position and the relative distance change between the moving body 50 and the vehicle 64 by moving the auxiliary line extending in the horizontal direction of the first recognition wall image 83 in the height direction. , Or either one can be displayed. For example, the control unit 33 can move the auxiliary line extending in the horizontal direction faster in the height direction when the distance between the moving body 50 and the vehicle 64 is narrowing. The control unit 33 may move the auxiliary line faster in the height direction as the distance between the moving body 50 and the vehicle 64 is shorter. The control unit 33 may move the auxiliary line extending in the horizontal direction of the first recognition wall image 83 in the height direction regardless of the position of the detection target and the distance from the detection target.

The color of the first recognition wall image 83 can be different depending on the distance in the depth direction from the moving body 50 to the vehicle 64. For example, as shown in FIG. 20, when the distance between the moving body 50 and the vehicle 64 is larger than a predetermined distance, the control unit 33 may display the color of the first recognition wall image 83 in blue. it can. When the distance from the vehicle 64 is shortened due to the backward movement of the moving body 50, the control unit 33 may change the color of the first recognition wall image 83 from blue to yellow and from yellow to red.

When the distance between the moving body 50 and the vehicle 64 is more than a predetermined distance, the control unit 33 may not display the first recognition wall image 83. When the moving body 50 and the vehicle 64 are relatively close to each other and are within a predetermined distance, the control unit 33 displays, for example, the first recognition wall image 83 after highlighting the vehicle 64 in the second image. To do. The highlighting includes a thick line display of the outline of the vehicle 64, blinking of the outline of the vehicle 64, filling of an image of the vehicle 64, and the like. The control unit 33 may change the display mode according to the change in the relative position with respect to the vehicle 64. For example, when the vehicle 64 moves relatively away from the moving body 50, the first recognition wall image 83 can not be displayed.

FIG. 21 is a diagram showing an example of a second image in a state where the distance between the moving body 50 and the vehicle 64 is narrowed due to the backward movement of the moving body 50. The first recognition wall image 83 is displayed in red. As a result, it is possible to warn the target person 60 that there is a risk of colliding with the vehicle 64 if the subject moves backward any further.

When the detection object and the guide wall image 80 overlap, the control unit 33 can change the display of the guide wall image 80. For example, as shown in FIG. 21, the control unit 33 can prevent the guide wall image 80 that overlaps with the vehicle 64 in the second image from being displayed at least partially. More specifically, the control unit 33 displays on the display device 40 a guide wall image 80 that overlaps with the vehicle 64 and is located farther from the moving body 50 in the depth direction than the vehicle 64 or the first recognition wall image 83. I won't let you. By doing so, the display device 40 can display the image on the subject 60 in an easy-to-see and intuitively easy-to-understand manner. The method of changing the display of the guide wall image 80 is not limited to hiding the overlapping portion. For example, the control unit 33 can make the transmittance of the guide wall image 80 of the portion overlapping the detection target object higher.

(Modification example)
The guide wall image 80 is not limited to the example shown in FIGS. 20 and 21. 22 to 24 show a modified example of the guide wall image 80.

(First modification)
In the example illustrated in FIG. 22, the horizontally extending auxiliary line 84b of the distance wall images 82a, 82b, 82c can display only one of the lowest positions in the height direction. The distance wall image 82a displaying only the auxiliary line 84b at the lowest position can facilitate the recognition of the ground surface height. The distance wall image 82a displaying only the auxiliary line 84b at the lowest position can easily grasp the distance from the recognition wall image. Auxiliary lines 84c extending in the height direction may be displayed at both ends of the distance wall images 82a, 82b, 82c in the horizontal direction. Auxiliary lines 84b located at the lower end and auxiliary lines 84c extending in the height direction located at both ends in the horizontal direction can be displayed on the outer periphery of the distance wall images 82a, 82b, 82c. The distance wall images 82a, 82b, 82c may be displayed by an auxiliary line 84b extending in the horizontal direction at the lower end of the outer circumference and an auxiliary line 84c extending in the height direction at both the left and right ends.

The distance wall images 82a, 82b, 82c can be displayed including a surface having a translucent color. In the distance wall images 82a, 82b, 82c, the transmittance can be increased or the color depth can be reduced as the distance from the road surface in the height direction increases. The auxiliary lines 84b and 84c located on the outer periphery of the distance wall image 82a closest to the moving body 50 may be displayed in a color different from that of the other auxiliary lines. The different colors can be, for example, red. In addition, other auxiliary lines can be white, but are not limited to this. The first recognition wall image 83 displayed in front of the vehicle 64 is displayed as a translucent surface of any color, so that the transmittance increases as the distance from the road surface increases in the height direction, or the color depth increases. May be displayed so that is lighter. The first recognition wall image 83 does not visually display the boundary of the upper end, but may be displayed so as to gradually disappear in the height direction.

In the example shown in FIG. 22, only two auxiliary lines 84a extending in the depth direction of the side wall images 81a and 81b of the guide wall image 80 are displayed at the upper and lower ends of the side wall images 81a and 81b, respectively. The lower auxiliary line 84a ₁ may be highlighted more than the upper auxiliary line 84a ₂ . Highlighting includes displaying using thicker lines, lines with lower transmittance, lines with higher brightness, and the like. The upper auxiliary line 84a ₂ and the lower auxiliary line 84a ₁ may be lines having the same display mode. A line of the same aspect is, for example, a line having the same thickness, transmittance, brightness, and the like.

The auxiliary line 84a extending in the depth direction may be displayed at the farthest position in the depth direction by using a display method in which the auxiliary line becomes thinner and disappears. The side wall images 81a and 81b may be displayed so that the end portion at the farthest position in the depth direction can be visually recognized.

On the side wall images 81a, 81b, the auxiliary line 84c extending in the height direction may be displayed at least at the frontmost end and at a position intersecting the distance wall images 82a, 82b, 82c. The auxiliary lines 84c extending in these height directions may be highlighted more than the auxiliary lines 84c extending in other height directions. The auxiliary lines 84c extending in the height direction may have different thicknesses. The auxiliary line 84c can be displayed thicker than the auxiliary line 84c displayed at the position where the auxiliary line 84c intersects with the distance wall images 82a and 82c.

(Second modification)
The guide wall image 80 shown in FIG. 23 is different from the guide wall image 80 of FIG. 22 in the side wall images 81a and 81b. In FIG. 23, two or more auxiliary lines 84a extending in the depth direction of the side wall images 81a and 81b of the guide wall image 80 may be displayed, including the top and bottom ones. The auxiliary line 84a extending in the depth direction may highlight the auxiliary line located at the lowest position in the height direction more than other auxiliary lines. The plurality of auxiliary lines 84a may all be displayed as lines of the same aspect. The guide wall image 80 of FIG. 23 may adopt a change in transmittance, a change in color, a color scheme, and the like that can be adopted in the guide wall image 80 of FIG.

(Third modification example)
The guide wall image 80 shown in FIG. 24 is different from the guide wall image 80 of FIG. 22 in the side wall images 81a and 81b. In FIG. 24, the auxiliary lines 84a extending in the depth direction of the side wall images 81a and 81b of the guide wall image 80 display only the auxiliary lines 84a located at the lower end. The auxiliary line 84a extending in the depth direction located at the lower end of the guide wall image 80 may correspond to the line on the road surface when the first image is projected in the real space together with the auxiliary line 84b extending in the horizontal direction. The guide wall image 80 of FIG. 24 may adopt a change in transmittance, a change in color, a color scheme, and the like that can be adopted in the guide wall image 80 of FIG. The auxiliary lines 84c extending in the height direction may have different thicknesses. The auxiliary lines 84c are arranged from the front side to the back side in the depth direction, and the auxiliary lines 84c may be thinner as the lines are located on the back side.

In the guide wall image 80 shown in each of FIGS. 20 to 24, the display device 40 does not have to display a part or all of the auxiliary lines 84a, 84b, 84c.

(7th example)
FIG. 25 shows a seventh example of the detection region 61 in the first image acquired by the image processing device 30 from the image pickup device 20. In the example shown in FIG. 25, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The display area 62 is not limited to a part of the detection area 61, and the entire detection area 61 may be used as the display area 62.

In the seventh example, the moving body 50 is a vehicle and can be assumed to be moving backward toward the parking space of the parking lot. It can be assumed that the image pickup device 20 is located behind the moving body 50 and images the rear side. From the detection area 61, the control unit 33 may detect the vehicle 64b and the pedestrian 63, which are outside the first predicted course 65 and are at least partially reflected in the display area 62, as detection targets. In the example shown in FIG. 25, the control unit 33 determines that the distance in the depth direction of the vehicle 64a reflected in the first predicted course 65 of the moving body 50 is longer than the predetermined distance, and does not detect it.

FIG. 26 shows an example of the second image corresponding to the display area 62 of the first image shown in FIG. 25. The control unit 33 recognizes the vehicle 64b and the pedestrian 63 to be detected in the display area 62 of the first image, and the first recognition which is a virtual plane on the moving body 50 side of the vehicle 64b and the pedestrian 63, respectively. The wall images 83a and 83b are displayed. The first recognition wall images 83a and 83b can be displayed as translucent surfaces parallel to the side wall images 81a and 81b extending in the height direction and the depth direction. The control unit 33 acquires the change in the distance between the vehicle 64b and the pedestrian 63 and the moving body 50, and may display the side wall images 81a and 81b only when the distance changes in the direction in which the distance becomes shorter. .. As described above, any algorithm can be adopted for estimating the distance between the moving body 50 and the detection target.

The control unit 33 estimates the distance in the horizontal direction from the guide wall image 80 of the vehicle 64a and the pedestrian 63 to be detected, and determines the display colors of the first recognition wall images 83a and 83b according to the estimated distance. You can. Here, the distance between the detection target and the guide wall image 80 is the detection target and the virtual guide wall when the virtual guide wall displayed by the guide wall image 80 exists in the real space on which the first image is projected. Is the distance to. The control unit 33 can estimate the first distance between the position where the guide wall image 80 is mapped in the real space on which the first image is projected and the position of the detection target. The control unit 33 sets a threshold value for the first distance, and for each of the first recognition wall images 83a and 83b, different display colors such as blue, yellow, and red are displayed in order from the longest to the shortest in the horizontal direction. It may be used and displayed on the display device 40. The first recognition wall images 83a and 83b may have auxiliary lines extending in the depth direction. The control unit 33 sets the auxiliary line extending in the depth direction of the first recognition wall images 83a and 83b to the frame of the first recognition wall image 83 in the same manner as the auxiliary line extending in the horizontal direction of the first recognition wall image 83 of FIG. Can be moved in the height direction within.

The control unit 33 estimates the first position as the position in the depth direction of the vehicle 64b and the pedestrian 63 to be detected. The control unit 33 may change the display of the side wall images 81a and 81b facing the vehicle 64b and the pedestrian 63, respectively, at least partially according to the first position. The mode of changing the display includes changing the color, brightness, saturation, etc. of the wall surface, changing the color, thickness, etc. of the border and / or auxiliary line, or a combination thereof. For example, in FIG. 26, the display colors of a part 81a ₁ of the side wall image 81a facing the vehicle 64b and a part 81b ₁ of the side wall image 81b facing the pedestrian 63 are changed. A part of the side wall images 81a and 81b may be a region divided by an auxiliary line 84c extending in the height direction on the side wall images 81a and 81b. Display color of a portion 81b ₁ of the part 81a ₁ and the sidewall image 81b of the sidewall image 81a is good as a different color in response to the first distance. When the detection target is an obstacle larger than the length represented by the side wall images 81a and 81b, the control unit 33 may change the overall display color of the side wall images 81a or 81b. Examples of large obstacles include large vehicles such as trailers and buildings.

When the detection target moves, the control unit 33 may estimate the second predicted course of the detection target and estimate the first position as the position where the second predicted course and the side wall images 81a and 81b intersect. The control unit 33 can estimate the first position as the position where the second predicted course in the real space on which the first image is projected intersects with the mapping of the side wall images 81a and 81b to the real space. The control unit 33 may change the display of the side wall images 81a and 81b at least partially according to the first position. Also in this case, the display colors of the partial 81a ₁ and 81b ₁ of the side wall images 81a and 81b whose display is changed may be different colors depending on the first distance. The control unit 33 may change the overall display color of the side wall image 81a or 81b when the detection target is a very large obstacle.

As another display mode, the control unit 33 may change the display of the side wall images 81a and 81b located on the back side in the depth direction from the first recognition wall images 83a and 83b. As yet another display mode, the control unit 33 may change the display of the side wall images 81a and 81b located on the back side in the depth direction from the position where the second predicted course and the side wall images 81a and 81b intersect. ..

The display of the first recognition wall images 83a and 83b may be changed according to the height from the road surface. Similar to the first recognition wall image 83 of the sixth example, the display of the first recognition wall images 83a and 83b is set to different values such as transmittance, saturation, or brightness according to the height from the road surface. sell.

As described above, the display system 10 can spatially and three-dimensionally express the distance and / or position of the detection target that may be an obstacle to the backward movement by using the guide wall image 80 and the first recognition wall images 83a and 83b. In addition, the position where there is a risk of collision can be displayed three-dimensionally from the predicted course of the detection target during movement. As described above, the display system 10 represents the guide wall image 80 and the first recognition wall images 83a and 83b as walls (or planes) having a height direction, thereby spatially expressing the detection target and the warning content in a spatial manner. Is possible. This facilitates the display and understanding of the warning content for the target person 60.

(8th example)
FIG. 27 shows an eighth example of the detection region 61 in the first image acquired by the image processing device 30 from the image pickup device 20. The image pickup device 20 may be arranged so as to image the rear of the moving body 50 which is a vehicle. In the example shown in FIG. 27, the detection region 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The control unit 33 detects the pedestrian 63 and the vehicle 64b, which are outside the display area 62 on the first image and are reflected in the second area inside the detection area 61, as detection targets, respectively. Good. In the eighth example, the control unit 33 determines that the distance in the depth direction of the vehicle 64a displayed in the display area 62 is farther than a predetermined distance, and does not set the vehicle 64a as a detection target.

FIG. 28 is a diagram schematically showing the arrangement of the moving body 50 and the pedestrian 63 and the vehicle 64b to be detected as seen from above along the height direction in the real space on which the first image is projected. The image pickup apparatus 20 can image a subject in a range included in the regions F ₁ and F ₂ in the real space. Area F ₁ and area F ₂ correspond to the detection area 61 of the first image. The area F ₁ is a first area corresponding to the display area 62 of the first image. The area F ₂ corresponds to a second area inside the detection area 61 of the first image and outside the display area 62.

In FIG. 28, for the sake of explanation, the virtual guide wall 90, which is a map of the guide wall image 80 displayed in the second image in the real space, is displayed by a broken line. The virtual side walls 91a and 91b correspond to the side wall images 81a and 81b displayed in the second image. The virtual distance walls 92a, 92b, 92c correspond to the distance wall images 82a, 82b, 82c displayed in the second image.

Further, in FIG. 28, for purposes of explanation, the virtual side wall 91a of the vehicle 50, the virtual line _95l 1 _{~95l 5} and _95r 1 _{~95r 5} showing the horizontal distance of the detection target with respect to 91b, indicated by dashed lines ing. This horizontal distance is also called the second distance. The virtual lines 95l _{1 to} 95l ₅ are located on the left side of the moving body 50. The virtual lines 95r _{1 to} 95r ₅ are located on the right side of the moving body 50. The intervals between the virtual lines 95l _{1 to} 95l ₅ and the virtual lines 95r _{1 to} 95r ₅ can be set arbitrarily. The intervals between the virtual lines 95l _{1 to} 95l ₅ and the virtual lines 95r _{1 to} 95r ₅ may be equal to each other. The control unit 33 determines which virtual line 95l _{1 to} 95l ₅ and 95r _{1 to} 95r ₅ is closest to the detection target from the first video, or which 2 of the virtual lines 95l _{1 to} 95l ₅ and 95r _{1 to} 95r ₅ It is possible to determine whether or not it is located in the area sandwiched between the two.

A second image corresponding to the first image of FIG. 27 is illustrated in FIG. In the eighth example, the guide wall image 80 may adopt the same display method as that described in the sixth and seventh examples. When the detection target is located in the second region of the first video, the control unit 33 synthesizes the second recognition wall images 89a and 89b at the end of the second video on the side where the detection target exists. The second recognition wall images 89a and 89b may include wall portions 96a and 96b extending in the height direction and the depth direction, and floor portions 97a and 97b extending in the depth direction and the horizontal direction, respectively. In FIG. 29, the wall portions 96a and 96b extend from the lower end to the upper end of the left and right end portions of the second image. The wall portions 96a and 96b may extend vertically along only a part of the left and right ends of the second image, not the entire portion. Similarly, the floor portions 97a and 97b may extend in the left-right direction along the whole or a part of the lower end portion of the second image. The floor portions 97a and 97b may be displayed between the guide wall image 80 and the lower end of the second image.

The control unit 33 may change the display of the second recognition wall images 89a and 89b according to the second distance of the detection target. Changes in display include changes in color, transmittance, saturation, lightness, size, etc., and changes in dynamic display methods such as blinking and movement. For example, in one of a plurality of embodiments, the change in display may be represented by a change in color. The display color may be red when the risk is high, yellow when the risk is medium, blue when the risk is low, and the like. The pedestrian 63 shown in FIG. 28 is located on the third virtual line 95r ₃ on the right side in the horizontal direction from the virtual side wall 91b of the moving body 50. In this case, the control unit 33 may set the risk level to medium and the color of the second recognition wall image 89b on the right side to be yellow. Further, the vehicle 64b shown in FIG. 28 is located away from the virtual side wall 91a of the moving body 50 with respect to the fifth virtual line 95l ₅ on the left side in the horizontal direction. In this case, the control unit 33 may determine that the risk is low and set the color of the second recognition wall image 89a on the left side to blue. In one of a plurality of embodiments, the change in display may include lateral movement of the vertical auxiliary lines 96a ₁ , 96b ₁ displayed on the walls 96a, 96b. The control unit 33 may change the speed of movement of the wall portions 96a and 96b according to the length of the second distance.

The second recognition wall images 89a and 89b can be translucent. The display characteristics of the wall portions 96a and 96b of the second recognition wall images 89a and 89b may be changed in the vertical direction. For example, the transmittance of the wall portions 96a and 96b of the second recognition wall images 89a and 89b may be lowered toward the lower part. Further, the second recognition wall images 89a and 89b may be more saturated toward the lower part. The floor portions 97a and 97b of the second recognition wall image 89a may change the display characteristics in the left-right direction. For example, the floor portions 97a and 97b of the second recognition wall image 89a may have a lower transmittance toward the left and right ends and a higher transmittance toward the center of the second image. Further, the floor portions 97a and 97b of the second recognition wall image 89a may be desaturated from the left and right ends toward the center of the second image. The saturation of an image can also be rephrased as the color depth of the image.

The control unit 33 may determine the type of the detection target and display the icon images 99a and 99b corresponding to the type on the floor portions 97a and 97b of the second recognition wall images 89a and 89b. The type of the detection target can be determined by a known method based on the image of the detection target detected from the first video. For example, the control unit 33 can determine the object type by matching the contour shape of the detection target with the pattern that serves as a model. For example, for pedestrian 63 and vehicle 64b, icons indicating people and vehicles can be selected, respectively. The control unit 33 can estimate a second distance, which is a horizontal distance between the virtual side walls 91a and 91b obtained by mapping the side wall images 81a and 81b in the real space and the detection target in the real space. The second distance, each a line virtual line _{95l 1 ~95l 5, 95r 1 ~95r} 5 of the real space mapped onto the detection area 61 of the first image, determines the relationship between the position of the image of each detected object sell. The control unit 33 can change the position where the icon images 99a and 99b are displayed according to the second distance.

In the display example of the second image as shown in FIG. 29, respectively to a virtual line _{95l 1 ~95l 5, 95r 1 ~95r} 5 corresponds in a real space, a first distance recognition line _{98l 1 ~98l 5, 98r 1 ~98r} 5 Is provided. The first distance identification lines 98l _{1 to} 98l ₅ are located on the left side of the side wall image 81a and are numbered in the order of 98l ₁ , 98l ₂ , 98l ₃ , 98l ₄ , 98l ₅ from the center to the left side. 98r _{1 to} 98r ₅ are located on the right side of the side wall image 81b and are numbered in the order of 98r ₁ , 98r ₂ , 98r ₃ , 98r ₄ , 98r ₅ from the center to the right side. The first distance identification lines 98l _{1 to} 98l ₅ and 98r _{1 to} 98r ₅ indicate the positions where the icon images 99a and 99b are displayed, and may not be displayed in the second video.

In FIG. 28, since the pedestrian 63 is located on the third virtual line 95r ₃ on the right side in the horizontal direction from the virtual side wall 91b of the moving body 50, the control unit 33 is the pedestrian 63 in the second image of FIG. The icon image 99b may be placed on the _third first distance identification line 98r ₃ on the right side. As shown in FIG. 29, the icon image 99b is three-dimensionally displayed as a surface having lengths in the depth direction and the height direction on the first distance identification line 98r ₃ extending in the depth direction on the second image. You can. When viewed stereoscopically, the icon image 99b may be displayed as a plane parallel to the side wall images 81a and 81b and the second recognition wall images 89a and 89b. Similar to the second recognition wall image 89b, the icon image 99b may have different display modes such as color, saturation, and brightness depending on the second distance. For example, the icon image 99b can be displayed in yellow.

In FIG. 28, the vehicle 64b is located farther from the virtual side wall 91a than the _fifth virtual line 95l ₅ on the left side in the horizontal direction from the virtual side wall 91a of the moving body 50. In this case, the control unit 33 may arrange the icon image 99a of the vehicle 64b in the second image of FIG. 29 on the first distance identification line 98l ₅ on the left side. Alternatively, the control unit 33 may be arranged so that when the second image is viewed three-dimensionally, it can be seen on the same surface as the wall portion 96a of the second recognition wall image 89a. Similar to the icon image 99b, the icon image 99a can have a different display mode depending on the second distance. For example, the icon image 99a can be displayed in blue.

When the relative position of the detection target as seen from the moving body 50 changes due to the movement of both or one of the moving body 50 and the detection target, the second recognition wall images 89a and 89b and the icon images 99a and 99b The position and display mode change. Further, when the moving body 50 enters the display area 62, the detection target may be displayed in the manner described in the seventh example. An example of a change in the display when the vehicle 64b moves from the area F ₂ to the area F ₁ in the real space shown in FIG. 28 will be described.

First, in the real space of FIG. 28, it is assumed that the vehicle 64b is located on the left side of the virtual line 95l ₅ . At this time, as described above, the second recognition wall image 89a is displayed in the second image of FIG. 29, and the icon image 99a is displayed on the leftmost side of the floor portion 98a of the second recognition wall image 89a. .. At this time, the second recognition wall image 89a and the icon image 99a can be displayed in blue.

In the real space of FIG. 28, as the vehicle 64b approaches the virtual guide wall 90, the second recognition wall image 89a displayed in the second image of FIG. 29 changes the display mode. For example, the color of the second recognition wall image 89a changes sequentially from blue to yellow to red. At the same time, the icon image 99a moves the display position from the first distance identification line 98l ₅ onto the first distance identification line 98l ₁ . The color of the icon image 99a may change sequentially from blue to yellow and red. As a result, the subject 60 can recognize that the vehicle 64b is approaching outside, which is not displayed in the second image. Further, by using the three-dimensional display method, the subject 60 can spatially grasp the approach of the vehicle 64b and can reduce the risk of information being overlooked.

The icon image 99a can move toward the center of the screen when the vehicle 64b is approaching the moving body 50. The subject 60 can recognize the moving direction of the vehicle 64b through the movement of the icon image 99a.

Further, when the vehicle 64b enters the area F ₂ to F ₁ in the real space of FIG. 28, the vehicle 64b enters the display target area of the first image, and is therefore included in the second image and displayed on the display device 40. .. At the same time, the display of the second recognition wall image 89a disappears, and as shown in FIG. 26, the first recognition wall image 83a is displayed on the guide wall image 80 side of the vehicle 64b. When the second recognition wall image 89a disappears, the control unit 33 may blink the second recognition wall image 89a a plurality of times in order to call the attention of the target person 60. The control unit 33 displays the first recognition wall image 83a on the moving body 50 side of the vehicle 64b. Further, the control unit 33 estimates the second predicted course of the vehicle 64b, and estimates the first position as the position where the second predicted course and the side wall images 81a and 81b intersect. The control unit 33 may change the display of a part 81a ₁ of the side wall image 81a including the first position. For example, the control unit 33 can sequentially change the color of a part 81a ₁ of the side wall image 81a to blue, yellow, and red as the vehicle 64b approaches. As a result, the subject 60 can easily grasp that the vehicle 64b is approaching and that the second predicted course of the vehicle 64b overlaps with the first predicted course 65 of the moving body 50. Become.

As described above, according to one of the plurality of embodiments of the present disclosure, the position of the detection target is displayed by three-dimensional display of the guide wall image, the first recognition wall image, the second recognition wall image, the icon image, and the like. It becomes easy to grasp the space of. Further, according to one of the plurality of embodiments of the present disclosure, it is possible to provide information such as cautions and warnings in an easy-to-understand manner by changing the display mode of each wall surface. Further, by using the three-dimensional display, it is possible to reduce the risk of information being overlooked.

(Modification example)
FIG. 30 shows another example of the second video corresponding to the first video of FIG. 27. In FIG. 30, view display areas 101 and 102 showing the selection status between the “normal view display” and the “wide view display” are added to the lower left. "Normal view display" and "wide view display" will be described later. In FIG. 30, the display modes of the guide wall image 80, the second recognition wall images 89a and 89b, and the icon images 99a and 99b may be partially different from those in FIG. 29. For example, in FIG. 30, a guide wall image 80 having a display mode similar to that in FIG. 22 is displayed.

The wall portions 96a, 96b of the second recognition wall images 89a, 89b can have _one or more auxiliary lines 96a ₁ , 96b ₁ extending in the height direction. In FIG. 30, the wall portions 96a and 96b have a plurality of auxiliary lines 96a ₁ and 96b ₁ , but only one of them is coded. The wall portions 96a and 96b of the second recognition wall images 89a and 89b represent the distance to the detection target in color, and the transmittance may be different from the front side to the back side in the depth direction. The wall portions 96a and 96b may gradually increase in transmittance in the height direction or gradually decrease in density and disappear at a predetermined position in the height direction.

The control unit 33 determines that the object exists within a predetermined distance range due to the transmittance or color density of the wall portions 96a and 96b and the change of the auxiliary lines 96a ₁ and 96a ₂ , and the detection target. The change in distance or distance to the object to be detected can be displayed. The change in the transmittance or the color depth of the wall portions 96a and 96b can also be said to be a change in gradation. For example, the control unit 33 can display that the detection target exists within a predetermined distance by moving the auxiliary lines 96a ₁ and 96a ₂ of the wall portions 96a and 96b. The control unit 33 darkens the color on the front side of the wall portions 96a and 96b to the darkest, and can display that the detection target exists within a predetermined distance by changing the gradation in the depth direction. Changes in the auxiliary lines 96a ₁ , 96b ₁ and transmittance of the wall portions 96a, 96b are not essential. The control unit 33 can prevent the wall portions 96a and 96b from displaying the auxiliary lines 96a ₁ and 96b ₁ . The control unit 33 may not have a change in transmittance (gradation) in the wall portions 96a and 96b.

In addition to the first distance identification lines 98l and 98r, one or more second distance identification lines 100l and 100r are provided on the floor portions 97a and 97b of the second recognition wall images 89a and 89b, respectively. In FIG. 30, there are a plurality of second distance identification lines 100l and 100r, but only one of each is indicated by a reference numeral. The first distance identification lines 98l and 98r and the second distance identification lines 100l and 100r do not have to be displayed on the display. The first distance identification lines 98l and 98r correspond to the horizontal distance from the moving body 50 to the detection target, whereas the second distance identification lines 100l and 100r can correspond to the distance in the depth direction. .. Each of the plurality of second distance identification lines 100l and 100r may correspond to the distance in the depth direction when the auxiliary lines 84c extending in the height direction on the side wall images 81a and 81b are mapped in the real space.

The subject 60 can grasp the positions in the horizontal and depth directions to the detection target according to the positions of the icon images 99a and 99b on the floors 97a and 97b in the horizontal and depth directions. As the icon images 99a and 99b, three-dimensional images corresponding to the type of detection target are used. The floor portions 97a and 97b may be arranged on the front side of the side wall images 81a and 81b from the foremost end portion and inside the line extending the side wall images 81a and 81b. The positional relationship between the second recognition wall images 89a and 89b and the guide wall image 80 is not limited to this, and can be changed as appropriate.

In FIG. 30, the display device 40 shows view display areas 101 and 102 showing the selection status of the “normal view display” and the “wide view display” in the lower right portion. .. The angle of view of the displayed image is different between the "normal view display" and the "wide view display". In one of the plurality of embodiments, the "normal view display" corresponds to the angle of view of the display area 62 of the second image shown in FIG. In one of the plurality of embodiments, the "wide view display" corresponds to the angle of view of the detection area 61 of the first image shown in FIG. The "wide view display" does not have to be the image of the entire detection area 61 of the first image, and an image having an angle of view including the "second image" may be cut out from the detection area 61. FIG. 30 and the second video shown before this are both normal view displays. Whether the image being displayed is "normal view display" or "wide view display" is indicated by changing the display of the view display areas 101 and 102 indicating the selection status. For example, changing the display of the area indicating the selection status includes changing the colors of the view display areas 101 and 102, changing the darkness, changing the color of characters, and the like.

The control unit 33 sets the "normal view display" and the "wide view display" according to the result of processing the image of the first video or by receiving the input from the target person 60 via the display device 40 or the like. You can switch. For example, the control unit 33 may switch the displayed image to the "wide view display" when it detects that the detection target is approaching in the "normal view display" state. When the control unit 33 detects that there is no detection target within a predetermined range in the "wide view display" state, the control unit 33 may switch the displayed image to the "normal view display". When the display device 40 includes a touch panel type screen, the control unit 33 receives a signal from the display device 40 indicating that the target person 60 touches and selects a portion of the view display area 101 or 102 on the screen. You may switch the display.

FIG. 31 is a diagram showing an example of an image in which the “normal view display” shown in FIG. 30 is switched to the “wide view display”. The angle of view of the image of the "wide view display" of FIG. 31 is wider in the horizontal direction than the angle of view of the image of the "normal view display" of FIG. In FIG. 31, the pedestrian 63 and the vehicle 64b located outside the display range of the “normal view display” displayed in FIG. 30 are displayed. Further, in the image displayed in FIG. 31, the angle of view in the horizontal direction is wider than that in the “normal view display” of FIG. 30, so that the distance between the left and right side wall images 81a and 81b of the guide wall image 80 is relative. It is narrowed down to.

In FIG. 31, the second recognition wall images 89a and 89b have disappeared because the pedestrian 63 and the vehicle 64b have entered the display range. The control unit 33 recognizes the vehicle 64b and the pedestrian 63 to be detected in the detection area 61 of the first image, and the first recognition which is a virtual plane on the moving body 50 side of the vehicle 64b and the pedestrian 63, respectively. Wall images 83a and 83b can be displayed. The control unit 33 estimates the positions of the vehicle 64b and the pedestrian 63 to be detected in the depth direction. The control unit 33 may change the display of the side wall images 81a and 81b facing the vehicle 64b and the pedestrian 63, respectively, based on the estimated position. In FIG. 31, the display of a part 81b ₁ of the side wall image 81b and a part 81a ₁ of the side wall image 81a is changed corresponding to each of the pedestrian 63 and the vehicle 64b. The modification of the display includes various aspects. For example, changing the display includes changing the color, changing the thickness of the auxiliary line surrounding the periphery, changing the line type, starting and stopping blinking, and changing the blinking cycle. The first recognition wall images 83a, 83b and a part of the side wall images 81a, 81b 81a ₁ , 81b ₁ can adopt the same display method and display mode as the example of FIG.

As shown in FIGS. 30 and 31, according to one of the plurality of embodiments, the images captured by the imaging device 20 are displayed on the display device 40 in a "normal view display" and a "wide view display" having different angles of view. It is possible to switch between and display. In the "normal view display", the subject 60 can easily determine the existence and the position of the detection target located outside the range of the displayed image by the second recognition wall images 89a and 89b and the icon images 99a and 99b. Can be grasped. When the display device 40 automatically or manually switches from the "normal view display" to the "wide view display", the subject 60 is the detection target detected outside the range of the "normal view display" image. Can be confirmed on the video. In the "wide view display", the guide wall image 80 and the first recognition wall image 83b can be displayed to three-dimensionally grasp the positional relationship with the detection target. As described above, the display system 10 in one of the plurality of embodiments can easily make the target person grasp the surrounding situation and detect the danger.

Although the present invention has been described based on the drawings and embodiments, it should be noted that those skilled in the art can easily make various modifications and modifications based on the present disclosure. Therefore, it should be noted that these modifications and modifications are included in the scope of the present invention. For example, the functions included in each means, each step, etc. can be rearranged so as not to be logically inconsistent, and a plurality of means, steps, etc. can be combined or divided into one. ..

For example, each component and function of the display system 10 according to the above-described embodiment may be rearrangeable. For example, a part or all of the configuration and functions of the image processing device 30 may be included in at least one of the image pickup device 20 and the display device 40.

Some of the components of the display system 10 according to the above-described embodiment may be located outside the moving body 50. For example, the image processing device 30 and the like may be realized as a communication device such as a mobile phone or an external server, and may be connected to other components of the display system 10 by wire or wirelessly.

10 Display system 20 Imaging device 21 Imaging optical system 22 Imaging element 23 Communication unit 24 Control unit 30 Image processing device 31 Communication unit 32 Storage unit 33 Control unit 40 Display device 50 Mobile 51 Network 60 Target person 61 Detection area 62 Display area 63 , 63a, 63b, 63c, 63d Pedestrians 64, 64a, 64b Vehicles 65 First predicted course 66 Guide line 67 Icon image 68 Third marker 69 Contour line 70 Area 71 First marker 72 Second marker 73 Fifth marker 73a Icon Image 73b Band image 73c Right end region 74 Obstacle image 75a, 75b, 75c 6th marker 80 Guide wall image 81a, 81b Side wall image 82a, 82b, 82c Distance wall image 83, 83a, 83b First recognition wall image 84a, 84b, 84c Auxiliary line 89a, 89b Second recognition wall image 90 Virtual guide wall 91a, 91b Virtual side wall 92a, 92b, 92c Virtual distance wall 95l _{1 to} 95l ₅ , 95r _{1 to} 95r ₅ Virtual line 96a, 96b Wall 97a, 97b Floor Part 98l _{1 to} 98l ₅ , 98r _{1 to} 98r ₅ First distance identification line 99a, 99b Icon image 100l, 100r Second distance identification line 101, 102 View display area

Claims (18)

An image sensor that captures the rear of the vehicle and generates the first image,
A control unit that synthesizes a guide image showing the predicted course of the vehicle is provided on the road surface portion included in the display area on the first image .
A first surface representing a surface extending a predetermined distance from the road surface in the height direction between the detection target in the first image and the guide image, and the transmittance increases as the distance from the road surface in the height direction increases. Combining recognition wall images ,
Imaging device. The imaging device according to claim 1.
The guide image includes two side images that extend in the depth direction and are displayed horizontally at predetermined intervals from each other.
The first recognition wall image has the surface facing the portion of the side image facing the detection target.
Imaging device. The imaging device according to claim 2 .
The guide image includes a distance image extending from one of the two side images toward the other.
In the first recognition wall image, the surface faces the portion of the distance image facing the detection target.
Imaging device. The imaging device according to any one of claims 1 to 3.
The first recognition wall image shows different colors depending on the distance from the vehicle to the detection target.
Imaging device. The imaging device according to claim 2.
The first recognition wall image shows different colors depending on the distance from the side image to the detection target.
Imaging device. The imaging device according to any one of claims 1 to 5 .
The first recognition wall image displays the detection target whose distance to the vehicle is within a predetermined range, and does not display the detection target whose distance to the vehicle is outside the predetermined range. .. The imaging device according to claim 3 .
The plurality of distance images include a first distance image displayed at a position closer to the vehicle than the other distance images.
The first distance image is different in color from the other distance images.
Imaging device. The imaging device according to any one of claims 1 to 7 .
The control unit changes the display of the portion of the guide image facing the detection target .
Imaging device. The imaging device according to any one of claims 1 to 8.
The control unit estimates the predicted course of the detection target , and
The display of the guide image corresponding to the position where the predicted course of the detection target and the guide image intersect is changed.
Imaging device. The imaging device according to claim 8 or 9 .
The control unit changes the color of the guide image when the distance between the vehicle and the detection target is within a predetermined range.
Imaging device. The imaging device according to any one of claims 1 to 10 .
The control unit does not synthesize the first recognition wall image with the first image when the vehicle moves relatively away from the detection target.
Imaging device. The imaging device according to any one of claims 1 to 11 .
The guide image extends in the height direction and the depth direction, and extends horizontally to two side images separated by a distance corresponding to the vehicle width of the vehicle, and extends in the height direction and the horizontal direction to extend the depth of the vehicle. Includes multiple distance images showing directional distances,
Imaging device. The imaging device according to claim 12 .
When the detection target is detected, the guide image shortens the length in the height direction.
Imaging device. The imaging device according to any one of claims 1 to 13 .
It has a detection means for detecting a detection target existing in an area not included in the display area, which is rear right or left rear of the vehicle.
The control unit detects that the shift gear of the vehicle is in a predetermined state, causes the display device to display the display area of the first image, and causes the display device to display the display area.
The guide image includes two side images that extend in the depth direction and are displayed horizontally at predetermined intervals from each other.
When the speed of the detection target is faster than that of the vehicle, an image showing the approach of the detection target to the vehicle is displayed outside the side image in the display area.
Imaging device. The imaging device according to claim 14.
The image showing the approach to the front of the vehicle is displayed on the side where the detection target exists in the display area.
Imaging device. The communication unit that acquires the first image of the rear of the vehicle,
A control unit that synthesizes a guide image showing the predicted course of the vehicle is provided on the road surface portion included in the display area on the first image.
A first surface representing a surface extending a predetermined distance from the road surface in the height direction between the detection target in the first image and the guide image, and the transmittance increases as the distance from the road surface in the height direction increases. Combining recognition wall images,
Image processing device. An image sensor that captures the rear of the vehicle and generates the first image,
A control unit that synthesizes a guide image showing the predicted course of the vehicle on the road surface portion included in the display area on the first image.
A display device for displaying an image of the display area in which the guide image is combined is provided.
A first surface representing a surface extending a predetermined distance from the road surface in the height direction between the detection target in the first image and the guide image, and the transmittance increases as the distance from the road surface in the height direction increases. Combining recognition wall images,
Display system. A vehicle equipped with a display system
The display system is
An image sensor that images the rear of the vehicle and generates a first image,
A control unit that synthesizes a guide image showing the predicted course of the vehicle on the road surface portion included in the display area on the first image.
A display device for displaying an image of the display area in which the guide image is combined is provided.
A first surface representing a surface extending a predetermined distance from the road surface in the height direction between the detection target in the first image and the guide image, and the transmittance increases as the distance from the road surface in the height direction increases. Combining recognition wall images,
vehicle.