JP2022052416A - Image processing device - Google Patents

Abstract

To provide an image processing device capable of appropriately warning an occupant of a mobile body.SOLUTION: An image processing device 1 comprises: an input section 11 for acquiring an image obtained by imaging a peripheral area of a mobile body; and a display determination section 13 for selecting a detection area to be visually displayed for an occupant of the mobile body from areas where an obstacle included in the peripheral area is detected, in accordance with a travel state of the mobile body. In the image processing device 1, the detection area includes: a first detection area being an area in front of the mobile body within the peripheral area; and a second detection area being an area behind the mobile body. The first detection area may be visually displayed for the occupant of the mobile body by overlapping with the scene of the peripheral area or the image of the peripheral area, and the second detection area may be displayed under the first detection area.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to an image processing apparatus.

In recent years, an image pickup device may be mounted on a moving body such as a vehicle. The vehicle is, for example, an automobile, and the image pickup device can take an image of a peripheral area that is difficult for the driver to see directly, and the image can be displayed on the display device for the driver. In addition, based on the captured image, people existing around the vehicle, obstacles such as other vehicles, lanes on the road surface, etc. are recognized, warning to the driver for avoiding vehicle collision, automatic braking control, and It may be used for driving support such as accelerator control for auto cruise control.

Here, in the driving support that highlights the images of obstacles such as people and other vehicles existing around the vehicle in the video and shows them to the driver, if all the obstacles are highlighted by highlighting, it really is. There was a problem that attention was reduced because attention was given to obstacles other than obstacles that should be noted. On the other hand, for example, Patent Document 1 discloses an image display control device that suppresses the display of the out-of-track area from the display of the expected travel area.

Japanese Unexamined Patent Publication No. 2009-40108

However, Patent Document 1 considers obstacles for a vehicle traveling straight ahead, and cannot be applied to the movement of a vehicle other than traveling straight ahead.

An object of the present disclosure made in view of the above-mentioned problems of the prior art is to provide an image processing apparatus capable of appropriately alerting a moving occupant.

The image processing apparatus according to the embodiment of the present disclosure is
An input unit that acquires an image of the peripheral area of a moving object, and
It is provided with a display determination unit that selects a detection area to be visually displayed to the occupant of the moving body from the area where an obstacle included in the peripheral area is detected according to the traveling condition of the moving body.

According to the present disclosure, it is possible to provide an image processing device capable of appropriately alerting a moving occupant.

FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating a vehicle equipped with an image pickup device. FIG. 3 is a diagram showing one display example of the detection area. FIG. 4 is a diagram showing one display example of the detection area. FIG. 5 is a diagram showing one display example of the detection area. FIG. 6 is a flowchart of an image processing method executed by the image processing apparatus. FIG. 7 is a diagram showing one display example of the detection area. FIG. 8 is a diagram showing one display example of the detection area. FIG. 9 is a flowchart illustrating an operating environment setting method.

FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus 1 according to an embodiment of the present disclosure. The image processing device 1 according to the present embodiment is mounted on the moving body 100 (see FIG. 2), and an image that calls attention for safe driving is superimposed on an image obtained by capturing an image of the peripheral area of the moving body 100. Display.

The image processing device 1 includes an input unit 11, a recognition unit 12, a display determination unit 13, and a superimposition processing unit 14. Details of each functional block of the image processing device 1 will be described later. Further, FIG. 1 is an example, and the image processing apparatus 1 may not include all of the functional blocks shown in FIG. 1, and may further include functional blocks not shown in FIG.

The image processing device 1 transmits and receives necessary information to and from the image pickup device 101, the ECU 102, and the external device 110. In the present embodiment, the image processing device 1 acquires an image obtained by capturing an image of the peripheral region of the moving body 100 from the image pickup device 101 (so-called vehicle-mounted camera) attached to the moving body 100. FIG. 2 is a diagram illustrating a mobile body 100 equipped with an image pickup device 101. The image pickup apparatus 101 is one or more, and is installed at various places of the moving body 100. In the present embodiment, the image pickup apparatus 101a may image the traveling direction (hereinafter referred to as “forward”) of the moving body 100 and may be arranged at or near the front bumper. The image pickup apparatus 101b may take an image in a direction opposite to the traveling direction of the moving body 100 (hereinafter referred to as “rearward”) and may be arranged at or near the rear bumper. The image pickup apparatus 101c may image the direction of the side surface on the left side with respect to the traveling direction of the moving body 100 (hereinafter referred to as the left side), and may be arranged at or near the door mirror on the left side of the moving body 100. The image pickup apparatus 101d may image the direction of the side surface on the right side with respect to the traveling direction of the moving body 100 (hereinafter referred to as the right side), and may be arranged at or near the door mirror on the right side of the moving body 100. The fan-shaped region shown in FIG. 2 exemplifies a region that can be imaged by the image pickup device 101a, the image pickup device 101b, the image pickup device 101c, and the image pickup device 101d, respectively. The image pickup apparatus 101 can take an image of the peripheral region of the moving body 100 in motion. Here, the number of the image pickup apparatus 101 may be four as in the present embodiment, but may be one or two, or may be another number.

Here, in the present disclosure, the mobile body 100 includes a vehicle, a ship, and an aircraft. Vehicles include, but are not limited to, automobiles and industrial vehicles, and may include railroad vehicles, living vehicles, and fixed-wing aircraft traveling on runways. Automobiles include, but are not limited to, passenger cars, trucks, buses, motorcycles, trolley buses and the like, and may include other vehicles traveling on the road. Industrial vehicles include industrial vehicles for agriculture and construction. Industrial vehicles include, but are not limited to, forklifts and golf carts. Industrial vehicles for agriculture include, but are not limited to, tractors, tillers, transplanters, binders, combines, and lawnmowers. Industrial vehicles for construction include, but are not limited to, bulldozers, scrapers, excavators, mobile cranes, dump trucks, and road rollers. Vehicles include those that run manually. The classification of vehicles is not limited to the above. For example, an automobile may include an industrial vehicle capable of traveling on a road, and the same vehicle may be included in a plurality of categories. Vessels in the present disclosure include marine jets, boats and tankers. Aircraft in the present disclosure include fixed-wing aircraft and rotary-wing aircraft. In the present embodiment, the moving body 100 will be described as a vehicle.

Further, the image processing device 1 acquires information on the traveling status of the moving body 100 from the ECU (Electronic Control Unit) 102 of the moving body 100. The information regarding the traveling state of the moving body 100 acquired from the image processing device 1 from the ECU 102 includes, for example, the speed of the moving body 100 and the like. The image processing device 1 and the ECU 102 may be connected by, for example, CAN (Control Area Area Network).

Further, the image processing device 1 outputs information of an image that calls attention for safe driving and displays it on the external device 110. The image output from the image processing device 1 includes an image in a detection area described later. In the present embodiment, the external device 110 includes a head-up display (HUD: Head-Up Display). The external device 110 projects an image on the windshield (windshield) of the moving body 100 so that the occupant of the moving body 100 can see it, and the windshield on which the image is projected corresponds to the display unit 111.

Further, in the present embodiment, the external device 110 includes a navigation device. The route information from the navigation device to the destination is projected as an image on the windshield by the head-up display. That is, in the present embodiment, the navigation device and the head-up display operate in conjunction with each other. As another example, the external device 110 may be only a head-up display or a navigation device, and may further include devices other than these. For example, the external device 110 is a navigation device, and an image calling attention for safe driving may be displayed on a display unit 111 such as a liquid crystal display (LCD: Liquid Crystal Display) or an organic EL (Electro-luminescence) display. ..

The image processing device 1 acquires information on the traveling status of the moving body 100 from the external device 110. The information regarding the traveling state of the moving body 100 acquired by the image processing device 1 from the external device 110 includes, for example, the distance to the place where the moving body 100 turns right or left, the map information, the current position of the moving body 100, and the like. Here, the image processing device 1 may configure an image processing system mounted on the moving body 100 together with the image pickup device 101, the ECU 102, and the external device 110.

(Configuration of image processing device)
The input unit 11 acquires an image of the peripheral region of the moving body 100 from the image pickup device 101. In this embodiment, the number of image pickup devices 101 is four. Correspondingly, the image pickup device 101 may be composed of four parts for acquiring a captured image from each image pickup device 101. The input unit 11 has a connector according to the transmission method of the input image signal. For example, the input unit 11 includes a physical connector. The physical connector includes an electric connector corresponding to transmission by an electric signal, an optical connector corresponding to transmission by an optical signal, and an electromagnetic connector corresponding to transmission by an electromagnetic wave. The electrical connector includes a connector compliant with IEC60603, a connector compliant with the USB standard, a connector corresponding to the RCA terminal, a connector corresponding to the S terminal specified in EIAJ CP-1211A, and a D terminal specified in EIAJ RC-5237. Includes connectors that support HDMI, HDMI (Registered Trademark) standards, and connectors that support coaxial cables, including BNC. Optical connectors include various connectors according to IEC 61754. The input unit 11 can include a wireless communication device. The wireless communication device may include a wireless communication device conforming to each standard including Bluetooth (registered trademark) and 802.11.

The recognition unit 12 executes object recognition for the image acquired by the input unit 11. Object recognition is a process of identifying an object contained in an image and recognizing what the object is. In the present disclosure, an object in object recognition includes a person, another vehicle, and the like. For object recognition, a known method may be used, for example, a pre-learned deep learning model may be used.

Further, the recognition unit 12 determines whether or not the recognized object includes an obstacle. In the present disclosure, the obstacle refers to an object that hinders the progress of the moving body 100 and an object that may hinder the progress of the moving body 100. Objects that can be obstacles include, for example, motorcycles, automobiles and people. When the obstacle is recognized, the recognition unit 12 calculates the coordinates in the image of the obstacle, the actual positional relationship with the moving body 100, the size, and the like. The recognition unit 12 outputs the calculated coordinates of the obstacle to the display determination unit 13 as obstacle information. Here, the processing of the recognition unit 12 may be executed by the display determination unit 13. That is, the recognition unit 12 may be configured to be included in the display determination unit 13.

The display determination unit 13 visually displays the detection area, which is a part of the peripheral area of the mobile body 100 imaged by the image pickup apparatus 101, to the occupant of the mobile body 100 via the superimposition processing unit 14. Here, the detection area is a peripheral area imaged by the image pickup apparatus 101, which is notified to the driver of the mobile body 100 by changing the display mode of the detection area when an obstacle is recognized. Is. If an obstacle exists in an area other than the detection area, the obstacle is displayed in the image of the surrounding area. Although it is possible to notify the existence and type of obstacles in an area other than the detection area, it is not necessary to notify the driver from the viewpoint of preventing the driver's attention from being lowered. The display determination unit 13 acquires information on the traveling status of the mobile body 100 from the ECU 102 and the external device 110. The display determination unit 13 may select a detection area from the peripheral area according to the traveling condition of the moving body 100. Further, as described above, the display determination unit 13 acquires obstacle information from the recognition unit 12. When an obstacle is detected, the display determination unit 13 changes the display mode of at least a part of the detection area including the detected obstacle. In the present embodiment, the display determination unit 13 displays the detection area when the moving body 100 approaches a place where the moving body 100 turns right or left. In this way, the display determination unit 13 determines the display or non-display of the detection area and the display mode of the detection area. The details of the detection area and the details of the control regarding the detection area executed by the display determination unit 13 will be described later.

The superimposition processing unit 14 causes the detection area to be superimposed and displayed on the display image of the external device 110 according to the determination of the display determination unit 13. The superimposition processing unit 14 adjusts the display position, size, and the like of the detection area so that the detection area is appropriately displayed on the windshield, for example, when superimposing the image on the display image of the head-up display. As another example, the superimposition processing unit 14 detects so that the detection area is appropriately displayed together with the map displayed on the display and the icon indicating the position of the moving body 100 when the superimposition processing unit 14 is superposed on the display of the navigation device. Adjust the display position, size, etc. of the area. As yet another example, when the superimposition processing unit 14 superimposes and displays on the display of the navigation device, the detection area is appropriately displayed in the image obtained by capturing the traveling direction of the moving body 100 displayed on the display. Adjust the display position, size, etc. of the detection area.

In this embodiment, the number of image pickup devices 101 is four. Correspondingly, the superimposition processing unit 14 may be composed of four parts so that the superimposition display processing can be independently executed for the image pickup region of each image pickup apparatus 101. Here, the processing of the superimposition processing unit 14 may be executed by the display determination unit 13. That is, the superimposition processing unit 14 may be configured to be included in the display determination unit 13.

Here, the image processing device 1 may be configured to include one or more processors. The "processor" in the present disclosure may include a dedicated processor specialized for a specific process, and a general-purpose processor that performs a specific function by reading a specific program. Dedicated processors may include DSPs (Digital Signal Processors) and application specific integrated circuits (ASICs). The processor may include a programmable logic device (PLD). The PLD may include an FPGA (Field-Programmable Gate Array). The image processing device 1 may be either a System (System-on-a-Chip) in which one or a plurality of processors cooperate, or a SiP (System In a Package). The processor may include programs for various processes and one or more memories for storing information in operation. The memory includes a volatile memory and a non-volatile memory.

Each functional block of the image processing apparatus 1 may be a hardware module or a software module. The operation executed by each functional block can be rephrased as that executed by the image processing device 1. The operation executed by the image processing device 1 can be rephrased as being executed by at least one processor constituting the image processing device 1. The functions of each functional block may be distributed and executed by a plurality of processors. Also, a single processor may perform the functions of multiple functional blocks.

(Detection area)
As described above, the detection region is a part of the peripheral region of the mobile body 100 imaged by the image pickup apparatus 101. The detection area is an area to be paid special attention to for safe driving, and is displayed to the occupants of the mobile body 100 including the driver.

FIG. 3 is a diagram showing one display example of the detection area. In FIG. 3, FIG. 4, FIG. 5, FIG. 7 and FIG. 8 described later, an image of the detection area (detection area image 31), a right / left turn arrow, etc. An example shows a state in which a guide image is superimposed and displayed. The display of these images means that by projecting the image onto the windshield of the moving body 100 and displaying it, the driver recognizes it as a virtual image at an appropriate position. That is, these figures show an example of displaying the detection area from the driver's point of view. FIG. 3 shows a left turn arrow 32, which is a display image of the head-up display, and a detection area image 31 superimposed and displayed by the image processing device 1. The left turn arrow 32 is for the head-up display to acquire image information from the navigation device and display it. Further, in the detection area image 31, the head-up display acquires image information from the image processing device 1 and displays the image information in an superimposed manner. In the example of FIG. 3, the detection area image 31 includes a front detection area image 31a and a rear detection area image 31b. That is, in the example of FIG. 3, as the detection area, not only the front but also the rear, which should be noted when turning left or right of the moving body 100, is selected. In other words, the detection region in the example of FIG. 3 includes a first detection region which is a region in front of the mobile body 100 and a second detection region which is a region behind the mobile body 100. The detection region image 31a and the detection region image 31b are shown in a fan shape corresponding to the imageable region of the image pickup apparatus 101a and the image pickup apparatus 101b, respectively (see FIG. 2). The detection area image 31b corresponding to the second detection area is displayed at the lower part of the detection area image 31a corresponding to the first detection area. Further, in the example of FIG. 3, the detection area image 31a has different colors for near and far so that the driver can easily grasp the distance, but may be further subdivided, and the whole is displayed in one color. It's okay.

FIG. 3 shows a display example of the detection area in a situation where the moving body 100 is approaching an intersection where the moving body 100 turns left. FIG. 3 includes a left view, a center view, and a right view, and the display changes in this order as the moving body 100 approaches the intersection. First, as shown in the left figure of FIG. 3, when the moving body 100 approaches an intersection where a left turn is made, a left turn arrow 32 is indicated by the navigation device. When the left turn arrow 32 is indicated by the navigation device, the fan-shaped detection area image 31a and the detection area image 31b are displayed, respectively. Then, the position of the detection area changes according to the traveling condition from the point where the detection area is displayed to the point where the right turn or the left turn of the moving body 100 ends, and the display image corresponding to the detection area also changes. As shown in the center view and the right figure of FIG. 3, when the moving body 100 approaches the intersection and turns left, the display mode of the detection area changes. Specifically, as the detection region moves toward the bending direction of the moving body 100, the fan-shaped regions of the detection area image 31a and the detection area image 31b move toward the left side, which is the bending direction of the moving body 100. It will decrease. Here, when the moving body 100 makes a right turn, the fan-shaped areas of the detection area image 31a and the detection area image 31b decrease toward the right side. Such a change in the shape of the detection area image 31 corresponds to a change in the area that the driver should pay particular attention to when the moving body 100 turns left (or right). For example, when the moving body 100 is making a left turn (or a right turn), the driver should pay particular attention to whether there is an obstacle at the end of the turn and behind the turn direction side with respect to the oncoming lane. Therefore, the shape change of the detection area image 31 as shown in FIG. 3 makes it possible to appropriately indicate the detection area in which the driver should pay attention to obstacles according to the situation of the left turn (or right turn) operation of the moving body 100. To.

When the recognition unit 12 recognizes an obstacle and the display determination unit 13 determines that the obstacle is included in the detection area, the image processing device 1 further alerts the driver to the detection area image 31. Can be displayed.

FIG. 4 shows a display example of a detection area in a situation where the moving body 100 approaches an intersection where a left turn is made and an obstacle is detected in the detection area behind. FIG. 4 includes a left view, a center view, and a right view, and the display changes in this order as the moving body 100 approaches the intersection. The detection area image 31a, the detection area image 31b, and the left turn arrow 32 are the same as those in FIG. When an obstacle is detected in the detection area, the driver can be notified of the presence of the obstacle by changing the display color of the detection area. In the example of FIG. 4, there is an obstacle in the rear detection area from the time when the moving body 100 approaches the intersection until the left turn, and the display color of the detection area image 31b corresponding to the rear detection area is the default. It has changed from the color of (hereinafter referred to as the standard color). In order to draw the driver's attention, the display color when an obstacle is detected is set to a color that stands out from the reference color. For example, when the reference color is blue, the display color when an obstacle is detected may be yellow. Here, in the example of FIG. 4, since no obstacle is detected in the front detection area, the display color of the detection area image 31a remains the reference color.

FIG. 5 shows a display example of a detection area in a situation where the moving body 100 approaches an intersection where a left turn is made and an obstacle is detected in front of the intersection. FIG. 5 includes a left view, a center view, and a right view, and the display changes in this order as the moving body 100 approaches the intersection. The detection area image 31a, the detection area image 31b, and the left turn arrow 32 are the same as those in FIG. The example of FIG. 5 shows a change in the detection area image 31 in a situation where an obstacle is detected at a turning point (left side of the intersection) while the moving body 100 is approaching the intersection. At this time, as shown in the central figure and the right figure corresponding to the state in which an obstacle is detected in the front detection area, the display color of the detection area image 31a corresponding to the front detection area changes from the reference color. The driver can be informed of the presence of obstacles. Here, in the example of FIG. 5, the display color of the detection area image 31b corresponding to the detection area behind the area where no obstacle is detected remains the reference color. Here, as a method of notifying the driver of the existence of an obstacle, a mode such as issuing a warning sound may be taken.

Here, the display color of the detection area image 31, which changes when an obstacle is detected, may change depending on the type of the obstacle. For example, when the obstacles are a two-wheeled vehicle, an automobile, and a person, the display colors of the detection area image 31 may change to green, red, and yellow, respectively. At this time, the driver can easily grasp the object to be noted. Further, the change in the detection area image 31 that occurs when an obstacle is detected is not limited to the change in the display color. That is, the display mode may be changed so as to call the driver's attention to at least a part of the detection area including the obstacle. As another example, the detection area image 31 may blink, or the brightness of the detection area image 31 may change. The color of only the portion of the detection area image 31 where an obstacle exists may be changed. Further, the frame-shaped image may be displayed so that the periphery of the obstacle is surrounded by the frame-shaped image and can be recognized by the driver.

(Image processing method)
FIG. 6 is a flowchart of an image processing method executed by the image processing apparatus 1. The image processing apparatus 1 may be configured to read and execute a program recorded on a non-temporary computer-readable medium for the processing described below. Non-temporary computer-readable media include, but are not limited to, magnetic storage media, optical storage media, photomagnetic storage media, and semiconductor storage media. Magnetic storage media include magnetic disks, hard disks, and magnetic tapes. The optical storage medium includes an optical disk such as a CD (Compact Disc), a DVD, and a Blu-ray disc (Blu-ray (registered trademark) Disc). The semiconductor storage medium includes a ROM (Read Only Memory), an EEPROM (Electrically Erasable Programmable Read-Only Memory), and a flash memory.

The image processing device 1 determines whether the moving body 100 is approaching a place where the moving body 100 turns right or left (step S1). The image processing device 1 may execute the determination in step S1 based on the information regarding the traveling state of the moving body 100 acquired from the ECU 102 and the external device 110. The image processing device 1 may use, for example, the information of the distance to the place where the right turn or the left turn is obtained from the navigation device. For example, the information that the navigation device is displaying the right turn or left turn guidance and the speed information acquired from the ECU 102 may be used. In the image processing device 1, for example, when the speed of the moving body 100 is low (15 km / h or less as an example) and the distance to the place where the right or left turn is short (within 700 m as an example), the moving body 100 is used. It may be determined that the vehicle is approaching a place to turn right or left.

Here, when the image processing device 1 acquires map information and information on the current position of the moving body 100 (as an example, coordinates such as latitude and longitude) as information on the traveling status of the moving body 100, the image processing device 1 is based on these. The determination in step S1 may be executed. That is, the image processing device 1 may calculate the distance between the place where the right turn or the left turn is made and the current position, and determine whether or not the image processing device 1 is approaching the place where the right turn or the left turn is made according to the calculated distance.

When the image processing device 1 determines that it is approaching a place to turn right or left (Yes in step S1), the image processing device 1 proceeds to the process of step S2. When the image processing apparatus 1 determines that it is not approaching a place to turn right or left (No in step S1), the image processing apparatus 1 repeats the process of step S1 until it approaches.

The image processing device 1 displays the detection area when the moving body 100 approaches a place where the moving body 100 turns right or left (step S2). For example, as shown in the left view of FIGS. 3 to 5, the detection area is displayed by a fan-shaped detection area image 31a and a detection area image 31b that are spread to the left and right.

The image processing device 1 detects an obstacle (step S3). Specifically, the input unit 11 of the image processing device 1 acquires the captured image from the image pickup device 101, the recognition unit 12 recognizes an obstacle from the captured image, and the display determination unit 13 includes the obstacle in the detection area. Is determined.

When the image processing apparatus 1 detects an obstacle in the detection area (Yes in step S3), the image processing apparatus 1 changes the display color of the detection area image 31 corresponding to the detection area in which the obstacle exists (step S4). For example, when the image processing device 1 detects an obstacle in the rear detection area, the display color of the detection area image 31b is changed as shown in FIG. 4, and when the obstacle is detected in the front detection area, FIG. The display color of the detection area image 31a is changed as in 5.

When the image processing apparatus 1 does not detect an obstacle in the detection area (No in step S3), the image processing device 1 displays the detection area image 31 in the first display color (reference color) (step S5). Here, when an obstacle once detected is separated and cannot be detected, the display color of the detection area image 31 corresponding to the detection area in which the obstacle exists is returned to the reference color by the process of step S5. Is done. Here, when an obstacle is detected in a region other than the detection region, the detection region image 31 may be displayed in the first display color (reference color).

The image processing device 1 determines whether the moving body 100 has completed a right turn or a left turn (step S6). The image processing device 1 may execute the determination in step S6 based on the information regarding the traveling state of the moving body 100 acquired from the ECU 102 and the external device 110. The image processing device 1 may use, for example, information indicating that the right turn or left turn display (arrow as an example) acquired from the navigation device has been erased. The image processing device 1 may determine that the moving body 100 has completed the right turn or the left turn, for example, when a short time (3 seconds as an example) has elapsed after the right turn or left turn arrow of the navigation device disappears. Further, the image processing device 1 may use, for example, the speed information of the moving body 100 acquired from the ECU 102. The image processing device 1 calculates the mileage based on, for example, the elapsed time since the detection area is displayed and the speed of the moving body 100, and determines the end of the right turn or the left turn of the moving body 100 from the calculated mileage. You can do it.

Here, the image processing device 1 may execute the determination in step S6 based on the map information and the information on the current position of the moving body 100 as the information on the traveling state of the moving body 100.

When the image processing device 1 determines that the moving body 100 has finished turning right or left (Yes in step S6), the image processing device 1 hides the detection area (step S7).

When the image processing apparatus 1 determines that the moving body 100 has not completed the right turn or the left turn (No in step S6), the image processing apparatus 1 changes the shape of the detection area image 31 (step S8), and performs the processing in step S3. return. For example, as shown in FIGS. 3 to 5, in the area of the detection area image 31, the fan-shaped area decreases in the bending direction of the moving body 100 as the moving body 100 approaches the place where the moving body 100 turns right or left.

As described above, the image processing device 1 according to the present embodiment causes the occupants of the moving body 100 to recognize the detection area selected from the peripheral areas according to the traveling condition of the moving body 100. For example, when a display that calls attention to obstacles existing in the entire peripheral area is displayed, there is a problem that attention is drawn to other than the obstacles that should be really noted and the attention is reduced. With the configuration of, it is possible to appropriately alert the occupants of the moving body 100.

Although the embodiments according to the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or modifications based on the present disclosure. It should be noted, therefore, that these modifications or modifications are within the scope of this disclosure. For example, the functions included in each component or each step can be rearranged so as not to be logically inconsistent, and a plurality of components or steps can be combined or divided into one. Is. Although the embodiment according to the present disclosure has been mainly described with respect to the apparatus, the embodiment according to the present disclosure can also be realized as a method including steps executed by each component of the apparatus. The embodiments according to the present disclosure can also be realized as a method, a program, or a storage medium on which a program is recorded, which is executed by a processor included in the apparatus. It should be understood that these are also included in the scope of this disclosure.

In the above embodiment, the detection area image 31 and the left turn arrow 32 are displayed by projecting them onto the windshield of the moving body 100 using a head-up display. However, an image in the traveling direction of the moving body 100 may be displayed on a display configured by an LCD or the like, and the detection area image 31 and the left turn arrow 32 may be superimposed on the image. The display may also serve as a display for a navigation device.

In the above embodiment, the detection area image 31 is fan-shaped, but may be shown in another shape such as a circle or a polygon. Further, the detection area image 31 may be shown in a shape different from the fan shape when an obstacle is detected. FIG. 7 shows a display example of a detection area in a situation where a bicycle 33, which is an example of an obstacle, is detected on the left side of the front. In the example of FIG. 7, the area to be imaged is blocked by the bicycle 33, so that the blind spot area is generated on the left side in front. The recognition unit 12 cannot recognize an obstacle existing in the blind spot region, or the possibility of being able to recognize it is less than a predetermined value. In such a case, the display determination unit 13 of the image processing apparatus 1 may display the detection area image 31a in a shape in which a portion corresponding to the blind spot area is missing. By seeing the detection area image 31 different from the normal shape (fan shape), the occupant of the moving body 100 can grasp that the field of view is blocked by the bicycle 33 and the object beyond it cannot be detected. In this way, the display determination unit 13 may display the position of the blind spot region in the detection region so that the occupant can visually display it by changing the shape of the detection region image 31. Here, blind spot areas can also be formed by utility poles, guardrails, walls and vegetation.

In the above embodiment, when an obstacle is detected, the display color of the portion corresponding to at least a part of the detection area including the detected obstacle is changed in the detection area image 31. Here, the display color of the detection area image 31 may change depending on the traveling condition or the environment of the moving body 100. For example, as shown in FIG. 8, when the moving body 100 turns right or left (while the moving body 100 is turning slightly before turning), the display color of the detection area is changed in order to call the driver more attention. You may let me. Further, the reference color (for example, usually blue) may be changed according to the time zone so that the driver can easily see the detection area. For example, the reference color may be changed to yellow at night. Further, the reference color may be changed to red in the daytime. At this time, the display determination unit 13 may acquire time information from an external device 110 such as a navigation device. Further, the color of the detection area image 31 may be adjusted to a color that is easy to see according to the color of the superimposed landscape (for example, a road).

In the above embodiment, the detection area image 31 includes a front detection area image 31a and a rear detection area image 31b. Further, the detection area image 31 was displayed on the head-up display. Here, the image processing device 1 is not limited to such a configuration, and can be applied to an in-vehicle system having various configurations. For example, the image processing device 1 may execute a process for selecting a detection area and a display device (hereinafter referred to as an operating environment setting) before executing the above image processing method. FIG. 9 is a flowchart illustrating an operating environment setting method.

The image processing device 1 detects the number of image pickup devices 101 attached to the moving body 100 (step S11). When the number of image pickup devices 101 is four, it is detected whether the angle of view of the side cameras (right and left image pickup devices 101) is 90 ° or more (step S12).

When the angle of view of the side camera is 90 ° or more (Yes in step S12), the image processing device 1 may set the image pickup areas of the four image pickup devices 101 as the detection areas (step S13). The image processing device 1 detects the image pickup areas of the two image pickup devices 101 when the angle of view of the side camera is not 90 ° or more (No in step S12) and when the number of image pickup devices 101 is two. It may be set in the area (step S14). At this time, the front and rear imaging regions may be selected as the detection region, or another combination of imaging regions such as front and left may be selected. Further, when the number of image pickup devices 101 is one, the image processing device 1 may set the image pickup area of one image pickup device 101 as the detection area (step S15).

The image processing device 1 detects the type of the external device 110 that displays the detection area image 31 (step S16). When the external device 110 to be displayed is a head-up display, the image processing device 1 is set so that the detection area image 31 can be displayed on the head-up display (step S17). Specifically, the superimposition processing unit 14 of the image processing device 1 adjusts the display position, size, and the like of the detection area so that the detection area is appropriately displayed on the windshield. Further, when the external device 110 to be displayed is a navigation device, the image processing device 1 is set so that the detection area image 31 can be displayed on the display of the navigation device (step S18). The image processing device 1 can be flexibly applied to the in-vehicle system of the mobile body 100 by executing, for example, the operating environment setting method as shown in FIG. 9 before the image processing method.

In the above embodiment, when the moving body 100 makes a right turn or a left turn, the oncoming lane is excluded from the detection area (see FIG. 3). Here, with respect to the oncoming lane, the detected oncoming vehicle or the like may be shown to the extent that the driver's attention is not reduced. For example, the oncoming vehicle may be indicated by a display color frame (as an example, a dark red frame) that is less conspicuous than the detection area image 31.

In the above embodiment, the detection area is displayed when the moving body 100 makes a right turn or a left turn. Here, the detection area may be displayed when the moving body 100 moves backward, for example, when parking. At this time, since the rearward confirmation is important, it is preferable that the detection area image 31 includes the rearward detection area image 31b. The image processing device 1 may acquire information on the gear of the moving body 100 as information on the traveling state of the moving body 100 acquired from the ECU 102, and may determine from the gear information that the moving body 100 is moving backward. ..

Here, when parking and starting, the moving body 100 often moves backward. When the image processing device 1 acquires map information and information on the current position of the moving body 100 as information on the traveling status of the moving body 100, the image processing device 1 determines whether the moving body 100 is about to park or start based on the map information and the information on the current position of the moving body 100. It's okay. For example, the image processing device 1 may determine that the moving body 100 is about to park when the current position of the moving body 100 is a parking lot. In this case, the image processing device 1 may change the display color of the rear detection area image 31b to red so as to call the driver more attention.

In the above embodiment, the image processing device 1 is a device independent of the external device 110. Here, the image processing device 1 may be configured as a part of one of the external devices 110 (for example, a navigation device).

1 Image processing device 11 Input unit 12 Recognition unit 13 Display judgment unit 14 Superimposition processing unit 31, 31a, 31b Detection area image 32 Left turn arrow 33 Bicycle 100 Mobile object 101, 101a, 101b, 101c, 101d Imaging device 102 ECU
110 External device 111 Display unit

Claims (12)

An input unit that acquires an image of the peripheral area of a moving object, and
Image processing including a display determination unit that selects a detection area to be visually displayed to the occupant of the moving body from the area where an obstacle included in the peripheral area is detected according to the traveling condition of the moving body. Device. The detection region includes a first detection region, which is a region in front of the moving body, and a second detection region, which is a region behind the moving body, in the peripheral region.
The first detection area is superimposed on the scenery of the peripheral area or the image of the peripheral area and visually displayed to the occupant of the moving body, and the second detection area is the lower part of the first detection area. The image processing apparatus according to claim 1. It is equipped with a recognition unit that recognizes obstacles existing in the peripheral area.
The detection area is a part of the peripheral area where the obstacle is recognized by the recognition unit.
A claim that, when the obstacle is detected in the detection area, the occupant of the moving body is notified, and when the obstacle is detected in a region other than the detection area, the notification is not performed. The image processing apparatus according to 1 or 2. The display determination unit
When the moving body approaches a place where it turns right or left, the detection area is displayed.
The invention according to any one of claims 1 to 3, wherein the position of the detection area is changed according to the traveling condition from the point where the detection area is displayed to the point where the right turn or the left turn of the moving body ends. Image processing equipment. The display determination unit
The image processing apparatus according to claim 4, wherein when the moving body makes a right turn or a left turn, the detection area is moved toward the bending direction of the moving body, and the size of the detection area is reduced and displayed. .. The display determination unit
The image processing apparatus according to claim 4 or 5, wherein when the moving body makes a right turn or a left turn, the display color of the detection area is changed. The display determination unit
When the obstacle is present in the imaged region and there is a blind spot region in the detection region where the obstacle cannot be detected, the position of the blind spot region in the detection region is set to the occupant. The image processing apparatus according to any one of claims 1 to 6, which is visually displayed. The display determination unit
The image processing apparatus according to any one of claims 1 to 7, wherein when the obstacle is detected, the display mode of at least a part of the detection area including the detected obstacle is changed. The display determination unit
The image processing apparatus according to claim 8, wherein when the obstacle is detected, the display color of at least a part of the detection area including the detected obstacle is changed. The display determination unit
The image processing according to claim 9, wherein when the obstacle is detected, at least a part of the detection area including the detected obstacle is displayed in a display color corresponding to the type of the obstacle. Device. The image processing apparatus according to any one of claims 1 to 10, wherein the detection area is superimposed and displayed on a display image of an external device. The image processing device according to claim 11, wherein the external device includes a head-up display.

Images