WO2014087606A1 - Photographic image display device and photographic image display method - Google Patents

Abstract

A photographic image display device comprises a mobile body detection unit (104), an index value detection unit (106), an arousal degree detection unit (108), a display mode determination unit (110), and a display control unit (112). If a mobile body is present in a photographic image which is photographed with a vehicle-mounted camera (10), the mobile body detection unit (104) detects at least one mobile body from the photographic image. The index value detection unit (106) detects an index value for each detected mobile body relating to ease of recognition by a driver. The arousal degree detection unit (108) detects a degree of arousal of the driver. The display mode determination unit (110) determines, on the basis of the index value of each mobile body and the degree of arousal, an enhanced display mode whereby each mobile body is displayed in an enhanced form. On the basis of the enhanced display mode of each mobile body, the display control unit (112) generates an image wherein each mobile body is displayed in the enhanced form in the photographed image and displays same in a display screen (14).

Description

Captured image display device and captured image display method Cross-reference of related applications

This disclosure is based on Japanese Patent Application No. 2012-264759 filed on December 3, 2012, the contents of which are incorporated herein.

The present disclosure relates to a photographed image display apparatus and a photographed image display method for detecting a moving body from a photographed image photographed by a vehicle-mounted camera and displaying a photographed image in which the movable body is highlighted.

Various driving support technologies have been developed for the purpose of enabling the driver to drive the vehicle safely and comfortably. As a representative example of such driving support technology, there is a technology for detecting a moving body such as a pedestrian or a vehicle existing in front of or around the vehicle and informing the driver (hereinafter referred to as a moving body detection notification technology). To do.

This moving object detection notification technology is not affected by the difference in the size of the moving object, the difference in the distance to the moving object, and the fact that part of the moving object is behind something. Therefore, it is necessary to be able to detect the moving body without leakage and correctly. Nevertheless, it is also required that the processing load for detection is not excessive, and various detection methods such as a method using pattern matching have been developed in order to satisfy these requests (for example, patent documents). 1).

In addition, a technique has been developed that can detect a moving object accurately even at night by taking an image with near-infrared light (Patent Document 2).

However, no matter how accurately the moving body can be detected, the final judgment for driving the vehicle is left to the driver. Therefore, in these moving body detection notification techniques, the driver is notified of the presence of the moving body by displaying a photographed image on a monitor screen and surrounding the area where the moving body is detected with a frame.

However, at present, the technology for notifying the driver of the presence of the detected moving body has not caught up with the advancement of the technology for detecting the moving body, and therefore the detection result of the moving body is used as driving assistance for the driver. There was a problem that it was not fully utilized. For example, currently, the detected moving object is displayed by enclosing the moving object in a frame on the in-vehicle camera image displayed on the monitor screen. However, if the moving object is small, it is enclosed in a frame. The driver cannot immediately recognize what the moving object is. In addition, even when a moving object is shown in a dark image (or a dark portion in the image), the driver cannot immediately recognize what the moving object is surrounded by the frame. When the driver is tired or feels sleepy, such a tendency becomes even more remarkable, and as a result, it becomes difficult to utilize the detection result of the moving body for driving assistance of the driver.

JP2007-087551A Japanese Patent No. 4611919

The present disclosure has been made in view of the above points, and provides a captured image display device and a captured image display method that allow a driver to recognize a detection result of a moving object regardless of a driver's condition. The purpose is to do.

The captured image display device according to the first aspect of the present disclosure includes a moving object detection unit, an index value detection unit, a wakefulness detection unit, a display mode determination unit, and a display control unit. The moving body detection unit detects at least one moving body from the captured image when the moving body is present in the captured image captured by the in-vehicle camera. The index detection unit detects an index value related to ease of recognition by the driver for each detected moving body. The awakening level detection unit detects the driver's awakening level. The display mode determination unit determines a highlighted display mode for highlighting and displaying each mobile body based on the index value of each mobile body and the arousal level. A display control part produces | generates the image which each moving body emphasized in the said picked-up image based on the emphasis display aspect of each moving body, and displays it on a display screen.

According to the above-mentioned photographed image display device, it is possible to make the driver recognize the detection result of the moving body regardless of the driver's condition.

The captured image display method according to the second aspect of the present disclosure detects at least one moving object from a captured image captured by an in-vehicle camera, and sets an index value related to the ease of recognition by the driver for each detected moving object. Detecting, detecting the driver's arousal level, determining an emphasis display mode for emphasizing and displaying each mobile unit based on the index value of each mobile unit and the arousal level, and highlighting each mobile unit Each moving body may generate an image displayed in an emphasized manner in the captured image and display the image on a display screen based on an aspect.

According to the above method, it is possible to make the driver recognize the detection result of the moving object regardless of the driver's condition.

The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. The drawing
FIG. 1 is an explanatory diagram of a vehicle equipped with a captured image display device according to an embodiment of the present disclosure. FIG. 2 is a block diagram of the captured image display device. FIG. 3 is a flowchart of the first half of the captured image display process of the present embodiment. FIG. 4 is a flowchart of the latter half of the captured image display process of the present embodiment. FIG. 5 is an explanatory diagram illustrating a captured image acquired from the in-vehicle camera. 6 (a) to 6 (d) are explanatory diagrams showing a method for detecting the size and contrast of the moving object, FIG. 7 is an explanatory diagram of a method of determining the recognition ease according to the size and contrast of the moving object. FIG. 8A and FIG. 8B are explanatory diagrams of a method for correcting the degree of ease of recognition of a moving object according to the driver's arousal level. FIG. 9A and FIG. 9B are explanatory diagrams illustrating the manner in which the range that is difficult to recognize changes depending on the driver's arousal level, FIG. 10 is an explanatory diagram illustrating a method of determining a highlight mode according to the recognition ease corrected by the arousal level, FIG. 11A and FIG. 11B are explanatory diagrams illustrating a method for displaying a moving object with enhanced contrast. FIG. 12A and FIG. 12B are explanatory views illustrating another method of displaying the contrast of the moving object with emphasis, FIG. 13A and FIG. 13B are explanatory diagrams illustrating a method of displaying a moving body with an enlarged size, FIG. 14A and FIG. 14B are explanatory diagrams illustrating a method of displaying a moving object by replacing it with a symbol image. FIG. 15 is an explanatory diagram illustrating a captured image displayed on the display monitor in a state where the moving object is highlighted, FIG. 16 is an explanatory diagram conceptually showing a method for determining the highlighted display mode of the moving object in the captured image display processing of the modified example.

Hereinafter, embodiments will be described with reference to the drawings in order to clarify the contents of the present disclosure described above.

FIG. 1 shows a rough configuration of a vehicle 1 on which the captured image display device 100 of the present embodiment is mounted. As shown in the drawing, the captured image display device 100 of the present embodiment displays an in-vehicle camera 10 that captures an image in the traveling direction (or surroundings) of the vehicle 1 and an image (captured image) captured by the in-vehicle camera 10. It is mounted on the vehicle 1 together with a display monitor 14 that is a display screen, a driver camera 12 that captures a face image of the driver of the vehicle 1, and the like.

The captured image display device 100 receives a captured image captured by the in-vehicle camera 10, detects a moving body (for example, a pedestrian or a vehicle) in the captured image and drives the moving body when the captured image includes a moving body. The captured image is displayed on the display monitor 14 in a state of being highlighted so as to be easily recognized by a person. Further, when determining the highlighting mode of the moving object, the driver's arousal level is detected from the face image captured by the driver camera 12, and the mode of highlighting is determined in consideration of the awakening level.

FIG. 2 shows an internal configuration of a photographed image display device (IMG DISP APPA) 100 that controls the above-described functions. As shown in the figure, the captured image display device 100 is connected to an in-vehicle camera (IN-V CAMERA) 10, a driver camera (DRIVER CAMERA) 12, a display monitor (DISP MONITOR) 14, and an image storage unit (IMG STORE). 102, moving object detection unit (MOVE OBJ DETC) 104, index value detection unit (INDEX DETC) 106, arousal level detection unit (WAKE LV DETC) 108, display mode determination unit (DISP MODE DECIDE) 110, display image generation unit (DISP IMG GENERATE) 112 and symbol image storage unit (SYMBOL IMG STORE) 114 are provided.

Note that these seven “parts” are those for which the captured image display device 100 is classified for the sake of convenience, and the inside of the captured image display device 100 needs to be physically divided into seven parts. There is no. Of these seven “units”, the image storage unit 102 and the symbol image storage unit 114 are realized by a memory (not shown), but the other five “units” are hardware using a logic circuit or a CPU. It can be realized as hardware, or can be realized as software as a computer program or a part of the program.

The image storage unit 102 receives the captured image from the in-vehicle camera 10 and stores it in the memory.

The moving body detection unit 104 reads the captured image stored in the image storage unit 102, and detects a moving body from the captured image when a moving body such as a pedestrian is reflected in the captured image. Various known methods such as a method using pattern matching can be used for detecting the moving object.

The index value detection unit 106 detects an index value (recognition level) indicating the ease of recognition for the driver for each of the mobile bodies detected by the mobile body detection unit 104. The ease of recognition will be described in detail later.

The arousal level detection unit 108 analyzes the driver's face image taken by the driver camera 12 and detects the driver's arousal level. When detecting the arousal level from the face image, a well-known method can be used, such as detecting the arousal level based on the frequency with which the driver moves his / her line of sight, the frequency of blinking, and the speed of the eyelid when blinking. . Note that the degree of arousal may be detected by measuring the driver's heart rate and breathing by incorporating a pressure sensor in the driver's seat, for example, without being limited to the face image photographed by the driver camera 12.

The display mode determination unit 110 displays a mode for highlighting the moving body for each moving body based on the index value (recognition level) detected by the index value detection unit 106 and the arousal level detected by the arousal level detection unit 108. decide. Further, in this embodiment, as one mode for highlighting the moving body, a mode is also set in which the image of the moving body is replaced with a symbol image corresponding to the moving body and displayed. The “symbol image corresponding to the moving object” is, for example, a graphic image obtained by simplifying the shape of a human if the moving object is a pedestrian, and the animal shape and face are simplified if the moving object is an animal. It is an image of a figure that has become These symbol images are stored in advance in the symbol image storage unit 114.

Then, when the display image generation unit 112 reads the captured image stored in the image storage unit 102, the moving body detected by the moving body detection unit 104 is highlighted in the highlighted display mode determined by the display mode determination unit 110. The displayed image is generated and output to the display monitor 14.

In this way, for each moving body detected from the captured image, if the moving body is highlighted and displayed on the display monitor 14 in a manner that considers the ease of recognition of the moving body and the driver's arousal level, Regardless of the condition, it is possible to cause the driver to recognize the detection result of the moving body and to realize appropriate driving support. In the following, a process performed inside the captured image display device 100 in order to realize this will be described in detail.

In this embodiment, the moving body detection unit 104 corresponds to the “moving body detection unit” in the present disclosure, the index value detection unit 106 corresponds to the “index value detection unit” in the present disclosure, and the arousal level detection unit 108. Corresponds to the “wakefulness detection unit” of the present disclosure, the display mode determination unit 110 corresponds to the “display mode determination unit” of the present disclosure, and the symbol image storage unit 114 corresponds to the “symbol image storage unit” of the present disclosure. Further, since the display monitor 14 merely displays the captured image according to the image data output from the display image generation unit 112, the display image generation unit 112 actually controls the display monitor 14 to display the captured image. You can think that you are. Therefore, the display image generation unit 112 of this embodiment is also referred to as a display control unit 112 and corresponds to the “display control unit” of the present disclosure.

3 and 4 show a flowchart of the captured image display process of the present embodiment executed by the captured image display apparatus 100. FIG. This process is a process executed at a constant cycle of about 30 times per second in synchronization with the movement of the in-vehicle camera 10 to capture an image.

When the captured image display process of the present embodiment is started, first, the image storage unit 102 in the captured image display device 100 stores the captured image output from the in-vehicle camera 10 (S100).

Subsequently, when a moving object is present in the captured image, the moving object detection unit 104 in the captured image display device 100 analyzes the captured image, thereby moving the moving object (pedestrian, vehicle, animal, etc.) in the captured image. Is detected (S102). The moving body in the captured image is detected as follows.

First, a typical luminance change pattern of a portion where a pedestrian is shown in the image is stored in advance. Similarly, a typical change pattern of a portion in which an image of a vehicle or an animal is captured is stored. Then, a portion where the luminance is changed in a pattern close to these typical change patterns is searched for in the captured image. For example, if a certain part in the photographed image is close to the pedestrian change pattern, it can be determined that a pedestrian is shown in that part. Similarly, if it is close to the change pattern of the vehicle, it can be determined that the vehicle is shown, and if it is close to the change pattern of the animal, it can be determined that the animal is shown.

Of course, since a distant moving object appears small and a nearby moving object appears large, the size of the moving object shown in the photographed image is not uniform. Accordingly, patterns of various sizes (however, the shapes are similar to each other) are prepared for the change pattern of the moving body, and a photographed image is searched using the plurality of change patterns. In this way, when a moving body exists in the captured image, it is possible to detect moving bodies of various sizes from a moving body that appears large in the captured image to a moving body that appears small.

Note that, as described above, the type of moving body (pedestrian, vehicle, animal, etc.) to be detected changes depending on which change pattern is used, such as for pedestrians, vehicles, and animals. Therefore, if various types of change patterns are stored in advance, it is possible to detect more types of moving bodies. Moreover, the classification of the detected moving body can be determined depending on which classification change pattern is used.

FIG. 5 shows an example in which a moving body is detected from a captured image when a moving body is present in the captured image. In the illustrated example, pedestrian images A to E and a vehicle image F are detected. In the figure, a dashed frame surrounding a pedestrian or vehicle in a rectangle represents the position of the change pattern when the moving body is detected. In other words, as described above, when detecting a moving body, a change pattern of various sizes is moved little by little in the captured image, and a portion where the luminance in the captured image changes in the same manner as the change pattern is searched. is doing. Accordingly, the broken-line rectangle surrounding the pedestrian image A in the figure changes the luminance in the captured image in the same manner as the change pattern when the pedestrian change pattern of that size is placed at that position. It represents that. The same applies to the broken-line rectangles surrounding the pedestrian images B to E. Note that the change patterns for pedestrians are similar in shape even if they are different in size, so that the broken-line rectangles surrounding the pedestrian images A to E are similar to each other.

Also, the above description applies to the image F of the vehicle in exactly the same manner. That is, a broken-line rectangle surrounding the vehicle image F indicates that the luminance in the captured image changes in the same manner as the change pattern when the change pattern for the vehicle of that size is placed at that position. ing.

When moving bodies in the captured image are detected in this way, the size and contrast of the moving bodies are detected, and an index value (recognition ease) indicating ease of recognition for the driver is determined for each moving body. (S104).

FIGS. 6A to 6D show examples of detecting the size and contrast of a moving body detected from a captured image. As an example, if it demonstrates using the image A of the pedestrian mentioned above, the magnitude | size of a pedestrian will be represented by the length of the diagonal line of the change pattern (dashed rectangle) which detected the pedestrian (FIG. 6 (a)). reference). If the length of the diagonal line is used, the size of the moving object can be expressed without distinguishing between the case of using a vertically long change pattern like a pedestrian and the case of using a horizontally long change pattern like a vehicle. Can do.

As for contrast, the contour of the pedestrian shown in the image A is detected by detecting an edge with respect to the image A of the pedestrian (dotted line in FIG. 6B), and the portion inside the contour ( The average luminance at the moving body part) is calculated (see FIG. 6C). Further, the average luminance in the portion outside the contour (background portion) is also calculated (see FIG. 6D). The difference in average luminance between the inside of the contour (moving body portion) and the outside of the contour (background portion) can be detected as the contrast of the moving body. Of course, the contrast may be detected using other known methods.

Thus, if the size and contrast of the moving body are known, the degree of ease of recognition of the moving body (how easily the moving body can be recognized by the driver) can be determined. For example, the driver can easily recognize the size of the moving body as the moving body becomes larger. Further, regarding the contrast of the moving body, it becomes difficult for the driver to recognize the contrast because the contrast between the moving body and the background becomes difficult to distinguish. Therefore, the driver can easily recognize the higher the contrast of the moving body.

However, if the size of the moving body is too small, the driver cannot easily recognize even if the contrast is high. On the other hand, even if the contrast is low, if the moving body is sufficiently large, the driver can recognize the moving body relatively easily.

FIG. 7 shows results obtained by experimentally determining the ease of recognition (recognition level) for moving bodies having various sizes and contrasts. If such data is obtained, it is possible to determine the ease of recognition of the moving object from the size and contrast detected for the moving object. In FIG. 7, the size of the moving body increases along the arrow direction, the contrast of the moving body increases along the arrow direction, and the recognition degree is high along the arrow direction, that is, the recognition becomes easy. .

If it demonstrates using the picked-up image of FIG. 5 as an example, since the pedestrian of the location which displayed "A" in the figure has sufficient size and contrast is high, a driver | operator can recognize easily. This is possible, and the degree of recognition is increased. In addition, the vehicle at the location indicated as “F” in the figure is sufficiently large but has a low contrast, so that the ease of recognition is slightly reduced. In the figure, the pedestrians indicated by “B” or “E” are not sufficiently large in size and low in contrast, so that the degree of recognition is further reduced. In addition, the pedestrian at the location indicated by “D” in the figure has a high contrast but a small size, so the driver cannot easily recognize it, and the degree of recognition becomes small. Furthermore, since the pedestrian at the location indicated by “C” in the drawing is very small in size and low in contrast, the degree of recognition is very small.

7 is stored in advance in a memory (not shown) of the photographed image display apparatus 100. When the index value detection unit 106 receives the size and contrast of the moving object from the moving object detection unit 104, the index value detection unit 106 refers to the data stored in the memory to determine the recognition ease for each moving object (see FIG. 3 S104).

Subsequently, the arousal level detection unit 108 of the captured image display device 100 detects the driver's arousal level by analyzing the driver's face image captured by the driver camera 12 (S106). Various known methods can be applied to detect the degree of arousal.

For example, the driver's pupil is detected from the face image, and the frequency with which the driver moves the line of sight is detected from the movement of the pupil position. When the driver frequently moves his / her line of sight, it can be determined that the degree of arousal is high, and as the frequency of moving the line of sight decreases, it can be determined that the degree of arousal has decreased. Alternatively, the arousal level may be detected by detecting the frequency with which the driver blinks, the movement speed of the eyelid when blinking, and the like. Furthermore, instead of analyzing the face image by the driver camera 12, the degree of arousal can be detected by monitoring the driver's breathing and pulse using a pressure sensor built in the driver's seat.

Subsequently, when a moving body exists in the captured image, one arbitrary moving body is selected from one or a plurality of moving bodies detected in the captured image (S108 in FIG. 3). Then, the recognition ease of the selected moving object is corrected based on the arousal level (S110). For example, a driver who is sufficiently awake can recognize a pedestrian in a portion displayed as “B” or “E” in the captured image of FIG. 5, but the driver's arousal level is reduced. Then, it becomes difficult to recognize these pedestrians. Therefore, the ease of recognition of the moving object is corrected in consideration of the influence of the driver's arousal level.

In the captured image display device 100, a correction coefficient corresponding to the driver's arousal level is set in advance in a memory (not shown). Then, when the display mode determination unit 110 receives the recognition ease and the arousal level from the index value detection unit 106 and the arousal level detection unit 108, the display mode determination unit 110 reads the correction coefficient corresponding to the arousal level and multiplies the recognition level by the correction coefficient. The corrected recognition degree is calculated.

FIG. 8 (a) shows an example in which a correction coefficient is set in advance according to the driver's arousal level. In the illustrated example, the arousal level is obtained in six stages, and a correction coefficient of 1.0 is set for the arousal level of 5 (completely awakened state). Also, the correction coefficient decreases as the awakening level decreases.

FIG. 8B conceptually shows an example in which the recognition ease determined by the size and contrast of the moving object is modified according to the arousal level.

As described above, when the recognition ease corrected in consideration of the arousal level is obtained, it is determined whether or not the recognition ease is equal to or greater than a predetermined allowable value (S112 in FIG. 4). As a result, it is expected that the driver cannot sufficiently recognize the moving body whose recognition ease is less than the allowable value even if it is displayed on the display monitor 14 as it is.

For example, FIG. 9A shows the degree of ease of recognition for the size and contrast of the moving object when the driver's arousal level is sufficiently high. It is expected that the driver cannot recognize the moving body in the area (indicated by slanted lines in the figure) even if the image captured by the in-vehicle camera 10 is displayed on the display monitor 14 as it is. FIG. 9B shows the degree of ease of recognition for the size and contrast of the moving body when the driver's arousal level is lowered. Since the correction is made, the area where the recognition ease is less than or equal to the allowable value (the area where the driver cannot be recognized even if displayed on the display monitor 14 as it is) becomes wider. 9 (a) and 9 (b), the size of the moving body increases along the direction of the arrow, the contrast of the moving body increases along the direction of the arrow, and the ease of recognition is increased in the direction of the arrow. It becomes high along the line, that is, it becomes easy to recognize.

Therefore, the display mode determination unit 110 of the photographed image display device 100 compares the corrected recognition ease with the allowable value and does not satisfy the allowable value (S112: NO), depending on the corrected easy recognition level. The highlighted display mode is determined (S114 in FIG. 4).

FIG. 10 exemplifies a highlight mode according to the degree of easy recognition after correction.

For example, when the degree of ease of recognition after correction does not satisfy the permissible value only slightly (that is, smaller than the permissible value but greater than or equal to the first threshold), the detected moving body is surrounded by a frame. Highlight. The first threshold value is set to a value smaller than the allowable value. In addition, the frame surrounding the moving body can be the size of the change pattern (see FIGS. 5 and 6A to 6D) used to detect the moving body.

If the moving object has a degree of ease of recognition that has been corrected in consideration of the driver's arousal level and is slightly less than the permissible value, the driver can reliably recognize the moving object simply by surrounding the moving object with a frame. It becomes possible to do.

In addition, for a moving body (moving body that is less than the first threshold value but greater than or equal to the second threshold value) that has been corrected in consideration of the driver's arousal level, the moving body is surrounded by a frame and contrast is increased. Is highlighted. The second threshold value is set to a value smaller than the first threshold value. For such a moving body, in addition to displaying the moving body surrounded by a frame, the contrast is also displayed with emphasis, so that the driver can surely recognize the moving body. A method for enhancing the contrast of the moving object will be described later.

For mobile objects that have a smaller level of recognition that has been corrected in consideration of the degree of arousal (moving objects that are less than the second threshold, but greater than or equal to the third threshold), the mobile object is surrounded by a frame and contrast is enhanced. Further, the image of the moving object is enlarged and displayed. The third threshold value is set to a value smaller than the second threshold value. In this way, it is possible to reliably recognize even a moving body that cannot be reliably recognized by the driver simply by enclosing the frame and enhancing the contrast. A method of enlarging and displaying the moving object image will be described later.

For moving objects that have a smaller recognition level that has been corrected in consideration of the arousal level (moving objects that do not satisfy the third threshold), images of moving objects taken with the in-vehicle camera 10 are classified according to the type of moving object. The symbol image to be replaced is displayed. For example, if the moving body is an image of a pedestrian, it is replaced with a simple figure (symbol image) representing a person and displayed. If the moving body is an animal, it is displayed by replacing it with a simple figure (symbol image) representing the animal. Since the symbol image is set to a simple and easy-to-understand graphic, the driver can easily recognize it even if it is displayed in a small size. The symbol image will be described in detail later.

As described above, the display mode determination unit 110 of the captured image display apparatus 100 determines that the value of the ease of recognition of the selected moving object corrected according to the arousal level is less than the allowable value (see FIG. 4). (S112: NO), an emphasis display mode when the moving body is displayed on the display monitor 14 is determined according to the corrected recognition ease (S114).

Subsequently, the display image generation unit 112 of the captured image display device 100 generates an image of the moving object highlighted in the determined manner, and generates an image (captured image) captured by the in-vehicle camera 10. The moving body image (highlighted image) is synthesized (S116).

For example, if the determined highlighting mode is a moving body (a moving body having a corrected recognition degree equal to or higher than the first threshold and lower than the allowable value) in a mode of being displayed surrounded by a frame, Synthesize an image of a frame that surrounds the moving object.

Further, when the determined highlighted display mode is a moving body (a moving body having a corrected recognition degree equal to or higher than the second threshold value and lower than the first threshold value) that is displayed in a frame and with the contrast emphasized. Generates an image of a moving body with enhanced contrast and synthesizes the obtained image with a captured image as follows.

11 (a), 11 (b), 12 (a), and 12 (b) exemplify a method for generating an image of a moving body with enhanced contrast. First, the example shown in FIGS. 11 (a) and 11 (b) will be described. In the example shown in FIG. 11 (a), a little bright (brightness is shown) on a dark (low brightness) background. High) moving object (pedestrian B) is shown.

When generating an image of a moving object with enhanced contrast, first, a luminance frequency distribution within a frame surrounding the moving object is detected. Then, in the obtained frequency distribution, a moving body part (pedestrian in the illustrated example) and a background part appear. Therefore, as shown in FIG. 11B, the luminance on the moving object side is increased with the background side as a reference. By so doing, it is possible to enhance the contrast of the moving object by expanding the luminance gradation range within the frame.

It should be noted that the rate of expanding the gradation range may be increased as the recognition ease corrected according to the arousal level is smaller.

In the example shown in FIG. 12A, a slightly dark (low luminance) moving body (pedestrian B) is shown on a bright (high luminance) background. In such a case, as shown in FIG. 12B, the luminance on the moving body side is decreased with the background side as a reference. In this way, the gradation range of luminance within the frame can be expanded, and the contrast of the moving object can be enhanced.

When the contrast of the image in the frame is emphasized by the method shown in FIGS. 11A, 11B, 12A, and 12B, the outside of the frame in which the contrast is not enhanced. There may be a place where the luminance changes unnaturally at the boundary of. However, by displaying a frame at the border, it is possible to make the luminance unnaturally change.

In addition, when the determined highlight display mode is a mode in which the moving body is surrounded by a frame, the contrast is enhanced, and the image is further enlarged (the second recognition level when the moving body is corrected is greater than or equal to the third threshold). If it is less than the threshold), the image of the moving object with the contrast enhanced by the above-described method is further enlarged as follows, and the finally obtained moving object image is combined with the captured image. .

13 (a) and 13 (b) illustrate several methods for enlarging a moving object image. For example, in the example shown in FIG. 13A, the image of the moving object is enlarged within the range of the frame surrounding the moving object. In this case, the portion that protrudes from the frame due to enlargement of the image of the moving object is not displayed. As a result, in the illustrated example, an image in a state where the upper end portion of the pedestrian's head and the tip portion of the foot are missing is displayed.

In the example shown in FIG. 13B, the image of the moving body is enlarged for each frame surrounding the moving body. In this case, the image around the frame before enlargement is overwritten with the image of the moving body enlarged together with the frame.

Note that, in any case of enlarging by any method, the ratio of enlarging the image may be increased as the recognition degree after correction is smaller. However, if the image of the moving object is enlarged, a distant moving object will appear to be in the vicinity, so it is desirable to set a predetermined upper limit for the enlargement ratio.

In addition, when the determined highlighting mode is a mode in which the moving body is replaced with a symbol image (when the ease of recognition after correction of the moving body is less than the third threshold), the captured image is displayed. The image of the part in which the moving body is shown is replaced with a symbol image corresponding to the type of the moving body to synthesize the captured image.

For example, as illustrated in FIG. 14A, if the moving body is a pedestrian, the symbol image 114s representing a person is overwritten on the image of the portion where the pedestrian is shown. If an animal is shown, a symbol image 114s representing the animal as illustrated in FIG. 14B is overwritten. These symbol images 114 s are stored in advance in the symbol image storage unit 114.

It should be noted that the moving object to be replaced with the symbol image 114s is considered to be very small like a pedestrian in the portion “C” in FIG. For this reason, if the size of the symbol image 114s is displayed in accordance with the size of the moving object, the symbol image 114s may be too small to be recognized by the driver. In view of this, the symbol image storage unit 114 stores one symbol image 114s large enough to be recognized by the driver for each type of moving object, and the size of the moving object shown in the captured image. Regardless of this, the symbol image 114s having the same size is replaced and displayed.

As described above, when the value obtained by correcting the recognition ease of the selected moving body based on the arousal level is less than the allowable value (S112: NO in FIG. 4), the recognition ease after the correction is increased. An image of the moving body highlighted in a corresponding manner is generated, and the generated image is combined with the captured image (S114, S116).

On the other hand, when the ease of recognition corrected by the moving body is equal to or greater than the allowable value (S112: YES), the processes of S114 and S116 are omitted.

Then, it is determined whether or not the above-described processing (S110 to S116) has been completed for all the moving bodies detected in the captured image (S118). If any unprocessed moving bodies remain (S118: NO) ), Returning to S108, selecting one new moving body, and executing the series of processes described above.

When it is determined that the processing for all the moving objects has been completed by repeating such processing (S118: YES), the display image generating unit 112 displays the image data of the finally obtained captured image on the display monitor 14. In step S120, the captured image display process of this embodiment is terminated.

As a result, the captured image is displayed on the display monitor 14 in a state where all the moving bodies detected by the moving body detection unit 104 can be easily recognized by the driver.

FIG. 15 illustrates a captured image displayed on the display monitor 14. Compared with the image captured by the in-vehicle camera 10 shown in FIG. 5, in the captured image displayed on the display monitor 14, even a moving body that is difficult for the driver to recognize is highlighted depending on the degree of difficulty. As a result, all the moving objects are displayed in a state where they can be easily recognized.

For example, the vehicle indicated by “F” in the figure is sufficiently large in the image taken by the in-vehicle camera 10 (the image in FIG. 5), but the contrast is low, so the driver may overlook it. However, the image on the display monitor 14 in FIG. 15 is surrounded by a frame and is thus easily recognized by the driver.

In addition, a pedestrian at a location indicated as “B” or “E” in the drawing may not be sufficiently large and has a low contrast in the image taken by the in-vehicle camera 10, so the driver may miss it. is there. However, in the image on the display monitor 14 of FIG. 15, the frame is displayed surrounded by a frame and the contrast is also emphasized, so that the driver can easily recognize it as much as the vehicle at the location “F”. .

Furthermore, the pedestrian at the location indicated by “D” in the figure is likely to be overlooked by the driver because it is small in size and low in the image taken by the in-vehicle camera 10. However, the screen on the display monitor 14 in FIG. 15 is displayed surrounded by a frame and emphasized in contrast, and is further magnified, so that it is compared with a pedestrian “B” or “E”. To the extent that it is not significantly different, it is easier for the driver to recognize.

In addition, since the pedestrian at the location indicated by “C” in the figure is very small and has a low contrast in the image taken by the in-vehicle camera 10, the driver can hardly recognize it, but on the display monitor 14 in FIG. In this screen, the symbol image 114s is replaced and displayed. The symbol image 114s is a simple figure and easily visible, so that the driver can easily recognize it.

In addition, since the pedestrian of the location indicated as “A” in the figure is sufficiently large and has high contrast, the driver can easily recognize the image taken by the in-vehicle camera 10 as it is.

As described above, in the image displayed on the display monitor 14, the driver can easily recognize any moving body. For this reason, in this embodiment, even a moving body that is difficult for the driver to recognize is highlighted and displayed according to the degree of ease of recognition of the moving body, so that the driver can recognize the presence of all detected moving bodies. It is possible. In addition, the ease of recognition of moving objects is corrected according to the driver's arousal level, and the highlighting mode is determined based on the corrected level of recognition, so it can be detected regardless of the driver's condition. It is possible to make the driver recognize all the moving bodies.

In the above embodiment, when there is a moving object in the captured image, the index value regarding the ease of recognition by the driver for each of the moving objects detected by the index value detection unit 106 from the captured image by the moving object detection unit 104. Is detected. Then, based on the obtained index value and the driver's arousal level, the display mode determination unit 110 determines the highlight mode for highlighting the mobile unit for each mobile unit, and the display image generation unit 112 determines An image in which the moving body is highlighted in the highlighted display mode is generated, and the generated image is displayed on the display device 14.

In this way, the moving body can be highlighted in a manner that takes into consideration not only the index value relating to the ease of recognition of the moving body but also the driver's arousal level. For this reason, even when the driver feels tired or sleepy, the driver can recognize the moving body detected from the captured image.

In the above embodiment, the index value detection unit 106 detects the size of the moving body on the captured image as an index value related to the ease of recognition by the driver.

The size of the moving body on the photographed image is largely related to the ease of recognition by the driver and is easy to detect. Therefore, the size of the moving body on the captured image can be suitably used as an index value relating to the ease of recognition by the driver.

In the above embodiment, the index value detection unit 106 determines the contrast of the moving body (that is, an index related to the luminance difference between the portion where the moving body appears on the captured image and the background portion of the moving body) by the driver. It is detected as an index value related to ease of recognition.

The magnitude of the contrast of the moving object on the photographed image is greatly related to the ease of recognition by the driver, and is easy to detect. Therefore, the contrast of the moving body on the captured image can also be suitably used as an index value related to the ease of recognition by the driver.

Further, in the above-described embodiment, the display mode determination unit 110 may display the mobile body in an enlarged display mode by highlighting the mobile body.

If the moving object is enlarged and displayed, the moving object can be easily highlighted on the photographed image.

Further, in the above-described embodiment, the display mode determination unit 110 may use a display mode in which the contrast between the moving body and the background portion of the moving body is emphasized and displayed with the moving body highlighted.

If the contrast between the moving object and the background portion of the moving object is displayed with emphasis, the moving object can be easily highlighted on the captured image.

In the above embodiment, the display mode determination unit 110 corresponds to the classification of the moving object from the symbol image storage unit 114 in which a predetermined symbol image is stored for each classification of the moving object (for example, a pedestrian or a vehicle). The symbol image to be acquired may be acquired, and the image of the moving object in the captured image may be replaced with the symbol image corresponding to the type of the moving object and displayed. That is, the display mode determination unit 110 may change the display of the moving body image to the symbol image corresponding to the type of the moving body and may display the highlighted moving body.

In this way, it is possible to easily highlight the moving object on the captured image by setting the image that is easy for the driver to recognize as the symbol image.

(Modification):
In the above-described embodiment, when a moving object is present in the captured image, it is determined whether to highlight the moving object based on the recognition ease of the moving object detected from the captured image, and it is determined to highlight the moving object. It has been described that the highlighting mode in the case is also determined based on the recognition degree. However, the highlighting mode may be determined based on the size and contrast of the moving body (used to determine the recognition ease) instead of the recognition ease.

FIG. 16 shows a concept for determining the highlighting mode of the moving object in the captured image display process of the modification.

For example, as indicated by “first group” in the figure, for a moving body having a large moving body and low contrast, the main reason for the ease of recognition being below the allowable value is mainly low contrast. You can think that there is. Therefore, as shown by the white arrow in the figure, the highlighted display mode that mainly emphasizes the contrast is determined.

In the example shown in FIG. 5, the vehicle shown in the place indicated by “F” in the captured image corresponds to the moving body of the first group.

In addition, as indicated by “second group” in the figure, the reason why the ease of recognition falls below the permissible value is the reason for the moving object having a medium size and contrast. And it can be considered that both of them are in contrast. Therefore, as indicated by the white arrow in the figure, the highlighted display mode is selected so that the moving body is enlarged and the contrast is also emphasized. However, if the moving body is enlarged, the perspective is distorted, so it is desirable to keep the expansion rate of the moving body low.

In the example shown in FIG. 5, a pedestrian reflected in a place indicated as “B” or “E” in the photographed image corresponds to the moving body of the second group.

In addition, as indicated by “third group” in the figure, for a moving body having a small moving body and a high contrast, the reason why the ease of recognition is below the allowable value is small. You can think that there is. Accordingly, although the size of the moving body is enlarged, there is a limit to enlargement because the perspective is distorted if it is enlarged too much. Therefore, as shown by the white arrow in the figure, the moving object is enlarged, and the highlighted display mode is determined by compensating for the portion that is insufficient only by the enlargement of the moving object by enhancing the contrast.

In the example shown in FIG. 5, a pedestrian who appears in a place where “D” is displayed in the captured image corresponds to a moving body of the third group.

Further, as indicated by “fourth group” in the figure, for a moving body having a small moving body size and low contrast, it is determined as an emphasis display mode in which the image of the moving body is replaced with the symbol image 114s. This is because, if an attempt is made to prevent the perspective from being distorted, the recognition ease cannot be increased to a level exceeding the allowable value even if the moving body is enlarged and the contrast is enhanced.

In the example shown in FIG. 5, a pedestrian that appears in a place indicated by “C” in the captured image corresponds to a moving body of the fourth group.

Further, with respect to a moving object having a higher contrast than the “third group” in FIG. 16, a portion that is insufficient only by enlargement of the moving object cannot be compensated for by contrast enhancement. You may determine to the highlight display aspect replaced with.

In the modification described above, since the highlighting mode of the moving body can be determined according to the cause that the moving body is difficult to be recognized by the driver, the moving body is displayed on the screen of the display monitor 14. A captured image highlighted in a more appropriate manner can be displayed.

Although the present disclosure has been described based on the embodiments, it is understood that the present disclosure is not limited to the embodiments and structures. The present disclosure includes various modifications and modifications within the equivalent range. In addition, various combinations and forms, as well as other combinations and forms including only one element, more or less, are within the scope and spirit of the present disclosure.

Claims (7)

1. A moving body detection unit (104) for detecting at least one moving body from the captured image when a moving body is present in the captured image captured by the in-vehicle camera (10);
   An index value detection unit (106) for detecting an index value relating to ease of recognition by the driver for each detected moving body;
   An arousal level detection unit (108) for detecting the driver's arousal level;
   A display mode determination unit (110) for determining a highlight mode for displaying each mobile unit in an emphasized manner based on the index value of each mobile unit and the arousal level;
   A captured image display comprising: a display control unit (112) that generates an image displayed by emphasizing the captured image in the captured image based on the highlighted display mode of each movable object and displays the image on the display screen (14). apparatus.
2. The photographed image display device according to claim 1, wherein the index value detection unit (106) detects the size of each moving body on the photographed image as the index value.
3. The captured image display device according to claim 1, wherein the index value detection unit (106) detects, as the index value, a contrast between each moving body and a background portion of the moving body on the captured image. .
4. The photographed image display device according to any one of claims 1 to 3, wherein the display mode determination unit (110) determines a mode in which each moving body is displayed in an enlarged manner as the highlight display mode.
5. The display mode determination unit (110) determines, as the highlight display mode, a mode in which the contrast between each moving body and the background portion of the moving body is emphasized and displayed. The captured image display device described in 1.
6. A symbol image storage unit (114) for storing a predetermined symbol image for each category of the moving body;
   The display mode determination unit (110) determines, as the highlight mode, a mode in which an image of each moving body is replaced with the symbol image corresponding to the type of the moving body. The captured image display device according to claim 1.
7. At least one moving object is detected from a photographed image photographed by the in-vehicle camera (10),
   For each detected moving body, an index value relating to ease of recognition by the driver is detected,
   Detecting the driver's arousal level,
   Based on the index value of each moving body and the arousal level, the highlighting display mode for emphasizing and displaying each moving body is determined,
   A captured image display method comprising: generating, based on an emphasis display mode of each moving object, an image in which each moving object is emphasized and displayed in the captured image and displaying the generated image on a display screen (14).

Images