JP2014116756A - Periphery monitoring system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a periphery monitoring system which allows for discriminating between an actual scene and a body, in an image captured by a camera.SOLUTION: The periphery monitoring system comprises: imaging means 23 which includes the body of the vehicle and the periphery of the vehicle in an imaging range; boundary information storage means 37 which stores therein boundary information representing the boundary between the body and the periphery of the vehicle, in image data captured by the imaging means; image processing means 36 which on the basis of the boundary information applies boundary-clarifying processing to the image data; and display means 21 for displaying the image data on which the image processing means has applied the boundary-clarifying processing.

Description

  The present invention relates to a periphery monitoring system that captures and displays the periphery of a vehicle.

  In order to confirm the situation from the rear side of the vehicle to the rear side, door mirrors are mainly used as mirrors attached to the outside of the vehicle. As an alternative to this door mirror, a peripheral monitoring system that attaches the camera to the left and right fenders and displays the image on the display in the passenger compartment (hereinafter referred to as displaying an image captured by the camera on the display is called an electronic mirror) is being considered. Has been. Since the electronic mirror is attached in front of the door mirror, it is possible to capture and display a wider angle of view than the door mirror. For this reason, there exists an advantage that it becomes easy for a driver | operator to confirm back.

  FIG. 1 is a diagram illustrating an example of a mounting position of a camera for an electronic mirror in a vehicle. 1A shows a side view and FIG. 1B shows a front view. When viewed from the side, a camera for an electronic mirror is mounted on the upper right (about 2 o'clock direction) of the left front wheel fender with the optical axis facing rearward. A camera for an electronic mirror is also attached to the right front wheel at the same position.

JP 2009-012652 A

  However, in the case of the electronic mirror, there is a problem that the actual scene is reflected in the body depending on the mounting position of the camera, and it is difficult to distinguish between the actual scene and the reflected image reflected in the body.

  FIG. 2 shows an example of an image taken by a camera for an electronic mirror, but the reflected image reflected on the actual scene and the body is almost line symmetric, making it difficult to understand the boundary between the two. Even in the case of a door mirror, the actual scene may be reflected in the body, but the door mirror is placed at a higher position than the body and the body is photographed from diagonally above, so a line-symmetric reflection image is rarely reflected in the body. . On the other hand, since the optical axis of the camera of the electronic mirror is almost parallel to the body and the road surface, the reflected image reflected on the actual scene and the body is almost line-symmetric as shown in FIG. This phenomenon becomes more conspicuous as the camera mounting position of the electronic mirror is moved to the front of the vehicle and moved downward from the position of the side mirror to widen the shooting range of the electronic mirror camera.

  In addition, when the brightness in the screen is made uniform by the gain control function of the camera, the brightness difference between the actual scene and the reflected image becomes small, and the discrimination becomes more difficult.

  If the electronic mirror camera is installed so that the body does not enter, such inconvenience does not occur. However, it is difficult for the driver to install the vehicle so that it does not enter the vehicle because a part of the body is copied to determine how far the other vehicle is from the vehicle.

  In order for the driver to easily discriminate between the actual scene and the reflected image reflected in the body, the image may be corrected. A technique for correcting an image of a vehicle body portion in an image has been widely used (see, for example, Patent Document 1). In Patent Document 1, an image captured with a distorted lens is enlarged at a magnification that increases nonlinearly depending on the distance from the X-axis, so that the edge of the vehicle body and the edge of the vehicle body are A peripheral monitoring device having parallel straight lines is disclosed.

  However, the technique disclosed in Patent Document 1 does not consider the reflected image reflected on the vehicle body, and the problem that the reflected image reflected on the actual scene and the body cannot be easily distinguished cannot be solved.

  In view of the above problems, an object of the present invention is to provide a periphery monitoring system capable of discriminating a real scene and a body in an image taken by a camera.

  The present invention relates to boundary information in which boundary information indicating a boundary between the vehicle body and the periphery of the vehicle in image data captured by the imaging means includes image capturing means including the vehicle body and the vehicle periphery of the host vehicle. Storage means, image processing means for performing processing for clarifying the boundary on the image data based on the boundary information, and display for displaying the image data for which the image processing means has performed processing for clarifying the boundary And a peripheral monitoring system.

  It is possible to provide a periphery monitoring system capable of discriminating a real scene and a body from an image taken by a camera.

It is a figure which shows an example of the mounting position of the camera for electronic mirrors in a vehicle. It is a figure which shows an example of the image which the camera for electronic mirrors image | photographed. It is an example of the figure explaining the schematic characteristic of the periphery monitoring system in this embodiment. An example of a side view of the vehicle is shown. It is an example of the figure which shows the imaging range of a side camera and a fender camera. It is an example of the figure explaining the display on which the image image | photographed with the camera for electronic mirrors is displayed. It is a figure which shows an example of a schematic block diagram of a periphery monitoring system, and a functional block diagram of a camera computer. It is an example of the flowchart figure which shows the detection procedure of a boundary. It is an example of the figure which illustrates the detection of a boundary typically. It is a figure which shows an example of boundary information. It is an example of the figure which illustrates clarification of a boundary typically. It is an example of the figure which shows typically the imaging range of the camera for electronic mirrors, and a rear camera. It is a flowchart figure which shows an example of the operation | movement procedure of a periphery monitoring system.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. However, the technical scope of the present invention is not limited to this embodiment.

FIG. 3 is an example of a diagram illustrating the schematic features of the periphery monitoring system in the present embodiment.
(1) FIG. 3A is a diagram showing an uncorrected image taken by the electronic mirror camera. For such an image, the periphery monitoring system detects the boundary (or boundary line) between the body and the actual scene in an environment where the boundary between the body and the actual scene becomes clear. It has been found that this boundary becomes clear by adjusting the brightness of ambient light and the color temperature. The boundary information is stored in the nonvolatile memory of the vehicle.
(2) The periphery monitoring system performs image processing for clarifying the boundary from the boundary line to the image on the body side. This image processing is processing for enabling a driver or a passenger (hereinafter simply referred to as a driver) to easily grasp at least the outer edge of the body. For example, image processing such as lowering (darkening) the luminance value of the pixel on the body side, or painting the pixel value of the pixel on the body side with the same color as the body known in advance is suitable. FIG. 3B shows an example of an image that has been subjected to processing for reducing the luminance value of the pixel on the body side. As shown in FIG. 3B, the reflection of the actual scene on the body is reduced, and the driver can easily grasp the outer edge of the body.

  Therefore, according to the periphery monitoring system of the present embodiment, an image with a wide angle of view is displayed by the electronic mirror, and the driver can easily discriminate between the actual scene and the body.

[Example of mounting position]
Fig.4 (a) shows an example of the side view of a vehicle. The electronic mirror camera is preferably attached to the fender portion 11 in front of the vehicle. In some vehicles, the fender panel is an independent part, but in many vehicles it is the panel around the wheel that is integrated with the body. In the present embodiment, the fender portion 11 is referred to without particular distinction.

  The electronic mirror camera is fixed to the upper portion 11a, the rear portion 11b, or the front portion 11c of the fender portion 11. A camera with the same angle of view is preferable as it is attached to the front of the vehicle because the photographing range around the vehicle becomes wider. Further, the height at which the electronic mirror camera is mounted can be determined in consideration of the photographing range, the design and design of the vehicle, the dust resistance / water resistance, the ease of contamination of the lens, and the like. In principle, the fender portion 11 can be attached in the range from the lowermost portion to the uppermost portion. When attached to the fender portion 11 integrated with the body, the electronic mirror camera is fixed to the vehicle body via a bracket.

  In addition, as shown in FIG. 4B, in a vehicle equipped with the over fender 12, an electronic mirror camera can be embedded in the over fender 12. Further, if the above bracket is formed in the shape of the over fender 12, a sense of unity with the vehicle is obtained, which may be preferable in design.

  FIG. 4C is an example of a diagram showing a photographing range of the electronic mirror camera. The electronic mirror camera 23 is attached substantially symmetrically on the left and right of the front wheels. It is not always necessary to attach it to the left and right, and it is possible to attach it only on the side far from the driver's seat. Since it is attached in front of the door mirror, the photographing range around the vehicle is widened, and the driver can easily confirm the rear.

  In this embodiment, the case where the electronic mirror camera 23 is mounted on the fender unit 11 as shown in FIG. 4A will be described as an example. However, the image processing described later is mounted on the side mirror 13 or the fender mirror 14. It is also effective for images taken by cameras (hereinafter referred to as side cameras and fender cameras).

  FIG. 5A shows an example of a drawing showing the shooting range of the side camera, and FIG. 5B shows an example of a drawing showing the shooting range of the fender camera. The images taken by the side camera and the fender camera include a body and a real scene. The side camera and fender camera are less likely to cause the above-mentioned problems because the optical axis is less likely to be parallel to the road surface, but the driver can make the outer edge of the body easier by applying image processing to clarify the boundary of the image. It becomes possible to distinguish.

[Display arrangement]
FIG. 6 is an example of a diagram illustrating a display on which an image captured by the electronic mirror camera 23 is displayed. The display 21 is, for example, a flat panel display (liquid crystal, organic EL, plasma, etc.). The display 21 is disposed in a place that does not block the driver's view. Moreover, it is preferable that the driver seated in the driver's seat is disposed in a place where the line of sight movement is small. Examples of this include a display arranged on a center cluster or dashboard where the navigation device displays a road map. The center cluster display 21 is generally fixed to the vehicle, but the dashboard display may be fixed or portable. In this embodiment, an image taken by the electronic mirror camera 23 can be displayed on either display. In the case of a portable display, not only a display integrated with a portable navigation device but also a display of an information processing device such as a smartphone or a PDA is included. The vehicle and the information processing apparatus may communicate with each other wirelessly and transmit image data.

  In many cases, only one navigation display 21 is mounted. Therefore, when electronic mirror cameras are arranged on the left and right sides of the vehicle, the screen is divided to display respective images. When the electronic mirror camera is disposed only on the left side, an image may be displayed on the entire surface or a part of the display 21. Since the display 21 is shared with a road map, a television screen, a reproduced video, a menu display screen, etc., an arbitrary screen can be displayed by a driver's operation.

  Considering that the electronic mirror camera 23 provides an image of the rear side of the vehicle instead of the side mirror 13 (or together with the side mirror 13), it is preferable to dispose the display 21 near the side mirror 13. The driver can view the image captured by the electronic mirror camera 23 by moving the line of sight in the same direction as when moving the line of sight to the side mirror 13. In the figure, the display 21 is attached to the front pillar (A pillar). Further, the line-of-sight movement can be reduced by arranging the display on the dashboard near the front pillar instead of the front pillar. This display may be retractable or detachable according to the driver's preference, and the driver can check the rear side of the side mirror 13 and the electronic mirror camera 23 with a desired image.

  Further, the display 21 may be embedded in the side mirror 13. That is, the side mirror 13 becomes the display 21 itself. The driver can view the image taken by the electronic mirror camera 23 with the same line of sight movement as when moving the line of sight to the side mirror 13. Further, since this image is a wider range than the image reflected on the side mirror 13, the driver can see a wider range of images with the same line-of-sight movement as when viewing the side mirror 13.

  In addition, even if a projection display (for example, HUD (Head Up Display), projector) is used as a display device instead of a flat panel display (or with a flat panel display), an image captured by the electronic mirror camera 23 is displayed. Good. When the HUD is used as a display device, an image is displayed near the front pillar of the windshield glass or near the front pillar of the front door glass. As a result, the driver can view the image taken by the electronic mirror camera 23 with less line-of-sight movement in the same direction as when moving the line of sight to the side mirror 13 without occupying the interior space of the display 21.

[Function configuration example]
7A shows an example of a schematic configuration diagram of the periphery monitoring system 100, and FIG. 7B shows an example of a functional block diagram of the camera computer 22. The periphery monitoring system 100 includes an electronic mirror camera 23, a camera computer 22, and a display 21, and is controlled by the camera computer 22. The camera computer 22 is an information processing apparatus having a CPU, RAM, ROM, nonvolatile memory, and various I / Os. The camera computer 22 may be configured integrally with the electronic mirror camera 23, the camera computer 22 and the display 21 may be configured integrally, or the camera computer 22, the electronic mirror camera 23, and the display 21 are integrated. It may be configured.

  The electronic mirror camera 23 includes an image sensor having a predetermined number of pixels such as a CCD and a CMOS, an optical lens, a diaphragm, an image processing unit, and the like. Either a monochrome camera capable of capturing only luminance information or a color camera including a color filter may be used.

  The electronic mirror camera 23 and the camera computer 22 are connected to a control line 24 by a video transmission line 25 such as NTSC (National Television System Committee). The control line 24 is, for example, a LAN, CAN (Controller Area Network), Ethernet (registered trademark), a dedicated line, or the like for controlling an audiovisual device for a vehicle. The NTSC of the video signal is an example, and other analog signals such as PAL (Phase Alternating Line) and SECAM (sequential color memory), digital signals such as GVIF (Gigabit Video Interface), HDMI, and DVI may be transmitted. . Further, when transmitting as a digital signal, it may be compressed by a compression algorithm such as MPEG2 or H.264. The camera computer 22 and the display 21 are also connected by a control line 24 and a video transmission line 25. The connection method is not particularly limited as long as it is suitable for the specifications of the display 21, and may be the same as or different from the connection method of the camera computer 22 and the electronic mirror camera 23.

  In the figure, the electronic mirror camera 23, the camera computer 22, and the display 21 are connected by wire, but may be connected wirelessly. For example, connection is made by IEEE802.11a / bg / n / ac or Bluetooth (registered trademark).

  The display 21 has an operation unit 26 and accepts operations of a driver, a manufacturer developer, a serviceman, and the like. The operation unit 26 is a hard key arranged around the display 21, a touch panel integrated with the display 21, a soft key displayed on the touch panel, and the like. Further, the content of the utterance when the steering switch is pressed may be analyzed and the operation may be received by voice. Further, the movement of the driver's hand and the like may be photographed and analyzed by the camera, and an operation corresponding to the movement of the hand may be received.

  In the present embodiment, the operation unit 26 receives start / stop of the periphery monitoring system 100, an instruction operation for detecting a boundary, how to clarify the boundary, and the like. When these operable menus are displayed on the display 21, a driver, a service person, or the like may operate a soft key according to a guide message. If it is a hard key, it is activated by a start / stop button operation, and a long press of the same hard key or the like is used as a boundary detection instruction operation.

  As shown in FIG. 7B, the camera computer 22 includes a video conversion unit 31, a VRAM 33, a video processing unit 36, a communication processing unit 32, a video analysis unit 34, a control unit 35, and a nonvolatile memory 37. These functions are realized by the hardware and the CPU executing a program stored in the ROM and cooperating with the hardware.

  The video conversion unit 31 converts the video signal transmitted from the electronic mirror camera 23. If necessary, an analog signal is converted to a digital signal before conversion. This digital signal is converted into a format suitable for video processing. For example, the format is converted by changing the resolution or changing the gradation. Depending on the video signal transmitted by the electronic mirror camera 23, analog / digital conversion or format conversion may not be necessary.

  In addition, in a video signal transmitted in an interlaced manner, progressive image data is created by combining even fields and odd fields, or supplementing pixel values in odd fields and even fields, respectively. Also, the aspect ratio (for example, 4: 3) is converted into an aspect ratio (for example, 16: 9) suitable for the display 21. Further, the size (number of vertical and horizontal pixels) of the electronic mirror camera 23 is converted to a size suitable for the size of the display 21.

  A VRAM 33 (Video RAM) is a high-speed memory that stores image data for each frame of video transmitted from the electronic mirror camera 23 or image data for each frame converted from video by the video conversion unit 31. The VRAM 33 may be provided in the camera computer 22 as a dedicated memory, or a RAM used as a working memory by a program may be used.

  The video analysis unit 34 detects the boundary between the body and the real scene from the image data, and writes the boundary information in the nonvolatile memory 37. The nonvolatile memory 37 is an EEPROM or a flash memory. The detection of the boundary may be performed once at the time of shipment or when the mounting position of the electronic mirror camera 23 itself or the electronic mirror camera 23 is changed. This is because the electronic mirror camera 23 is fixed to the vehicle and does not change greatly during traveling. On the other hand, since the video analysis unit 34 is mounted on the vehicle, the boundary can be detected by an explicit operation by a driver or the like. The detection of the boundary can be performed in an environment where the boundary becomes clear. Since the clear environment is mainly determined by brightness, it is performed in a space where brightness can be controlled (factory factory, dealer service factory, etc.). What kind of brightness is preferable can vary depending on the body color, floor surface (ground where the vehicle is placed) and the color of the road surface, so manufacturers and service personnel can change the brightness of the lighting. To detect the boundary.

  In the nonvolatile memory 37, not only this boundary information, but also the specifications (focal length, angle of view, F value, etc.) of the electronic mirror camera 23, the mounting position (height and optical axis direction) of the electronic mirror camera 23, etc. Is memorized.

  The video processing unit 36 uses the boundary information stored in the nonvolatile memory 37 to perform image processing for clarifying the boundary on the image data in the VRAM 33. For example, the brightness of all pixels on the body side from the boundary is lowered by a certain amount, or a fixed value is written to all pixels on the body side from the boundary. By doing this, the actual scene reflected on the body side becomes inconspicuous.

  The control unit 35 controls the entire camera computer 22 by instructing the processing timing of the entire function. For example, when the operation unit 26 receives an operation of detecting a boundary, the boundary clearing mode is changed to the boundary detection mode, and the video analysis unit 34 is instructed to detect the boundary at the timing when the image data is stored in the VRAM 33. Further, in the boundary clarification mode such as immediately after activation of the periphery monitoring system 100, the video processing unit 36 is instructed to perform image processing for clarifying the boundary at the timing when the image data is stored in the VRAM 33. When controlling the electronic mirror camera 23, the control unit 35 causes the communication processing unit 32 to transmit a control signal.

  The communication processing unit 32 transmits a control signal to the electronic mirror camera 23 according to the communication protocol of the control line 24. In this embodiment, ON / OFF of gain control is mainly transmitted as a control signal. In addition, it is also possible to control the shutter speed and aperture that can be controlled by a general camera. The gain control is a process of amplifying the image signal output from the image sensor, and in the case of a digital camera, the amplification factor can be designated by photographing sensitivity (ISO).

[Boundary detection procedure]
FIG. 8 shows an example of a flowchart showing a boundary detection procedure, and FIG. 9 shows an example of a diagram schematically explaining boundary detection. The process of FIG. 8 starts when the operation unit 26 receives an instruction to detect a boundary while the periphery monitoring system 100 is activated. Note that the vehicle is stopped.

  The control unit 35 shifts from the boundary clarification mode to the boundary detection mode (S10). During this time, the control unit 35 displays a message such as “being detected boundary” on the display 21.

  The control unit 35 transmits a control signal for turning off gain control to the electronic mirror camera 23 (S20). The electronic mirror camera 23 transmits the image taken with the gain control OFF to the camera computer 22. The reason why the gain control is turned off is that when the gain is controlled to make the brightness of the actual scene and the body in which the actual scene is reflected, the boundary becomes unclear. Thus, by turning off the gain control, image data before the boundary becomes unclear can be obtained.

  In this embodiment, the gain control is turned off, but in addition to this, image processing in which the boundary becomes unclear may be turned off, or image processing in which the boundary becomes clear may be turned on. For example, the function of adjusting the aperture and shutter speed may be turned off or on, and the function of changing the dynamic range may be turned off or on.

  Next, the video analysis unit 34 starts detecting the boundary (S30). As shown in FIG. 9A, pixel rows arranged in the horizontal direction (x-axis direction) are referred to as lines, and are processed along the lines. In the present embodiment, when there is a difference (edge) that is equal to or greater than a threshold value between the luminance values adjacent in the horizontal direction, the difference is determined as the boundary position. In other words, since the difference in color and brightness between the real scene and the body appears as an edge at the boundary, the edge is detected.

  Here, the entire horizontal line may be scanned for the presence or absence of a luminance value difference equal to or greater than a threshold value, but the mounting position (the direction of the optical axis) of the electronic mirror camera 23 and the size and shape of the body are designed. It has been decided. Therefore, a calculation boundary can be obtained. Then, the boundary estimation range may be about several tens to several hundreds of pixels in the horizontal direction around the calculated boundary. By limiting the edge detection range, it is possible to suppress detection of edges other than the boundary.

  FIG. 9B is a diagram illustrating an example of the boundary estimation range obtained from the design value. The boundary estimation range can be stored in advance in the nonvolatile memory 37, but may be obtained by the camera computer 22 in the process of FIG.

  Further, instead of calculating from the design value, the boundary estimation range may be fixed such as “several hundred pixels horizontally from the center of the image data”. In this case, a band-like boundary estimation range is obtained in the vertical direction. “Left in the horizontal direction from the center” is based on the premise that the electronic mirror camera 23 is attached so that there is a boundary on the left side of the center, so “center” depends on the mounting position of the electronic mirror camera 23. It is preferable to change them. In either case, the calculation load of the boundary estimation range by the camera computer 22 can be reduced.

  Moreover, a driver, a service person, etc. may designate the boundary estimation range. That is, a driver, a serviceman, etc. display image data on the display 21 and operate the operation unit 26 to designate a boundary estimation range. For example, the belt-like boundary estimation range can be designated by tracing the outer edge of the boundary estimation range with a finger on the touch panel and selecting the upper left vertex and the lower right vertex of the boundary estimation range with the mouse. Since the boundary estimation range is visually specified, the boundary can be detected stably even when the attachment error is large.

  Returning to FIG. 8, the video analysis unit 34 sequentially obtains the difference in luminance between adjacent pixels in the boundary estimation range, and determines whether or not the threshold a is equal to or greater (S40). The threshold value a is experimentally determined in advance.

  If the difference in luminance between adjacent pixels is not greater than or equal to the threshold value a (No in S40), it is determined that the pixel is not a boundary, and the difference in luminance between adjacent pixels (dots) is obtained (S50). Then, the process returns to step S40, and the determination as to whether or not the difference in luminance between adjacent pixels is greater than or equal to the threshold value a is repeated.

  If the difference in luminance between adjacent pixels is greater than or equal to the threshold value a (Yes in S40), the video analysis unit 34 stores the coordinates of the pixels on either the left or right of the adjacent pixels (horizontal coordinates of the line of interest) in a RAM or the like. Keep it.

  And the process of step S40, S50 is performed about the line adjacent to the perpendicular direction (y-axis direction) (S60). Note that if no adjacent pixel having a luminance difference equal to or greater than the threshold value a is found in steps S40 and S50, the video analysis unit 34 provisionally determines the center pixel in the horizontal direction as a boundary.

  The video analysis unit 34 performs the processes of steps S40 and S50 up to the last line (S70). Therefore, the coordinates (x, y) of the pixels serving as the boundary as many as the number of lines are obtained.

  Next, the video analysis unit 34 performs error correction on the coordinates (x, y) of the pixel serving as the boundary (S80). FIG. 9C is an example for explaining error correction. Even if the luminance difference is greater than or equal to the threshold value a, the correct boundary may not be obtained due to a part that is locally reflected on the body, an edge due to body parts, lens contamination, noise, and the like. For this reason, the video analysis unit 34 compares the x coordinates of the coordinates (x, y) of the pixels that are adjacent borders in the vertical direction, and corrects the x coordinates if they are shifted by a threshold value b or more. As shown in the figure, when the x coordinate of the line of interest is separated by more than the threshold value b with respect to both the x coordinates of the adjacent upper line and the lower line, the x coordinate of the line of interest is corrected.

  The corrected x-coordinate is, for example, the average value of the x-coordinates of the boundary between the lines before and after the line of interest. In FIG. 9C, since the x coordinate of the boundary of the line before and after the line of interest is the same, the corrected x coordinate is the same as the x coordinate of the boundary of the preceding and following lines. In this way, even if an error is included in the boundary detection result, it can be corrected to a probable position.

  Further, when calculating the average, not only before and after the line of interest, but also a plurality of front side (for example, about 3 to 10) and a plurality of rear side averages may be used as corrected x coordinates. Further, instead of the average, the x coordinate of the boundary of the line immediately before the line of interest or the x coordinate of the boundary of the line immediately after the target line may be used.

  Error correction is performed on all of the coordinates (x, y) of the pixels serving as the boundary. By correcting the x-coordinate, there is a possibility that the x-coordinate of the uncorrected line boundary deviates from the x-coordinate of the preceding and following line boundaries by a threshold value b or more. For this reason, the video analysis unit 34 repeatedly performs error correction until the x coordinate of the coordinates (x, y) of the pixel serving as the boundary does not deviate more than the threshold value b with respect to the preceding and following lines.

  When the error correction is completed, one piece of boundary information is obtained (S90).

  The control unit 35 transmits a control signal for turning on gain control to the electronic mirror camera 23 (S100).

  Then, the video analysis unit 34 writes the boundary information in the nonvolatile memory 37 (S110). FIG. 10 is a diagram illustrating an example of boundary information. The boundary information is the x coordinate associated with the line number (same as the y coordinate) or the coordinates (x, y) of the pixel serving as the boundary sorted in ascending or descending order of the y coordinate.

  As described above, the camera computer 22 creates boundary information. Since the image data does not change greatly when the vehicle is stopped, boundary information may be created from only one frame. Further, the boundary information may be obtained in the same manner in a plurality of frames (for example, several frames to several tens or several hundred frames), and the average of the x coordinate may be calculated for each line to obtain the final boundary information.

[Image processing to clarify boundaries]
FIG. 11 is an example of a diagram for schematically explaining the clarification of the boundary. The video processing unit 36 performs image processing for clarifying the boundary as follows, for example.
(i) As shown in FIG. 11A, the brightness of all pixels on the body side of the boundary (or including the boundary) is lowered. The boundary with the actual scene becomes clear as the brightness on the body side decreases. The luminance may be always reduced by a certain amount, or the ambient light may be measured by an illuminance sensor, and the decrease amount may be increased or decreased according to the illuminance. When the display 21 displays black and white image data (0 to 255 levels of luminance values, where 0 is a black pixel and 255 is a white pixel), the luminance value is decreased. When the display 21 displays an RGB color image, the R value, the G value, and the B value are converted into luminance values as follows.
Y = 0.299R + 0.587G + 0.114B
0.299, 0.587, and 0.114 are the ratios of RGB with respect to the luminance (added to approximately 1), and by reducing the R value, G value, and B value, respectively, the luminance can be lowered without changing the color. Further, only the G value having a large ratio contributing to the luminance may be reduced, or the R value and the G value may be reduced.
(ii) As shown in FIG. 11B, a fixed value is written to all pixels on the body side (or including the boundary) rather than the boundary regardless of the original pixel value. The fixed value is, for example, black or a dark color that can be regarded as black, so that the boundary with the actual scene becomes clear. When the display 21 displays a monochrome image, for example, “0” is written as a fixed value. When the display 21 displays an RGB color image, (R, G, B) = (0, 0, 0) is written. Also, a luminance value or color that is the same as or close to the body color may be written. That is, if the body color is black, the above may be performed, and if it is white, “255”, (R, G, B) = (255, 255, 255) are written as the luminance value. In the case of gray, a luminance value or RGB value corresponding to gray of the body color density is written.

  Further, for example, when the body color has a saturation such as yellow and the display 21 displays an RGB color image, for example, it is a yellow RGB value (R, G, B) = (255, 255, 0). Write. When the body color has a saturation such as yellow, but the display 21 displays a black and white image, a luminance value obtained by converting the body color into luminance is written. As a result, the body displayed on the display 21 is displayed with the same brightness as the photographed body, so that the driver feels less uncomfortable.

  (iii) Also, it is not necessary to reduce the overall brightness of the body or write a fixed value to the entire body. In FIG. 11C, the processing of (i) or (ii) is performed only for pixels having a predetermined number of pixels on the body side from the boundary between the body and the actual scene. The area where the process (i) or (ii) is performed is hereinafter referred to as an edge area 40. The width of the edge region 40 may be equal to or larger than the width that can be visually recognized by the driver.

  In addition to the processing (i) or (ii), a white luminance value or RGB value that becomes high luminance may be written in the edge region 40. Further, the width of the edge region 40 may be reduced to a minimum and blinking (white pixels and black pixels are alternately switched). By flashing, the driver can easily grasp the boundary between the body and the actual scene even if the edge region 40 is narrow. Further, the pixel value of the edge region 40 may be processed so that the luminance and color gradually change from the boundary to the body side (gradation processing). Thereby, the edge region 40 becomes clear and the uncomfortable feeling that the edge region 40 exists can be reduced. Further, the gradation process is not limited to the edge region 40 and may be performed on the entire body.

  The driver can select from the operation unit 26 which method (i) to (iii) is used to emphasize the boundary. Alternatively, it can be set in the vehicle via a server from a portable terminal or PC (Personal Computer).

No processing may be performed on the body side from the edge region 40, or the following processing may be performed.
a. For example, the situation where the driver moves his eyes to the electronic mirror, such as when another vehicle is approaching from behind when the blinker switch is turned on, or there is a risk of abnormally approaching an obstacle detected by sonar Displays useful warning information.
b. If the character is small and difficult to read, a highly alert color such as red or yellow may be flashed.
c. Further, when the host vehicle is equipped with a rear camera that captures the rear, the image data captured by the rear camera may be combined with the image data captured by the electronic mirror camera 23.

  FIG. 12A is an example of a diagram schematically showing the photographing range of the electronic mirror camera 23 and the rear camera 16. The range in which the body is photographed with respect to the angle of view of the electronic mirror camera 23 is the blind spot of the electronic mirror camera 23. Although the vehicle B is photographed by the electronic mirror camera 23, the vehicle A is not reflected. On the other hand, the rear camera 16 is photographing the vehicle A in order to photograph a predetermined rear range including the blind spot. Therefore, the body side of the image data of the electronic mirror camera 23 can be supplemented with the image data of the rear camera 16.

  FIG. 12B is an example of a diagram illustrating the synthesis of image data. Image 1 shows image data of the rear camera 16, and image 2 shows image data of the electronic mirror camera 23. The video processing unit 36 performs processing for enhancing the edge region 40 on the image 1. Then, the blind spot portion 17 corresponding to the blind spot (body part) of the electronic mirror camera 23 is extracted from the image 1. The blind spot portion 17 is specified from the specification and attachment position of the electronic mirror camera 23 and the specification and attachment position of the rear camera 16. Alternatively, the driver or the like may arbitrarily select an appropriate synthesis position. Since the blind spot portion 17 is reduced or enlarged as compared with the image data of the electronic mirror camera 23 due to the difference in position and specification between the electronic mirror camera 23 and the rear camera 16, the video processing unit 36 performs size correction. Further, since the image quality such as brightness and color tone is different due to the difference in gain and shutter speed, the video processing unit 36 performs image quality correction. The blind spot portion 17 subjected to such image processing is synthesized with the image 2 (that is, the pixel value of the blind spot portion 17 subjected to the image processing is written in the body portion of the image 2).

  By using the rear camera 16 in this way, the driver can grasp the boundary between the body part and the actual scene from the edge region 40 and can grasp the existence of the vehicle other than the blind spot of the electronic mirror camera 23.

  Note that the driver can select from the operation unit 26 which image processing of ac is performed on the body part. Alternatively, it can be set in the vehicle via a server from a mobile terminal or a PC.

[Operation procedure]
FIG. 13 is a flowchart showing an example of the operation procedure of the periphery monitoring system 100. The process of FIG. 13 starts when the periphery monitoring system 100 is activated. Immediately after startup, the boundary is clarified.

  First, the video processing unit 36 reads a setting value from the nonvolatile memory 37 (S210). In this setting value, whether or not the boundary is clarified, how the boundary is clarified, and the image processing method in the range where the body is photographed are set.

  The electronic mirror camera 23 captures image data at a predetermined frame rate and sequentially transmits it to the camera computer 22 (S220).

  The control unit 35 determines whether or not an operation for shifting to the boundary detection mode is detected (S230). When the transition operation to the boundary detection mode is detected (Yes in S230), the video analysis unit 34 executes the process of the boundary detection mode described with reference to FIG. 8 (S250).

  When the transition operation to the boundary detection mode is not detected (No in S230), the video processing unit 36 performs a process of clarifying the boundary on the image data based on the boundary information read from the nonvolatile memory 37 (S240).

  As described above, the periphery monitoring system 100 according to the present embodiment can clearly identify the boundary between the actual scene and the reflected image reflected in the body even when the electronic mirror is mounted on the vehicle.

21 Display 22 Camera Computer 23 Electronic Mirror Camera 31 Video Conversion Unit 32 Communication Processing Unit 33 VRAM
34 Video analysis unit 35 Control unit 36 Video processing unit 37 Non-volatile memory 100 Perimeter monitoring system

Claims (8)

Photographing means including the body of the host vehicle and the vehicle periphery in the photographing range;
Boundary information storage means in which boundary information indicating the position of the boundary between the vehicle body and the periphery of the vehicle in the image data photographed by the photographing means is stored;
Image processing means for performing processing for clarifying the boundary on the image data based on the boundary information;
Display means for displaying the image data subjected to processing for clarifying the boundary by the image processing means;
A peripheral monitoring system characterized by comprising:   The image processing means reduces the luminance of the pixels in the range on the vehicle body side from the boundary of the image data, or sets a predetermined pixel value to the pixels in the range on the vehicle body side from the boundary of the image data. 2. The periphery monitoring system according to claim 1, wherein writing is performed.   The image processing means performs a process of reducing luminance or a process of writing a predetermined pixel value only to a part of pixels in the range on the vehicle body side including the boundary of the image data. The perimeter monitoring system according to claim 1.   The periphery monitoring system according to claim 3, wherein the image processing means displays warning information on pixels in a range on the vehicle body side other than the part.   The image processing unit synthesizes image data of a range that is captured by a rear imaging unit that captures the rear and is in a blind spot by the vehicle body, with pixels in the range on the vehicle body side other than the part. The periphery monitoring system according to claim 3. A boundary detection means for specifying a horizontal position in association with a line number when there is an edge strength equal to or greater than a first threshold in the horizontal direction, with the horizontal pixel row of the image data as one line,
The boundary detection unit corrects the horizontal position of the line of interest based on the horizontal position of the preceding and following lines when the horizontal position of the line of interest is shifted by a second threshold or more with respect to the horizontal position of the previous and subsequent lines. The perimeter monitoring system according to any one of claims 1 to 5.   The periphery monitoring system according to claim 6, further comprising a control unit that sets a gain control function of the photographing unit to OFF while the boundary detection unit detects the boundary.   The perimeter monitoring system according to claim 1, wherein the imaging unit is attached to a fender portion of a front wheel with an optical axis directed from the rear to the rear side.

Images