WO2020017230A1 - 電子制御装置及び電子制御方法 - Google Patents

電子制御装置及び電子制御方法 Download PDF

Info

Publication number
WO2020017230A1
WO2020017230A1 PCT/JP2019/024497 JP2019024497W WO2020017230A1 WO 2020017230 A1 WO2020017230 A1 WO 2020017230A1 JP 2019024497 W JP2019024497 W JP 2019024497W WO 2020017230 A1 WO2020017230 A1 WO 2020017230A1
Authority
WO
WIPO (PCT)
Prior art keywords
notification target
display
image
unit
captured image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2019/024497
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
鈴木　健介
健志 仲村
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Denso Corp
Original Assignee
Denso Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Denso Corp filed Critical Denso Corp
Publication of WO2020017230A1 publication Critical patent/WO2020017230A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

  • the disclosure in this specification relates to an electronic control device and an electronic control method.
  • Patent Document 1 discloses an object detection device in which a camera provided in a vehicle captures an image of a rear area of the vehicle, and a captured image captured by the camera is displayed on a display provided in a room of the vehicle.
  • an object present in a rear area of the vehicle is detected using a plurality of captured images, and a moving object and a vehicle to be notified to an occupant of the vehicle are detected from the detected objects.
  • a stationary object that is likely to move is selected.
  • the captured image and a frame for emphasizing the notification target in the captured image are superimposed and displayed on the display.
  • the photographed image is to be displayed on the display in the vehicle cabin
  • the smaller the display area of the display the smaller the notification target existing in the photographed image is displayed on the display.
  • an object to be disclosed is to provide an electronic control device and an electronic control method that can easily recognize a notification target.
  • An electronic control device is a notification target that detects a notification target to be notified to an occupant of a moving body, which is present in the captured image, based on a shot image captured by a shooting unit provided on the moving body.
  • a detection unit a display unit that displays a captured image provided in the room of the moving object, a size calculation unit that calculates a size at which the notification target detected by the notification target detection unit is displayed, and a size calculation unit.
  • a display control unit that, when the calculated size is equal to or smaller than the predetermined threshold, causes the display unit to display an enlarged image obtained by enlarging the specific notification target that is the notification target whose size is equal to or smaller than the predetermined threshold.
  • an electronic control method detects a notification target to be notified to an occupant of a moving body, which is present in the captured image, based on a shot image taken by a shooting unit provided on the moving body. And calculating a size on which a notification target detected by the detection is displayed on a display unit that displays a captured image provided in the room of the moving object, and a size calculated by the calculation. Is smaller than or equal to a predetermined threshold, and an enlarged image obtained by enlarging the specific notification target whose size is equal to or smaller than the predetermined threshold is displayed on the display unit.
  • the size of the notification target existing in the captured image and to be notified to the occupant is calculated on the display unit.
  • the display unit displays an enlarged image obtained by enlarging the specific notification target that is the notification target whose size is equal to or smaller than the predetermined threshold.
  • FIG. 1 is a schematic configuration diagram of an electronic control system according to a first embodiment
  • FIG. 2 is a block diagram illustrating a configuration of the electronic control device according to the first embodiment.
  • FIG. 3 is an example when a captured image, a support image, and a normal emphasized image are displayed
  • FIG. 4 is an example when a captured image, a support image, and a captured enhancement image are displayed
  • FIG. 5 is an example when an enlarged image is displayed
  • FIG. 6 is an example when an enlarged and emphasized image is displayed
  • FIG. 7 is a flowchart showing a display control process according to the first embodiment
  • FIG. 3 is an example when a captured image, a support image, and a normal emphasized image are displayed
  • FIG. 4 is an example when a captured image, a support image, and a captured enhancement image are displayed
  • FIG. 5 is an example when an enlarged image is displayed
  • FIG. 6 is an example when an enlarged and emphasized image is displayed
  • FIG. 7 is a flowchar
  • FIG. 8 is a block diagram illustrating a configuration of an electronic control device according to the second embodiment.
  • FIG. 9 is an example of displaying a captured image, a support image, an enlarged image, and an enlarged emphasized image according to the second embodiment
  • FIG. 10 is a flowchart showing a display control process according to the second embodiment
  • FIG. 11 is a flowchart showing a display control process according to the second embodiment
  • FIG. 12 is an example when the captured image, the support image, the enlarged image, and the enlarged emphasized image according to the first modification are displayed.
  • FIG. 13 is a block diagram illustrating a configuration of an electronic control device according to the third modification.
  • the moving body is the vehicle 100, but the moving body may be a train, a ship, an airplane, or the like.
  • FIG. 1 is a schematic configuration diagram of an electronic control system 1 according to the first embodiment.
  • the electronic control system 1 includes an electronic control device 2, an external camera 3 that captures an image around the vehicle 100, and a display 4.
  • the vehicle 100 is equipped with the electronic control unit 2, the camera 3 outside the vehicle, and the display 4.
  • the outside camera 3 is attached to the front and rear of the vehicle 100 in order to photograph the periphery of the vehicle 100, and photographs the front area and the rear area of the vehicle 100.
  • a captured image 40 captured by the camera 3 outside the vehicle is input to the electronic control device 2.
  • This photographed image 40 is composed of a set of a plurality of pixels.
  • the installation position of the camera 3 outside the vehicle may be either one of the front and rear of the vehicle 100, and the photographing area may be one of the front area and the rear area.
  • the camera 3 outside the vehicle corresponds to a photographing unit.
  • the electronic control device 2 outputs the captured image 40 in which the front area is captured to the display 4, and the vehicle 100 moves backward.
  • the captured image 40 obtained by capturing the rear area is output to the display 4. That is, the captured image 40 is displayed on the display 4.
  • FIG. 2 is a block diagram illustrating a configuration of the electronic control device 2 according to the first embodiment.
  • FIG. 2 shows only components necessary for explaining the features of the first embodiment, and omits general components.
  • the electronic control unit 2 is connected to the camera 3 outside the vehicle, the display 4, the indoor input unit 7, and the steering angle sensor 8.
  • the display 4 is a display that can be input by a touch operation, and has a touch panel 70.
  • the touch panel 70 is provided on a display surface of the display 4 that displays an image.
  • the display 4 corresponds to a display unit.
  • the input may be performed by providing a button (not shown) on the display 4.
  • the indoor input unit 7 is a haptic device, lever, or button that is provided inside the vehicle 100 and that can be operated by an occupant of the vehicle 100.
  • selection of content such as audio and navigation displayed on the display 4 and selection of a specific notification target 23 described later can be performed.
  • the steering angle sensor 8 is attached to a steering shaft (not shown) of the vehicle 100, and measures a rotation angle of a steering wheel.
  • the electronic control unit 2 includes a control unit 5 and a storage unit 6.
  • the storage unit 6 is, for example, a semiconductor memory device such as a RAM (Random Access Memory), a flash memory, or a storage device such as a hard disk or an optical disk, and stores the captured image 40. Further, the storage unit 6 stores in advance the resolution of the captured image 40 captured by the external camera 3, the resolution of the display 4, the resolution ratio between the captured image 40 and the display 4 described later, and the pixel pitch of the display 4. .
  • a semiconductor memory device such as a RAM (Random Access Memory), a flash memory, or a storage device such as a hard disk or an optical disk
  • the pixel pitch is a distance between a pixel and an adjacent pixel on the display 4 and indicates a size per pixel.
  • the storage unit 6 stores the pixel pitch in the width direction of the display 4 and the pixel pitch in the height direction of the display.
  • the control unit 5 is, for example, a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), and controls the entire electronic control device 2.
  • the control unit 5 includes an input detection unit 51, an image acquisition unit 54, a course estimation unit 55, an object detection unit 56, a notification target detection unit 57, a size calculation unit 58, and a display control unit 59. ing.
  • the input detection unit 51 detects an input to the indoor input unit 7 and the touch panel 70.
  • the input detection unit 51 detects, for example, coordinates on the touch panel 70 touched by an occupant as inputs to the touch panel 70. Then, the input detection unit 51 outputs information regarding the detected input to the selection determination unit 60 included in the display control unit 59 described below as input information.
  • the image acquisition unit 54 is, for example, an input port, and acquires the captured image 40 captured by the camera 3 outside the vehicle. Note that the external camera 3 outputs the captured image 40 to the image acquisition unit 54 according to the set frame rate. Then, the image acquisition unit 54 outputs the acquired captured image 40 to the display control unit 59 and the storage unit 6.
  • the course estimation unit 55 acquires the steering angle of the steering wheel detected by the steering angle sensor 8.
  • the course estimating unit 55 estimates the course of the vehicle 100, that is, the predicted trajectory, based on the steering angle of the steering wheel. Then, the course estimation unit 55 outputs predicted trajectory information including the estimated predicted trajectory to the notification target detection unit 57.
  • the object detection unit 56 detects an object such as a stationary object and a moving object based on the captured image 40 stored in the storage unit 6. Specifically, the object detection unit 56 extracts an optical flow, which is a movement vector connecting the same feature points, from the captured images 40 of a plurality of frames arranged in a time series acquired from the storage unit 6, and extracts the extracted optical flow. The object is detected based on.
  • an optical flow which is a movement vector connecting the same feature points
  • the object detection unit 56 detects a stationary object and a moving object as objects.
  • the object detection unit 56 detects the upper, lower, left, and right ends of the detected object as information necessary for the size calculator 58 to calculate later. Then, the object detection unit 56 forms a rectangle including the detected upper, lower, left, and right ends, and detects the number of pixels in the width direction and the height direction of the formed rectangle.
  • the object detection unit 56 may detect the contour of the detected object and detect the number of pixels in a region surrounded by the contour as information necessary for the size calculation unit 58 described later to calculate. .
  • the object detection unit 56 associates the information indicating whether the detected object is a stationary object or a moving object with the information indicating the number of pixels in the width direction and the height direction of the formed rectangle, and obtains the object information. , To the notification target detection unit 57.
  • the notification target detection unit 57 notifies the occupant from the objects detected by the object detection unit 56 based on the predicted trajectory information output from the route estimation unit 55 and the object information output from the object detection unit 56.
  • the notification target to be notified is detected.
  • the notification target is different between the first embodiment and a second embodiment to be described later.
  • the notification target in the first embodiment is one of a moving object and a stationary object that may come into contact with the vehicle 100, This is the closest object.
  • Notification targets in a second embodiment described later are all moving objects and stationary objects that may come into contact with the vehicle.
  • an object closest to the vehicle 100 may be set as the notification target, or an object closest to the vehicle 100 among the stationary objects that may contact the vehicle 100 may be set as the notification target. It may be.
  • the notification target detection unit 57 detects, among the stationary objects detected by the object detection unit 56, a stationary object existing in the predicted trajectory as a stationary object that may come into contact with the vehicle 100. Then, the notification target detection unit 57 determines, from among the stationary objects that may contact the detected vehicle 100 and the moving objects detected by the object detection unit 56, the object that is closest to the vehicle 100. Detected as
  • the method of measuring the distance between the object and the vehicle 100 includes a first method of measuring the distance to the object from the parallax between the left and right cameras using a stereo camera, and a photographed image 40 based on the angle and the installation position of the camera 3 outside the vehicle. There is a second method in which a distance to the vehicle 100 is set for each coordinate.
  • the outside camera 3 is a stereo camera
  • the notification target detection unit 57 captures images of the detected moving object and the stationary object that may come into contact with the vehicle 100 by the left and right cameras, respectively.
  • the distance to the vehicle 100 is measured from the parallax of the captured image 40.
  • the outside camera 3 is a stereo camera
  • two cameras photograph the front and rear of the vehicle 100. In the above-described extraction of the optical flow, the photograph was taken by one of the left and right cameras.
  • a plurality of captured images 40 may be used.
  • the storage unit 6 stores distance information in which the distance to the vehicle 100 is set in advance for each coordinate of the captured image 40.
  • the notification target detection unit 57 measures the distance to the vehicle 100 by comparing the distance information stored in the storage unit 6 with the coordinates of the moving object and the static object in the captured image 40.
  • the notification target detection unit 57 may perform the first contact with the vehicle 100 among the moving object and the stationary object that may contact the vehicle. Objects with a high level are detected. Thus, the driver can instantaneously determine a dangerous object.
  • the electronic control system 1 using the first method will be described.
  • the notification target detection unit 57 outputs the notification target information having the object information on the detected notification target and the predicted trajectory information received from the route estimation unit 55 to the size calculation unit 58.
  • the notification target information includes position information including a distance between the notification target and the vehicle 100, information indicating whether the notification target is a stationary object or a moving object, and upper, lower, left, and right ends of the notification target. This is information relating a rectangle to information indicating the number of pixels in the width direction and the height direction.
  • the size calculation unit 58 calculates the size of the notification target detected by the notification target detection unit 57 displayed on the display 4.
  • the size is the length in the width direction and the height direction of a rectangle including the upper, lower, left, and right ends of the notification target displayed on the display 4.
  • the size of the notification target displayed on the display 4 may be a rectangular area including the upper, lower, left, and right ends of the notification target displayed on the display 4.
  • the size calculation unit 58 calculates the resolution of the captured image 40 captured by the external camera 3 stored in the storage unit 6, the resolution of the display 4, and the resolution ratio between the captured image 40 and the display 4 described later. , And the pixel pitch of the display 4.
  • the size calculation unit 58 calculates the number of pixels in the width direction and height direction of the rectangle including the upper, lower, left, and right ends of the notification target acquired from the notification target detection unit 57, and the captured image 40 acquired from the storage unit 6.
  • the size at which the notification target is displayed on the display 4 is calculated from the resolution ratio of the display 4 and the pixel pitch of the display 4.
  • the resolution ratio here is the ratio of the number of pixels between the long side of the display 4 and the long side of the captured image 40, or the ratio of the number of pixels between the short side of the display 4 and the short side of the captured image 40.
  • the reason for using the resolution ratio is to display the captured image 40 on the display 4 as much as possible.
  • the resolution of the captured image 40 is 1280 ⁇ 1024 and the resolution of the display 4 is 800 ⁇ 480, the resolution of the captured image 40 and the display 4 are different. Is displayed on the display 4 in a state in which both the width direction and the height direction of the captured image 40 are partially cut off.
  • the notification target in the photographed image 40 may be cut off.
  • the captured image 40 is reduced in accordance with the ratio of the number of pixels between the long side of the captured image 40 and the long side of the display 4, a part of the captured image 40 in the height direction can be cut off.
  • the captured image 40 is displayed on the display 4 without being cut off in the direction.
  • the resolution ratio shown in Expressions 1 and 2 is the ratio of the number of pixels between the long side of the captured image 40 and the long side of the display 4 (1.6).
  • the width direction of the captured image 40 is reduced by the ratio of the number of pixels on the long side of the display 4 to the captured image. Then, the height direction of the captured image 40 may be reduced according to the ratio of the number of pixels of the captured image 40 and the short side of the display 4.
  • the resolution ratio shown in Expression 1 is the ratio of the number of pixels between the long side of the captured image 40 and the long side of the display 4 (1.6)
  • the resolution ratio shown in Expression 2 is the short side of the captured image 40
  • the size calculation unit 58 outputs to the display control unit 59 the size information, which is the information on the calculated size of the notification target, and the notification target information and the predicted trajectory information received from the notification target detection unit 57.
  • the operation of the display control unit 59 will be described. Since the notification target detected by the notification target detection unit 57 is different between the first embodiment and the second embodiment, the operation of the display control unit 59 is also the first embodiment. The embodiment differs from the second embodiment.
  • the display control unit 59 has two functions.
  • the first function is a function of determining whether a specific notification target 23 described later and a normal notification target 22 described later exist.
  • the second function is to display an image on the display 4.
  • the display control unit 59 controls the notification target calculated by the size calculation unit 58 based on the notification target information that is the detection result of the notification target detection unit 57 and the size information that is the calculation result of the size calculation unit 58. It is determined whether there is a specific notification target 23 that is a notification target whose size is equal to or smaller than a predetermined threshold.
  • the display control unit 59 determines that the normal notification target 22 whose notification target size is larger than the predetermined threshold is present.
  • the predetermined threshold value is a total value of the size in the width direction and the size in the height direction, for example, 60 mm, and the width direction and the height of the notification target in the captured image 40 displayed on the display 4. If the size in the vertical direction is 30 mm or less, it is determined to be a specific notification target.
  • the predetermined threshold is not limited to the above description.
  • the predetermined threshold value that is determined to be the normal notification target 22 is a size that allows the occupant of the vehicle 100 to easily recognize the notification target in the captured image 40 in a situation where the captured image 40 is displayed on the display 4. Any value as long as it is indicated.
  • the predetermined threshold value determined to be the specific notification target may be a value indicating a size in which the notification target cannot be recognized unless the display 4 is watched in the situation where the captured image 40 is displayed on the display 4.
  • the predetermined threshold value is the size in the width direction, It is a value obtained by multiplying the size in the height direction.
  • the display control unit 59 acquires the captured image 40 captured by the camera 3 outside the vehicle via the image acquisition unit 54.
  • the display control unit 59 converts the captured image 40, a later-described support image, a later-described normal enhanced image 41, a later-described captured enhanced image 42, a later-described enlarged image 43, and a later-described enlarged emphasized image 44, respectively. Display on the display 4 under different conditions.
  • the display pattern of the display control unit 59 will be described below with reference to FIGS. FIGS. 3 to 6 show images displayed on the display 4 when the vehicle 100 is moving backward.
  • FIG. 3 is an example when the captured image 40, the support image, and the normal emphasized image 41 are displayed on the display 4.
  • the support image is an image that is displayed on the display 4 and assists the occupant in operating the vehicle 100, and is, for example, an image including the prediction line 20 and the vehicle width line 21.
  • the normal emphasis image 41 is an image that emphasizes the normal notification target 22 in the captured image 40.
  • the display control unit 59 generates the prediction line 20 as the support image based on the predicted trajectory included in the predicted trajectory information, and uses the vehicle width line 21 as the support image based on the width of the vehicle 100 stored in the storage unit 6 in advance.
  • the prediction line 20 is a boundary line of a prediction trajectory of the vehicle 100 in a belt shape, and is, for example, a pair of lines.
  • the vehicle width line 21 is a line indicating the vehicle width when the vehicle 100 is moving forward or backward straight when the steering wheel of the vehicle 100 is at a fixed position. Note that the width of the pair of prediction lines 20 corresponds to the width of the vehicle 100, but can be adjusted to be wider than the width of the vehicle 100.
  • the display control unit 59 generates the normal emphasized image 41 that emphasizes the normal notification target 22 in the captured image 40 based on the notification target information.
  • the normal emphasized image 41 is a rectangular frame including the upper, lower, left, and right ends of the notification target. Note that the normal emphasized image 41 may be a line surrounding the outline of the notification target. Also, the line surrounding the rectangular frame or the outline of the notification target may be a broken line.
  • the display control unit 59 determines that the normal notification target 22 exists as the first function, the display control unit 59 superimposes the captured image 40, the support image, and the normal emphasized image 41 on the display 4.
  • the normal emphasized image 41 may not be displayed.
  • the reason is that the normal notification target 22 is displayed larger than a predetermined threshold value on the display 4, and therefore, it is highly likely that the occupant can easily recognize the normal notification target 22 without emphasizing it.
  • FIG. 4 is an example when the captured image 40, the support image, and the captured enhanced image 42 are displayed on the display 4.
  • the display control unit 59 generates the photographing emphasized image 42 that emphasizes the specific notification target 23 in the photographed image 40 differently from the normal emphasized image 41 based on the notification target information.
  • the photographed emphasized image 42 is a rectangular frame including the upper, lower, left, and right ends of the notification target.
  • a color different from that of the normal emphasized image 41 a solid line or a broken line is used.
  • the photographing emphasized image 42 may be a line surrounding the outline of the notification target.
  • the display control unit 59 determines that the specific notification target 23 exists as the first function, the display control unit 59 superimposes and displays the captured image 40, the support image, and the captured enhancement image 42 on the display 4.
  • FIG. 5 is an example when the captured image 40, the support image, the captured enhanced image 42, and the enlarged image 43 are displayed on the display 4.
  • the display control unit 59 includes a selection determination unit 60 that determines whether or not the occupant has selected the specific notification target 23 displayed on the display 4 based on the input information detected by the input detection unit 51. I have.
  • the selection determination unit 60 is displayed on the display 4. It is determined that the specific notification target 23 has been selected.
  • the display control unit 59 determines whether the occupant is displayed on the display 4 by the selection determining unit 60 in a state where the captured image 40 is displayed on the display 4 by the pattern 1 or the pattern 2.
  • an enlarged image 43 obtained by enlarging a predetermined area 45 including the selected specific notification target 23 is displayed on the screen of the display 4 together with the captured image 40.
  • the display control section 59 superimposes and displays the enlarged image 43 on the captured image 40 displayed on the display 4 as shown in FIG. At this time, when superimposing the display, the display control unit 59 displays the enlarged image 43 on the display 4 so as not to cover the specific notification target 23 present in the captured image 40 displayed on the screen of the display 4. Is preferred.
  • the enlarged image 43 and the captured image 40 may be displayed on the display 4 without overlapping, or the enlarged image 43 may be displayed on the entire surface of the display 4.
  • the display control unit 59 changes the display position of the enlarged image 43 based on the predicted trajectory information estimated by the course estimation unit 55. Specifically, when the enlarged image 43 is superimposed on the captured image 40, the display controller 59 enlarges the enlarged image 43 in the left-right direction opposite to the direction of the prediction line 20 estimated by the path estimation unit 55. 43 is displayed. This is to reduce the overlap between the predicted trajectory determined by the pair of prediction lines 20 and the enlarged image 43.
  • the display control unit 59 converts the enlarged image 43 into the left and right ends of the captured image 40. One of them is displayed.
  • the display controller 59 may cause the display 4 to display the enlarged image 43 when the size calculated by the size calculator 58 is equal to or smaller than a predetermined threshold value, regardless of whether or not the occupant is selected.
  • the ⁇ predetermined area 45 is an area that is extended up, down, left, and right by a predetermined number of pixels with respect to a rectangle including the upper, lower, left, and right ends of the specific notification target 23 selected by the passenger. It is preferable that the predetermined area 45 is an area that is wider in the left-right direction than in the vertical direction. For example, with respect to a rectangle including the upper, lower, left, and right ends of the notification target, the pixel is expanded by 10 pixels in the vertical direction and 30 pixels in the horizontal direction.
  • the predetermined area 45 may be an area surrounding the outline of the specific notification target 23.
  • magnification for enlarging the predetermined area 45 may be appropriately set.
  • the predetermined area 45 is enlarged so as to be 1 / of the total area of the display 4.
  • a fourth pattern in which the display control unit 59 causes the display 4 to superimpose the enlarged image 43 and the enlarged emphasized image 44 on the display 4 will be described with reference to FIG.
  • the magnified and emphasized image 44 is an image that emphasizes the specific notification target 23 in the magnified image 43.
  • the enlarged and emphasized image 44 is an image obtained by enlarging the photographed and emphasized image 42 at the same ratio as the enlargement ratio at which the predetermined area 45 including the specific notification target 23 is enlarged. Therefore, the enlarged emphasized image 44 is a rectangular frame including the upper, lower, left, and right ends of the specific notification target 23 in the enlarged image 43.
  • the enlarged emphasized image 44 may be a line surrounding the outline of the specific notification target 23 in the enlarged image 43. Further, the line surrounding the outline of the specific notification target 23 in the rectangular frame or the enlarged image 43 may be a broken line.
  • FIG. 7 is a flowchart illustrating a procedure of a display control process performed by the control unit 5 of the first embodiment. Note that “S” in the flow of the display control processing means each step.
  • This display control process is started when the shift lever is put into reverse or when the shift lever is put into drive.
  • the process may be started when the vehicle 100 is moving forward, when the vehicle 100 is moving backward, or when the ignition of the vehicle 100 is turned on.
  • step S1 the image acquisition unit 54 acquires the captured image 40 captured by the camera 3 outside the vehicle, and proceeds to step S2. At this time, the image acquisition unit 54 outputs the acquired captured image 40 to the display control unit 59 and the storage unit 6.
  • the object detection unit 56 detects an object such as a stationary object and a moving object based on the captured image 40 stored in the storage unit 6. Then, if no object is detected, the object detection unit 56 ends the processing. On the other hand, when detecting the object, the object detection unit 56 outputs the object information to the notification target detection unit 57, and proceeds to S3.
  • the course estimation unit 55 calculates a predicted trajectory based on the steering angle of the steering wheel, and shifts to S4.
  • the notification target detection unit 57 selects one of the objects detected by the object detection unit 56. A notification target to be notified to the occupant is detected. Then, when the notification target detection unit 57 does not detect the notification target, the processing ends. On the other hand, when detecting the notification target, the notification target detection unit 57 outputs the notification target information and the predicted trajectory information to the size calculation unit 58, and shifts to S5.
  • the size calculator 58 calculates the size of the notification target displayed on the display 4, and proceeds to S6. At this time, the size calculation unit 58 outputs to the display control unit 59 the size information that is the information on the calculated size of the notification target, the notification target information, and the predicted trajectory information.
  • the display control unit 59 determines whether or not the specific notification target 23 exists based on the size information output from the size calculation unit 58 and the notification target information. When the specific notification target 23 does not exist, the display control unit 59 determines that the normal notification target 22 exists, and shifts to S7. On the other hand, when the display control unit 59 determines that the specific notification target 23 exists, the process proceeds to S8.
  • the display control unit 59 superimposes the captured image 40, the normal emphasized image 41, and the support image on the display 4, and ends the process.
  • the display control unit 59 superimposes the captured image 40, the captured enhanced image 42, and the support image on the display 4, and proceeds to S9.
  • the selection determination unit 60 determines whether or not the occupant has selected the specific notification target 23 displayed on the display 4 based on the input information detected by the input detection unit 51. When determining that the occupant has selected the specific notification target 23, the selection determination unit 60 proceeds to S10. When the selection determination unit 60 determines that the specific notification target 23 has not been selected, the process ends.
  • the display control unit 59 causes the display 4 to display an enlarged image 43 in which a predetermined area including the selected specific notification target 23 is enlarged, and proceeds to S11.
  • the display control unit 59 causes the display 4 to superimpose and display the enlarged image 43 and the enlarged emphasized image 44.
  • the display control section 59 determines whether or not the shift lever is reverse. If the shift lever is reverse, the display control process is continued, and if the shift lever is no longer reverse, the process ends. Note that the timings of the processes of S10 and S11 may be aligned. Further, the processing in S11 may be omitted.
  • the display control unit 59 causes the display 4 to display the enlarged image 43 and the enlarged emphasized image 44 for a certain period of time.
  • the certain time here may be a time during which the occupant can recognize the specific notification target 23 in the enlarged image 43, and is, for example, 5 seconds.
  • the size of the notification target present in the captured image 40 displayed on the display 4 is calculated.
  • the display 4 displays an enlarged image 43 obtained by enlarging the specific notification target 23 whose size is equal to or smaller than the predetermined threshold.
  • the specific notification target 23 displayed on the display 4 with a size equal to or smaller than the predetermined threshold is displayed on the display 4 in an enlarged state, so that the occupant can easily recognize the notification target.
  • the enlarged image 43 is an image obtained by enlarging a predetermined area 45 including the specific notification target 23. Thereby, the situation around the specific notification target 23 can be easily recognized.
  • the display control unit 59 sets the specific notification target 23 in the captured image 40 determined to be selected by the selection determination unit 60 as the enlarged image 43 together with the captured image 40 on the screen of the display 4. Display.
  • the specific notification target 23 selected by the occupant on the display 4 as the enlarged image 43, it is possible to prevent the enlarged image 43 from being displayed on the display 4 unnecessarily.
  • the enlarged image 43 and the captured image 40 are displayed together on the screen of the display 4, it is easy to recognize which area of the captured image 40 is enlarged.
  • the display control unit 59 causes the display 4 to display the enlarged image 43 so as not to cover the specific notification target 23 existing in the captured image 40 displayed on the screen of the display 4. Accordingly, the specific notification target 23 existing in the captured image 40 displayed on the screen of the display 4 and the specific notification target 23 in the enlarged image 43 are both displayed on the display 4. The occupant can easily grasp the positional relationship between the specific notification target 23 existing in the captured image 40 displayed on the screen 4 and the specific notification target 23 in the enlarged image 43.
  • the display control unit 59 determines that the specific notification target 23 exists from the detection result of the notification target detection unit 57 and the calculation result of the size calculation unit 58, the display control unit 59 displays the display on the display 4.
  • the display-enhanced image 42 that emphasizes the specific notification target 23 in the captured image 40 to be displayed is displayed on the display 4.
  • the display control unit 59 determines that the normal notification target 22 exists from the detection result of the notification target detection unit 57 and the calculation result of the size calculation unit 58, the display control unit 59 displays the display on the display 4.
  • the display 4 displays a normal emphasized image 41 that emphasizes the normal notification target 22 in the photographed image 40 different from the photographed emphasized image 42.
  • the specific notification target 23 displayed on the display 4 and the normal notification target 22 can be easily distinguished.
  • the display control unit 59 changes the display position of the enlarged image 43 based on the estimation result of the route estimation unit 55. This can prevent the occupant from recognizing the route of the vehicle 100.
  • the display control unit 59 causes the display 4 to display an enlarged and emphasized image 44 that emphasizes the specific notification target 23 in the enlarged image 43. This makes it easier to recognize the specific notification target 23 present in the enlarged image 43.
  • the notification target is the object closest to the vehicle 100 among the moving objects and the stationary objects that may come into contact with the vehicle 100. Accordingly, the notification target displayed on the display 4 is limited, and it is possible to prevent the occupant from being hindered by providing more information than necessary to the occupant.
  • FIG. 8 is a block diagram illustrating a configuration of an electronic control device 92 according to the second embodiment.
  • a notification target detection unit 957, a size calculation unit 958, and a display control unit 959 included in the control unit 95 are different from those of the first embodiment.
  • the notification target detection unit 957, the size calculation unit 958, and the display control unit 959 will be described below.
  • the notification target detection unit 957 notifies the occupant from the objects detected by the object detection unit 56 based on the predicted trajectory information output from the course estimation unit 55 and the object information output from the object detection unit 56.
  • the notification target to be notified is detected.
  • the notification targets detected by the notification target detection unit 957 are all moving objects and all stationary objects that may come into contact with the vehicle.
  • the notification target detection unit 957 outputs the notification target information including the object information regarding the detected notification target and the predicted trajectory information received from the route estimation unit 55 to the size calculation unit 958.
  • the notification target information includes position information including the distance between the notification target and the vehicle 100, information indicating whether the notification target is a stationary object or a moving object, and the upper, lower, left, and right ends of the notification target. This is information relating the number of detected notification targets in addition to information indicating the number of pixels in the width direction and the height direction for a rectangle including a portion.
  • the size calculation unit 958 calculates the size of the notification target detected by the notification target detection unit 957 displayed on the display 4. At this time, when a plurality of notification targets are detected, the size calculation unit 958 calculates the size of each notification target. Then, the size calculation unit 958 outputs to the display control unit 959 the size information that is the information on the calculated size of the notification target, the notification target information received from the notification target detection unit 957, and the predicted trajectory information. .
  • the display control unit 959 determines whether there is a plurality of notification targets detected by the notification target detection unit 957 based on the notification target information as a third function in addition to the first function and the second function. Is determined. Further, the display control unit 959 performs different operations depending on whether there are a plurality of notification targets or only one notification target.
  • the display control unit 959 determines that there is only one notification target, the display control unit 959 performs the same operation as the display control unit 59 in the first embodiment. That is, when determining that there is only one notification target, the display control unit 959 determines whether or not the specific notification target 23 exists based on the size information.
  • the display control unit 959 determines whether the specific notification target 23 and the normal notification target 22 exist in the captured image 40 based on the size information. That is, when the display control unit 959 determines that there are a plurality of notification targets, the display control unit 959 determines whether the specific notification target 23 exists, and further determines whether the normal notification target 22 exists.
  • the display control unit 959 performs the first to fourth patterns when there is only one notification target. However, for the third pattern, an operation of displaying the captured image 40 darker than the enlarged image 43 is added to the operation of the display control unit 59 of the first embodiment.
  • the display control unit 959 determines that one notification target detected by the notification target detection unit 957 and that the notification target is the specific notification target, Also makes the captured image 40 dark.
  • the specific notification target 23 emphasized by the photographing emphasized image 42 shown in FIG. 9 is also displayed dark because it is a part of the photographed image 40.
  • the display control unit 959 does not darken the captured image 40 when displaying only the enlarged image 43 on the display 4 as in the first embodiment.
  • the display control unit 959 highlights the specific notification target 23 or the normal notification target 22 existing in the captured image 40, or the specific notification target 23 and the normal notification target 22.
  • the display control unit 959 when only a plurality of normal notification targets 22 exist, the display control unit 959 superimposes and displays the normal emphasis image 41 for each of the normal notification targets 22 and the captured image 40. In addition, when only a plurality of specific notification targets 23 exist, the display control unit 959 superimposes the captured image 40 and the captured image 40 for each specific notification target 23. When both the normal notification target 22 and the specific notification target 23 exist, the display control unit 959 superimposes and displays the normal emphasized image 41, the photographed emphasized image 42, and the photographed image 40.
  • FIGS. 10 and 11 are flowcharts showing the procedure of the display control process executed by the control unit 95 of the second embodiment. Note that “S” in the flow of the display control processing means each step.
  • the course estimation unit 55 calculates a predicted trajectory based on the steering angle of the steering wheel, and proceeds to S19.
  • the notification target detection unit 957 detects all moving objects and stationary objects that may come into contact with the vehicle as notification targets based on the predicted trajectory information and the object information.
  • the notification target detection unit 957 When the notification target detection unit 957 does not detect the notification target, the process ends. On the other hand, when detecting the notification target, the notification target detection unit 957 outputs the notification target information and the predicted trajectory information to the size calculation unit 958, and proceeds to S20.
  • the size calculation unit 958 calculates the size of the notification target detected by the notification target detection unit 957 to be displayed on the display 4, and proceeds to S21.
  • the display control unit 959 determines whether there are a plurality of notification targets detected by the notification target detection unit 957. When the display control unit 959 determines that there are a plurality of notification targets, the process proceeds to S23. On the other hand, when the target determining unit 961 determines that there is not a plurality of notification targets, the process proceeds to S6.
  • step S22 the display control unit 959 causes the display 4 to display an enlarged image 43 obtained by enlarging a predetermined area including the selected specific notification target 23, and proceeds to step S11. At this time, when the enlarged image 43 and the captured image 40 are displayed in a superimposed manner, the display control unit 959 displays the captured image 40 darker than the enlarged image 43.
  • the display control unit 959 determines whether or not the specific notification target 23 exists. When the display control unit 959 determines that the specific notification target 23 does not exist, the process proceeds to S24, and when the display control unit 959 determines that the specific notification target 23 exists, the process proceeds to S25.
  • the display control unit 959 superimposes and displays the normal emphasized image 41 for each normal notification target 22 and the captured image 40. Then, the display control section 59 determines whether or not the shift lever is reverse. If the shift lever is reverse, the display control process is continued. If the shift lever is no longer reverse, the process ends.
  • the display control unit 959 determines whether or not the normal notification target 22 exists. When the display control unit 959 determines that the normal notification target 22 does not exist, the process proceeds to S26, and when the display control unit 959 determines that the normal notification target 22 exists, the process proceeds to S27.
  • the display control unit 959 superimposes and displays the photographed image 40 and the photographed image 40 for each specific notification target 23, and proceeds to S28.
  • the display control unit 959 displays the normal emphasized image 41, the photographed emphasized image 42, and the photographed image 40 in a superimposed manner because the normal notice target 22 and the specific notice target 23 are both present in the photographed image 40. .
  • the selection determination unit 60 determines whether the occupant has selected the specific notification target 23 displayed on the display 4 based on the input information detected by the input detection unit 51, as in S9. When determining that the occupant has selected the specific notification target 23, the selection determination unit 60 proceeds to S29. On the other hand, when the selection determining unit 60 determines that the occupant has not selected the specific notification target 23, the process ends.
  • the display control unit 59 causes the display 4 to display an enlarged image 43 obtained by enlarging a predetermined area including the selected specific notification target 23, and shifts to S30.
  • the display control unit 59 causes the display 4 to superimpose and display the enlarged image 43 and the enlarged emphasized image 44. Then, the display control section 59 determines whether or not the shift lever is reverse. If the shift lever is reverse, the display control process is continued. If the shift lever is no longer reverse, the process ends. Note that the timings of the processes of S29 and S30 may be aligned. Further, the processing of S30 may be omitted.
  • the display control unit 959 determines that both the normal notification target 22 and the specific notification target 23 exist in the captured image 40, the display control unit 959 sets the captured image 40 and the normal enhanced image 41 and the photographing emphasized image 42 are superimposed and displayed. Thereby, the specific notification target 23 in the captured image 40 and the normal notification target 22 can be easily distinguished.
  • the display control unit 959 determines that one notification target detected by the notification target detection unit 957 and that the notification target is the specific notification target,
  • the captured image 40 is displayed dark. This makes it easier to recognize the specific notification target 23 in the enlarged image 43.
  • the display control unit 59 may superimpose and display the enlarged image 43 and the specific notification target 23 in the captured image 40 as shown in FIG.
  • overlap refers to a case where the enlarged image 43 covers the specific notification target 23 in the captured image 40 and a case where the enlarged image 43 overlaps a part of the specific notification target 23 in the captured image 40.
  • the occupant since the specific notification target 23 in the captured image 40 is not emphasized by eliminating the processes in S8 and S9, the occupant may not be able to recognize the specific notification target 23 existing in the captured image 40. Therefore, the position of the specific notification target 23 in the captured image 40 is easily recognized by displaying the enlarged image 43 at the position where the specific notification target 23 is displayed in the captured image 40.
  • the enlarged image 43 is displayed after the photographed emphasized image 42 and the photographed image 40 or the photographed emphasized image 42, the normal emphasized image 41, and the photographed image 40 are superimposed on the display 4 in S26, S27, and S28.
  • This is a process of determining whether or not to make the setting.
  • the processes of S26, S27, and S28 may be omitted, and the display control unit 959 may display the enlarged image 43 on the display 4 when determining that the specific notification target 23 exists. This eliminates the need for the occupant to perform an operation of selecting the specific notification target 23, thereby improving convenience.
  • each specific notification target 23 may be displayed as an enlarged image 43 in a predetermined order.
  • the predetermined order is, for example, an order in which the distance to the vehicle 100 is short among the specific notification targets 23.
  • only the specific notification target 23 that is closest to the vehicle 100 may be displayed.
  • a plurality of enlarged images 43 may be displayed simultaneously.
  • a predetermined number of enlarged images 43 is displayed. This is to limit information to be reported at one time, thereby preventing the driver's judgment from being hindered by unnecessary report information.
  • the selection determination unit 60 determines whether or not the occupant has selected the specific notification target 23 based on an input to the indoor input unit 7 or the touch panel 70.
  • FIG. 13 is a block diagram showing an internal configuration of an electronic control unit 82 according to the third modification.
  • the electronic control unit 82 includes a control unit 85 and the storage unit 6.
  • a vehicle speed sensor 9 and an in-vehicle camera 10 are connected to the electronic control unit 82.
  • the in-vehicle camera 10 is provided in the room of the vehicle 100 and captures an occupant's face.
  • control unit 85 newly includes the stop determination unit 52 and the line-of-sight detection unit 53.
  • the stop determination unit 52 determines from the vehicle speed sensor 9 whether or not the vehicle 100 is stopped, and outputs information regarding the determination result to the gaze determination unit 80 newly provided in the display control unit 859 as stop information. Note that the stop determination unit 52 may determine whether the vehicle 100 is stopped from an acceleration sensor (not shown).
  • the line-of-sight detection unit 53 detects the direction of the line of sight of the occupant from the occupant's face image captured by the in-vehicle camera 10.
  • the line-of-sight detection unit 53 outputs information on the detected line-of-sight direction of the occupant to the gaze determination unit 80 as line-of-sight information.
  • the gaze determination unit 80 determines whether or not the occupant is gazing at the specific notification target 23 based on the stop information and the gaze information. Specifically, the gaze determination unit 80 gazes at the specific notification target 23 displayed on the display 4 when the vehicle 100 is stopped and the direction of the line of sight of the occupant is the direction of looking at the display 4. Is determined.
  • a flowchart or a process of the flowchart described in the present disclosure is composed of a plurality of units (or steps), and each unit is expressed as, for example, S1. Further, each part can be divided into a plurality of sub-parts, while a plurality of parts can be combined into one part. Furthermore, each part configured in this way can be referred to as a circuit, a device, a module, and a means.
  • Each or a combination of the above-mentioned plurality of units includes not only (i) a software unit combined with a hardware unit (eg, a computer), but also (ii) hardware (eg, an integrated circuit, As a part of the wiring logic circuit), it can be realized with or without including the function of the related device. Further, the hardware part may be configured inside the microcomputer.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Traffic Control Systems (AREA)
  • Image Analysis (AREA)
PCT/JP2019/024497 2018-07-19 2019-06-20 電子制御装置及び電子制御方法 Ceased WO2020017230A1 (ja)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018-136017 2018-07-19
JP2018136017A JP6927167B2 (ja) 2018-07-19 2018-07-19 電子制御装置及び電子制御方法

Publications (1)

Publication Number Publication Date
WO2020017230A1 true WO2020017230A1 (ja) 2020-01-23

Family

ID=69163497

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/024497 Ceased WO2020017230A1 (ja) 2018-07-19 2019-06-20 電子制御装置及び電子制御方法

Country Status (2)

Country Link
JP (1) JP6927167B2 (https=)
WO (1) WO2020017230A1 (https=)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7551699B2 (ja) * 2022-07-05 2024-09-17 キヤノン株式会社 カメラシステムおよびその制御方法、プログラム
WO2024190042A1 (ja) * 2023-03-16 2024-09-19 パナソニックホールディングス株式会社 情報処理方法、プログラム、および、情報処理装置
JP7794162B2 (ja) * 2023-03-31 2026-01-06 横河電機株式会社 装置、方法およびプログラム

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007280203A (ja) * 2006-04-10 2007-10-25 Nissan Motor Co Ltd 情報提示装置、自動車、及び情報提示方法
JP2009264835A (ja) * 2008-04-23 2009-11-12 Panasonic Corp ナビゲーション装置、方法及びプログラム
JP2010258705A (ja) * 2009-04-23 2010-11-11 Honda Motor Co Ltd 車両周辺監視装置
JP2012222776A (ja) * 2011-04-14 2012-11-12 Isuzu Motors Ltd 画像処理装置、画像処理方法、及び運転支援システム
JP2014109958A (ja) * 2012-12-03 2014-06-12 Denso Corp 撮影画像表示装置、および撮影画像表示方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2007280203A (ja) * 2006-04-10 2007-10-25 Nissan Motor Co Ltd 情報提示装置、自動車、及び情報提示方法
JP2009264835A (ja) * 2008-04-23 2009-11-12 Panasonic Corp ナビゲーション装置、方法及びプログラム
JP2010258705A (ja) * 2009-04-23 2010-11-11 Honda Motor Co Ltd 車両周辺監視装置
JP2012222776A (ja) * 2011-04-14 2012-11-12 Isuzu Motors Ltd 画像処理装置、画像処理方法、及び運転支援システム
JP2014109958A (ja) * 2012-12-03 2014-06-12 Denso Corp 撮影画像表示装置、および撮影画像表示方法

Also Published As

Publication number Publication date
JP6927167B2 (ja) 2021-08-25
JP2020013393A (ja) 2020-01-23

Similar Documents

Publication Publication Date Title
JP6443559B2 (ja) 車両用表示装置及び車両用表示方法
JP5099451B2 (ja) 車両周辺確認装置
JP7077616B2 (ja) 表示制御装置および表示制御方法
JP6669019B2 (ja) 車両用表示制御装置、車両用表示システム、車両用表示制御方法およびプログラム
KR101611194B1 (ko) 차량 주변 이미지 생성 장치 및 방법
US20160212384A1 (en) Image generation apparatus
JPWO2019216087A1 (ja) 画像処理装置、移動装置、および方法、並びにプログラム
US20200267353A1 (en) Display processing device
JP2018142884A (ja) 俯瞰映像生成装置、俯瞰映像生成システム、俯瞰映像生成方法およびプログラム
WO2020017230A1 (ja) 電子制御装置及び電子制御方法
JP2018142885A (ja) 車両用表示制御装置、車両用表示システム、車両用表示制御方法およびプログラム
JP2009183473A (ja) 視線方向検出装置及び視線方向検出方法
JP6589796B2 (ja) ジェスチャ検出装置
JP2014048978A (ja) 移動体警告装置、および移動体警告方法
US11394926B2 (en) Periphery monitoring apparatus
JP2013132976A (ja) 障害物警報装置
JP2018074411A (ja) 物体検出装置及び物体検出方法
CN108431866B (zh) 显示控制装置以及显示控制方法
WO2016185677A1 (ja) 車両周辺表示装置及び車両周辺表示方法
CN116915944A (zh) 显示控制装置、显示装置及车辆
KR20130029962A (ko) 차량용 영상 표시 장치 및 차량용 카메라로부터 입력된 급격한 영상 변화 표시 방법
JP2018142882A (ja) 俯瞰映像生成装置、俯瞰映像生成システム、俯瞰映像生成方法およびプログラム
US10926700B2 (en) Image processing device
JP2021033872A (ja) 車両用表示制御装置、車両用表示制御方法、およびプログラム
KR20150005072A (ko) 장애물 경고 기능을 구비한 주차 지원 장치

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19838665

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19838665

Country of ref document: EP

Kind code of ref document: A1