WO2019138771A1 - 車両用画像合成装置 - Google Patents

車両用画像合成装置 Download PDF

Info

Publication number
WO2019138771A1
WO2019138771A1 PCT/JP2018/045894 JP2018045894W WO2019138771A1 WO 2019138771 A1 WO2019138771 A1 WO 2019138771A1 JP 2018045894 W JP2018045894 W JP 2018045894W WO 2019138771 A1 WO2019138771 A1 WO 2019138771A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
boundary
similarity
camera
driver
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2018/045894
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
洋介 酒井
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Denso Corp
Original Assignee
Denso Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Denso Corp filed Critical Denso Corp
Priority to DE112018006812.9T priority Critical patent/DE112018006812T5/de
Priority to CN201880085804.2A priority patent/CN111566698A/zh
Publication of WO2019138771A1 publication Critical patent/WO2019138771A1/ja
Priority to US16/921,311 priority patent/US11270440B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/12—Edge-based segmentation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/136—Segmentation; Edge detection involving thresholding
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/174—Segmentation; Edge detection involving the use of two or more images
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • G06V20/597—Recognising the driver's state or behaviour, e.g. attention or drowsiness
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • B60W2040/0818—Inactivity or incapacity of driver
    • B60W2040/0827—Inactivity or incapacity of driver due to sleepiness
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/08—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to drivers or passengers
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/32—Indexing scheme for image data processing or generation, in general involving image mosaicing
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • G06T2207/30201—Face
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30268—Vehicle interior
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/16—Image acquisition using multiple overlapping images; Image stitching

Definitions

  • the present disclosure relates to an image synthesizing device for a vehicle.
  • An object of the present disclosure is to provide a vehicular image synthesizing device capable of obtaining an image obtained by photographing a driver's face favorably when the driver's face is photographed by a camera.
  • a plurality of cameras mounted on a vehicle for photographing a driver, and a boundary detection unit for comparing a plurality of images photographed by the plurality of cameras and detecting a boundary suitable for synthesis And a combining unit configured to combine the plurality of images at the boundary.
  • FIG. 1 is a block diagram of a vehicular image synthesis apparatus showing a first embodiment
  • Fig. 2 is a diagram showing steering, a meter, a camera, etc.
  • Fig. 3 is a longitudinal side view showing a steering, a meter, a camera, a driver, etc.
  • FIG. 4 is a diagram showing an upper camera image
  • FIG. 5 is a diagram showing a lower camera image
  • FIG. 6 is a diagram showing a composite image
  • FIG. 7 is a diagram for explaining a row area image of the upper camera image
  • FIG. 8 is a diagram for explaining a row area image of the lower camera image
  • FIG. 1 is a block diagram of a vehicular image synthesis apparatus showing a first embodiment
  • Fig. 2 is a diagram showing steering, a meter, a camera, etc.
  • Fig. 3 is a longitudinal side view showing a steering, a meter, a camera, a driver, etc.
  • FIG. 4 is a diagram showing an
  • FIG. 9 is a diagram showing a synthesized image synthesized with the line area image having the highest degree of similarity;
  • FIG. 10 is a flowchart of image synthesis control;
  • FIG. 11 is a flowchart of combination determination control;
  • FIG. 12 is a flowchart of synthesis condition search control;
  • FIG. 13 shows a second embodiment and is a diagram showing steering, a meter, a camera and the like.
  • FIG. 14 is a top view showing a camera, a driver, etc.
  • FIG. 15 is a diagram showing an upper camera image
  • FIG. 16 is a diagram showing a lower camera image
  • FIG. 17 is a diagram showing an image obtained by rotating the lower camera image
  • FIG. 18 is a diagram showing a composite image;
  • FIG. 19 shows a third embodiment and is a view showing a composite image;
  • FIG. 20 is a flowchart of synthesis condition search control according to the fourth embodiment.
  • FIG. 1 shows the electrical configuration of a vehicular image synthesis apparatus 1.
  • the vehicle image synthesizing device 1 includes an upper camera 2, a lower camera 3, a tilt position sensor 4, a steering angle sensor 5, and a control device 6.
  • the upper camera 2 constitutes a first camera, and as shown in FIG. 2 and FIG. 3, is disposed at the center of the upper side of the meter 7 provided on the instrument panel of the vehicle.
  • the upper camera 2 continuously captures the driver's face and transmits the captured image data to the control device 6.
  • the upper camera 2 has a function of continuously shooting at, for example, about 30 frames / s.
  • the lower camera 3 constitutes a second camera, and as shown in FIGS. 2 and 3, is disposed at the center of the lower side of the meter 7 of the vehicle.
  • the upper camera 2 and the lower camera 3 are disposed in the vertical direction, that is, in the vertical direction, at a set distance.
  • the upper camera 2 and the lower camera 3 are disposed to face the front of the driver's face.
  • the lower camera 3 continuously captures the driver's face and transmits the captured image data to the control device 6.
  • the lower camera 3 has a function of continuously shooting at a speed of, for example, about 30 frames / s.
  • the upper camera 2 takes an image of an area shown by a solid line and a broken line, and the taken upper camera image becomes an image as shown in FIG. 4 and the upper part of the driver's face is the steering wheel Hiding in 8
  • the upper camera 2 can shoot well without being hidden by the steering wheel 8 from the central part to the lower part of the driver's face, that is, the area shown by the solid line.
  • the lower camera 3 takes an image of an area shown by the solid line and the broken line, and the taken lower camera image becomes an image as shown in FIG. 5 and the lower part of the driver's face is hidden by the steering column 9 .
  • the lower camera 3 can shoot well without being hidden by the steering column 9 from the central portion to the upper side of the driver's face, that is, the area shown by the solid line.
  • the tilt position sensor 4 detects the position of the steering wheel 8 when the steering wheel 8 is moved by the tilt mechanism of the steering column 9, and transmits a detection signal of the position of the steering wheel 8 to the control device 6.
  • the steering angle sensor 5 detects the rotation angle of the steering wheel 8 when the steering wheel 8 is rotated, and transmits a detection signal of the rotation angle of the steering wheel 8 to the control device 6.
  • the control device 6 has a function of controlling the entire vehicle image combining device 1, and includes an image combining processing unit 10 and an image recognition processing unit 11.
  • the image synthesis processing unit 10 includes a boundary detection unit 12 and a synthesis unit 13.
  • the image synthesis processing unit 10 inputs image data from the upper camera 2 and the lower camera 3, detection signals from the tilt position sensor 4 and the steering angle sensor 5, and vehicle information from the vehicle.
  • the boundary detection unit 12 of the image synthesis processing unit 10 receives the upper camera image (see FIG. 4) photographed by the upper camera 2 and the lower camera image (see FIG. 5) photographed by the lower camera 3; It has a function of detecting a boundary suitable for synthesizing one image so that there is no part hidden by the steering wheel 8 or the steering column 9 or the like. Then, the combining unit 13 of the image combining processing unit 10 has a function of creating a combined image as shown in FIG. 6 by combining the upper camera image and the lower camera image at the detected boundary.
  • the image recognition processing unit 11 receives a composite image synthesized by the image synthesis processing unit 10 or a camera image captured by the cameras 2 and 3 and performs image recognition processing on the input image, that is, the face image of the driver. It has a function.
  • the image recognition processing unit 11 has a function of determining, for example, a nap or looking aside as a driver's state by image recognition processing, and determining whether a safe driving can be performed.
  • the upper camera image is divided into a plurality of line area images 15 in line units in the vertical direction as shown in FIG. 7, and the lower camera images are shown in FIG. 8 as a plurality of lines in line units in the vertical direction. It is divided into area images 16. Then, the line area image 15 obtained by dividing the upper camera image is compared with the line area image 16 obtained by dividing the lower camera image to calculate the similarity. The calculation of the degree of similarity is configured to be performed for all the row region images 15 of the upper camera image and for all the row region images 16 of the lower camera image.
  • the boundary is detected by setting the line area images 15 and 16 having the highest degree of similarity as the boundary, that is, the boundary area or the boundary line. Subsequently, the upper camera image and the upper camera image are synthesized at the boundary. In this case, for the upper camera image, an image portion 17 below the row region image 15 having the highest degree of similarity, and for the lower camera image, an image portion above the row region image 16 having the highest degree of similarity. And 18 are synthesized. Thus, the composite image shown in FIG. 9 is created. One of the row region images 15 and 16 having the highest degree of similarity is adopted.
  • 10 to 12 show the contents of control of the control device 6, that is, the image combining processing unit 10 and the image recognition processing unit 11.
  • step S10 in FIG. 10 the driver's face is photographed by the upper camera 2 and the lower camera 3, and the image synthesis processing unit 10 detects the upper camera image and the lower camera photographed by the upper camera 2 and the lower camera 3. Enter an image. Subsequently, the process proceeds to step S20, and the image synthesis processing unit 10 executes a synthesis determination process of determining whether or not the input upper camera image and lower camera image are to be synthesized. The combination determination process will be described later.
  • step S30 it is determined whether the combination determination is OK.
  • the process returns to step S10.
  • step S30 when the composition determination is OK (YES), the process proceeds to step S40 to set the composition condition of the image. Specifically, for example, the composition condition searched in the composition determination process of step S20. Line area images 15 and 16 of the detected boundary area are set.
  • step S50 the driver's face is photographed by the upper camera 2 and the lower camera 3, and the image synthesis processing unit 10 inputs the upper camera image and the lower camera image photographed by the upper camera 2 and the lower camera 3. Do.
  • the composition of the image photographed in step S10 is not performed, it is preferable to skip the photographing in step S50 once.
  • step S60 the process proceeds to step S60, and the upper camera image and the lower camera image are combined by the combining unit 13 of the image combining processing unit 10 with the line region images 15 and 16 of the set boundary region as boundaries.
  • step S70 the image recognition processing unit 11 executes an image recognition process on the face image of the driver appearing in the composite image, and determines the state of the driver. It is preferable to use image recognition processing of a known configuration for this face image recognition processing.
  • step S80 a process of determining whether to reset the combination condition is performed.
  • a process of determining whether to reset the combination condition is performed.
  • it is detected from the tilt position sensor 4 or the steering angle sensor 5 that the steering wheel 8 has moved or rotated it is determined that resetting is necessary.
  • step S90 it is determined whether the composition condition is to be reset.
  • the process returns to step S50, and the above-described processing, that is, the photographing process by the upper camera 2 and the lower camera 3 and the image combining process are repeatedly executed.
  • step S90 when setting the composition condition again (YES), the process returns to step S10, and the above-described processing, that is, the photographing processing by the upper camera 2 and the lower camera 3, the composition determination process, the composition condition again Setting processing, image combining processing, and the like are repeatedly performed.
  • step S110 of FIG. 11 it is determined whether the upper camera image has no hidden portion due to the steering wheel 8 or the like.
  • the process proceeds to step S120, and only the upper camera image is set as the combination condition.
  • step S130 the combination determination is set as OK.
  • the control of FIG. 11 is ended, the process returns to the control of FIG. 10, and the process proceeds to step S30.
  • step S110 when there is a hidden portion in the upper camera image (NO), the process proceeds to step S140 to determine whether there is a hidden portion due to the steering wheel 8 or the like in the lower camera image.
  • step S140 determines whether there is a hidden portion due to the steering wheel 8 or the like in the lower camera image.
  • step S150 if there is no hidden portion in the lower camera image (YES), the process proceeds to step S150, and only the lower camera image is set as the combination condition. Subsequently, the process proceeds to step S160, and the combination determination is set as OK.
  • the control of FIG. 11 is ended, the process returns to the control of FIG. 10, and the process proceeds to step S30.
  • step S140 when there is a hidden portion in the lower camera image (NO), the process proceeds to step S170, and a process of searching for a combination condition is performed.
  • the synthesis condition search process will be described later.
  • the control of FIG. 11 is ended, the process returns to the control of FIG. 10, and the process proceeds to step S30.
  • step S170 The combination condition search process of step S170 will be described with reference to the flowchart of FIG.
  • the synthesis condition search processing is executed by the boundary detection unit 12 of the image synthesis processing unit 10.
  • an outline of the synthesis condition search process will be described. First, one camera image, for example, the first line area image of the upper camera image, is cut out, and this line area image is compared with the other camera image, for example, the first line area image of the lower camera image That is, the similarity score is calculated. Then, this comparison and similarity calculation are all performed from the first line area image of the lower camera image to the last line area image.
  • the next line area image of the upper camera image is cut out, and the line area image is also compared with all line area images of the lower camera image to calculate the similarity.
  • comparison and similarity calculation are sequentially performed up to the last line area image of the upper camera image.
  • the line area image in which the similarity of the maximum value among the calculated similarity is obtained is selected as the boundary.
  • variable INIT_i is the initial value of the cropping area of the upper camera image
  • STEP_i is the update value of the cropping area of the upper camera image
  • max_i is the end value of the cropping area of the upper camera image j is lower
  • the variable INIT_j of the comparison line area of the camera image is an initial value of the comparison line area of the lower camera image
  • STEP_j is an update value of the comparison line area of the lower camera image max_j is an end value of the comparison line area of the lower camera image.
  • step S210 of FIG. 12 an initial value INIT_i is input to the variable i of the cutout area of the upper camera image.
  • step S220 the i-line area image of the upper camera image is cut out, that is, read out and acquired.
  • the process proceeds to step S230, and the initial value INIT_j is input to the variable j of the comparison line area of the lower camera image.
  • step S240 the i-line area image of the upper camera image and the j-line area image of the lower camera image are compared to calculate the similarity.
  • step S250 the calculated similarity is stored in the memory in the control device 6. In this case, a variable i of the i-line area image of the upper camera image and a variable j of the j-line area image of the lower camera image are added and stored as the similarity.
  • step S260 the process proceeds to step S260, and the variable j is updated by the update value STEP_j. Then, the process proceeds to step S270, and it is determined whether the variable j has become larger than the end value max_j.
  • j is not larger than max_j (NO)
  • the process returns to step S240, and the above-described processing is repeatedly executed.
  • step S270 when j becomes larger than max_j (YES), the process proceeds to step S280, and the variable i is updated by the update value STEP_i. Then, the process proceeds to step S290, and it is determined whether the variable i has become larger than the end value max_i.
  • i is not larger than max_i (NO)
  • the process returns to step S220, and the above-described processing is repeatedly executed.
  • step S290 when i becomes larger than max_i (YES), the process proceeds to step S300, and it is determined whether the calculated maximum value of similarity is equal to or more than a preset threshold for combination determination. To judge.
  • the process proceeds to step S310, and the line region image of the upper camera image and the lower camera image of the similarity maximum value Select and set the combination of line area image of. Then, the process proceeds to step S320, and the combination determination is set as OK. As a result, the present control is ended, and the control of FIG. 11 and finally the control of FIG. 10 are returned.
  • step S300 when the maximum value of the degree of similarity is not equal to or larger than the threshold value for combination determination (NO), the process proceeds to step S330, and the combination determination is set as NG. In this way, the present control is ended, and the control of FIG. 11 and the control of FIG. 10 are subsequently returned, and the photographing is performed again, and the composition determination is performed again.
  • two camera images captured by cameras 2 and 3 are compared to detect a boundary suitable for combining, that is, a row region image 15 or 16 having the highest similarity.
  • the two camera images are configured to be combined in the row region image 15 or 16 having the highest degree of similarity.
  • two images captured by two cameras can be favorably combined into one image. Therefore, when the driver's face is photographed by the cameras 2 and 3, an image in which the driver's face is photographed well, that is, a favorable composite image can be obtained.
  • the line area image 15 obtained by dividing the upper camera image taken by the upper camera 2 into a plurality of line areas and the line obtained by dividing the lower camera image taken by the lower camera 3 into a plurality of line areas The similarity to the area image 16 is calculated, and the line area image having the highest similarity is detected as the boundary. According to this configuration, the boundary of image composition can be detected easily and reliably with a simple configuration.
  • the boundary detection unit detects the boundary when the steering wheel is moved by the tilt mechanism or when the set time has elapsed since the image combining process or the image recognition process is performed. I configured it to run again. According to this configuration, the process of detecting the boundary of the image synthesis is executed only at the minimum necessary, and is not performed each time at the time of the image synthesis process, that is, the boundary condition is fixed at the image synthesis process for about the set time. Therefore, when combining a large number of composite images, the time required for combining can be significantly reduced.
  • the range of similarity calculation in the upper camera image or the lower camera image may be changed according to the position or angle of the steering wheel 8 or the like.
  • the update values STEP_i and STEP_j of the row area image may be changed as appropriate.
  • Second Embodiment 13 to 18 show a second embodiment.
  • the same components as those in the first embodiment are denoted by the same reference numerals.
  • two cameras 2 and 3 are disposed offset in the horizontal direction.
  • the lower camera 3 is disposed at a predetermined distance from the upper camera 2 in the horizontal direction.
  • the upper camera 2 is disposed to face the front of the driver's face. In the case of this configuration, it is configured to generate an image in which the lower camera image is rotationally moved so as to eliminate the deviation, and then to combine the two images.
  • the image synthesis processing unit 10 of the control device 6 has a function as an image generation unit.
  • the upper camera image captured by the upper camera 2 is an image as shown in FIG.
  • the lower camera image photographed by the lower camera 3 is an image as shown in FIG.
  • the driver's face of the upper camera image faces the front, but the driver's face of the lower camera image faces a little diagonally. If this is the case, combining the two images results in an unnatural face.
  • the control device 6 rotationally moves the lower camera image, for example, by performing affine transformation, creates a rotational movement image as shown in FIG. 17, and stores the rotational movement image in the memory.
  • the control apparatus 6 is comprised so that the said rotational movement image and lower camera image of a lower camera image may be synthesize
  • a good composite image as shown in FIG. 18 can be obtained.
  • the configuration of the second embodiment other than that described above is the same as the configuration of the first embodiment. Therefore, also in the second embodiment, substantially the same effect as that of the first embodiment can be obtained.
  • the second embodiment in the case where two cameras 2 and 3 are arranged to be shifted in the horizontal direction, an image in which a camera image is rotationally moved so as to eliminate the shift is generated. Since two images are configured to be combined, a good combined image can be combined.
  • FIG. 19 shows a third embodiment.
  • the same components as those in the first embodiment are denoted by the same reference numerals.
  • the image combining processing unit 10 of the control device 6 has a function as an image reducing unit.
  • the upper camera image and the lower camera image are combined using the image combining control of the first embodiment, and a combined image 21 as shown in FIG. 19 is created.
  • the size and position of the face image are detected by performing image recognition processing of the driver's face on the composite image 21. Then, based on the information of the size and position of the detected face image, the size of the image necessary for the image synthesis and the image recognition process is set as shown by a dashed rectangular area in FIG. Make it smaller than 21.
  • the size of the partial image corresponding to the portion indicated by the rectangular area of the broken line or the size slightly larger than this partial image for each of the upper camera image and the lower camera image Using the partial image of (1), a composition process substantially similar to the image composition control of the first embodiment is executed. And when performing an image recognition process with respect to the synthesize
  • the configuration of the third embodiment other than that described above is the same as the configuration of the first embodiment. Therefore, also in the third embodiment, substantially the same effect as that of the first embodiment can be obtained.
  • the processing load of the control device 6 can be reduced because the image data used for the image synthesis and the image recognition processing is reduced.
  • FIG. 20 shows a fourth embodiment.
  • the same components as those in the first embodiment are denoted by the same reference numerals.
  • the similarity is equal to or more than the threshold for boundary determination set in advance.
  • the line area image in which the degree of similarity is equal to or higher than the threshold for boundary determination is detected as the boundary.
  • steps S210 to S250 in FIG. 20 are performed as in the first embodiment, and after step S250, the process proceeds to step S410.
  • step S410 it is determined whether the calculated similarity is equal to or more than the threshold for boundary determination.
  • the process proceeds to step S260, and thereafter, the process up to step S330 is performed as in the first embodiment.
  • step S410 when the similarity is equal to or higher than the threshold for boundary determination (YES), the process proceeds to step S420, and the upper camera image of the similarity which is equal to or higher than the threshold for boundary determination as a boundary area of the combination condition.
  • the combination of the line area image of and the line area image of the lower camera image is selected and set.
  • step S320 the combination determination is set as OK, and the present control ends.
  • the configuration of the fourth embodiment other than that described above is the same as the configuration of the first embodiment. Therefore, also in the fourth embodiment, substantially the same effect as that of the first embodiment can be obtained.
  • the fourth embodiment when the similarity becomes equal to or higher than the threshold for boundary determination during calculation of the similarity, the calculation of the similarity is stopped and the similarity is for boundary determination. Since the line area image having the threshold value or more is detected as the boundary, the time required for the boundary detection process, and hence the time required for the image synthesis can be shortened.
  • the line area image of the upper camera image and the line area image of the lower camera image are compared to calculate the similarity, but the present invention is not limited to this.
  • the row region images may be compared.
  • the row region image is a region obtained by dividing the camera image into a plurality of row units in the horizontal direction.
  • the upper camera image and the lower camera image may be divided into a plurality of rectangular image areas of a predetermined size, and the divided rectangular image areas may be compared.
  • the present invention is applied to a configuration in which two cameras 2 and 3 are disposed, but the present invention is not limited to this. Three or more cameras are disposed, and photographing is performed by three or more cameras The present invention may be applied to a configuration for combining three or more camera images.
  • the similarity is calculated and the row region images 15 and 16 having the highest similarity are set as the boundaries, but the present invention is not limited to this.
  • the tilt position of the steering wheel In order to detect boundaries using steering angles, vehicle parameters included in vehicle information, for example, various vehicle parameters such as camera installation conditions, steering shapes, driver positions, etc., information of face recognition results of the image recognition processing unit 11, etc.
  • vehicle parameters included in vehicle information for example, various vehicle parameters such as camera installation conditions, steering shapes, driver positions, etc., information of face recognition results of the image recognition processing unit 11, etc.
  • the image areas of the upper camera image and the lower camera image for calculating the degree of similarity may be narrowed. With this configuration, the detection time of the boundary and the processing load of the control device 6 can be reduced.
  • stop calculation of the degree of similarity and detect and set boundaries suitable for synthesis by calculation based on the tilt position of the steering wheel, steering angle, and vehicle parameters, that is, line region images that become boundaries suitable for synthesis It may be configured as follows. According to this configuration, since the similarity is not calculated, the time required to detect the boundary can be significantly reduced.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)
PCT/JP2018/045894 2018-01-10 2018-12-13 車両用画像合成装置 Ceased WO2019138771A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
DE112018006812.9T DE112018006812T5 (de) 2018-01-10 2018-12-13 Fahrzeugbildsynthesevorrichtung
CN201880085804.2A CN111566698A (zh) 2018-01-10 2018-12-13 车辆用图像合成装置
US16/921,311 US11270440B2 (en) 2018-01-10 2020-07-06 Vehicular image synthesis apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2018-001883 2018-01-10
JP2018001883A JP6939580B2 (ja) 2018-01-10 2018-01-10 車両用画像合成装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/921,311 Continuation US11270440B2 (en) 2018-01-10 2020-07-06 Vehicular image synthesis apparatus

Publications (1)

Publication Number Publication Date
WO2019138771A1 true WO2019138771A1 (ja) 2019-07-18

Family

ID=67218596

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/045894 Ceased WO2019138771A1 (ja) 2018-01-10 2018-12-13 車両用画像合成装置

Country Status (5)

Country Link
US (1) US11270440B2 (https=)
JP (1) JP6939580B2 (https=)
CN (1) CN111566698A (https=)
DE (1) DE112018006812T5 (https=)
WO (1) WO2019138771A1 (https=)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2023036288A (ja) * 2021-09-02 2023-03-14 豊田合成株式会社 操舵装置
JP7750206B2 (ja) * 2022-10-19 2025-10-07 トヨタ自動車株式会社 車室内状態認識装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003208620A (ja) * 2002-01-17 2003-07-25 Fujitsu Ltd 生体情報を用いた本人認証方法、プログラム、記録媒体及び装置
JP2007328582A (ja) * 2006-06-08 2007-12-20 Matsushita Electric Ind Co Ltd 画像合成装置およびその方法
JP2008199515A (ja) * 2007-02-15 2008-08-28 Fujifilm Corp 同乗者着座姿勢検出・判定装置及び方法
JP2009011005A (ja) * 2008-10-10 2009-01-15 Panasonic Corp 画像処理装置
JP2010171777A (ja) * 2009-01-23 2010-08-05 Toyota Motor Corp 撮像装置
JP2017112486A (ja) * 2015-12-16 2017-06-22 オムロンオートモーティブエレクトロニクス株式会社 画像処理装置

Family Cites Families (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7054857B2 (en) 2002-05-08 2006-05-30 Overture Services, Inc. Use of extensible markup language in a system and method for influencing a position on a search result list generated by a computer network search engine
JP2005202599A (ja) * 2004-01-14 2005-07-28 Advanced Telecommunication Research Institute International 居眠り防止装置
JP4380412B2 (ja) * 2004-05-10 2009-12-09 株式会社デンソー 撮像制御装置及びプログラム
JP4739870B2 (ja) 2005-09-05 2011-08-03 アイシン精機株式会社 サングラス検出装置及び顔中心位置検出装置
JP2007069680A (ja) 2005-09-05 2007-03-22 Toyota Motor Corp 顔画像撮影カメラの搭載構造
JP4606322B2 (ja) * 2005-12-27 2011-01-05 アルパイン株式会社 車両運転支援装置
JP2007234060A (ja) * 2007-05-31 2007-09-13 Advanced Telecommunication Research Institute International 居眠り防止装置
JP4981700B2 (ja) * 2008-01-29 2012-07-25 パナソニック株式会社 顔認証装置
JP4989762B2 (ja) 2008-03-18 2012-08-01 パナソニック株式会社 運転者監視装置、運転者監視方法及び車両
EP2482557B1 (en) * 2009-09-24 2020-03-18 Panasonic Intellectual Property Management Co., Ltd. Driving support display device
JP5419757B2 (ja) 2010-03-09 2014-02-19 セコム株式会社 顔画像合成装置
EP2892230A4 (en) * 2012-08-30 2015-09-02 Fujitsu Ltd IMAGE PROCESSING DEVICE, IMAGE PROCESSING AND PROGRAM
JP6024581B2 (ja) * 2013-04-15 2016-11-16 株式会社デンソー 車両用画像処理装置
JP6452235B2 (ja) * 2014-11-20 2019-01-16 国立大学法人静岡大学 顔検出方法、顔検出装置、及び顔検出プログラム
CN104590130A (zh) * 2015-01-06 2015-05-06 上海交通大学 基于图像识别的后视镜自适应调节方法
KR101730321B1 (ko) * 2015-08-03 2017-04-27 엘지전자 주식회사 운전자 보조 장치 및 그 제어방법
JP6594180B2 (ja) * 2015-11-25 2019-10-23 キヤノン株式会社 撮像装置、撮像装置の制御方法及びプログラム
KR101949358B1 (ko) * 2016-04-04 2019-02-18 엘지전자 주식회사 차량용 어라운드뷰 제공 장치 및 이를 구비한 차량
JP6809262B2 (ja) * 2017-02-08 2021-01-06 トヨタ自動車株式会社 ドライバ状態検出装置

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2003208620A (ja) * 2002-01-17 2003-07-25 Fujitsu Ltd 生体情報を用いた本人認証方法、プログラム、記録媒体及び装置
JP2007328582A (ja) * 2006-06-08 2007-12-20 Matsushita Electric Ind Co Ltd 画像合成装置およびその方法
JP2008199515A (ja) * 2007-02-15 2008-08-28 Fujifilm Corp 同乗者着座姿勢検出・判定装置及び方法
JP2009011005A (ja) * 2008-10-10 2009-01-15 Panasonic Corp 画像処理装置
JP2010171777A (ja) * 2009-01-23 2010-08-05 Toyota Motor Corp 撮像装置
JP2017112486A (ja) * 2015-12-16 2017-06-22 オムロンオートモーティブエレクトロニクス株式会社 画像処理装置

Also Published As

Publication number Publication date
JP6939580B2 (ja) 2021-09-22
US11270440B2 (en) 2022-03-08
JP2019121270A (ja) 2019-07-22
CN111566698A (zh) 2020-08-21
DE112018006812T5 (de) 2020-09-17
US20200334822A1 (en) 2020-10-22

Similar Documents

Publication Publication Date Title
JP5792091B2 (ja) 物体検出装置及び物体検出方法
US10499014B2 (en) Image generation apparatus
JP5421072B2 (ja) 接近物体検知システム
US10979614B2 (en) Control apparatus, method of controlling imaging device, and non-transitory computer-readable storage medium that control an imaging device for tracking a tracking target
US10896336B2 (en) Parking compartment recognition apparatus
KR102429490B1 (ko) Bvm 영상 제공 기능을 가지는 svm 시스템 및 그 동작 방법
JP7000383B2 (ja) 画像処理装置および画像処理方法
WO2015122124A1 (ja) 車両周辺画像表示装置、車両周辺画像表示方法
JP5302730B2 (ja) 運転者監視装置
JP2018142884A (ja) 俯瞰映像生成装置、俯瞰映像生成システム、俯瞰映像生成方法およびプログラム
JP6890288B2 (ja) 画像処理装置、画像表示システムおよび画像処理方法
WO2019138771A1 (ja) 車両用画像合成装置
JP2018074411A (ja) 物体検出装置及び物体検出方法
US10867400B2 (en) System for processing an image obtained by using stereo photography
JP2014154062A (ja) 物体検知装置
JP2006318272A (ja) 車両用物体検出装置、および方法
JP6407596B2 (ja) 画像処理装置、及び、運転支援システム
US12333852B2 (en) Face detection device
JP2017065351A (ja) 駐車支援装置及び駐車支援方法
JP2017126835A (ja) 車載カメラのレンズ異常検出装置
JP2020063934A (ja) 物体認識装置およびこれを用いた車両制御装置並びに物体認識方法およびこれを用いた車両制御方法
JP4529768B2 (ja) 車載物体検出装置、および物体検出方法
JP2019133445A (ja) 区画線検出装置、区画線検出システム、及び区画線検出方法
JP7137356B2 (ja) 車載用故障検出装置、及び故障検出方法
JP6781010B2 (ja) 画像表示装置及び画像処理方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18900460

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 18900460

Country of ref document: EP

Kind code of ref document: A1