JP6519499B2 - Vehicle system and computer program - Google Patents

Description

  The present invention relates to a vehicle system and a computer program.

  If, for example, a part of the body such as a hand or face or an object such as a luggage projects from the vehicle compartment to the outside of the vehicle via the window during travel, the vehicle may come in contact with other vehicles or buildings. is there. Under such circumstances, a vehicle system for determining whether or not an object protrudes from a vehicle compartment is desired. As a system for this type of vehicle, a configuration is conceivable in which the position of an object is specified using a stereo camera, and it is determined whether the object protrudes from the vehicle interior. In a configuration using a stereo camera, if the distance from the stereo camera to the object is calculated using the parallax of the image and the distance from the stereo camera to the object is larger than the distance to the car window, Identify the object as sticking out of the cabin. Further, as another configuration for specifying the position of an object, Patent Document 1 discloses a configuration for specifying the position of an object by using a single-eye camera and a sensor such as a sonar or a radar in combination.

JP, 2014-127032, A

  However, in the configuration using a stereo camera or using a single-eye camera and a sensor in combination, there is a problem that the cost is increased as compared with a configuration using only a single-eye camera.

  The present invention has been made in view of the above-described circumstances, and an object thereof is to appropriately determine whether an object protrudes from a vehicle compartment without causing an increase in cost, and the object protrudes from the vehicle interior. An object of the present invention is to provide a vehicle system and a computer program capable of appropriately notifying that there is an event.

  According to the invention described in claim 1, the photographing unit (3) is constituted by a single-eye camera, and at least a range including an openable and closable car window (14 to 17) and a mirror surface of a side mirror that reflects the periphery of the car window Photograph an area including (12a, 13a). The image analysis unit (5) analyzes an image captured by the imaging unit. The determination unit (5c) analyzes the analysis result by the image analysis unit of the direct image in which the range including the car window is captured by the imaging unit and the analysis result by the image analysis unit of the indirect image in which the range including the mirror surface is captured by the imaging unit. And determine whether the object protrudes from the vehicle interior through the vehicle window to the vehicle exterior. The notification command unit (5d) notifies the determination result of the determination unit by the notification unit (4).

  Use the analysis results of the direct image in which the area including the car window is photographed and the analysis result of the indirect image in which the area including the mirror surface of the side mirror is photographed to determine whether the object protrudes from the vehicle interior I made it. Unlike the configuration that uses a stereo camera or a combination of a single-eye camera and a sensor, the configuration using only a single-eye camera makes it possible to appropriately determine whether the object protrudes from the vehicle interior without causing an increase in cost. It is determined that the object protrudes from the vehicle compartment can be appropriately notified.

Functional block diagram showing an embodiment of the present invention The figure which shows the aspect by which the camera is arrange | positioned in a vehicle interior Diagram showing the principle of specifying the position of an object using a stereo camera Diagram showing the principle of locating an object using a monocular camera and a mirror flowchart Sequence Diagram A diagram showing the situation where an object does not protrude from the vehicle cabin A diagram showing the situation where an object protrudes from the vehicle cabin

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The system 1 for vehicles has the apparatus 2 for vehicles, the camera 3 (equivalent to a imaging | photography part), and the alerting | reporting device 4 (equivalent to an alerting | reporting part). The camera 3 is a monocular camera with a 360 degree direction as a field of view (i.e. angle of view), and is disposed above the driver's seat on the ceiling of the vehicle interior of the vehicle 11 with the right handle as shown in FIG. . That is, the camera 3 is disposed at a position close to the driver's eyes, and photographs the range including the car window and the range including the mirror surfaces 12a and 13a of the side mirrors 12 and 13 attached to the vehicle body side surface of the vehicle 11. Do. The side mirrors 12 and 13 are door mirrors attached to the driver's seat door and the passenger's seat door, respectively.

  The mirror surfaces 12a and 13a of the side mirrors 12 and 13 can be adjusted in angle so that the rear of the vehicle can be viewed so that the driver can check the rear of the vehicle even with a small movement of the eyes from the front of the vehicle. In this case, if the angles of the mirror surfaces 12a and 13a are adjusted so that the driver can check the rear of the vehicle including a part of the side surface of the vehicle, the camera 3 reflects the part of the side surface of the vehicle , 13a will be taken. That is, if the driver adjusts the angle of the mirror surface 12 a so that a part of the car window 14 of the driver's seat and a part of the car window 15 of the right rear seat appear on the mirror surface 12 a of the right side mirror 12. The camera captures a mirror surface 12a that reflects a part of the car window 14 of the driver's seat and a part of the car window 15 of the right rear seat. In FIG. 2, the range in which the mirror surface 12 a of the right side mirror 12 is shown is indicated by broken lines A1 and A2.

  The same applies to the left side mirror 13, and the driver sets the angle of the mirror surface 13a so that a part of the front window 16 and a part of the left rear window 17 are reflected on the mirror surface 13a of the left side mirror 13. If it is adjusted, the camera 3 takes an image of a mirror surface 13a that reflects a part of the passenger window 16 and a part of the left rear seat 17. In FIG. 2, the range in which the mirror surface 13 a of the left side mirror 13 is shown is indicated by broken lines B1 and B2. As described above, the camera 3 directly shoots the range including the car windows 14 to 17 and indirectly shoots the range including the car windows 14 to 17 via the mirror surfaces 12a and 13a. The vehicle windows 14 to 17 can be opened and closed in the vertical direction by the occupants including the driver performing predetermined opening and closing operations.

  When the camera 3 performs a photographing operation, the camera 3 outputs an image signal including the photographed image to the vehicular device 2. The image signal output from the camera 3 to the vehicular apparatus 2 includes a direct image in which the camera 3 directly captures the range including the car windows 14 to 17 and a range including the camera window 14 to 17 as the mirror surface 12a. , 13a and an indirect image taken indirectly. In FIG. 2, broken lines C1 and C2 indicate ranges where the camera 3 directly shoots the car window 15 of the right rear seat, and broken lines D1 and D2 indicate ranges where the camera 3 directly shoots the car window 17 of the left rear seat. ing. The output of the image signal from the camera 3 to the vehicular apparatus 2 may be performed by data communication by wired connection or by data communication by wireless connection. For data communication by wire connection, for example, USB (Universal Serial Bus) may be used. For data communication by wireless connection, for example, Bluetooth (registered trademark), BLE (Bluetooth Low Energy), WiFi (Wireless Fidelity) (registered trademark) or the like may be used. Further, the camera 3 may continuously output the image signal to the vehicular device 2 or may intermittently output the image signal at a predetermined cycle (for example, a cycle of several millisecond intervals).

  The notification device 4 is a display, a speaker, or the like disposed in the vehicle compartment, and performs notification operations such as displaying notification information or outputting a sound when a notification command signal is input from the vehicular device 2. The output of the notification command signal from the vehicular device 2 to the notification device 4 may also be performed by data communication by wired connection or by data communication by wireless connection.

  The vehicular device 2 includes a control unit 5. The control unit 5 is configured by a microcomputer having a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), and an input / output (I / O). The control unit 5 executes the processing corresponding to the computer program by executing the computer program stored in the non-transitional tangible recording medium, and controls the overall operation of the vehicular device 2.

  The control unit 5 includes, as functions related to the present invention, a background component removal unit 5a, an image analysis unit 5b, a determination unit 5c, and a notification command unit 5d. These units 5a to 5d are configured by computer programs executed by the control unit 5, and are realized by software. The background component removing unit 5a processes the image signal input from the camera 3 to the vehicular apparatus 2 to extract a direct image and an indirect image, and applies an optical flow method to the direct image and the indirect image. Remove background components. Optical flow is a method of representing the motion of an object in an image as a vector. In this case, specifically, the background component removing unit 5a removes a region in which a vector of motion in the direction opposite to the traveling direction of the vehicle is generated as a background component.

  The image analysis unit 5b analyzes the direct image from which the background component has been removed and the indirect image from which the background component has been removed. The determination unit 5c specifies the position of the object using the analysis result by the image analysis unit 5b of the direct image and the analysis result by the image analysis unit 5b of the indirect image. It is judged whether or not it protrudes outside the vehicle through 17. When it is determined by the determination unit 5c that the object protrudes from the vehicle interior, the notification command unit 4d causes the notification device 4 to output a notification command signal, and causes the notification device 4 to perform a notification operation.

  Here, the principle of determining the position of an object by the determination unit 5c will be described. First, the principle of specifying the position of an object with a stereo camera will be described with reference to FIG. As shown in FIG. 3, the position of the first camera is M, the position of the second camera is N, the distance between the position of the first camera and the position of the second camera is B, any desired object The position of the point is P, the focal length of the first camera and the second camera is f, the position of an arbitrary point of the object in the image in which the first camera has photographed the object is pl, and from the image edge to pl The distance is xl, the position of an arbitrary point of the object in the image taken by the second camera is pr, the distance from the image edge to pr is xr, the position of the first camera and the position of the second camera Assuming that the distance from the position of the middle point to the position of an arbitrary point on the object is Z, the following equation holds from the relationship between parallax and depth.

| Xr-xl | / B = f / Z
| Xr-xl | is an absolute value of xr-xl. In this case, since B and f are known, Z can be calculated by calculating | xr−xl |, which is parallax, by image analysis.

  In this embodiment, in the principle shown in FIG. 3, this is established by replacing one of the two cameras with the mirror surface of the side mirror. That is, as shown in FIG. 4, the position of the mirror surface is M, the position of the camera is N, the distance between the position of the mirror surface and the position of the camera is B, the position of an arbitrary point of the object is P, The focal length is f, the position of an arbitrary point of the object in the image taken by the camera is pr, the distance from the image edge to pr is xr, and the position of the midpoint between the position of the mirror surface and the position of the camera Assuming that the distance to the position of an arbitrary point of the object is Z, and the position of an arbitrary point of the object in the image imaged by the mirror surface is pl, the distance from the image end to pl is xl / B, and the parallax and depth From the relationship, the following equation holds.

| Xr-xl / B | / B = f / Z
| Xr−xl / B | is an absolute value of xr−xl / B. Also in this case, since B and f are known, Z can be calculated by calculating | xr−xl / B |, which is parallax, by image analysis.

  Next, the operation of the above configuration will be described with reference to FIGS. 5 to 8. The control unit 5 performs object protrusion determination processing in relation to the present invention. When the object protrusion determination process is started, the control unit 5 receives an image signal from the camera 3 (S1), and performs signal processing on the input image signal to directly extract an image and an indirect image (S2). Next, the control unit 5 applies the optical flow method to the extracted direct image and indirect image to remove the background component (S3). Then, the control unit 5 analyzes the direct image from which the background component has been removed and the indirect image from which the background component has been removed (S4).

  Here, as shown in FIGS. 7 and 8, the control unit 5 directly images the range including the car window 15 of the right rear seat by the camera 3 and the mirror surface 12 a of the right side mirror 12 by the camera 3. The case of analyzing the indirect image whose captured range is included will be described. The broken line A shown in FIGS. 7 and 8 indicates an extension of the position of the car window 15, and indicates the boundary between the vehicle interior and the vehicle exterior. The control unit 5 calculates the parallax | xr−xl / B | according to the principle shown in FIG. 4 as a result of analyzing the direct image from which the background component has been removed and the indirect image from which the background component has been removed. The distance Z from the middle point (corresponding to the reference position) between the position of the mirror surface 12a and the position of the camera 3 to the position of an arbitrary point of the object is calculated as the object reaching distance (S5). The position of the mirror surface 12a is, for example, the central position of the mirror surface 12a, and the position of the camera 3 is, for example, the central position of the lens.

  The control unit 5 determines whether the calculation of the object reach distance is successful (S6), and when it is determined that the calculation of the object reach distance is successful (S6: YES), the calculated object reach distance is Based on the size of the car window 15 of the seat (that is, the length in the front-rear direction and the length in the height direction), it is compared with the setting range stored in advance to determine whether the object reach distance is within the setting range S7). If the control unit 5 determines that the calculation of the object arrival distance has failed (S6: NO), or if it determines that the object arrival distance is not within the set range (that is, outside the set range) (S7: NO), the right rear part It is specified that no object exists around the car window 15 of the seat (S8), and the object protrusion determination process is ended. On the other hand, when control unit 5 determines that the object reach distance is within the set range (S7: YES), it is one of the situation shown in FIG. 7 and the situation shown in FIG. It specifies that an object is present in the surroundings (S9).

  Next, the control unit 5 selects, for example, xr as the direction determination reference distance from among the above-described xr and xl / B (S10). The control unit 5 collates the selected direction determination reference distance with a setting value stored in advance based on the position of the vehicle window 15 of the right rear seat, and determines whether the direction determination reference distance is equal to or more than the setting value. To do (S11). In this case, as shown in FIG. 7, if there is an object around the window 15 of the right rear seat and there is an object in the vehicle compartment, that is, the object does not protrude from the vehicle interior, xr selected as the direction determination reference distance Is relatively long, and the direction from the position of the camera 3 to the position of an arbitrary point on the object is the direction toward the inside of the passenger compartment. On the other hand, as shown in FIG. 8, if an object exists around the car window 15 of the right rear seat and outside the vehicle, that is, the object protrudes from the vehicle interior, xr selected as the direction determination reference distance It becomes relatively short, and the direction from the position of the camera 3 to the position of an arbitrary point on the object is the direction inclined to the outside of the passenger compartment.

  When the control unit 5 determines that the direction determination reference distance is equal to or greater than the set value (S11: YES), the control unit 5 determines that an object is present in the vehicle compartment, that is, the object does not protrude from the vehicle compartment (S12), The object protrusion determination process ends. On the other hand, when control unit 5 determines that the direction determination reference distance is not less than the set value but less than the set value (S11: NO), an object is present outside the vehicle, that is, the object protrudes from the vehicle interior And (S13). Then, the control unit 5 outputs a notification command signal to the notification device 4 (S14), and ends the object protrusion determination process. At this time, when the notification device 4 receives a notification command signal from the vehicular device 2, the notification device 4 performs a notification operation such as displaying notification information or outputting a sound, and the driver indicates that the object protrudes from the vehicle interior. Alert to

  If the driver looks at the mirror surface 12a of the side mirror 12, the driver can recognize that the object protrudes from the vehicle interior. However, for example, the driver does not routinely look at the mirror surface 12a of the side mirror 12 Even if it is a situation that is difficult to visually check, it is possible to recognize that the object protrudes from the vehicle interior by the notification device 4 performing the notification operation as described above. In the above-described series of processes, S4 corresponds to the image analysis procedure, S5 to S13 correspond to the determination procedure, and S14 corresponds to the notification instruction procedure.

  Although the above illustrates the case where xr is selected as the direction determination reference distance among xr and xl / B, xl / B may be selected as the direction determination reference distance. Alternatively, both xr and xl / B may be selected as the direction determination reference distances, and the accuracy may be increased to determine whether the object protrudes from the vehicle interior by comparing each with the set value.

  Also, the above illustrates the case where it is determined whether an object protrudes from the passenger compartment to the outside of the passenger compartment via the window 15 of the right rear seat, but the range including the passenger window 17 of the left rear seat is photographed by the camera 3 Whether the object protrudes from the vehicle interior through the vehicle window 17 to the outside of the vehicle by analyzing the direct image taken and the indirect image in which the range including the mirror surface 13a of the left side mirror 13 is photographed by the camera 3 It is also possible to determine the Similarly, by adjusting the angle of the mirror surface 12a so that the vehicle window 14 of the driver's seat appears on the mirror surface 12a, it is determined whether an object protrudes from the vehicle interior through the vehicle window 14 of the driver's seat or not. It is also possible. Further, by adjusting the angle of the mirror surface 13a so that the vehicle window 16 of the front passenger seat appears on the mirror surface 13a, it is determined whether an object protrudes from the vehicle interior through the vehicle window 16 of the front passenger seat. It is also possible.

As described above, according to the present embodiment, the following effects can be obtained.
In the vehicle system 1, an analysis result of the direct image in which the range including the car windows 14 to 17 is captured by the camera 3 and an indirect image in which the range including the mirror surfaces 12a and 13a of the side mirrors 12 and 13 is captured by the camera 3 It is determined whether or not the object protrudes from the vehicle interior using the analysis result of Unlike the configuration that uses a stereo camera or a combination of a single-eye camera and a sensor, the configuration using only a single-eye camera makes it possible to appropriately determine whether the object protrudes from the vehicle interior without causing an increase in cost. It is determined that the object protrudes from the vehicle compartment can be appropriately notified. In addition, by applying the optical flow method to the direct image or the indirect image to remove the background component, it is possible to enhance the accuracy of detecting the object when analyzing the image, and the object protrudes from the vehicle interior. It is possible to improve the accuracy of determining whether the

The present invention is not limited to those exemplified in the above-described embodiment, and can be arbitrarily modified or expanded without departing from the scope thereof.
Although the side mirrors 12 and 13 illustrated the structure which is a door mirror, it may be a fender mirror attached to the front of the bonnet of a vehicle body. In the configuration in which the side mirrors 12 and 13 are fender mirrors, the distance from the camera 3 to the fender mirror is larger than the distance to the door mirror, and thus the image analysis capability may be enhanced.

  The camera 3 is not limited to the configuration in which the direction of 360 degrees is the field of view, but may be the range in which the range including at least the car windows 14 to 17 and the range including the mirror surfaces 12a and 13a of the side mirrors 12 and 13 are the field of view. Just do it. Also, the camera 3 may be disposed at a position close to the driver's eyes, and may be disposed, for example, at the side of a headrest at the driver's seat if the driver's head does not interfere with the field of view up to the mirror surfaces 12a and 13a. May be

  Although the configuration for performing the process of removing the background component by applying the optical flow method to both the direct image and the indirect image has been illustrated, it may be performed on either the direct image or the indirect image. In addition, the optical flow method may be applied as necessary, and the process of removing the background component may not be performed without applying the optical flow method.

  In the drawings, 1 is a vehicle system, 3 is a camera (shooting unit), 4 is a notification device (notification unit), 5 is a control unit, 5a is a background component removal unit, 5b is an image analysis unit, 5c is a determination unit, 5d Is a notification command unit, 12 and 13 are side mirrors, 12a and 13a are mirror surfaces, and 14 to 17 are car windows.

Claims (5)

A vehicle system (1) mounted on a vehicle equipped with a side mirror (12, 13) for reflecting an openable / closable car window (14 to 17),
An imaging unit (3) configured by a single-eye camera and imaging an area including at least the car window and an area including the mirror surface (12a, 13a) of the side mirror;
An image analysis unit (5b) that analyzes an image captured by the imaging unit;
The analysis result by the image analysis unit of the direct image in which the range including the car window is captured by the imaging unit by the image analysis unit, and the analysis result by the image analysis unit of the indirect image in which the range including the mirror surface is captured by the imaging unit A determination unit (5c) that determines whether an object protrudes out of the vehicle compartment to the outside of the vehicle via the vehicle window, using
And a notification command unit (5d) for notifying the determination result by the determination unit by a notification unit (4). In the vehicle system according to claim 1,
The vehicle system according to claim 1, wherein the determination unit determines whether the object protrudes from the vehicle interior through the vehicle window to the outside by specifying the position of the object relative to the position of the vehicle window. In the vehicle system according to claim 2,
The determination unit determines the distance from a reference position depending on the position of the imaging unit and the position of the side mirror to the position of the object, at least one of the position of the imaging unit and the position of the mirror surface. A system for a vehicle that determines the position of the object relative to the position of the car window using a direction toward the position of the object. The vehicle system according to any one of claims 1 to 3.
The background component removing unit (5a) removes background components of at least one of the direct image and the indirect image;
The system for a vehicle, wherein the image analysis unit analyzes at least one of the direct image from which a background component has been removed and the indirect image from which a background component has been removed. It is mounted on a vehicle equipped with a side mirror (12, 13) that mirrors the circumference of an openable car window (14 to 17) and includes at least a range including the car window and a mirror surface (12a, 13a) of the side mirror In the control unit (5) of the vehicle system (1) having a photographing unit (3) for photographing the range and a notification unit (4),
An image analysis procedure for analyzing a direct image in which the range including the car window is captured by the imaging unit and an indirect image in which the range including the mirror surface is captured;
A determination procedure for determining whether or not an object protrudes out of a vehicle compartment through the vehicle window to the outside of the vehicle using the analysis result of the image analysis procedure;
A computer program that causes a notification command procedure to notify the determination result of the determination procedure by the notification unit.

Images