JP7103773B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing apparatus and an image processing method.

In recent years, various security functions for vehicles have been proposed. For example, when a vehicle is injured, such as a broken window glass, the vehicle is equipped with a device that sounds an alarm such as a siren or blinks a lamp or the like. Further, for example, Patent Document 1 proposes a security function in which, when an approaching person is detected with respect to a vehicle, the approaching person is photographed by an in-vehicle camera and the image is transmitted to the mobile terminal of the owner of the vehicle.

Japanese Unexamined Patent Publication No. 2011-207444

However, when a person or other object approaching the vehicle is photographed from the viewpoint inside the vehicle as in the conventional technique, a part of the vehicle such as a window frame or a body may be recorded on the image so as to overlap with the object. be. That is, there is a problem that the whole picture of the object may not be grasped. Therefore, when the vehicle is harmed, there is a concern that even if a person or another object approaching the vehicle is recorded as an image, the image cannot be effectively used.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique capable of obtaining an effective image in grasping a situation and responding after the fact when a vehicle is harmed. ..

The image processing device according to the present invention is an image processing device that is mounted at least outside the vehicle and processes images taken by an in-vehicle camera that captures the entire circumference of the vehicle , and an occupant is inside the vehicle. The approach detection unit that predicts or detects external contact with the vehicle on which the in-vehicle camera is mounted when it is detected that the image is not present, and a predetermined position inside the vehicle as a viewpoint position, as viewed from the viewpoint position. The approach is provided with an image generation unit that generates an image outside the vehicle including the image of the vehicle as a transmission image with the direction of the contact position as the line-of-sight direction, and a communication unit that transmits the image outside the vehicle to an external device. When the detection unit predicts or detects the contact with the vehicle, the image generation unit generates the vehicle exterior image and transmits the vehicle exterior image to the external device via the communication unit (No. 1). 1 configuration).

Further, in the image processing device having the first configuration, the approach detection unit has a configuration (second configuration) of detecting an object approaching the vehicle and predicting the contact of the object with the vehicle (second configuration). You may.

Further, in the image processing device having the second configuration, the approach detection unit is based on the object by staying within a predetermined range from the vehicle for a predetermined time after the object approaches the vehicle. The configuration may be such that the contact with the vehicle is predicted (third configuration).

Further, in the image processing apparatus having the first to third configurations, when the approach detection unit predicts the contact with the vehicle, the image generation unit starts generating the image outside the vehicle, and further, the above. When the approach detection unit detects the contact with the vehicle, the communication unit may transmit the image outside the vehicle to the external device (fourth configuration).

Further, in the image processing apparatus having the first to fourth configurations, when the image generation unit determines that a person has invaded the inside of the vehicle at a timing different from the generation of the image outside the vehicle, Further, the vehicle interior image obtained by capturing the inside of the vehicle may be generated, and the communication unit may further transmit the vehicle interior image to the external device (fifth configuration).

Further, the image processing apparatus having the first to fifth configurations may have a configuration ( sixth configuration) including a storage unit capable of recording the image outside the vehicle.

Further, in the image processing device having the sixth configuration, the image outside the vehicle is stored in the storage unit based on the fact that the approach detection unit predicts the contact with the vehicle, and is stored in the storage unit. The outside image is stored in the storage unit when the approach detection unit detects the contact with the vehicle, and the storage is stored when the approach detection unit no longer predicts the contact with the vehicle. It may be a configuration ( seventh configuration) that is discarded from the unit.

Further, in the image processing devices having the first to seventh configurations, the communication unit may have a configuration ( eighth configuration) of transmitting a command related to recording the image outside the vehicle to the external device. ..

Further, the image processing method according to the present invention is an image processing method that processes an image taken by an in-vehicle camera that is mounted at least outside the vehicle and captures the entire circumference of the vehicle, and is inside the vehicle. The process of predicting or detecting external contact with the vehicle equipped with the in-vehicle camera when it is detected that there is no occupant, and the above -mentioned when external contact with the vehicle is predicted or detected. A step of generating an outside image including the image of the vehicle as a transmission image by setting a predetermined position inside the vehicle as the viewpoint position and the direction of the contact position as seen from the viewpoint position as the line-of-sight direction, and the outside image as the outside. It is a configuration ( nineth configuration) including a step of transmitting to the device.

According to the configuration of the present invention, for example, the face and behavior of a person approaching a vehicle can be confirmed as an image. That is, when the vehicle is harmed, it is possible to obtain an image that is effective in grasping the situation and responding after the fact.

Block diagram showing the configuration of the image processing apparatus of the embodiment The figure which illustrates the position where the in-vehicle camera is arranged in a vehicle Diagram to explain the method of generating a virtual viewpoint image A flowchart showing an example of a processing flow related to generation and transmission of an image outside the vehicle according to the first embodiment. Explanatory drawing showing the situation of external contact with the vehicle Schematic diagram showing the first example of the image outside the vehicle Schematic diagram showing an external device on which the first example of an outside vehicle image is displayed. Schematic diagram showing an example of an in-vehicle image of a modified example of the first embodiment Schematic diagram showing an example of an external device on which an image outside the vehicle and an image inside the vehicle are displayed. A flowchart showing an example of a processing flow related to generation and transmission of an image outside the vehicle according to the second embodiment. Explanatory drawing showing the situation of approaching the vehicle from the outside Schematic diagram showing the second example of the outside image A flowchart showing an example of a processing flow related to generation and transmission of an image outside the vehicle according to the third embodiment. A flowchart showing an example of a processing flow related to generation and transmission of an image outside the vehicle according to the fourth embodiment.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. The present invention is not limited to the contents of the embodiments shown below.

Further, in the following description, the direction from the driver's seat to the steering wheel, which is the straight-ahead traveling direction of the vehicle, is referred to as "forward direction (forward)". The direction in which the vehicle travels straight from the steering wheel to the driver's seat is defined as the "rear direction (rear)". The direction from the right side to the left side of the driver who is facing forward, which is the direction perpendicular to the straight line and the straight line of the vehicle, is defined as the "left direction". The direction from the left side to the right side of the driver who is facing forward, which is the direction perpendicular to the straight line and the straight line of the vehicle, is defined as the "right direction".

<1. Outline of image processing device>
FIG. 1 is a block diagram showing the configuration of the image processing device 1 of the embodiment. The image processing device 1 can be incorporated into a vehicle peripheral monitoring system including, for example, a photographing unit 2, a sensor unit 3, and an external device 4. The vehicle peripheral monitoring system is a system for monitoring the peripheral conditions of a vehicle equipped with a photographing unit 2 (vehicle-mounted camera).

The image processing device 1 is a device that processes a captured image captured by an in-vehicle camera. The image processing device 1 is provided for each vehicle equipped with an in-vehicle camera. In the present embodiment, the image processing device 1 acquires a photographed image from the photographing unit 2 and processes it. Further, the image processing device 1 acquires information from the sensor unit 3 and makes a determination regarding image processing based on the acquired information.

The photographing unit 2 is provided for the purpose of monitoring the conditions around the vehicle and inside the vehicle interior. The photographing unit 2 includes, for example, five cameras 21 to 25. The five cameras 21 to 25 are in-vehicle cameras. FIG. 2 is a diagram illustrating a position where the vehicle-mounted cameras 21 to 25 are arranged in the vehicle 5.

The in-vehicle camera 21 is provided at the front end of the vehicle 5. Therefore, the vehicle-mounted camera 21 is also referred to as a front camera 21. The optical axis 21a of the front camera 21 extends along the front-rear direction of the vehicle 5 in a plan view from above. The front camera 21 photographs the front direction of the vehicle 5. The vehicle-mounted camera 23 is provided at the rear end of the vehicle 5. Therefore, the in-vehicle camera 23 is also referred to as a back camera 23. The optical axis 23a of the back camera 23 extends along the front-rear direction of the vehicle 5 in a plan view from above. The back camera 23 photographs the rear direction of the vehicle 5. The front camera 21 and the back camera 23 are preferably mounted at the center of the left and right sides of the vehicle 5, but may be slightly deviated from the center of the left and right sides in the left-right direction.

The vehicle-mounted camera 22 is provided on the right door mirror 51 of the vehicle 5. Therefore, the in-vehicle camera 22 is also referred to as a right side camera 22. The optical axis 22a of the right side camera 22 extends along the left-right direction of the vehicle 5 in a plan view from above. The right side camera 22 photographs the vehicle 5 in the right direction. The vehicle-mounted camera 24 is provided on the left side door mirror 52 of the vehicle 5. Therefore, the in-vehicle camera 24 is also referred to as a left side camera 24. The optical axis 24a of the left side camera 24 extends along the left-right direction of the vehicle 5 in a plan view from above. The left side camera 24 photographs the vehicle 5 in the left direction.

When the vehicle 5 is a so-called door mirrorless vehicle, the right side camera 22 is provided around the rotation axis (hinge portion) of the right side door without passing through the door mirror, and the left side camera 24 is the rotation axis of the left side door. It is provided around the (hinge part) without using a door mirror.

The vehicle-mounted camera 25 is provided at the front end of the vehicle interior of the vehicle 5. Therefore, the in-vehicle camera 25 is also referred to as an in-vehicle camera 25. The in-vehicle camera 25 is arranged, for example, above the front side of the driver's seat or the passenger seat, or above the front side between the driver's seat and the passenger seat. The optical axis 25a of the in-vehicle camera 25 extends along the front-rear direction of the vehicle 5 in a plan view from above. The in-vehicle camera 25 photographs the inside of the vehicle interior from the front to the rear.

As the lenses of the in-vehicle cameras 21 to 25, for example, a fisheye lens is used. Since the in-vehicle cameras 21 to 24 have an angle of view θ of 180 degrees or more in the horizontal direction, the vehicle-mounted cameras 21 to 24 can photograph the entire circumference of the vehicle 5 in the horizontal direction. Similarly, since the in-vehicle camera 25 also has a horizontal angle of view θ of about 180 degrees, it is possible to photograph the entire area of the vehicle interior of the vehicle 5 in the horizontal direction.

Returning to FIG. 1, the sensor unit 3 has a plurality of sensors that detect information about the vehicle 5 on which the vehicle-mounted cameras 21 to 25 are mounted. The information about the vehicle 5 may include information about the vehicle itself and information around the vehicle. In the present embodiment, the sensor unit 3 is, for example, a seating sensor that detects the seating of a vehicle occupant on a seat, a shift sensor that detects the operating position of a shift lever of a vehicle transmission, and a door that detects the opening and closing of a vehicle door. Includes open / close sensors, vibration sensors that detect vehicle vibrations, tilt sensors that detect vehicle tilt, ultrasonic sensors and radars that detect people, animals, vehicles, and other objects around the vehicle. In addition to ultrasonic sensors and radars, optical sensors using infrared rays or the like may be used to detect humans and the like.

The external device 4 is a device that receives a display image output from the image processing device 1. The external device 4 includes, for example, a terminal, a server, or the like owned by the owner or the like of the vehicle 5. Examples of the terminal include a mobile terminal such as a smartphone, a stationary information terminal such as a personal computer, and the like. In the present embodiment, the external device 4 is, for example, a mobile terminal such as a smartphone, and is a display device including a display unit 41 that displays a display image output from the image processing device 1 on the screen. The display unit 41 may be, for example, a liquid crystal display panel, which may include an input unit such as a touch panel system, and may have a configuration in which information can be input from the outside.

<2. Details of image processing device>
The image processing device 1 includes an image generation unit 11, a control unit 12, a storage unit 13, and a communication unit 14.

The image generation unit 11 processes the captured image captured by the photographing unit 2 to generate a display image. In the present embodiment, the image generation unit 11 is configured as a hardware circuit capable of performing various image processing. In the present embodiment, the image generation unit 11 generates a virtual viewpoint image showing the periphery of the vehicle 5 as seen from a virtual viewpoint, based on the captured images taken by the vehicle-mounted cameras 21 to 24 mounted on the vehicle 5. Further, the image generation unit 11 generates a display image to be displayed by the external device 4 which is a display device based on the virtual viewpoint image. The details of the method for generating the virtual viewpoint image will be described later.

The control unit 12 is a so-called microcomputer including a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) (not shown). The storage unit 13 is a non-volatile memory. The control unit 12 processes and transmits / receives information based on the program stored in the storage unit 13. The control unit 12 is connected to the photographing unit 2, the sensor unit 3, and the external device 4 by wire or wirelessly.

The control unit 12 includes an image acquisition unit 121, an image control unit 122, and an approach detection unit 123. The functions of each of these constituent elements included in the control unit 12 are realized by the CPU performing arithmetic processing according to a program.

The image acquisition unit 121 acquires captured images taken by the vehicle-mounted cameras 21 to 25. In the present embodiment, the number of the vehicle-mounted cameras 21 to 25 is 5, and the image acquisition unit 121 acquires the captured images taken by the vehicle-mounted cameras 21 to 25, respectively.

The image control unit 122 controls the image processing executed by the image generation unit 11. For example, the image control unit 122 instructs the image generation unit 11 various parameters and the like necessary for generating the virtual viewpoint image and the display image. Further, the image control unit 122 controls when the display image generated by the image generation unit 11 is output to the external device 4.

The approach detection unit 123 predicts or detects external contact with the vehicle 5 on which the vehicle-mounted cameras 21 to 24 are mounted. Specifically, the approach detection unit 123 detects people, animals, vehicles, and other objects approaching the vehicle 5, and also detects contact by them with the vehicle 5.

More specifically, the approach detection unit 123 is based on detection signals from, for example, a seating sensor, a shift sensor, and a door open / close sensor of the sensor unit 3 prior to the prediction or start of detection of external contact with the vehicle 5. , Detects that there are no occupants inside the vehicle 5. For example, when the occupant is not seated and the ignition switch is OFF, it is detected that there is no occupant inside the vehicle 5. Further, for example, when the occupant is not seated and the door is closed, it is detected that there is no occupant inside the vehicle 5. Further, for example, when the occupant is not seated and the shift lever is in any of the "P", "N", and "R" positions, it is detected that there is no occupant inside the vehicle 5. The means for detecting that there is no occupant inside the vehicle 5 is not limited to these. For example, when the ignition switch is turned off, then the driver's door is closed and then closed, that is, when it is estimated that the driver has disembarked and all the doors are closed and locked, the inside of the vehicle 5 It may be detected that there is no occupant.

Subsequently, the approach detection unit 123 acquires a detection signal indicating the degree of vibration of the vehicle from, for example, a vibration sensor, and acquires a detection signal indicating the degree of inclination of the vehicle from the inclination sensor. Based on the acquisition of these detection signals, the approach detection unit 123 detects contact from the outside with the vehicle 5. Then, the image processing device 1 transmits a command to the photographing unit 2 to instruct the in-vehicle cameras 21 to 24 to photograph the surroundings of the vehicle. The approach detection unit 123 detects people, animals, vehicles, and other objects around the vehicle by image recognition based on the captured images around the vehicle taken by the vehicle-mounted cameras 21 to 24. As a result, the approach detection unit 123 detects the presence / absence, direction, and position of a person or vehicle approaching or contacting the vehicle 5.

It should be noted that, for example, the in-vehicle cameras 21 to 24 are periodically activated at regular intervals without using a vibration sensor, an inclination sensor, or the like as an activation trigger of the in-vehicle cameras 21 to 24, and the in-vehicle cameras 21 to 24 are used to display the surroundings of the vehicle. You may let me take a picture.

Further, for detecting people, animals, vehicles, and other objects around the vehicle, for example, an ultrasonic sensor or radar of the sensor unit 3 may be used. Ultrasonic sensors and radars are embedded in a plurality of locations such as the front bumper, the rear bumper, and the door of the vehicle 5. Ultrasonic sensors and radars transmit transmitted waves toward the periphery of the vehicle and receive reflected waves reflected by people and other vehicles to detect the presence, direction, and position of people and other vehicles. ..

The storage unit 13 is a non-volatile memory such as a flash memory, and stores various types of information. The storage unit 13 stores, for example, a program as firmware, and various data for the image generation unit 11 to generate a virtual viewpoint image and a display image. In addition, the storage unit 13 stores various data required for the image acquisition unit 121 and the approach detection unit 123 to execute the process.

The communication unit 14 is connected to the external device 4 by radio, for example. The image processing device 1 can output the display image generated by the image generation unit 11 to the external device 4 via the communication unit 14.

<3. Generation of virtual viewpoint image>
A method in which the image generation unit 11 generates a virtual viewpoint image showing the state of the surroundings of the vehicle 5 as seen from the virtual viewpoint will be described. FIG. 3 is a diagram for explaining a method for generating a virtual viewpoint image.

The front camera 21, the right side camera 22, the back camera 23, and the left side camera 24 simultaneously acquire four captured images P21 to P24 indicating the front, right side, rear side, and left side of the vehicle 5. These four captured images P21 to P24 include data on the entire circumference of the vehicle 5. The image generation unit 11 acquires these four captured images P21 to P24 via the image acquisition unit 121.

The image generation unit 11 projects the data (values of each pixel) included in these four captured images P21 to P24 onto the projection surface TS which is a three-dimensional curved surface in a virtual three-dimensional space. The projection surface TS has, for example, a substantially hemispherical shape (bowl shape), and its central portion (bottom portion of the bowl) is defined as the position of the vehicle 5.

On the projection surface TS, the data of the captured image is projected on the area outside the area of the vehicle 5. The correspondence between the position of each pixel included in the captured images P21 to P24 and the position of each pixel on the projection surface TS is determined in advance. The table data showing this correspondence is stored in the storage unit 13. The value of each pixel of the projection surface TS can be determined based on this correspondence and the value of each pixel included in the captured images P21 to P24.

Next, the image generation unit 11 sets the virtual viewpoint VP for the three-dimensional space under the control of the image control unit 122. The virtual viewpoint VP is defined by the viewpoint position and the line-of-sight direction. The image generation unit 11 can set a virtual viewpoint VP at an arbitrary viewpoint position and an arbitrary line-of-sight direction in a three-dimensional space. The image generation unit 11 cuts out the data projected on the area included in the field of view seen from the set virtual viewpoint VP in the projection surface TS as an image. As a result, the image generation unit 11 generates a virtual viewpoint image viewed from an arbitrary virtual viewpoint VP.

For example, as shown in FIG. 3, assuming a virtual viewpoint VPa with the viewpoint position directly above the vehicle 5 and the line-of-sight direction directly below, a virtual viewpoint image (overhead image) CPa that gives a bird's-eye view of the vehicle 5 and the surroundings of the vehicle 5 is obtained. Can be generated. Further, assuming a virtual viewpoint VPb in which the viewpoint position is the inside of the vehicle 5 and the line-of-sight direction is the left front of the vehicle 5, a virtual viewpoint image CPb showing the periphery of the left front of the vehicle 5 as seen from the inside of the vehicle 5 is generated. can do.

The image 5p of the vehicle 5 shown in the virtual viewpoint image CPa is prepared in advance as data such as a bitmap and stored in the storage unit 24. When the virtual viewpoint image is generated, the data of the image 5p of the vehicle 5 having a shape corresponding to the viewpoint position and the line-of-sight direction of the virtual viewpoint VP of the virtual viewpoint image is read out and included in the virtual viewpoint image CPa. Be done.

The virtual viewpoint image CPb of FIG. 3 will be described in more detail. In the virtual viewpoint image CPb, the viewpoint position is near the driver's viewpoint, which is a predetermined position inside the vehicle 5, and the left front of the vehicle 5 is the line-of-sight direction. It is an outside image of the vehicle 5 based on the virtual viewpoint. In the following description, a virtual viewpoint image in which a predetermined position inside the vehicle 5 is a viewpoint position and a direction outside the vehicle 5 as seen from the viewpoint position is a line-of-sight direction may be referred to as an outside vehicle image Px.

The virtual viewpoint image CPb, which is the external image Px, includes the transmission image 5t of the vehicle 5 in the virtual viewpoint image generated by the image generation unit 11. The vehicle exterior image Px is an image expressing a situation closer to the actual scene seen in the line-of-sight direction when the inside of the vehicle 5 is set as the viewpoint position of the virtual viewpoint, and the positional relationship between the vehicle 5 and the surrounding objects is also understood. Easy to do. In the vehicle exterior image Px, the image of the vehicle 5 is set as a transmission image 5t in order to prevent an object around the vehicle 5 from overlapping with a part of the vehicle 5 and hiding.

The transmission image 5t of the vehicle 5 shown in the vehicle exterior image Px is prepared in advance as data such as a bitmap and stored in the storage unit 24. When the vehicle exterior image is generated, the data of the transmission image 5t having a shape corresponding to the viewpoint position and the line-of-sight direction of the virtual viewpoint VP of the vehicle exterior image is read out and included in the vehicle exterior image Px.

In this way, the image generation unit 11 can generate virtual viewpoint images CPa and CPb that have a realistic sensation by using the virtual three-dimensional projection surface TS. Further, the image generation unit 11 sets a predetermined position inside the vehicle 5 as the viewpoint position of the virtual viewpoint VP, sets the direction outside the vehicle 5 as seen from the viewpoint position as the line-of-sight direction, and sets the image of the vehicle 5 as the transmission image 5t. Generate a vehicle exterior image Px including.

<4. Processing related to generation and transmission of images outside the vehicle>
In the present embodiment, the image processing device 1 generates the vehicle exterior image Px and transmits the vehicle exterior image Px to the external device 4.

<4-1. First Embodiment>
FIG. 4 is a flowchart showing an example of a processing flow related to the generation and transmission of the vehicle exterior image Px of the first embodiment. The processing related to the generation and transmission of the vehicle exterior image Px of the first embodiment will be described with reference to FIG.

As shown in FIG. 4, first, in the vehicle 5, the contact from the outside with the vehicle 5 is monitored by the approach detection unit 123. (Step S101). When the predetermined start condition is satisfied, the image processing device 1 starts monitoring the contact from the outside to the vehicle 5. The predetermined start condition may be, for example, when there are no occupants inside the vehicle 5, as described above. For example, when the occupant is not seated and the ignition switch is OFF, it is detected that there is no occupant inside the vehicle 5, the monitoring mode is entered, and the monitoring of the external contact with the vehicle 5 is started. To. The monitoring mode is canceled when a legitimate user starts driving the vehicle. For example, if the door is unlocked by a key, or if the door is unlocked by a transmitter of a keyless entry device (not shown), it is determined that a legitimate user starts driving the vehicle.

Next, it is determined whether or not an external contact with the vehicle 5 has been detected (step S102). The external contact with the vehicle 5 is determined based on, for example, a detection signal indicating the degree of vibration of the vehicle acquired from the vibration sensor of the sensor unit 3. Further, the external contact with the vehicle 5 is determined based on, for example, a detection signal indicating the degree of inclination of the vehicle acquired from the inclination sensor of the sensor unit 3. Further, for detecting an external contact with the vehicle 5, for example, an ultrasonic sensor or radar of the sensor unit 3 may be used.

If no external contact with the vehicle 5 is detected (No in step S102), the process returns to step S101 to continue monitoring the contact of the object with the vehicle 5.

When an external contact with the vehicle 5 is detected (Yes in step S102), a peripheral image of the vehicle 5 is acquired (step S103). FIG. 5 is an explanatory diagram showing a state of contact with the vehicle 5 from the outside. As shown in FIG. 5, in the present embodiment, for example, it is assumed that the human Tr comes into contact with the parked vehicle 5. The contact position is the left front portion 5f of the vehicle 5. The peripheral image of the vehicle 5 can be acquired by having the in-vehicle cameras 21 to 24 take a picture of the periphery of the vehicle. The image processing device 1 transmits a command to the photographing unit 2 instructing the in-vehicle cameras 21 to 24 to photograph the surroundings of the vehicle. The image processing device 1 acquires captured images taken by the in-vehicle cameras 21 to 24 via the image acquisition unit 121 and stores them in the storage unit 13.

Returning to FIG. 4, the contact position and contact direction are then detected (step S104). For example, the contact position and the contact direction are detected based on the captured images around the vehicle taken by the vehicle-mounted cameras 21 to 24. The approach detection unit 123 detects the contact position and contact direction of the person Tr that comes into contact with the vehicle 5 from the outside. Further, for example, the contact position and the contact direction can be detected by using an ultrasonic sensor or a radar of the sensor unit 3.

Next, a virtual viewpoint image is generated (step S105). The image generation unit 11 generates a virtual viewpoint image showing the state of the surroundings of the vehicle 5 as seen from a virtual viewpoint, based on a plurality of captured images around the vehicle acquired in step S103 and stored in the storage unit 13.

Next, the image generation unit 11 generates an external image Px including the transmission image 5t of the vehicle 5 and stores it in the storage unit 13 (step S106). FIG. 6 is a schematic view showing a first example of the vehicle exterior image Px. For example, in the present embodiment, it is assumed that in step S104, an external contact with the vehicle 5 is detected at the left front portion 5f of the vehicle 5. As a result, the image generation unit 11 sets the vicinity of the driver's viewpoint, which is a predetermined position inside the vehicle 5, as the viewpoint position, and the left front portion 5f of the vehicle 5, which is the direction of the contact position as seen from the viewpoint position, is the line-of-sight direction. The vehicle exterior image Px shown in FIG. 6 is generated based on the virtual viewpoint VPx (see FIG. 5).

The vehicle exterior image Px includes an image of the vehicle 5 as a transmission image 5t in the virtual viewpoint image generated by the image generation unit 11. The virtual viewpoint image of the vehicle exterior image Px includes the image Tp of the person Tr in contact with the vehicle 5. Further, the image generation unit 11 generates a display image of the vehicle exterior image Px for display by the external device 4 based on the virtual viewpoint image of the vehicle exterior image Px.

In FIG. 6 and FIG. 7 described later, the image Tp of the person Tr in contact with the vehicle 5 included in the external image Px is drawn in a simplified manner, but in reality, the person Tr in contact is a suspicious person. It is displayed to the extent that it can be identified. Further, the same applies to the image Tp of the human Tr in other drawings described later.

Returning to FIG. 4, the communication unit 14 then transmits the vehicle exterior image Px to the external device 4 (step S107). The image processing device 1 creates, for example, an e-mail or the like to which a display image of the vehicle exterior image Px is attached, and transmits the e-mail to the external device 4 via the communication unit 14. In the e-mail or the like, for example, a message notifying that the human Tr has contacted the vehicle 5 is described.

FIG. 7 is a schematic view showing an external device 4 on which the first example of the vehicle exterior image Px is displayed. For example, in the external device 4 which is a mobile terminal such as a smartphone, the display unit 41 can display the image Px outside the vehicle. When the vehicle exterior image Px is transmitted to the external device 4, the processing flow shown in FIG. 4 is terminated.

The image Px outside the vehicle can be transmitted to the external device 4 by attaching the image data to an e-mail or the like, or by transmitting, for example, an address on the network in which the image Px outside the vehicle is stored. be. In this case, the vehicle exterior image Px is stored in the storage unit 13 of the image processing device 1 or another external device (server or the like) on the network.

Further, the vehicle exterior image Px that can be displayed by the external device 4 is an image (still image) that has already been captured, or even a video (moving image) of the captured vehicle exterior image Px generated by the image generation unit 11. good. Further, the video (moving image) of the external image Px may be a real-time (live broadcast) video showing the current situation. In these cases, the e-mail or the like transmitted from the image processing device 1 to the external device 4 contains a message notifying the network address, viewing method, etc. for accessing the video which is the real-time external image Px. ..

Further, the still image and the moving image of the vehicle exterior image Px can be recorded and recorded. The recording and recording of the still image and the moving image of the external image Px may be performed by the storage unit 13 of the image processing device 1 or may be performed by the storage unit (not shown) of the external device 4. When the external device 4 is to record the external image Px, the communication unit 14 transmits a command related to the recording of the external image Px to the external device 4 in addition to transmitting the external image Px itself.

In addition, the display unit 41 of the external device 4 can perform operations such as enlargement and reduction of the image Px outside the vehicle and its real-time image. In this case, for example, the operation can be performed using a touch panel or the like provided on the display unit 41.

Further, on the display unit 41 of the external device 4, the viewpoint of the external image Px and its real-time image may be freely switched. In this case, for example, the viewpoint can be switched by operating using a touch panel or the like provided on the display unit 41.

Specifically, on the display unit 41 of the external device 4, a new viewpoint position and line-of-sight direction are determined using a touch panel or the like. The external device 4 transmits the determined viewpoint position and line-of-sight direction information (hereinafter referred to as line-of-sight information) to the communication unit 14. The image control unit 122 causes the image generation unit 11 to generate a new vehicle exterior image Px or real-time image based on the line-of-sight information received via the communication unit 14, and transmits the new vehicle exterior image Px or real-time image from the communication unit 14 to the external device 4. In this way, when the viewpoint position and the line-of-sight direction are changed by the external device 4, the changed new external image Px or real-time image is displayed on the display unit 41 of the external device 4. Further, a plurality of vehicle exterior image Px having different viewpoint positions may be displayed on the display unit 41 at the same time.

As described above, when the image processing device 1 of the present embodiment detects an external contact with the vehicle 5, it generates an external image Px including the image of the vehicle 5 as a transmission image 5t in the virtual viewpoint image. The vehicle exterior image Px is transmitted to the external device 4. As a result, the face and behavior of the person Tr who has come into contact with the vehicle 5 can be confirmed as the image Px outside the vehicle. That is, when the vehicle 5 is harmed, it is possible to obtain an external image Px that is effective in grasping the situation and responding after the fact.

Further, the approach detection unit 123 detects contact from the outside with the vehicle 5 based on the vibration or inclination of the vehicle 5. Thereby, the timing of the contact of the object with the vehicle 5 can be appropriately determined. Immediately after the object comes into contact with the vehicle 5, the face and behavior of the person Tr who has come into contact with the vehicle 5 can be confirmed as an external image Px.

Further, the approach detection unit 123 detects that there is no occupant inside the vehicle 5, and further starts detecting contact with the vehicle 5 from the outside. As a result, the image processing device 1 does not operate when there is an occupant inside the vehicle 5, and can be operated at an appropriate timing.

Further, the vehicle exterior image Px transmitted to the external device 4 is a still image or a moving image. Further, the moving image of the outside vehicle image Px is a live broadcast image. As a result, the face and behavior of the person Tr who has come into contact with the vehicle 5 can be seen, for example, by still images, moving images, and live broadcast images. Therefore, it is more effective in grasping the situation and responding after the fact when the vehicle 5 is harmed.

Further, the image processing device 1 includes a storage unit 13 capable of recording an image Px outside the vehicle. Thereby, it is possible to see the vehicle exterior image Px of the image processing device 1 at an arbitrary timing.

Further, in addition to transmitting the vehicle exterior image Px itself, the communication unit 14 transmits a command related to recording the vehicle exterior image Px to the external device 4. As a result, the vehicle exterior image Px can be recorded in the external device 4. Therefore, in the external device 4, it is possible to see the vehicle exterior image Px of the image processing device 1 at an arbitrary timing.

<4-1-1. Modification example of the first embodiment>
FIG. 8 is a schematic view showing an example of an in-vehicle image Py of a modified example of the first embodiment. In the image processing device 1 of the modified example, the image generation unit 11 generates the in-vehicle image Py. The image generation unit 11 further generates an in-vehicle image Py in which the inside of the vehicle 5 is photographed in addition to the in-vehicle image Px.

As the in-vehicle image Py, a photographed image taken by the in-vehicle camera 25 is used. For example, when a human Tr invades the inside of the vehicle 5, the in-vehicle image Py includes the image Tp of the human Tr invading the vehicle 5 in the image representing the inside of the vehicle 5. Further, the image generation unit 11 generates a display image of the vehicle interior image Py to be displayed by the external device 4 based on the vehicle interior image Py. Whether or not a human Tr has invaded the inside of the vehicle 5 can be determined by detecting that the door is opened without being unlocked during the monitoring mode, but the present invention is not limited to this.

The communication unit 14 further transmits the in-vehicle image Py to the external device 4 in addition to the in-vehicle image Px. The image processing device 1 creates, for example, an e-mail or the like to which a display image of the vehicle exterior image Px and a display image of the vehicle interior image Py are attached, and transmits the e-mail to the external device 4 via the communication unit 14. In the e-mail or the like, for example, a message notifying that the human Tr has invaded the inside of the vehicle 5 is described.

FIG. 9 is a schematic view showing an example of an external device 4 on which the vehicle exterior image Px and the vehicle interior image Py are displayed. For example, in the external device 4 which is a mobile terminal such as a smartphone, the display unit 41 can display the vehicle exterior image Px and the vehicle interior image Py.

The display unit 41 of the external device 4 may display both the outside image Px and the inside image Py at the same time, or may individually select and display either the outside image Px or the inside image Py. good.

According to this configuration, the face and behavior of the person Tr who has invaded the inside of the vehicle 5 can be confirmed as an in-vehicle image Py. That is, when the vehicle 5 is harmed, it becomes possible to obtain an outside image Px and an inside image Py that are even more effective in grasping the situation and responding after the fact.

<4-2. Second Embodiment>
FIG. 10 is a flowchart showing an example of a processing flow related to the generation and transmission of the vehicle exterior image Px of the second embodiment. Since the basic configuration of the second embodiment is the same as that of the first embodiment described above, the configuration different from the first embodiment will be described below, and the configuration common to the first embodiment will be described below. A detailed description will be omitted.

In the image processing device 1 of the second embodiment, the approach detection unit 123 detects an object approaching the vehicle 5 and predicts the contact of the object with the vehicle 5. That is, as shown in FIG. 10, first, in the vehicle 5, a person, an animal, a vehicle, or another object approaching the vehicle 5 from the outside is monitored by the approach detection unit 123 (step S201). In the image processing device 1, when a predetermined start condition such as when there is no occupant inside the vehicle 5 is satisfied, the monitoring mode is set and the monitoring of the approach to the vehicle 5 from the outside is started.

Next, it is determined whether or not the approach to the vehicle 5 from the outside is detected (step S202). Regarding the detection of an object approaching the vehicle 5, in detail, the approach detection unit 123 predicts the contact of the object with the vehicle 5 by entering the object within a predetermined range from the vehicle 5. As a predetermined range from the vehicle 5, for example, a range such as within a radius of 3 m can be arbitrarily set. The detection of people, animals, vehicles, and other objects around the vehicle is determined based on, for example, image recognition based on the ultrasonic sensor of the sensor unit 3, the detection signal of the radar, or the captured image of the cameras 21 to 24.

When the approach to the vehicle 5 from the outside is detected (Yes in step S202), the peripheral image of the vehicle 5 is acquired (step S203). Hereinafter, detection of the approaching position and approaching direction (step S204), generation of a virtual viewpoint image (step S205), generation and storage of an external image Px including a transmission image 5t of the vehicle 5 (step S206), an external device of the external device Px. The transmission to 4 (step S207) is sequentially performed.

In the present embodiment, an object approaching the vehicle 5 is detected and the contact of the object with the vehicle 5 is predicted. Therefore, the predicted contact position is the position of the object approaching the vehicle 5. FIG. 11 is an explanatory diagram showing a situation of approaching the vehicle 5 from the outside. FIG. 12 is a schematic view showing a second example of the vehicle exterior image Px. .. 11 and 12 are diagrams showing a situation immediately before the contact of the human Tr with the vehicle 5 in the first example of the vehicle exterior image Px described with reference to, for example, FIGS. 5 and 6. That is, in the second example of the external image Px, the position of the person Tr approaching the vehicle 5 is the left front portion 5f of the vehicle 5.

Therefore, the image generation unit 11 has the viewpoint position near the driver's viewpoint, which is a predetermined position inside the vehicle 5, and the left front portion 5f of the vehicle 5, which is the position of the person Tr approaching the vehicle 5 as seen from the viewpoint position. The vehicle exterior image Px is generated based on the virtual viewpoint with the direction of the line of sight as the line-of-sight direction (step S206). The virtual viewpoint image of the vehicle exterior image Px includes an image Tp of a person Tr approaching the vehicle 5 in addition to the transmission image 5t of the vehicle 5.

The details of steps S203 to S205 and S207 are the same as those in the first embodiment, and thus the description thereof will be omitted.

As described above, in the image processing device 1 of the second embodiment, the approach detection unit 123 detects an object (human Tr) approaching the vehicle 5 and predicts the contact of the object (human Tr) with the vehicle 5. .. As a result, when the human Tr approaches the vehicle 5, the face and behavior of the human Tr can be confirmed as an external image Px. That is, after that, when the vehicle 5 is harmed, it becomes possible to obtain an external image Px that is effective in grasping the situation and responding after the fact.

<4-3. Third Embodiment>
FIG. 13 is a flowchart showing an example of a processing flow related to the generation and transmission of the vehicle exterior image Px of the third embodiment. Since the basic configuration of the third embodiment is the same as that of the first and second embodiments described above, configurations different from these embodiments will be described below and are common to these embodiments. A detailed description of the configuration to be performed will be omitted.

In the image processing device 1 of the third embodiment, as shown in FIG. 13, first, in the vehicle 5, a person, an animal, a vehicle, or another object approaching the vehicle 5 from the outside is monitored by the approach detection unit 123. (Step S301). In the image processing device 1, when a predetermined start condition such as when there is no occupant inside the vehicle 5 is satisfied, the monitoring mode is set and the monitoring of the approach to the vehicle 5 from the outside is started.

Next, it is determined whether or not the approach to the vehicle 5 from the outside is detected (step S302). Regarding the detection of an object approaching the vehicle 5, in detail, the approach detection unit 123 predicts the contact of the object with the vehicle 5 by entering the object within a predetermined range (for example, within a radius of 3 m) from the vehicle 5. .. Detection of people, animals, vehicles, and other objects around the vehicle is determined based on, for example, image recognition based on the ultrasonic sensor of the sensor unit 3, the detection signal of the radar, or the captured image of the cameras 21 to 24.

When the approach to the vehicle 5 from the outside is detected (Yes in step S302), it is determined whether or not a predetermined time has elapsed while the object is approaching the vehicle 5 (step S303). Specifically, the approach detection unit 123 predicts the contact of the object with the vehicle 5 by staying within a predetermined range from the vehicle 5 for a predetermined time after the object approaches the vehicle 5. As the predetermined time for staying the object, a time such as 1 minute can be arbitrarily set. The detection of an object around the vehicle is subsequently determined based on, for example, an ultrasonic sensor of the sensor unit 3, a radar detection signal, or the like.

When the object is approaching the vehicle 5 and the predetermined time has not elapsed (No in step S303), the process returns to step S302 and the monitoring of the approaching state of the object is continued. Further, when the approach of the object to the vehicle 5 is not detected, or when the approach of the object to the vehicle 5 is not detected (No in step S302), the process returns to step S301 to monitor the approach of the object to the vehicle 5. Is continued.

When a predetermined time elapses with the object approaching the vehicle 5 (Yes in step S303), the peripheral image of the vehicle 5 is acquired (step S304). Hereinafter, detection of the approaching position and approaching direction (step S305), generation of a virtual viewpoint image (step S306), generation and storage of an external image Px including a transmission image 5t of the vehicle 5 (step S307), an external device of the external device Px. The transmission to 4 (step S308) is sequentially performed. The details of steps S304 to S308 are the same as those of the first and second embodiments, and thus the description thereof will be omitted.

As described above, in the image processing device 1 of the third embodiment, the approach detection unit 123 depends on the object by allowing the object to stay within a predetermined range from the vehicle 5 for a predetermined time after the object approaches the vehicle 5. Predict contact with vehicle 5. Thereby, for example, when a person approaches the vehicle 5 and stays in the vicinity of the vehicle for a long time, the face and behavior of the person can be confirmed as an outside image Px. Then, for example, when a person, an animal, a vehicle, or another object simply passes around the vehicle, the external image Px can be prevented from being generated or transmitted. Therefore, it is possible to efficiently obtain an external image Px that is effective in grasping the situation and responding after the fact when the vehicle 5 is harmed.

<4-4. Fourth Embodiment>
FIG. 14 is a flowchart showing an example of a processing flow related to the generation and transmission of the vehicle exterior image Px of the fourth embodiment. Since the basic configuration of the fourth embodiment is the same as that of the first, second, and third embodiments described above, configurations different from these embodiments will be described below, and these embodiments will be described. Detailed description of the configuration common to the form will be omitted.

In the image processing device 1 of the fourth embodiment, as shown in FIG. 14, first, in the vehicle 5, a person, an animal, a vehicle, or another object approaching the vehicle 5 from the outside is monitored by the approach detection unit 123. (Step S401). Hereinafter, determination of the approach of an object to the vehicle 5 (step S402), acquisition of a peripheral image of the vehicle 5 (step S403), detection of an approach position and approach direction (step S404), generation of a virtual viewpoint image (step S405), Generation and storage (step S406) of the external image Px including the transmission image 5t of the vehicle 5 is sequentially performed. The details of steps S401 to S406 are the same as those of the first and second embodiments, and thus the description thereof will be omitted.

Next, it is determined whether or not an external contact with the vehicle 5 has been detected (step S407). External contact with the vehicle 5 is determined based on detection signals of, for example, a vibration sensor, an inclination sensor, an ultrasonic sensor, a radar, or the like of the sensor unit 3.

When an external contact with the vehicle 5 is detected (Yes in step S407), the communication unit 14 transmits the vehicle exterior image Px to the external device 4 (step S408). When the vehicle exterior image Px is transmitted to the external device 4, the processing flow shown in FIG. 14 is terminated.

When no external contact with the vehicle 5 is detected (No in step S407), it is determined whether or not the object is in a state of approaching the vehicle 5 (step S409). Subsequently, the detection of an object around the vehicle is determined based on, for example, the ultrasonic sensor of the sensor unit 3, the detection signal of the radar, and the like. When the object is in a state of approaching the vehicle 5 (Yes in step S409), the process returns to step S407 to determine the contact of the object with the vehicle 5.

When the approach of the object to the vehicle 5 is no longer detected (No in step S409), the outside image Px generated in step S406 is discarded (step S410). That is, the image processing device 1 erases the vehicle exterior image Px temporarily stored in the storage unit 13. Then, returning to step S401, monitoring of the approach of the object to the vehicle 5 is continued.

As described above, in the image processing device 1 of the fourth embodiment, when the approach detection unit 123 predicts the external contact with the vehicle 5, the image generation unit 11 starts generating the external image Px, and further. When the approach detection unit 123 detects an external contact with the vehicle 5, the communication unit 14 transmits the vehicle exterior image Px to the external device 4. As a result, for example, when a person approaches the vehicle 5, the face and behavior of the person can be quickly confirmed as an external image Px. Further, only when the vehicle actually touches the vehicle 5, the vehicle exterior image Px can be transmitted to the external device 4. Therefore, it becomes possible to obtain an external image Px that is more efficient and effective in grasping the situation and responding after the fact when the vehicle 5 is harmed.

Further, the external image Px is stored in the storage unit 13 based on the fact that the approach detection unit 123 predicts the contact with the vehicle 5, that is, the detection of the approach to the vehicle 5, and is stored in the storage unit 13. The vehicle exterior image Px is stored in the storage unit 13 when the approach detection unit 123 detects contact with the vehicle 5, and the approach detection unit 123 no longer predicts contact with the vehicle 5, that is, the approach to the vehicle 5. When is no longer detected, it is discarded from the storage unit 13. Therefore, the storage capacity of the storage unit 13 can be effectively utilized.

<5. Others>
The various technical features disclosed herein can be modified in addition to the above embodiments without departing from the spirit of the technical creation. That is, it should be considered that the above embodiment is exemplary in all respects and is not restrictive. It should be understood that the technical scope of the present invention is indicated by the scope of claims, not the description of the above embodiment, and includes all modifications belonging to the meaning and scope equivalent to the scope of claims. Is. Moreover, the above-mentioned plurality of embodiments and examples may be carried out in combination to the extent possible.

For example, in the second, third, and fourth embodiments, in addition to the vehicle exterior image Px, the vehicle interior image Py may be further transmitted to the external device 4. As a result, the in-vehicle image Py can be effectively used in grasping the situation of the vehicle 5.

Further, in the above embodiment, it has been explained that various functions are realized by software by the arithmetic processing of the CPU according to the program, but some of these functions are realized by the electric hardware circuit. You may. On the contrary, some of the functions realized by the hardware circuit may be realized by software.

1 Image processing device 3 Sensor unit 4 External device 5 Vehicle 11 Image generation unit 12 Control unit 13 Storage unit 14 Communication unit 21 Front camera (vehicle-mounted camera)
22 Right side camera (vehicle-mounted camera)
23 Back camera (vehicle-mounted camera)
24 Left side camera (vehicle-mounted camera)
123 Approach detector

Claims (9)

An image processing device that processes at least an image taken by an in-vehicle camera that is mounted outside the vehicle and captures the entire circumference of the vehicle.
An approach detection unit that predicts or detects external contact with the vehicle equipped with the in-vehicle camera when it detects that there is no occupant inside the vehicle.
An image generation unit that generates an outside image including the image of the vehicle as a transmission image, with a predetermined position inside the vehicle as the viewpoint position and the direction of the contact position as seen from the viewpoint position as the line-of-sight direction.
A communication unit that transmits the outside image to an external device is provided.
When the approach detection unit predicts or detects the contact with the vehicle, the image generation unit generates the vehicle exterior image and transmits the vehicle exterior image to the external device via the communication unit. Processing equipment. The image processing device according to claim 1, wherein the approach detection unit detects an object approaching the vehicle and predicts the contact of the object with the vehicle. The approach detection unit according to claim 2 predicts the contact of the object with the vehicle by staying within a predetermined range from the vehicle for a predetermined time after the object approaches the vehicle. The image processing apparatus described. When the approach detection unit predicts the contact with the vehicle, the image generation unit starts generating the outside image, and further
The image processing device according to any one of claims 1 to 3, wherein when the approach detection unit detects the contact with the vehicle, the communication unit transmits the image outside the vehicle to the external device. The image generation unit further generates an in-vehicle image in which the inside of the vehicle is photographed when it is determined that a person has invaded the inside of the vehicle at a timing different from the generation of the outside image .
The image processing device according to any one of claims 1 to 4, wherein the communication unit further transmits the in-vehicle image to the external device. The image processing device according to any one of claims 1 to 5 , further comprising a storage unit capable of recording an image outside the vehicle. The outside image is stored in the storage unit based on the fact that the approach detection unit predicts the contact with the vehicle.
The image of the outside of the vehicle stored in the storage unit is
When the approach detection unit detects the contact with the vehicle, it is stored in the storage unit.
The image processing device according to claim 6 , wherein when the approach detection unit does not predict the contact with the vehicle, the image processing device is discarded from the storage unit. The image processing device according to any one of claims 1 to 7 , wherein the communication unit transmits a command for recording an image outside the vehicle to the external device. It is an image processing method that processes an image taken by an in-vehicle camera that is mounted at least outside the vehicle and captures the entire circumference of the vehicle.
A process of predicting or detecting external contact with the vehicle equipped with the in-vehicle camera when it is detected that there is no occupant inside the vehicle.
When an external contact with the vehicle is predicted or detected, the predetermined position inside the vehicle is set as the viewpoint position, and the direction of the contact position viewed from the viewpoint position is set as the line-of-sight direction, and the image of the vehicle. And the process of generating an image of the outside of the vehicle including as a transmission image
An image processing method including a step of transmitting an image outside the vehicle to an external device.

Images