JP2020145612A - Image processing apparatus, image processing system, image processing method, and program - Google Patents

Abstract

To provide an image processing apparatus, an image processing system, an image display method, and a program, capable of transmitting an image according to operation of a vehicle.SOLUTION: An image processing apparatus 100 includes: an image acquisition unit 101 for acquiring one or a plurality of images captured by an imaging unit 2 disposed in a vehicle 4; a vehicle information acquisition unit 102 for acquiring vehicle information related to running of the vehicle 4; a transmission image generation unit 104 for generating a transmission image by performing at least one of cutting out a part of the captured image based on the vehicle information, selecting a transmission image from at least one of the captured images, and synthesizing the plurality of captured images; and a communication unit 105 for transmitting a transmission image by wireless communication.SELECTED DRAWING: Figure 3

Description

The present invention relates to an image processing apparatus, an image processing system, an image processing method, and a program.

Patent Document 1 discloses a vehicle information processing device that performs vehicle-to-vehicle communication with other vehicles. A user in a vehicle specifies the traveling direction and angle width of another vehicle in order to reduce the amount of communication data. The other vehicle cuts out an image in a range corresponding to the specified traveling direction and angle width and transmits it.

Japanese Unexamined Patent Publication No. 2006-115360

In the vehicle information processing device of Patent Document 1, a user of another vehicle specifies a traveling direction and an angle width. There is a problem that it is not possible to acquire an image corresponding to the movement of the vehicle.

In view of the above problems, an object of the present disclosure is to provide an image processing device, an image processing system, and an image processing method capable of transmitting an image according to the operation of a vehicle.

The image processing apparatus according to the present embodiment includes an image capturing image acquisition unit that acquires one or a plurality of captured images captured by an imaging unit arranged in the vehicle, and a vehicle information acquisition unit that acquires vehicle information related to the running of the vehicle. By performing at least one of cutting out a part of the captured image, selecting a transmission image from the plurality of captured images, and synthesizing the plurality of captured images based on the vehicle information. , A transmission image generation unit that generates a transmission image, and a communication unit that transmits the transmission image by wireless communication are provided.

The image processing method according to the present embodiment includes a step of acquiring one or a plurality of captured images captured by an imaging unit arranged in the vehicle, a step of acquiring vehicle information regarding the traveling of the vehicle, and the vehicle information. Based on this, a transmission image is generated by cutting out a part of the captured image, selecting a transmission image from the plurality of captured images, and synthesizing the plurality of captured images. It includes a step and a step of transmitting the transmitted image by wireless communication.

The program according to the present embodiment is based on a step of acquiring one or a plurality of captured images captured by an imaging unit arranged in a vehicle, a step of acquiring vehicle information regarding the running of the vehicle, and the vehicle information. A step of generating a transmission image by cutting out a part of the captured image, selecting a transmission image from the plurality of captured images, and synthesizing the plurality of captured images. , The step of transmitting the transmitted image by wireless communication, and the step of causing the computer to execute.

According to the present disclosure, it is possible to provide an image processing device, an image processing system, an image processing method, and a program capable of transmitting an image according to the operation of a vehicle.

It is a figure which shows the image processing system which concerns on this embodiment. It is a figure for demonstrating the arrangement example of the camera of the image pickup part. It is a figure which shows typically the captured image which imaged the vehicle interior. It is a figure which shows typically the captured image which imaged the front of the vehicle. It is a control block diagram of the image processing system which concerns on this embodiment. It is a figure for demonstrating the transmission image generated based on the posture information. It is a flowchart which shows the image processing method which concerns on this Embodiment. It is a figure for demonstrating the arrangement example of the camera of the image pickup part. It is a figure for demonstrating the transmission image when the image size is changed according to the vehicle information. It is a figure for demonstrating the transmission image at the time of changing the cut-out position according to the vehicle information. It is a top view for demonstrating the arrangement example of a camera and the imaging range.

Hereinafter, an example of the embodiment of the present invention will be described with reference to the drawings. The following description shows a preferred embodiment of the present invention, and the technical scope of the present invention is not limited to the following embodiments. In each drawing, the same or corresponding elements are designated by the same reference numerals, and duplicate description will be omitted as necessary for the sake of clarity of description.

Embodiment 1
FIG. 1 is a diagram schematically showing an image processing system 1 according to the first embodiment. The image processing system 1 includes an image pickup unit 2, VR (Virtual Reality) goggles 3, an image processing device 100 provided in the vehicle 4, and a server device 5.

The image pickup unit 2 includes one or a plurality of cameras, and captures an image of the outside or the inside of the vehicle 4. The imaging unit 2 is arranged in the vehicle 4. The image pickup unit 2 is, for example, an in-vehicle camera such as a front camera, a rear camera, or a side camera installed on the outside of the vehicle 4, and captures an image of the surroundings of the vehicle 4. Alternatively, the image pickup unit 2 may be a camera of a drive recorder installed in the vehicle of the vehicle 4. Further, the imaging unit 2 may be an electronic device having a camera. For example, the imaging unit 2 may be a camera such as a smart phone or a tablet computer.

The image pickup unit 2 may be a wide-angle camera using a fisheye lens or the like. Further, the imaging unit 2 may be a 360 ° camera provided on the upper part of the vehicle 4 or inside the vehicle in order to image the entire surroundings of the vehicle 4. Further, the imaging unit 2 may be a camera provided so as to be rotatable by 360 °. Alternatively, a plurality of cameras may be arranged in different directions. By using such a camera as the imaging unit 2, it is possible to image a wider range outside the vehicle 4.

The vehicle 4 is provided with an image processing device 100 described later, and processes the captured image captured by the imaging unit 2. Specifically, the image processing device 100 generates a transmission image by processing the captured image. The processing by the image processing apparatus 100 will be described later.

The image processing device 100 transmits to the server device 5. The server device 5 is, for example, a distribution server that distributes a transmitted image to the VR goggles 3. The VR goggles 3 is a head-mounted display or the like, and displays a received transmitted image. The VR goggles 3 function as an image display device for displaying a transmitted image. That is, the user wearing the VR goggles 3 can visually recognize the image inside or outside the vehicle 4. In the following description, a person wearing the VR goggles 3 is identified as a user, and a person riding in the vehicle 4 is identified as a driver or a passenger.

The VR goggles 3 include a gyro sensor, an azimuth sensor, an acceleration sensor, and the like, and detects a posture. Then, the VR goggles 3 acquire the posture detected by the sensor as posture information. Specifically, the acceleration sensor detects the direction of gravity to detect the vertical direction of the user's head. In addition, the orientation sensor and the gyro sensor detect the direction or left-right direction of the user's face. By detecting the orientation of these heads, the posture information of the user is obtained. Further, even if the VR goggles 3 are not used, the posture of the user can be detected by an electronic device having a similar sensor, for example, a smartphone. The VR goggles 3 transmit the posture information to the vehicle 4 via the server device 5. The display device that receives and displays the transmitted image is not limited to the VR goggles 3, and may be a smartphone, a tablet computer, or the like. It may be a display device such as a monitor or a projector that does not have a VR function. The communication between the vehicle 4 and the VR goggles 3 may be P2P (Peer To Peer) communication that does not go through the server device 5.

Next, the imaging unit 2 will be described. FIG. 2 is a diagram for explaining an arrangement example of a camera provided in the imaging unit 2. FIG. 2 is a view of the driver's cab inside the vehicle 4 viewed from the driver's seat in the front direction of the vehicle 4. The vehicle 4 includes a steering wheel 10, a dashboard 12, a windshield 14, a center console 16, a cluster panel 18 for displaying the traveling speed of the vehicle, an engine speed, and the like. Further, the center console 16 may be provided with a center display unit 20 for displaying a navigation screen or the like.

A smart phone 210 is arranged on the dashboard 12. The smart phone 210 functions as an imaging unit 2. The smart phone 210 includes an in-camera 211, an out-camera 212, a display screen 213, and the like. The in-camera 211 and the display screen 213 are provided on the same surface of the smart phone 210. The out-camera 212 is provided on a surface opposite to the surface on which the in-camera 211 is provided.

The in-camera 211 faces the rear and images the driver's cab. As shown in FIG. 3, the in-camera 211 images the driver's cab including the driver sitting in the driver's seat. The out-camera 212 faces the front and captures the scenery in front of the vehicle 4. FIG. 4 is a diagram schematically showing an image 50 captured by the out-camera 212. As shown in FIG. 4, the out-camera 212 captures a landscape including roads, buildings, vehicles, etc. in front of the vehicle 4 through the windshield 14. The captured image may be a moving image or a plurality of still images that are continuous in time.

In addition, an image processing device 100 is provided inside the dashboard 12. The image processing device 100 performs predetermined image processing on the captured images captured by the in-camera 211 and the out-camera 212. The installation position of the image processing device 100 is not limited to the inside of the dashboard. The image processing device 100 is not limited to the in-vehicle device, but may be a portable device. That is, the image processing device 100 may be a device fixed to the vehicle 4 or a device that can be taken out from the vehicle 4.

Further, the image processing device 100 is not limited to a physically single device, and may be composed of a plurality of devices. That is, the processing of the image processing apparatus 100 may be performed by the distributed processing apparatus. For example, the in-vehicle device installed in the vehicle 4 may perform a part of the processing, and the portable device may perform the remaining processing. The connection between the plurality of devices may be a wired connection or a wireless connection such as Bluetooth (registered trademark) or WiFi (registered trademark). Further, the smart phone 210 may function with the image processing device 100. That is, the smart phone 210 may perform part or all of the image processing.

FIG. 5 is a control block diagram of the image processing system according to the present embodiment. The image processing system 1 includes an image pickup unit 2, an image processing device 100, and VR goggles 3. The image processing device 100 and the VR goggles 3 are connected to each other via the network 6.

First, the image processing device 100 will be described. The image processing device 100 includes a captured image acquisition unit 101, a vehicle information acquisition unit 102, a transmission image generation unit 104, and a communication unit 105. There may be a form including the posture information acquisition unit 103.

The captured image acquisition unit 101 acquires the captured image from the imaging unit 2. For example, as shown in FIG. 2, when the imaging unit 2 has two cameras, the captured image acquisition unit 101 acquires images captured by the respective cameras. That is, the captured image acquisition unit 101 acquires the captured image by the in-camera 211 and the captured image by the out-camera 212. The connection between the image processing device 100 and the image pickup unit 2 may be a wired connection or a wireless connection.

The vehicle information acquisition unit 102 acquires vehicle information related to the traveling of the vehicle 4. The vehicle information may include, for example, the speed and acceleration of the vehicle 4. Further, the vehicle information may include the operation information of the vehicle 4. The operation information includes information on the operation of the gear, the steering wheel, the brake, and the accelerator. Here, it is assumed that the operation information includes the gear information related to the gear operation of the automatic transmission vehicle. For example, the gear information indicates, for example, forward movement (D range), reverse movement (R range), parking (P range), and the like.

The vehicle information acquisition unit 102 may acquire vehicle information via CAN (Control Area Network). That is, the image processing device 100 can be connected to the CAN. Alternatively, in the image processing device 100, the vehicle information acquisition unit 102 may acquire vehicle information from a car navigation system or the like connected to CAN by Bluetooth (registered trademark) or the like.

Further, when the image processing device 100 is equipped with an acceleration sensor, a gyro sensor, or the like, the vehicle information acquisition unit 102 can acquire vehicle information based on these sensor information. For example, by using an acceleration sensor provided in the smartphone 210, the vehicle information acquisition unit 102 can acquire vehicle information such as speed, acceleration, and traveling direction (forward, backward, left turn, right turn).

The posture information acquisition unit 103 acquires the posture information detected by the posture detection unit 32 of the VR goggles 3. The transmission image generation unit 104 generates a transmission image by processing the captured image. Specifically, the transmission image generation unit 104 cuts out a part of the captured image based on the vehicle information, selects the transmission image from the plurality of captured images, and at least synthesizes the plurality of captured images. Do one and process the image. Further, the transmission image generation unit 104 may process the captured image based on the posture information. For example, the transmission image generation unit 104 cuts out a part of the captured image based on at least one of the vehicle information and the posture information, selects a transmission image from the plurality of captured images, and synthesizes the plurality of captured images. Do at least one of the things to generate the transmitted image.

The communication unit 105 has a telecommunications function. Specifically, the communication unit 105 is a modulation circuit that modulates and outputs the transmission image generated by the transmission image generation unit 104 to an antenna for outputting a radio signal. The communication unit 105 may be configured to include an antenna. Further, it may be configured to include a demodulation circuit of the signal received by the antenna. The antenna receives the received signal and radiates the transmitted signal into space. The demodulation circuit demodulates the received signal received by the antenna. The modulation circuit modulates the transmitted signal and outputs it to the antenna. For example, the communication unit 105 performs communication processing according to the communication standards of mobile phones such as 3G, LTE (registered trademark), 4G, and 5G. The communication standard of the communication unit 105 is not particularly limited.

The wireless communication by the communication unit 105 enables data communication to the network 6 outside the vehicle. The network 6 is an external network such as the Internet. The communication unit 105 transmits the transmission image to the VR goggles 3 via the network 6. Further, the communication unit 105 receives the posture information transmitted from the VR goggles 3 via the network 6. For example, the network 6 includes the server device 5 shown in FIG. The communication unit 105 may perform wireless communication for a voice call.

Next, the configuration of the VR goggles 3 will be described. The VR goggles 3 include a display unit 31, a posture detection unit 32, a control unit 33, and a communication unit 35.

The display unit 31 displays an image to the user wearing the VR goggles 3. The display unit 31 includes a flat panel display such as a liquid crystal monitor or an organic EL (Electro-Luminescence) monitor. Further, the display unit 31 includes an optical system for guiding the display image displayed on the display to the user's eyes. The display unit 31 may include an optical system for the left eye and an optical system for the right eye, respectively.

The posture detection unit 32 detects the posture of the user wearing the VR goggles 3. Specifically, the posture detection unit 32 has a head tracking function. The posture detection unit 32 has a sensor that detects the posture state, such as a 3-axis sensor and a 6-axis sensor. Further, the posture detection unit 32 includes a circuit for estimating the movement amount from the sensor output. The posture detection unit 32 generates a movement amount signal according to the detection result of each axis. The VR goggles 3 can change the display image by detecting the position and orientation of the user's head by the posture detection unit 32.

The communication unit 35 has a telecommunications function like the communication unit 105. For example, the communication unit 105 performs communication processing according to the communication standards of mobile phones such as 3G, LTE (registered trademark), 4G, and 5G. The communication standard of the communication unit 105 is not particularly limited. Alternatively, the communication unit 35 may be a short-range wireless communication function such as Bluetooth communication or WiFi communication. Further, the communication unit 35 may be a wired connection interface such as HDMI (registered trademark). In these cases, the communication unit 35 is connected to a WiFi router or the like capable of communicating with the network 6.

The communication by the communication unit 35 enables the VR goggles 3 to perform data communication with the network 6. The communication unit 35 transmits the posture information to the image processing device 100 via the network 6. Further, the communication unit 35 receives the transmitted image transmitted from the image processing device 100 via the network 6. Further, the communication unit 35 may perform wireless communication for a voice call.

The control unit 33 comprehensively controls the entire VR goggles 3. For example, the transmission image received by the communication unit 35 is input to the control unit 33. The control unit 33 generates a video signal and a control signal based on the transmitted image. The display of the display unit 31 generates a display image based on the video signal and the control signal. For example, when the image size of the transmitted image and the display size of the display are different, the control unit 33 converts the transmitted image into the display size of the display. The display unit 31 can display a display image based on the transmitted image. The transmitted image and the displayed image are not limited to the same image. For example, the control unit 33 may add information to the transmitted image as appropriate to generate a display image.

Part or all of the processing in the image processing apparatus 100 can be realized by a computer program. Similarly, part or all of the processing in VR goggles 3 can be realized by a computer program. The image processing device 100 and the VR goggles 3 include a memory and a processor. The memory stores processing programs, various parameters, data, and the like. The processor executes a processing program stored in memory. Each process is executed when the processor executes the process program. The processor may be, for example, a CPU (Central Processing Unit), an FPGA (Field-Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a GPU (Graphics Processing Unit), or the like. ..

The processing in the transmission image generation unit 104 will be described in detail. First, an example in which the transmission image generation unit 104 generates a transmission image based on the vehicle information will be described. Specifically, the transmission image generation unit 104 selects the transmission image based on vehicle information on whether or not the vehicle 4 is stopped, for example, gear information of the vehicle 4. The captured image acquisition unit 101 acquires, for example, the captured image of the in-camera 211 and the captured image of the out-camera 212, as shown in FIG. The captured image of the in-camera 211 is an image of the driver's cab as shown in FIG. 3, and the captured image of the out-camera 212 is a view in front of the vehicle 4 as shown in FIG. The transmission image generation unit 104 may determine the cutting range of the transmission image based on the vehicle information.

When the gear is in the D range, that is, when the forward operation is in progress, the transmission image generation unit 104 selects the captured image of the out-camera 212 that captures the front view. When the gear is in the P range, that is, when the vehicle is parked, the transmission image generation unit 104 selects an image captured by the in-camera 211 that images the driver's cab. In this way, the transmission image generation unit 104 selects one transmission image from the plurality of captured images based on the vehicle information. Then, the communication unit 105 transmits the selected transmission image to the VR goggles 3. As a result, the VR goggles 3 can display a display image based on the selected image.

Since the user visually recognizes the display image displayed on the VR goggles 3, he / she can enjoy the scenery while driving. That is, the user can visually recognize the same scenery as the scenery in front of the driver. In addition, while parking, the driver's face can be visually recognized in the same manner as a videophone. This allows the user to have a simulated experience as if traveling with the driver.

The vehicle information is not limited to the gear information, and may be any information indicating whether or not the vehicle 4 is stopped. For example, the transmission image generation unit 104 or the vehicle information acquisition unit 102 may determine whether or not the vehicle is stopped according to the acceleration or speed detected by the sensor. For example, the transmission image generation unit 104 or the vehicle information acquisition unit 102 may determine that the vehicle is stopped depending on whether the engine is stopped or the position information has not changed. Then, when the vehicle information indicates that the vehicle is stopped, the transmission image generation unit 104 selects the image captured by the in-camera 211. Further, when the vehicle information indicates that the vehicle is traveling forward, the transmission image generation unit 104 may select the out-camera 212.

In the above description, the image captured by the out-camera 212 is used as the transmission image while the vehicle is running, and the image captured by the in-camera 211 is used as the transmission image while the vehicle is stopped. However, the selection of the transmission image is not limited to this example. .. When the imaging unit 2 has three or more cameras, the transmission image generation unit 104 can appropriately select the captured image according to the vehicle information. When the imaging unit 2 is a 360 ° camera, the transmission image generation unit 104 may appropriately cut out a part of the captured image according to the vehicle information. Further, the transmission image generation unit 104 selects one or a plurality of images from three or more captured images to obtain a transmission image.

For example, when the gear is in the R range, that is, when the vehicle is moving backward, the image captured by the rear camera that images the rear of the vehicle 4 may be used as the transmission image. The user can visually recognize the rear of the vehicle 4. As a result, the user can confirm the safety during the reverse operation at a remote location. For example, when there is an obstacle behind the vehicle 4, the user can notify the driver of the presence of the obstacle by voice call. Therefore, safety can be enhanced.

In this way, the transmission image generation unit 104 generates the transmission image by processing the captured image based on the vehicle information. For example, the transmission image generation unit 104 generates a transmission image by selecting a transmission image from a plurality of captured images. As a result, the image processing device 100 can transmit an image according to the operation of the vehicle 4. In addition, the amount of data of the transmitted image can be reduced. Therefore, since the communication delay can be suppressed, the VR goggles 3 can perform a highly responsive display.

Next, an example in which the transmission image generation unit 104 generates a transmission image based on the posture information will be described. Here, a transmitted image when the user changes the direction of the head while the vehicle is moving forward will be described. As shown in FIG. 4, the transmission image generation unit 104 has acquired the captured image captured by the out-camera 212. The posture detection unit 32 detects the orientation or position of the user's head. Then, the VR goggles 3 transmit the direction or position of the head as posture information to the image processing device 100.

When the image processing device 100 receives the posture information transmitted by the VR goggles 3, the transmission image generation unit 104 generates a transmission image according to the posture information. Here, the transmission image generation unit 104 cuts out a part of the captured image according to the posture information.

For example, as shown in FIG. 6, the captured image 50 includes a view in front of the vehicle 4 as in FIG. The transmission image generation unit 104 determines the cutting position of the transmission image based on the posture information. For example, when the user is facing the front front, the transmission image generation unit 104 cuts out a part of the central portion of each of the top, bottom, left, and right of the captured image 50 to obtain the transmission image 60C. When the user faces the left side, the transmission image generation unit 104 cuts out a part of the left side of the captured image 50 to obtain the transmission image 60L. When the user faces upward, the transmission image generation unit 104 cuts out a part of the upper side of the captured image 50 to obtain the transmission image 60U.

In this way, the transmission image generation unit 104 generates the transmission image by processing the captured image based on the posture information. The transmission image generation unit 104 cuts out a part of the captured image to obtain a transmission image. As a result, the image size of the transmitted image can be made smaller than the image size of the captured image, so that the amount of transmitted data can be reduced. Therefore, since the communication delay can be suppressed, the VR goggles 3 can perform a highly responsive display.

In particular, when the imaging unit 2 can image 360 ° around the vehicle 4, the image size can be reduced by cutting out a part of the captured image as a transmission image. Therefore, the transmitted image can be transmitted without reducing the resolution. In addition, the processing time for decompressing the compressed image on the VR goggles 3 side can be shortened. High-quality distribution with low delay is possible, and real-time performance can be improved.

Further, the transmission image generation unit 104 determines the cutting position based on the posture information. The transmission image generation unit 104 calculates the pixel address for cutting out the transmission image in the captured image based on the posture information. Specifically, when the direction of the user's head detected by the VR goggles 3 is the front direction, the transmission image generation unit 104 sets the central portion of the image captured by the out-camera 212 as the cutout position. When the direction of the user's head detected by the VR goggles 3 is backward, that is, when the user faces backward, the transmission image generation unit 104 sets the central portion of the image captured by the in-camera 211 as the cutout position. Further, when cutting out the image of the 360 ° camera, the center of the cutout position can be freely changed up, down, left and right according to the orientation of the user's head. Therefore, the transmission image generation unit 104 can generate the scenery in the direction desired by the user wearing the VR goggles 3 as the transmission image. Therefore, the user can experience the experience of traveling with the driver while sitting in the passenger seat. Further, the user of the VR goggles 3 may have a conversation with the driver or passenger of the vehicle 4 by voice call. As a result, the driver or passenger can receive advice such as directions and tourist information from the user. Further, the user can point out a driving danger that is difficult for the driver to notice.

Further, when the user wearing the VR goggles 3 faces backward, the transmission image generation unit 104 may switch to the image captured by the in-camera 211. That is, when the user faces backward, the transmission image generation unit 104 may use the captured image of the driver's cab as the transmission image.

In the above description, an example of generating a transmission image based on posture information or vehicle information has been described separately, but the transmission image generation unit 104 generates a transmission image based on both posture information and vehicle information. You may. In this case, for example, the transmission image generation unit 104 may generate a transmission image based on the posture information while the vehicle is moving forward, and may generate a transmission image based on the vehicle information while the vehicle is stopped. The cutout position of the transmitted image as shown in FIG. 6 may be adjusted.

The image processing method according to the first embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart of the image processing method. First, the captured image acquisition unit 101 acquires the captured image captured by the imaging unit 2 (S11). The vehicle information acquisition unit 102 acquires vehicle information (S12). The posture information acquisition unit 103 acquires the posture information of the user of the VR goggles 3 (S13).

The transmission image generation unit 104 generates a transmission image from the captured image (S14). That is, the transmission image generation unit 104 processes the captured image based on the vehicle information or the posture information. For example, the transmission image generation unit 104 selects one transmission image from a plurality of captured images. Alternatively, the transmission image generation unit 104 cuts out a part of the captured image to generate a transmission image. Since the specific processing is the same as above, the description thereof will be omitted. As a result, a transmitted image is generated. The communication unit 105 transmits the transmitted image (S15).

By repeating the above processing by the image processing device 100, the VR goggles 3 can display a moving image. Therefore, the user can have a simulated experience as if he / she is riding in the vehicle 4. The image processing device 100 can transmit a high-quality transmitted image with low delay, and can improve real-time performance. When the frame interval of the captured image does not match the update interval of the vehicle information and the attitude information, the image processing device 100 may generate a transmission image using the latest vehicle information and the attitude information. Further, the transmission image generation unit 104 may generate a transmission image based on only one of the posture information and the vehicle information.

Embodiment 2.
In the second embodiment, the vehicle information acquisition unit 102 acquires speed information indicating the speed of the vehicle as vehicle information. For example, the vehicle information acquisition unit 102 can acquire speed information using the vehicle speed pulse output from CAN. Alternatively, the vehicle information acquisition unit 102 may acquire speed information by communicating with the acceleration sensor provided in the vehicle 4 or the smart phone 210.

FIG. 8 is a diagram showing an arrangement example of the imaging unit 2 according to the present embodiment. In this embodiment, a drive recorder 220 is provided instead of the smart phone 210. The drive recorder 220 is installed on the windshield 14 which is a windshield. The image pickup unit 2 includes a camera 221 of the drive recorder 220. The camera 221 captures the scenery in front of the out-camera 212 of the first embodiment. FIG. 9 is a diagram showing an image captured by the camera 221.

The transmission image generation unit 104 changes the range of the image to be cut out, for example, the image size to be cut out (also referred to as the cutout size) according to the speed information. FIG. 9 shows a transmission image 61 at a speed V1 and a transmission image 62 at a speed V2. The speed V1 is faster than the speed V2, and the transmitted image 61 has an image size smaller than that of the transmitted image 62. The transmission image generation unit 104 reduces the size of the image to be cut out as the vehicle speed indicated by the speed information increases. By doing so, it is possible to obtain a display image with a high sense of presence. The transmitted image 61 and the transmitted image 62 have the same aspect ratio, but differ in the number of vertical × horizontal pixels. At this time, it is preferable that the center points of the cut out transmitted images 61 and 62 are the same. It is more preferable that the center points of the transmitted images 61 and 62 are the points that the user is gazing at or the center of the area that the user is gazing at, depending on the posture information.

Further, as the moving speed of the vehicle becomes faster, the communication becomes unstable and the communication speed may decrease. Also, when the speed of the vehicle increases, the transmitted image is compressed by an MPEG (Moving Picture Experts Group) encoder or the like and transmitted. The change in the captured image becomes large, and the data size of the compressed data obtained by image compression may become large. Therefore, as the vehicle speed increases, the communication delay can be suppressed by reducing the cutout size. The VR goggles 3 can perform a highly real-time display.

Further, the transmission image generation unit 104 may change the range of the image to be cut out, for example, the cutout position, according to the speed information. For example, the transmission image generation unit 104 can change the cutout position of the transmission image in the captured image in the vertical direction according to the speed information. FIG. 10 shows a transmission image 61 at a speed V1 and a transmission image 62 at a speed V2. The speed V1 is faster than the speed V2, and the transmitted image 61 is above the transmitted image 62. The transmission image generation unit 104 raises the cutting position as the vehicle speed indicated by the speed information increases. By doing so, the faster the vehicle speed, the more it is possible to generate a transmission image that captures a position farther from the vehicle 4. Therefore, it is possible to obtain a display image with a higher sense of presence.

The transmission image generation unit 104 sets the cutting position upward as the vehicle speed increases. By doing so, communication delay can be suppressed. For example, as the vehicle speed increases, the transmitted image does not include roads. It is assumed that the transmitted image is compressed by an MPEG encoder or the like and transmitted. Since the transmitted image not including the road has a small amount of change in the image, the compression rate is higher than the compression rate of the transmitted image including the road with a large amount of change in the image. Therefore, the image processing device 100 compresses the transmitted image that does not include the road, so that the amount of data of the transmitted image can be further reduced. Further, as the moving speed of the vehicle becomes faster, it is more likely that a problem such as dizziness of the user will occur by gazing at the road where the amount of change in the image is large. Therefore, it is preferable to use the cutout position as information. Further, the transmitted image may be an image that does not include the road at all or an image that includes a part of the road. By reducing the area of the road in the transmitted image, the transmitted image can be generated with a higher compression rate. As a result, the communication delay can be suppressed, and the VR goggles 3 can perform a display with high real-time performance.

In the above description, an example in which the transmission image generation unit 104 changes the image size and the cutout position according to the speed information has been described separately, but the transmission image generation unit 104 has described the image size and the image size according to the speed information. Both cutout positions may be changed. For example, as the vehicle speed indicated by the speed information increases, the image size may be reduced and the cutout position may be set upward.

The second embodiment and the first embodiment may be combined. For example, the transmission image generation unit 104 can determine the cutting position based on the attitude information and the speed information.

Embodiment 3.
In the third embodiment, the vehicle information acquisition unit 102 acquires acceleration information indicating the acceleration of the vehicle as vehicle information. For example, the vehicle information acquisition unit 102 can acquire acceleration information using the vehicle speed pulse output from CAN. Alternatively, the vehicle information acquisition unit 102 may acquire acceleration information by an acceleration sensor provided in the vehicle 4 or the smartphone 210.

The transmission image generation unit 104 changes the size of the image to be cut out according to the acceleration information. The transmission image generation unit 104 reduces the image size of the transmission image as the acceleration increases. In FIG. 9, the transmission image 61 is a transmission image in the case of acceleration A1, and the transmission image 62 is a transmission image in the case of acceleration A2. The acceleration A1 is larger than the acceleration A2.

In this way, by changing the image size (cutout size) according to the acceleration information, the VR goggles 3 can display a display image with a higher sense of presence. For example, when the vehicle 4 starts, the acceleration becomes large, so that the image size becomes smaller. The user wearing the VR goggles 3 can visually recognize the displayed image with a high sense of presence.

Alternatively, the transmission image generation unit 104 may change the cutting position according to the acceleration information. The transmission image generation unit 104 raises the cutout position of the transmission image as the acceleration increases. In FIG. 10, the transmission image 61 is a transmission image in the case of acceleration A1, and the transmission image 62 is a transmission image in the case of acceleration A2. The acceleration A1 is larger than the acceleration A2.

In this way, by changing the cutout position according to the acceleration information, the VR goggles 3 can display a display image with a higher sense of presence. For example, when the vehicle 4 starts, the acceleration becomes large, so that the cutting position becomes higher. The user wearing the VR goggles 3 can visually recognize the displayed image with a high sense of presence.

In the above description, an example in which the transmission image generation unit 104 changes the image size and the cutout position according to the speed information or the acceleration information has been described separately, but the transmission image generation unit 104 has the acceleration information or the speed. Both the image size and the cropping position may be changed according to the information. For example, as the vehicle speed indicated by the speed information increases, or as the acceleration indicated by the acceleration information increases, the image size may be reduced and the cutout position may be set upward. Further, the transmission image generation unit 104 may process the captured image according to both the acceleration information and the velocity information.

The third embodiment may be combined with the first embodiment. For example, the transmission image generation unit 104 can determine the cutting position based on the posture information and the acceleration information.

Embodiment 4.
In the image processing system according to the fourth embodiment, the image pickup unit 2 has a plurality of cameras. An example of arranging the cameras in the imaging unit 2 will be described with reference to FIG. FIG. 11 is a schematic view of the vehicle 4 viewed from above. In order to cover the entire circumference of the vehicle 4 by 360 °, the imaging unit 2 includes a front camera 231, a rear camera 232, a left side camera 233, and a right side. It is equipped with a camera 234. FIG. 11 shows the imaging ranges of the front camera 231, the rear camera 232, the left side camera 233, and the right side camera 234 with broken lines.

The front camera 231 and the rear camera 232, the left side camera 233, and the right side camera 234 are wide-angle cameras to cover a wide imaging range. The imaging unit 2 images the entire periphery of the vehicle 4.

The front camera 231 images the front of the vehicle 4, and the rear camera 232 images the rear of the vehicle 4. The left side camera 233 images the left side of the vehicle 4, and the right side camera 234 images the right side of the vehicle 4. A part of the imaging range of the front camera 231 overlaps with a part of the imaging range of the left side camera 233. A part of the image pickup range of the front camera 231 overlaps with a part of the image pickup range of the right side camera 234. Similarly, a part of the imaging range of the rear camera 232 overlaps with a part of the imaging range of the left side camera 233 and a part of the imaging range of the right side camera 234.

The posture information acquisition unit 103 acquires posture information indicating the posture of the user. Here, it is assumed that the user is facing diagonally forward to the left. At this time, the range to be displayed in the transmitted image is set to the range 65 as shown in FIG. The left and right ends of the range 65 are the left end 66L and the right end 66R. The range 65 includes a part of the image pickup range by the front camera 231 and a part of the image pickup range by the left side camera 233. The left end 66L of the range 65 is included only in the imaging range of the left side camera 233, and the right end 66R is included only in the imaging range of the front camera 231.

The transmission image generation unit 104 cuts out a part of the captured image of the front camera 231. Specifically, the transmission image generation unit 104 cuts out the captured image in the cropping range 67 on the left side of the imaging range of the front camera 231. Further, the transmission image generation unit 104 cuts out a part of the captured image of the left side camera 233. Specifically, the transmission image generation unit 104 cuts out the captured image in the cropping range 68 on the front side of the captured image of the left side camera 233. The cutout range 67 and the cutout range 68 partially overlap. The transmission image generation unit 104 combines the two cut-out images into a transmission image.

The transmission image generation unit 104 synthesizes the two captured images. That is, the transmitted image generation unit 104 synthesizes the captured images captured by the two cameras to obtain a transmitted image. Further, the transmission image generation unit 104 may cut out a part of the image captured by one camera and combine it with the entire image captured by the other camera. Of course, the two captured images to be combined are synchronized. That is, the transmission image generation unit 104 synthesizes two captured images captured by different cameras at the same timing. Further, the transmission image generation unit 104 may generate a transmission image by synthesizing the images captured by three or more cameras.

The above embodiments 1 to 4 can be appropriately combined. The transmission image generation unit 104 performs at least one of cutting out a part of the captured image based on the vehicle information, selecting the transmission image from the plurality of captured images, and synthesizing the plurality of captured images. With, the transmission image can be generated. By doing so, the image processing device 100 can transmit an image according to the operation of the vehicle 4. In addition, since the amount of data of the transmitted image can be reduced, the VR goggles 3 can display high image quality and high real-time performance.

In the above embodiment, the transmission image generation unit 104 determines the cutout position in the captured image based on the posture information, but the cutout position may be determined based on information other than the posture information. For example, the user may change the viewing direction by voice input using a microphone or operation input using a button or the like. That is, when the user specifies a desired direction by voice input or operation input, the transmission image generation unit 104 changes the cutout position so that the direction is included in the transmission image. Further, the camera may be switched or the image size may be changed by voice input or operation input by the user.

The image processing device 100 and the VR goggles 3 are composed of a processor such as a CPU (Central Processing Unit), a storage device such as a memory, a user interface, and various peripheral circuits. That is, the image processing device 100 and the VR goggles 3 have a function as a computer. Further, in the image processing device 100, the processor executes a program stored in the storage device, so that the captured image acquisition unit 101, the vehicle information acquisition unit 102, the attitude information acquisition unit 103, the transmission image generation unit 104, and the communication unit Components such as 105 may be realized. The VR goggles 3 may realize components such as a display unit 31, a posture detection unit 32, a control unit 33, and a communication unit 35 by executing a program stored in the storage device by a processor.

Further, each component of the image processing device 100 and the VR goggles 3 is not limited to being realized by software by a program, but may be realized by any combination of hardware, firmware, and software. .. Further, each component of the image processing apparatus 100 may be realized by using an integrated circuit programmable by the user, such as an FPGA (field-programmable gate array) or a microcomputer. In this case, this integrated circuit may be used to realize a program composed of each of the above components. This also applies to other embodiments described later.

In addition, the program for executing the above display control method can be stored and supplied to the computer using various types of non-transitory computer readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-temporary computer-readable media include magnetic recording media (eg, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (eg, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W, semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)) is included. The program may also be supplied to the computer by various types of transient computer readable media. Examples of temporary computer-readable media include electrical, optical, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

1 System 2 Imaging unit 3 VR goggles 4 Vehicle 5 Server device 6 Network 31 Display unit 32 Attitude detection unit 33 Control unit 35 Communication unit 50 Captured image 61 Transmission image 62 Transmission image 100 Image processing device 101 Captured image acquisition unit 102 Vehicle information acquisition Unit 103 Attitude information acquisition unit 104 Transmission image generation unit 105 Communication unit 210 Smartphone 211 In-camera 212 Out-camera 220 Drive recorder 221 Camera 231 Front camera 232 Rear camera 233 Left side camera 234 Right side camera

Claims (7)

An image acquisition unit that acquires one or more images captured by an image pickup unit arranged in a vehicle, and an image acquisition unit.
A vehicle information acquisition unit that acquires vehicle information related to the running of the vehicle,
Based on the vehicle information, at least one of cutting out a part of the captured image, selecting a transmission image from the at least one captured image, and synthesizing a plurality of the captured images is performed. By doing so, the transmission image generator that generates the transmission image and
An image processing device including a communication unit that transmits the transmitted image by wireless communication. The captured image acquisition unit acquires at least an image of the inside of the vehicle and obtains an image.
The vehicle information includes information indicating whether or not the vehicle is stopped.
When the vehicle is stopped, a part of the captured image inside the vehicle is cut out, a transmission image is selected from at least one captured image inside the vehicle, and the captured image inside the vehicle is captured. The image processing apparatus according to claim 1, wherein a transmission image is generated by performing at least one of combining the plurality of captured images including the image. The vehicle information includes speed information or acceleration information indicating the speed of the vehicle.
The image processing apparatus according to claim 1 or 2, wherein the transmission image generation unit changes the cutout range of the transmission image in the captured image based on the velocity information or the acceleration information. The image processing apparatus according to claim 3, wherein the transmission image generation unit changes the cutout size of the transmission image in the captured image based on the velocity information or the acceleration information. The image processing apparatus according to any one of claims 1 to 4,
An image processing system including a display device that receives and displays the transmitted image. The step of acquiring one or more captured images captured by the imaging unit arranged in the vehicle, and
The step of acquiring vehicle information regarding the running of the vehicle and
Based on the vehicle information, at least one of cutting out a part of the captured image, selecting a transmission image from the at least one captured image, and synthesizing a plurality of the captured images is performed. By doing so, the steps to generate the transmitted image,
An image processing method comprising a step of transmitting the transmitted image by wireless communication. The step of acquiring one or more captured images captured by the imaging unit arranged in the vehicle, and
The step of acquiring vehicle information regarding the running of the vehicle and
Based on the vehicle information, at least one of cutting out a part of the captured image, selecting a transmission image from the at least one captured image, and synthesizing a plurality of the captured images is performed. By doing so, the steps to generate the transmitted image,
A program that causes a computer to execute a step of transmitting the transmitted image by wireless communication.

Images