JP2018106313A - Image converter for mobile body - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a compression and decompression technique for image data, which is different from conventional one.SOLUTION: An image converter for a mobile body, which is used for detecting an obstacle on a road, comprises a filter for degrading resolution of an original image and a filter for upgrading resolution. A compressed image with degraded resolution is stored in a server, and a restored image is generated by receiving the compressed image and upgrading resolution for decompression. A difference image between the restored image and a captured image enables detection of an obstacle.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image conversion apparatus mounted on a moving body such as an automobile.

  Automobiles handle image data for autonomous driving control, drive recorder or security purposes. These automobiles include an image conversion device for communication and recording of image data (see, for example, Patent Document 1). The image conversion apparatus compresses or decompresses image data. Conventionally, many image compression / decompression techniques including JPEG (Joint Photographic Experts Group) or MPEG (Moving Pictures Experts Group) are known.

JP 2012-151711 A JP2015-115747A

  A technique for compressing / decompressing image data different from conventional ones is provided.

  The moving body image conversion device of one embodiment of the present invention is mounted on a moving body and converts a first image into a second image by super-resolution processing.

  Another aspect of the present invention is an image conversion apparatus for a moving body that is mounted on a moving body and converts a fifth image into a sixth image by resolution degradation processing.

  The method according to one embodiment of the present invention includes a first resolution lower than the fourth resolution by performing resolution degradation processing on the data of the fourth image having the fourth resolution and the fourth data size in a data server outside the mobile body. A first image data having a first data size smaller than the fourth data size is generated, and a second resolution higher than the first resolution is obtained by performing super-resolution processing on the first image data in the moving body. A second image having a resolution is generated.

  The resolution degradation process may include applying a fourth filter to the data of the fourth image. The super-resolution processing may include applying a first filter corresponding to the fourth filter to the data of the first image.

  Further, the data server may acquire information indicating the current geographical position of the mobile object. The data of the fourth image may be extracted from data of a plurality of images stored in a database based on information indicating the geographical position in the data server. In the moving body, third image data may be acquired from a camera installed on the moving body. The moving body may further include recognizing an obstacle around the moving body based on at least the data of the second image and the data of the third image.

  Moreover, you may acquire the information which shows the weather environment around the said mobile body in the said mobile body. The mobile unit may further include determining to perform the super-resolution processing based on information indicating the weather environment.

  The method according to another aspect of the present invention provides a sixth resolution and the fifth data lower than the fifth resolution by performing resolution degradation processing on the data of the fifth image having the fifth resolution and the fifth data size in the moving body. Generating data of a sixth image having a sixth data size smaller than the size, and subjecting the data of the sixth image to super-resolution processing in the data server to have a seventh resolution higher than the sixth resolution. 7 generating images.

  The resolution degradation process may include applying a fifth filter to the data of the fifth image. The super-resolution processing may include applying a sixth filter corresponding to the fifth filter to the data of the sixth image.

  The data of the fifth image may be acquired from a camera installed on the moving body. Information indicating the current geographical position of the mobile unit may be acquired in the data server. The data server may extract fourth image data from a plurality of image data stored in a database based on the information indicating the geographical position. The data server may further include recognizing an obstacle around the moving body based on at least the data of the seventh image and the data of the fourth image.

  Moreover, you may acquire the information which shows the weather environment around the said mobile body in the said mobile body. The mobile unit may further include determining the execution of the resolution deterioration process based on information indicating the weather environment.

  A moving body refers to a manned or unmanned artifact that can move on the ground, in the ground, on the sea, in the sea, and in the air. For example, the mobile body includes an automobile, a train, a locomotive, a special vehicle, a ship, a submersible, an aircraft, a rocket, and an artificial satellite.

  The image includes a still image and a moving image.

  The data size is a physical quantity expressed in bits or bytes.

  ADVANTAGE OF THE INVENTION According to this invention, the compression / decompression technique of the data of the image different from the past can be provided.

Conceptual diagram of image conversion processing The block diagram which shows the system which concerns on 1st Embodiment of this invention. The flowchart which shows the obstruction recognition process of the moving body which concerns on 1st Embodiment of this invention. The flowchart which shows the image processing of the data server which concerns on 1st Embodiment of this invention. The block diagram which shows the system which concerns on 2nd Embodiment of this invention. The flowchart which shows the obstruction recognition process of the moving body which concerns on 2nd Embodiment of this invention. The flowchart which shows the obstruction recognition process of the data server which concerns on 2nd Embodiment of this invention.

(Overview of image conversion process)
The outline of the image conversion process will be described with reference to FIG. The image conversion process includes a super-resolution process, a resolution deterioration process, and an obstacle recognition process.

  The original image 11, the degraded image 17 and the restored image 23 have the same aspect ratio and the same number of pixels. The resolution degradation filter 15 and the super-resolution filter 21 have the same aspect ratio and the same number of pixels. Each pixel value of the filter 15 is expressed by a gray scale of 256 gradations, for example. Similarly, each pixel value of the filter 21 is represented by a gray scale of 256 gradations, for example. The distribution of the pixel values of the filter 15 is determined so as to degrade the resolution of the image. Conversely, the distribution of the pixel values of the filter 21 is determined so as to increase the resolution of the image. That is, the distribution of the pixel values of the filter 15 has a corresponding relationship with the distribution of the pixel values of the filter 21. For example, the distribution of the pixel values of the filter 15 has an inverse relationship with the distribution of the pixel values of the filter 21. The filter 21 may be a filter based on a point spread function described in JP-A-2015-115747. Since the filter 15 has an inverse relationship with the filter 21, the filter 15 is also a filter based on a point spread function.

  The resolution degradation process includes an iterative calculation process that repeats a plurality of filtering processes. Each filtering process includes a process of generating a new image data by convolving a filter with the image data. For example, in the first filtering process, the filter 15 is convolved with the data of the original image 11 to generate intermediate image data. In the second filtering process, the filter 15 is convolved with the intermediate image data to generate data of the degraded image 17. The number of times the filtering process is repeated is not limited to two, and may be more than this. Further, the filter may be changed each time. The convolution calculation includes a process of multiplying and calculating each pixel value of the pixel block 13 and each pixel value of the filter 15 to generate a new pixel value of the center pixel of the pixel block 13. The resolution of the degraded image 17 is lower than the resolution of the original image 11. In addition, the data size of the degraded image 17 is lower than the data size of the original image 11. By appropriately designing the characteristics of the filter 15 and the number of times the filtering process is repeated, the data size of the degraded image 17 can be reduced from 1/10 to 1/20 of the data size of the original image 11.

  The super-resolution processing is the same processing as the resolution deterioration processing except for the filter characteristics. For example, in the first filtering process, the filter 21 is convolved with the data of the degraded image 17 to generate intermediate image data. In the second filtering process, the filter 21 is convolved with the intermediate image data to generate the restored image 23 data. The convolution calculation includes a process of generating a new pixel value of the center pixel of the pixel block 19 by multiplying and calculating each pixel value of the pixel block 19 and each pixel value of the filter 21. The resolution of the restored image 23 is higher than the resolution of the degraded image 17. Note that the resolution of the restored image 23 may be higher than the resolution of the original image 11 depending on the number of repetitions of the iterative calculation process.

  The super-resolution processing is described in detail in, for example, JP-A-2015-115747. The super-resolution processing of this embodiment also uses a similar mechanism. However, in Japanese Patent Laid-Open No. 2015-115747, the image degradation process is unknown, whereas in this embodiment, the image degradation process is known. Therefore, the super-resolution processing of the present embodiment can generate a plausible restored image with a smaller number of iterations.

  In the present embodiment, the filter 15 and the filter 21 are each 3 pixels in the vertical and horizontal directions. However, the present invention is not limited to this, and may be 5 pixels in the vertical and horizontal directions, 7 pixels in the vertical and horizontal directions, or more.

  The resolution degradation process may include a reduction in image size by thinning out pixels instead of or in addition to the filtering process. The super-resolution processing may include enlargement of the image size by pixel interpolation instead of or in addition to the filtering processing.

  The obstacle recognition process includes generating a difference image 27 between the restored image 23 and the captured image 25. The original image 11 and the restored image 23 are also background images that do not include an obstacle. The captured image 25 is a background image including an obstacle. The difference image 27 includes an obstacle and does not include a background. The difference image 27 can be used for obstacle recognition. The original image 11 is a past photographed image or a processed image based thereon, and is stored in a database in association with information indicating a geographical position such as longitude and latitude (hereinafter “geographical position information”). . In the field of auto driving, this is known as 3D map information. The captured image 25 is the current captured image. The captured image 25 is obtained from a camera mounted on a moving body, for example. Geographic location information may include depth or altitude in addition to longitude and latitude. Thereby, the position of the moving body that moves in the sea, the ground, or the air can be accurately indicated.

(First embodiment)
The system shown in FIG. 2 includes a moving body 101 and a data server 131 that can communicate with the moving body 101.

  The moving body 101 includes a camera 103, a geographical position sensor 105, a weather environment sensor 107, a transmission / reception unit 109, a drive system 111, and a controller 113. The data server 131 includes a transmission / reception unit 133, a database 135, and a controller 137.

  The camera 103 captures at least one of the front, rear, right side, left side, upper side or lower side of the moving body 101 to generate data of a third image (captured image). The camera 103 has sensitivity in at least one wavelength region of visible light, infrared light, and ultraviolet light.

  Although not limited to this, the geographical position sensor 105 is a GPS (Global Positioning System) sensor, and detects the geographical position including the longitude and latitude of the moving object. The geographical position sensor 105 may detect depth or altitude in addition to longitude and latitude.

  The weather environment sensor 107 indicates a weather environment by directly or indirectly detecting a weather environment that may adversely affect the image quality of the third image captured by the camera 103, for example, rain, fog, snow, sandstorm, or smog. Information (hereinafter “meteorological environment information”) is generated. The water sensor can directly detect rain, fog and snow. By monitoring the operation of the windshield wiper of an automobile, rain, fog and snow can be detected indirectly.

  The transmission / reception unit 109 includes a transmission unit and a reception unit capable of wireless communication or wired communication.

  The drive system 111 is a mechanism for controlling the position and speed of the moving body 101. For example, in the case of an automobile, the drive system 111 includes a steering wheel, an accelerator, and a brake.

  The controller 113 includes a processor 115 and a memory 117. The processor 115 implements super-resolution processing (first image processing), obstacle recognition, and weather environment recognition by executing a program stored in the memory 117. The memory 117 includes a volatile memory and a nonvolatile memory, and stores the first filter 119 and the predetermined weather condition 121. The first filter 119 is a filter for super-resolution processing based on the first point spread function, and is, for example, the above-described filter 21. The predetermined weather condition provides a criterion for selecting the obstacle recognition process from a plurality of options. In one embodiment, the obstacle recognition process utilizes both the image of the camera 103 and the image of the data server 131 when the weather environment may adversely affect the image quality of the image of the camera 103; Only the image of the camera 103 is used. In this aspect, weather environment information that may adversely affect the image quality of the image of the camera 103 is stored in the memory 117 as the predetermined weather condition. Such weather conditions are, for example, rain, fog, snow, sandstorm or smog as described above.

  The controller 113 is not limited to a program and a processor that executes the program, and may be realized by hardware such as an SoC (System on Chip) or an FPGA (Field Programmable Gate Array).

  The transmission / reception unit 133 includes a transmission unit and a reception unit capable of wireless communication or wired communication.

  The database 135 stores photographed images photographed at a plurality of different geographical positions or processed images based thereon. These captured images or processed images are associated with geographical position information. That is, the database 135 is configured to be able to extract specific image data from a plurality of image data based on the geographical position information.

  The controller 137 includes a processor 139 and a memory 141. The processor 139 implements resolution degradation processing by executing a program stored in the memory 141. The memory 141 includes a volatile memory and a nonvolatile memory, and stores a fourth filter 143 corresponding to the first filter. The fourth filter 143 is a resolution degradation filter based on the fourth point spread function, and is, for example, the above-described filter 15. The fourth point spread function corresponds to the first point spread function.

  The controller 137 is not limited to a program and a processor that executes the program, and may be realized by hardware such as an SoC (System on Chip) or an FPGA (Field Programmable Gate Array).

  FIG. 3 is a flowchart showing the obstacle recognition process of the moving object 101.

  The controller 113 acquires the third image data from the camera 103 (S11), and acquires geographical position information from the geographical position sensor 105 (S13). The data of the third image and the geographical position information are associated with each other.

  The controller 113 acquires the weather environment information from the weather environment sensor 107 (S15), and determines whether or not the weather environment indicated by the weather environment information satisfies the predetermined weather condition 121 stored in the memory 117 (S17). . The predetermined weather condition 121 is, for example, rain, fog, snow, sandstorm, and smog. In this case, if the weather environment indicated by the weather environment information is clear, the predetermined weather condition 121 is not satisfied. If the weather environment indicated by the weather environment information is rain, the predetermined weather condition 121 is satisfied.

  When the weather environment does not satisfy the predetermined weather condition (S17, No), the controller 113 performs the first obstacle recognition process (S19). The first obstacle recognition process is a process for performing obstacle recognition using only the third image taken by the camera 103. Since such processing is well known, description thereof is omitted.

  When the weather environment satisfies the predetermined weather condition (S17, Yes), the controller 113 transmits the geographical position information to the data server 131 (S25). Corresponding to this, the data server 131 transmits the data of the first image corresponding to the geographical position information (details will be described later with reference to FIG. 4). The first image is, for example, the above-described degraded image 17. The controller 113 waits until the first image data is received (S27, No). If the controller 113 receives the first image data (S27, Yes), the controller 113 performs super-resolution processing on the first image data to generate the second image data (S29). The first image has a first resolution. The second image has a second resolution that is higher than the first resolution. A specific example of the super-resolution processing is as described with reference to FIG. The second image is, for example, the restored image 23 described above. Thereby, the 2nd image for raising the accuracy of image recognition is obtained.

  The controller 113 performs a second obstacle recognition process (S31). The second obstacle recognition process is a process for recognizing an obstacle using both the third image of the camera 103 and the second image of the data server 131. A specific example of the second obstacle recognition process is as described with reference to FIG.

  If the controller 113 finds an obstacle (S21, Yes), the controller 113 controls the drive system 111 to avoid the obstacle. The control of the drive system 111 includes, for example, deceleration, acceleration, and course change. The process of the controller 113 returns to the acquisition of the third image from the camera 103 (S11) after the control of the drive system 111. Also, when no obstacle is found (S21, No), the process of the controller 113 returns to the acquisition of the third image from the camera 103 (S11).

  FIG. 4 is a flowchart showing image processing of the data server 131.

  The controller 137 is on standby until it receives geographical position information from the moving body 101 (S33, No). If the controller 137 acquires the geographical position information from the moving body 101 (S33, Yes), the controller 137 extracts the fourth image from the plurality of images stored in the database 135 (S35). The fourth image is an image associated with the geographical position information received from the mobile object 101.

  The controller 137 performs resolution degradation processing on the data of the fourth image to generate data of the first image (S37), and transmits the data of the first image to the moving body 101 (S39). The fourth image has a fourth resolution and a fourth data size. The first image has a first resolution lower than the fourth resolution and a first data size smaller than the fourth data size. A specific example of the resolution deterioration process is as described with reference to FIG.

  As described above, the data server 131 can generate data of the first image (degraded image) by performing resolution degradation processing on the data of the fourth image (original image). The fourth image has a fourth resolution and a fourth data size. The first image has a first resolution lower than the fourth resolution and a first data size smaller than the fourth data size. The moving body 101 can generate data of the second image (restored image) by performing super-resolution processing on the data of the first image. The second image has a second resolution that is higher than the first resolution. Since the first image is transmitted from the data server 131 to the mobile unit 101, the amount of data communication can be reduced.

  The resolution degradation process includes applying the fourth filter 143 to the data of the fourth image. The super-resolution processing includes applying the first filter 119 corresponding to the fourth filter 143 to the data of the first image. The first filter 119 may have a characteristic opposite to that of the fourth filter 143. Since the first filter 119 corresponds to the fourth filter 143, a plausible restored image can be generated with a smaller number of iterations.

  The moving body 101 selects an image recognition process from a plurality of options. The first image recognition process does not require communication between the moving object 101 and the data server 131. The second image recognition process requires communication between the moving object 101 and the data server 131. In this way, by properly using the image recognition processing, the entire data communication amount can be reduced.

(Second Embodiment)
In the first embodiment, the mobile object performs the obstacle recognition process, but in the second embodiment, the data server performs the obstacle recognition process. The same components as those in the first embodiment are denoted by the same reference numerals and description thereof is omitted.

  The system shown in FIG. 5 includes a moving body 201 and a data server 231 that can communicate with the moving body 201.

  The moving body 201 includes a controller 213 that stores a fifth filter 219. The fifth filter 219 is a filter for resolution deterioration processing based on the fifth point spread function, for example, the above-described filter 15. The data server 231 includes a controller 237 that stores a sixth filter 243. The sixth filter is a filter for super-resolution processing based on the sixth point spread function, for example, the filter 21 described above. The sixth point spread function corresponds to the fifth point spread function. The processor 115 implements resolution degradation processing (second image processing) by executing a program stored in the memory 117.

  FIG. 6 is a flowchart showing the obstacle recognition process of the moving object 201.

  The controller 213 acquires the data of the fifth image from the camera 103 (S51), and acquires geographical position information from the geographical position sensor 105 (S53). The fifth image is an image photographed by the camera 103. The data of the fifth image and the geographical position information are associated with each other.

  The controller 213 acquires weather environment information from the weather environment sensor 107 (S55), and determines whether or not the weather environment indicated by the weather environment information satisfies the predetermined weather condition 121 stored in the memory 117 (S57). .

  When the weather environment does not satisfy the predetermined weather condition (S57, No), the controller 213 performs a first obstacle recognition process (S59).

  When the weather environment satisfies the predetermined weather condition (S57, Yes), the controller 213 performs resolution degradation processing on the fifth image data to generate sixth image data. A specific example of the resolution deterioration process is as described with reference to FIG. The fifth image has a fifth resolution and a fifth data size. The sixth image has a sixth resolution lower than the fifth resolution and a sixth data size smaller than the fifth data size.

  The controller 213 transmits the data of the sixth image and the geographical position information to the data server 231 (S67, S69). In response to this, the data server 231 performs the second obstacle recognition process and transmits the result (details will be described later with reference to FIG. 7). The controller 213 waits until it receives the result of the second obstacle recognition process (S71, No).

  If the controller 213 receives the result of the first obstacle recognition process (S59) or the result of the second obstacle recognition process (S71, Yes) and finds an obstacle (S61, Yes), the controller 213 avoids the obstacle. The drive system 111 is controlled. The process of the controller 213 returns to the acquisition of the fifth image from the camera 103 (S51) after the control of the drive system 111. Also, when no obstacle is found (S61, No), the process of the controller 213 returns to the acquisition of the fifth image from the camera 103 (S51).

  FIG. 7 is a flowchart showing the obstacle recognition process of the data server 231.

  The controller 237 is on standby until it receives the geographical position information and the sixth image data from the moving body 201 (S73, No). If the controller 237 acquires the geographical position information from the moving body 201 (S75, Yes), the controller 237 extracts the fourth image from the plurality of images stored in the database 135 (S75). The fourth image is an image associated with the geographical position indicated in the geographical position information received from the moving body 201.

  The controller 237 performs super-resolution processing on the sixth image data to generate seventh image data (S77). A specific example of the super-resolution processing is as described with reference to FIG. The sixth image has a sixth resolution. The seventh image has a seventh image that is higher than the sixth image.

  The controller 237 performs the second obstacle recognition process (S79), and transmits the result to the moving body 201 (S81). The second obstacle recognition process is a process for recognizing an obstacle using both the fifth image of the camera 103 and the fourth image of the data server 231. A specific example of the second obstacle recognition process is as described with reference to FIG.

  As described above, the moving body 201 can generate data of the sixth image (degraded image) by performing resolution degradation processing on the data of the fifth image (original image). The fifth image has a fifth resolution and a fifth data size. The sixth image has a sixth resolution lower than the fifth resolution and a sixth data size smaller than the fifth data size. The moving body 201 transmits the data of the fifth image, and the data server 231 receives the data of the fifth image. The data server 231 can generate data of the seventh image (restored image) by performing super-resolution processing on the data of the fifth image. The seventh image has a seventh resolution higher than the sixth resolution. Since the sixth image is transmitted from the moving body 201 to the data server 231, the amount of data communication can be reduced.

  The resolution deterioration process includes applying a fifth filter 219 to the data of the fifth image. The super-resolution processing includes applying a sixth filter 243 corresponding to the fifth filter 219 to the fifth image. The sixth filter 243 may have a characteristic opposite to that of the fifth filter 219. Since the sixth filter 243 corresponds to the fifth filter 219, a plausible restored image can be generated with a smaller number of iterations.

  The present invention is not limited to the configuration and processing disclosed in the first and second embodiments. You may change suitably in the range of the meaning of this invention.

  The resolution degradation process may be used as a pre-process for existing compression / decompression techniques including JPEG or MPEG. In an example, a moving body performs resolution degradation processing on an original image to generate a degraded image, applies a compression technique to the degraded image to generate a compressed image, and transmits the compressed image. The data server receives the compressed image, applies the existing decompression technique to the compressed image, reproduces the deteriorated image, performs super-resolution processing on the deteriorated image, and generates a restored image.

  The present invention is applicable to image data compression / decompression techniques.

DESCRIPTION OF SYMBOLS 101,201 Mobile body 103 Camera 105 Geographical position sensor 107 Weather environment sensor 109 Transmission / reception part 111 Drive system 113, 213 Controller 131,231 Data server 133 Transmission / reception part 135 Database 137,237 Controller

Claims (11)

  A moving body image conversion apparatus that is mounted on a moving body and converts a first image into a second image by super-resolution processing. The first image is transmitted from a data server outside the mobile body,
The mobile image conversion device according to claim 1, wherein the mobile image conversion device includes a receiving unit that receives the first image.   The mobile image conversion apparatus according to claim 1, wherein the super-resolution processing includes repetitive calculation processing. The moving body includes a camera,
The said moving body image conversion apparatus has an obstruction recognition part which recognizes the obstruction around the said moving body based on the 3rd image and said 2nd image which were image | photographed with the said camera at least. The image conversion apparatus for moving bodies according to any one of the above.   The image conversion apparatus for a mobile body includes a weather environment recognition unit that recognizes a weather environment around the mobile body, and determines to perform the super-resolution processing based on a recognition result of the weather environment recognition unit. 5. The moving body image conversion apparatus according to 1 to 4.   The image conversion apparatus for moving body includes a memory for storing a filter based on a point spread function, and a first image processing unit that generates the second image by applying the filter to the first image. The image conversion apparatus for moving bodies described in 1.   An image conversion apparatus for a moving body that is mounted on a moving body and converts a fifth image into a sixth image by resolution degradation processing. The moving body includes a camera that images the outside and generates the fifth image;
The said moving body image converter is a moving body image converter of Claim 7 which has a transmission part which transmits the said 6th image to an external data server.   The moving image conversion apparatus according to claim 7, wherein the resolution deterioration process includes an iterative calculation process.   The said image conversion apparatus for moving bodies is provided with the weather environment recognition part which recognizes the weather environment around the said moving body, and determines implementation of the said resolution degradation process based on the recognition result of the said weather environment recognition part. 10. The moving body image conversion device according to any one of items 1 to 9.   The said moving body image converter includes a memory for storing a filter based on a point spread function, and a second image processing unit for generating the sixth image by applying the filter to the fifth image. The image conversion apparatus for moving bodies described in 1.