WO2017086029A1

WO2017086029A1 - Image processing device, image processing method, moving body, wearable electronic device, and computer program

Info

Publication number: WO2017086029A1
Application number: PCT/JP2016/078678
Authority: WO
Inventors: 小曽根　卓義; 洋司山本
Original assignee: ソニー株式会社
Priority date: 2015-11-20
Filing date: 2016-09-28
Publication date: 2017-05-26

Abstract

[Problem] To provide an image processing device capable of performing image processing with little delay, for captured images. [Solution] Provided is an image processing device comprising: an imaging element capable of reading any data in pixel group units comprising a pixel or a plurality of adjacent pixels; and a control unit that determines a reading order for the data, for the imaging elements, and reads data from the imaging elements on the basis of the determined reading order. The image processing device is capable of processing captured images, with little delay, as a result of reading data from imaging elements on the basis of the determined reading order.

Description

Image processing apparatus, image processing method, moving object, body-mounted electronic device, and computer program

The present disclosure relates to an image processing device, an image processing method, a moving body, a body-mounted electronic device, and a computer program.

By using a camera equipped with a lens with a special structure, an image including the entire circumference of the camera can be obtained. Various techniques for processing and using such images have been proposed. For example, Patent Document 1 describes a technique for converting an image captured using a fisheye lens into a developed image developed on a cylindrical surface.

JP 2012-226645 A

In order to transform an image captured by a lens with a large distortion that is not a central projection method, such as a fisheye lens, into a form that the observer does not feel uncomfortable, the image is stored once in memory and then subjected to arbitrary deformation processing However, depending on the deformation process, the required memory capacity increases, and it takes time to write and read images from the memory, resulting in a delay from imaging to output. Similarly, when a part of an image is cut out and output, a delay from imaging to output occurs in the same manner.

Therefore, in the present disclosure, a new and improved image processing apparatus, image processing method, moving object, and computer program capable of performing image processing on a captured image with a small delay and a small memory capacity. Propose.

According to the present disclosure, an image sensor capable of arbitrarily reading data in units of pixels or a pixel group composed of a plurality of adjacent pixels, a data read order determined for the image sensor, and the determined read order And a control unit that reads data from the imaging device based on the image processing device.

Further, according to the present disclosure, a moving object including the image processing device is provided. According to the present disclosure, there is provided a body-mounted electronic device including the image processing device.

Further, according to the present disclosure, a data reading order is determined for an image sensor that can arbitrarily read data in units of pixels or a pixel group composed of a plurality of adjacent pixels, and the data reading order is determined based on the determined reading order. An image processing method is provided that includes reading data from an imaging device.

Further, according to the present disclosure, a data reading order is determined for an image sensor that can arbitrarily read data in units of pixels or a pixel group composed of a plurality of adjacent pixels, and the data reading order is determined based on the determined reading order. A computer program for causing a computer to read data from an image sensor is provided.

As described above, according to the present disclosure, a new and improved image processing apparatus, image processing method, moving object, and computer program capable of performing image processing with little delay on a captured image are provided. Can be provided.

Note that the above effects are not necessarily limited, and any of the effects shown in the present specification, or other effects that can be grasped from the present specification, together with or in place of the above effects. May be played.

It is explanatory drawing which shows the function structural example of the image processing apparatus 10 which correct | amends the image with distortion. It is explanatory drawing which shows the function structural example of image processing apparatus 10 'which correct | amends an image with distortion. FIG. 3 is an explanatory diagram illustrating a functional configuration example of an image processing apparatus 100 according to an embodiment of the present disclosure. It is explanatory drawing which shows the structural example of the 1st chip | tip 120 of CIS101. It is explanatory drawing which shows the structural example of the 2nd chip | tip 130 of CIS101. 5 is a flowchart illustrating an operation example of the image processing apparatus 100 according to an embodiment of the present disclosure. 6 is an explanatory diagram for explaining an operation of the image processing apparatus 100 according to the embodiment. FIG. 6 is an explanatory diagram for explaining an operation of the image processing apparatus 100 according to the embodiment. FIG. 6 is an explanatory diagram for explaining an operation of the image processing apparatus 100 according to the embodiment. FIG. It is explanatory drawing which shows another reading method of the image data from CIS101. It is explanatory drawing which shows another reading method of the image data from CIS101. It is explanatory drawing explaining a mode that the image output from the image processing apparatus 100 changes based on operation information. It is explanatory drawing explaining a mode that the image output from the image processing apparatus 100 changes based on operation information. FIG. 11 is an explanatory diagram illustrating an application example of an image processing device 100 according to an embodiment of the present disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be made in the following order.
1. Embodiment of the present disclosure 1.1. Background 1.2. Functional configuration example 1.3. Example of operation 1.4. Application example Summary

<1. Embodiment of the present disclosure>
[1.1. background]
First, before describing the embodiment of the present disclosure in detail, the background of the embodiment of the present disclosure will be described.

Using a camera equipped with a lens with a special structure such as a fisheye lens, an image including the entire circumference of the camera can be obtained. Since an image captured by a lens having a large distortion such as a fisheye lens has a large distortion, an observer feels uncomfortable with the image as it is captured. Accordingly, it is necessary to perform processing for transforming an image having a large distortion into a shape that does not cause the viewer to feel uncomfortable.

In order to transform an image having a large distortion, generally, an image of the entire region including the distortion output from the image sensor is first stored in a frame memory. Then, an image is read from the frame memory, subjected to a deformation process for eliminating the distortion, and output, or a process for rewriting the frame memory.

FIG. 1 is an explanatory diagram showing a functional configuration example of an image processing apparatus 10 that corrects a distorted image. Image data output from a CIS (CMOS Image sensor) 11 is temporarily stored in the DRAM 13 via the bus bridge 12 in units of frames. In the image data stored in the DRAM 13, the coordinates to be read are designated by the coordinate generation unit 18, and are stored in an SRAM (Static Random Access Memory) 15 via the bus bridge 14.

In the image data stored in the SRAM 15, the coordinates to be corrected are designated by the coordinate generation unit 18 and stored in the pixel interpolation unit 16. The pixel interpolation unit 16 corrects the distortion by moving the position of the image, performing interpolation processing, and the like, and outputs the corrected image data to the SRAM 17.

FIG. 2 is an explanatory diagram showing a functional configuration example of the image processing apparatus 10 ′ that corrects a distorted image. Image data in a predetermined range within the frame output from the CIS 11 is stored in the SRAM 21. In the image data stored in the SRAM 21, the coordinates to be corrected are designated by the coordinate generation unit 18 and stored in the pixel interpolation unit 16. The pixel interpolation unit 16 corrects the distortion by moving the position of the image, performing interpolation processing, and the like, and outputs the corrected image data to the SRAM 17.

However, even in the case where it is desired to display only a partial area of an image with large distortion, the image of the entire area is output from the image sensor in the conventional processing. Further, when the image needs to be deformed in the horizontal or vertical direction, a memory for holding the coordinate relationship before and after the deformation is necessary.

In addition, it is difficult to reduce the capacity of the frame memory because the image of the entire area including distortion is once stored in the frame memory, and if the deformation is limited, the capacity of the memory can be reduced. In this case, a memory corresponding to the amount of deformation in the vertical direction (line memory for deformation in the vertical direction) was required. Further, since the image output from the image sensor is once stored in the frame memory, there is a delay from imaging to output.

Therefore, in view of the background described above, the present disclosure has intensively studied a technique for performing a deformation process with a small memory capacity and a small delay on an image captured by a lens having a large distortion. As a result, as will be described below, the present disclosure uses an image sensor that can read image data from an arbitrary location, thereby reducing delay with respect to an image captured by a lens with large distortion, and The inventors have devised a technology that can perform deformation processing with a small memory capacity.

The background of the embodiment of the present disclosure has been described above. Subsequently, an embodiment of the present disclosure will be described in detail.

[1.2. Functional configuration example]
FIG. 3 is an explanatory diagram illustrating a functional configuration example of the image processing apparatus 100 according to the embodiment of the present disclosure. FIG. 3 shows an example of the functional configuration of the image processing apparatus 100 that performs correction processing on an image captured by a lens with large distortion. Hereinafter, a functional configuration example of the image processing apparatus 100 according to the embodiment of the present disclosure will be described with reference to FIG.

As illustrated in FIG. 3, the image processing apparatus 100 according to the embodiment of the present disclosure includes a CMOS image sensor (CIS) 101, an image processing unit 102, and an input unit 103. The image processing unit 102 includes

SRAMs

111 and 113, an image transformation unit 112, and a coordinate generation unit 114.

The CIS 101 is a solid-state image sensor that converts light collected through a lens, for example, a fish-eye lens that is not a central projection method, into an electrical signal and outputs the electrical signal. The CIS 101 is provided with a color filter having a Bayer array and other arrays.

In this embodiment, the CIS 101 is configured to output data in an arbitrary pixel unit or an arbitrary pixel group (block) unit based on an external command. Here, a configuration example of the CIS 101 will be described.

The CIS 101 according to the present embodiment has a structure in which two semiconductor chips are stacked. In the following description, the upper chip of the stacked structure is a first chip, and the lower chip is a second chip.

The first chip is a pixel chip in which a pixel array unit in which unit pixels including photoelectric conversion elements are two-dimensionally arranged on a matrix is formed.

The second chip includes a driving unit that drives each pixel of the pixel array unit formed on the first chip, and a signal processing unit that performs signal processing such as converting an analog signal read from each pixel of the pixel array unit into a digital signal. A memory that stores data that has been subjected to signal processing by the signal processing unit, a data processing unit that outputs data written in the memory based on an external command, and the like can be included.

FIG. 4 is an explanatory diagram showing a configuration example of the first chip 120 of the CIS 101. FIG. 5 is an explanatory diagram showing a configuration example of the second chip 130 of the CIS 101.

4, the pixel array unit 121 in which unit pixels including photoelectric conversion elements are two-dimensionally arranged in a matrix,

pads

122 a and 122 b that are electrically connected to the outside, and the second chip 130 are electrically connected. A first chip 120 provided with vias (VIA) 123 is shown.

Further, FIG. 5 illustrates a signal processing unit 131 that performs signal processing such as converting an analog signal read from each pixel of the pixel array unit into a digital signal and writes the signal to the memory, and drives each pixel formed in the first chip 120.

Peripheral circuits

132a and 132b, an output circuit 133 that outputs data written to the signal processing unit 131 based on a command from the outside, and pixel data from each pixel of the pixel array unit 121 is written to the memory or read from the memory. 2 shows a second chip 130 provided with

column recorders

134a and 134b for designating column addresses and row addresses.

In this embodiment, pixel units each having a predetermined number of pixels as one unit are two-dimensionally arranged in a matrix, and a via 123 is formed for each pixel unit. The number of pixels constituting the pixel unit can be arbitrary.

The CIS 101 according to the present embodiment is configured to output pixel data to the outside in units of pixel units based on an external command. The order of outputting pixel data in units of pixel units can be arbitrarily set based on an external command. In the present embodiment, the coordinate generation unit 114 designates the order of reading pixel data from the CIS 101 and outputs the pixel data of the designated pixel unit to the image processing unit 102. The processing will be described in detail later.

Next, the configuration of the image processing unit 102 will be described. The image processing unit 102 may be configured by, for example, a CPU (Central Processing Unit), various ROMs (Read Only Memory), a RAM (Random Access Memory), and the like. The image processing unit 102 can function as an example of the image processing unit of the present disclosure.

The SRAM 111 is a memory that stores pixel data output from the CIS 101. The pixel data stored in the SRAM 111 is output to the image transformation unit 112 with the output target specified by the coordinate generation unit 114.

Here, it is sufficient that the SRAM 111 has a capacity sufficient to store pixel data for a line to be output after correcting the distortion by the image deforming unit 112 in the subsequent stage. That is, the SRAM 111 does not need a capacity for storing the pixel data for one frame, and may have a capacity for storing the pixel data for less than one frame.

The image deformation unit 112 uses the pixel data stored in the SRAM 111, but performs image deformation processing. The image transformation unit 112 can perform rotation, enlargement, reduction, or the like of at least a part of the range of the image captured by the CIS 101 in addition to processing for removing distortion of the image captured by the CIS 101 as image deformation processing.

The image transformation unit 112 performs an interpolation process using pixel data stored in the SRAM 111, for example, when generating an image from which distortion is removed. When generating an output image from which distortion has been removed from a distorted image, the output image is generated by moving pixel data in the distorted image. It becomes an image without. Therefore, it is necessary to generate pixel data in a place where there is no pixel data by performing interpolation processing using the pixel data after the pixel data is moved. The image transformation unit 112 performs this interpolation process.

The image transformation unit 112 may use the data of the coordinates specified by the coordinate generation unit 114 when performing the interpolation process. The image transformation unit 112 outputs the pixel data generated by performing the interpolation process to the SRAM 113.

The SRAM 113 is a memory that stores pixel data output from the image transformation unit 112. The pixel data stored in the SRAM 113 is output to the subsequent processing block at a predetermined timing and becomes the original data of the output image.

The coordinate generation unit 114 outputs information related to the coordinates to the CIS 101, the SRAM 111, and the image transformation unit 112. The coordinate generation unit 114 specifies a block for outputting pixel data to the CIS 101. In addition, the coordinate generation unit 114 specifies the coordinates of the output target of the pixel data to the image transformation unit 112 for the SRAM 111. Further, the coordinate generation unit 114 specifies the coordinates to be interpolated for the image deformation unit 112.

When the coordinate generation unit 114 specifies a block for outputting pixel data to the CIS 101, the coordinate generation unit 114 specifies from the upper left direction of the output image. For example, when a part of an image captured by the CIS 101 is cut out, rotated, and output, the coordinate generation unit 114 outputs pixel data from a block corresponding to the upper left location of the rotated image. It is specified in CIS101.

The input unit 103 includes, for example, a lever, a button, a touch panel, and the like, and outputs operation information generated based on the operation of these devices to the coordinate generation unit 114. For example, the input unit 103 may allow an operator to specify an arbitrary range in an image captured by the CIS 101, to enlarge or reduce the range, and to rotate the range.

The coordinate generation unit 114 generates information related to coordinates to be output to the CIS 101, the SRAM 111, and the image deformation unit 112 based on operation information sent from the input unit 103 based on the operation of the input unit 103. The coordinate generation unit 114 can dynamically change the distortion correction target by generating information about coordinates based on the operation of the input unit 103.

Of course, it goes without saying that the image processing unit 102 may include components other than those described above. In the above example, the SRAM 111 is included in the image processing unit 102. However, the SRAM 111 may be provided outside the image processing unit 102.

Since the image processing apparatus 100 according to the embodiment of the present disclosure has such a configuration, it is possible to perform a deformation process on an image captured by a lens having a large distortion with a small delay and a small memory capacity. It becomes.

The functional configuration example of the image processing apparatus 100 according to the embodiment of the present disclosure has been described above. Subsequently, an operation example of the image processing apparatus 100 according to the embodiment of the present disclosure will be described.

[1.3. Example of operation]
FIG. 6 is a flowchart illustrating an operation example of the image processing apparatus 100 according to the embodiment of the present disclosure. FIG. 6 shows an operation example of the image processing apparatus 100 when a target image is generated by correcting distortion of an image captured by the CIS 101 using data output from the CIS 101. Hereinafter, an operation example of the image processing apparatus 100 according to the embodiment of the present disclosure will be described with reference to FIG.

When the image processing apparatus 100 according to the present embodiment corrects distortion of an image captured by the CIS 101, the image processing apparatus 100 first performs coordinate calculation when performing distortion correction, and in the image captured by the CIS 101, A block to be subjected to distortion correction processing is designated (step S101). The coordinate generation unit 114 can execute the process of step S101.

The coordinate calculation when performing distortion correction is to calculate the correspondence between the coordinates of the image before distortion correction and the image after distortion correction. That is, the coordinate calculation when performing distortion correction is to calculate which coordinate (x ′, y ′) of the image after distortion correction corresponds to the coordinate (x ′, y ′) of the image after distortion correction. It is. If the correspondence between the coordinates of the image before distortion correction and the image after distortion correction is known, it can be determined which block of image data should be read from the CIS 101 when the target image is generated.

When the image processing apparatus 100 according to the present embodiment specifies a block for outputting pixel data to the CIS 101, the image processing apparatus 100 specifies from the upper left direction of the output image. For example, when a part of an image captured by the CIS 101 is cut out, rotated, and output, the image processing apparatus 100 according to the present embodiment starts from the block corresponding to the upper left location of the rotated image. The CIS 101 is designated to output pixel data.

When the coordinate calculation and the block designation are performed in step S101, the image processing apparatus 100 subsequently reads the image data designating the block from the CIS 101 based on the designation in step S101 (step S102). The coordinate generation unit 114 can execute the process in step S102.

When the image data is read by designating a block from the CIS 101, the image processing apparatus 100 stores the image data read from the CIS 101 in the SRAM 111 (step S103). At this time, the image data stored in the SRAM 111 only needs to be able to generate one line of the output target image. That is, the SRAM 111 does not need a capacity for storing the pixel data for one frame, and may have a capacity for storing the pixel data for less than one frame.

When the image data read from the CIS 101 is stored in the SRAM 111, the image processing apparatus 100 then reads the image data by designating coordinates from the SRAM 111 (step S104). The coordinate generation unit 114 can execute the process in step S104. Since the image data is stored in the SRAM 111 in units of blocks of the CIS 101, the image processing apparatus 100 reads from the SRAM 111 the image data necessary for the subsequent deformation process from the SRAM 111.

When the image data is read by designating the coordinates from the SRAM 111, the image processing apparatus 100 subsequently performs an image deformation process using the image data read from the SRAM 111 (step S105). The process in step S105 can be executed by the image transformation unit 112. The image deformation processing in step S105 may include processing for correcting image distortion, image rotation processing, image enlargement or reduction processing, and the like. When the image processing apparatus 100 performs image deformation processing in step S <b> 105, the image processing apparatus 100 stores the deformed image data in the SRAM 113.

An operation example of the image processing apparatus 100 according to the embodiment of the present disclosure will be described in detail with reference to other drawings.

7, 8, and 9 are explanatory diagrams for explaining the operation of the image processing apparatus 100 according to the embodiment of the present disclosure. FIG. 7 shows an example of an image output from the CIS 101, which is an example of a distorted image 140 formed on the CIS 101 through a fisheye lens that is a lens that is not a central projection method.

The image 140 having such distortion is very uncomfortable when viewed by an observer. Therefore, the image processing apparatus 100 according to the embodiment of the present disclosure first deforms the range indicated by the reference numeral 141 in the image 140 including the uppermost line of the deformed image, thereby enabling the observer. Convert to an image that doesn't feel strange. At that time, the image processing apparatus 100 according to the embodiment of the present disclosure reads the image data from the CIS 101 by designating a block from which the image data is read. Thereafter, the image processing apparatus 100 reads image data from the CIS 101 by sequentially designating blocks corresponding to the downward direction from the topmost line of the deformed image.

FIG. 8 is an explanatory diagram for explaining that the image processing apparatus 100 reads the image data from the CIS 101 by designating a block from which the image data is read. FIG. 9 is an example of an image 150 that is generated by the image processing apparatus 100 according to the embodiment of the present disclosure and the image output from the CIS 101 is deformed so that the observer does not feel uncomfortable.

FIG. 8 shows a block 144 containing a line 143 corresponding to the top line 151 of the image 150 in FIG. The image processing apparatus 100 reads the image data of each block 144 shown in FIG. 8 from the CIS 101 in order to generate the highest line 143 of the converted image.

Thereafter, the image processing apparatus 100 reads out image data from the CIS 101 by sequentially designating blocks corresponding to the downward direction from the uppermost line of the deformed image. The designation of this block is performed by the coordinate generation unit 114 based on the generated coordinate information. Thus, by designating a block and reading image data from the CIS 101, the capacity of the SRAM 111 that stores the image data read from the CIS 101 can be reduced.

The image processing apparatus 100 can arbitrarily determine the order of reading image data from the CIS 101 according to the image to be generated. If the image to be generated is not rotated, the image processing apparatus 100 may select a block from the top to the bottom of the image output by the CIS 101. If there is no rotation in the image to be generated, the image processing apparatus 100 may select a block from which image data is read from the CIS 101 so that the generated image has a top-to-bottom direction.

Depending on the image generation status, unnecessary blocks may be generated after reading from the CIS 101. Therefore, the image processing apparatus 100 may delete unnecessary image data from the SRAM 111 when the image data is read from the CIS 101 in units of blocks.

The operation example of the image processing apparatus 100 according to the embodiment of the present disclosure has been described above. The image processing apparatus 100 according to the embodiment of the present disclosure performs deformation processing on an image captured by a lens with a large distortion with a small delay and a small memory capacity by executing the above-described operation. Is possible.

The image processing apparatus 100 according to the embodiment of the present disclosure can read image data from the CIS 101 by various methods by applying the reading of the image data from the CIS 101 in units of blocks described above.

10 and 11 are explanatory diagrams illustrating another method of reading image data from the CIS 101 by the image processing apparatus 100 according to the embodiment of the present disclosure. FIGS. 10 and 11 show that when reading image data from the CIS 101, the CIS 101 by the image processing apparatus 100 is used when the area divided in the horizontal direction is read sequentially from the nearest place in the vertical direction, not in units of blocks. This is an example of reading image data from.

FIG. 10 shows an example in which an image is divided into three regions (A phase, B phase, and C phase), and sequentially read from a location closest to the vertical direction. Also, FIG. 11 shows that the image is divided into two at the center of the image and then divided into three regions (A phase, B phase, C phase) so as to be line symmetric, and then in the vertical direction. This is an example of reading sequentially from the nearest location.

Thus, by reading out the image data from the CIS 101, the system of the image processing apparatus 100 can be simplified as compared with the case of reading out in units of pixels or blocks.

As described above, the image processing apparatus 100 according to the embodiment of the present disclosure is based on the operation information sent from the input unit 103 based on the operation of the input unit 103, with respect to the CIS 101, the SRAM 111, and the image deformation unit 112. Information regarding the coordinates to be output may be generated. Below, the image processing apparatus 100 which changes the image output based on operation information is demonstrated.

12 and 13 are explanatory diagrams for explaining how the image processing apparatus 100 changes the image output based on the operation information. In FIG. 12, the image processing apparatus 100 outputs the image 160 a that has been subjected to deformation processing on the image output from the CIS 101 and the state in which the display target is moved in the diagonally lower left direction based on the operation information. ,It is shown.

The reference numeral 161 shown in FIG. 12 is an example of a distorted image that is imaged on the CIS 101 through the fisheye lens and output from the CIS 101. Reference numerals 162a to 162d shown in FIG. 12 indicate the range of the image to be output, which is cut out from the image output by the CIS 101 and deformed.

When the display object moves in the diagonally downward left direction based on the operation information, the image processing apparatus 100 changes the block to be read from the CIS 101 in accordance with the movement of the display object. In FIG. 13, when the display target moves in the diagonally lower left direction based on the operation information, the block to be read from the CIS 101 changes, and the image finally output by the image processing apparatus 100 is displayed from the image 160a. It is explanatory drawing which shows the example which changes to the image 160d.

12 and 13, the upper side of the deformed and output images 160a to 160d corresponds to the upper right of the range 162a to 162d in FIG. Accordingly, when the display target moves in the diagonally downward left direction based on the operation information, the image processing apparatus 100 changes the block to be read first from the block 163a to the block 163d as shown in FIG. Go.

As described above, when the image processing apparatus 100 according to the embodiment of the present disclosure changes the image to be output based on the operation information sent from the input unit 103 based on the operation of the input unit 103, Based on the operation information, the block to be read from the CIS 101 is changed. Then, the CIS 101 outputs the image data of the designated block. The block reading order is coordinate information generated by the coordinate generation unit 114 according to the contents of the operation information sent from the input unit 103, as in the processing for removing distortion of the image captured by the fisheye lens described above. Determine based on.

The image processing apparatus 100 according to the embodiment of the present disclosure performs an image process according to operation information sent from the input unit 103 based on an operation of the input unit 103 by performing a deformation process on the image data output from the CIS 101. It becomes possible to output.

In the above-described example, the image output by the CIS 101 is an image formed through a special lens such as a fisheye lens, but the present disclosure is not limited to such an example. For example, the image processing apparatus 100 changes a block to be read from the CIS 101 even when a part of a normal image with little distortion is cut out and output based on the operation of the input unit 103. I can do it.

Subsequently, an application example of the image processing apparatus 100 according to the embodiment of the present disclosure will be described. The image processing apparatus 100 according to the embodiment of the present disclosure can be applied to, for example, automobiles, flying objects, and other mobile objects, and body-mounted electronic devices (wearable devices).

FIG. 14 is an explanatory diagram illustrating an application example of the image processing apparatus 100 according to the embodiment of the present disclosure, and illustrates an example in a case where the image processing apparatus 100 is provided to an automobile.

FIG. 14 shows

imaging units

2910, 2912, 2914, 2916, and 2918 and outside

information detection units

2920, 2922, 2924, 2926, 2928, and 2930.

The

imaging units

2910, 2912, 2914, 2916, and 2918 are provided at, for example, at least one of the front nose, the side mirror, the rear bumper, the back door, and the upper part of the windshield in the vehicle interior of the vehicle 2900. An imaging unit 2910 provided in the front nose and an imaging unit 2918 provided in the upper part of the windshield in the vehicle interior mainly acquire an image in front of the vehicle 2900. The

imaging units

2912 and 2914 provided in the side mirror mainly acquire an image on the side of the vehicle 2900. An imaging unit 2916 provided in the rear bumper or the back door mainly acquires an image behind the vehicle 2900. An imaging unit 2918 provided on the upper part of the windshield in the passenger compartment is mainly used for detecting a preceding vehicle or a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

FIG. 14 shows an example of shooting ranges of the

respective imaging units

2910, 2912, 2914, and 2916. The imaging range a indicates the imaging range of the imaging unit 2910 provided in the front nose, the imaging ranges b and c indicate the imaging ranges of the

imaging units

2912 and 2914 provided in the side mirrors, respectively, and the imaging range d The imaging range of the imaging unit 2916 provided in the rear bumper or the back door is shown. For example, by superimposing the image data captured by the

imaging units

2910, 2912, 2914, and 2916, an overhead image when the vehicle 2900 is viewed from above is obtained.

The vehicle outside

information detection units

2920, 2922, 2924, 2926, 2928, 2930 provided on the front, rear, side, corner, and upper windshield of the vehicle 2900 may be, for example, an ultrasonic sensor or a radar device. The vehicle outside

information detection units

2920, 2926, and 2930 provided on the front nose, the rear bumper, the back door, and the windshield in the vehicle interior of the vehicle 2900 may be, for example, LIDAR devices. These vehicle outside information detection units 2920 to 2930 are mainly used for detecting a preceding vehicle, a pedestrian, an obstacle, and the like.

Each of the

imaging units

2910, 2912, 2914, 2916, and 2918 in FIG. 14 is provided with the CIS 101 that can read image data from an arbitrary block, and the above-described image processing unit 102 is provided inside the vehicle 2900. Thus, an image around the vehicle can be presented to the driver of the vehicle 2900 with a low delay. In addition, since the SRAM 111 only needs to have a capacity for storing image data of less than one frame, it can contribute to reducing the cost of hardware for presenting an image around the vehicle to the driver of the vehicle 2900.

Of course, an apparatus to which the image processing apparatus 100 according to the embodiment of the present disclosure can be applied is not limited to an automobile. For example, the image processing apparatus 100 according to the embodiment of the present disclosure can also be applied to a body-mounted electronic device (wearable device) intended to be used by being worn on a human body.

For example, when the wearable device is provided with the CIS 101 and the image blur caused by the movement of the body is corrected, the above-described image processing unit 102 is provided in the wearable device, so that the image blur or distortion can be reduced. It is possible to correct the image with a low delay and present the corrected image to an observer who has captured the image with the wearable device. Further, since the SRAM 111 only needs to have a capacity for storing image data of less than one frame, it can contribute to reducing the cost of hardware for presenting an image captured by a wearable device.

For example, the image processing apparatus 100 according to the embodiment of the present disclosure can also be applied to a case where a camera is mounted on an autonomous flying object (drone) and an image from above is captured by the camera.

For example, when the CIS 101 is provided in a camera mounted on an autonomous flying vehicle and the image blur caused by the movement of the flying vehicle is corrected, the above-described image processing unit 102 is provided inside the flying vehicle. Thus, it is possible to correct image blurring and distortion with a low delay and present the corrected image to an observer who has captured the image with a flying object. Further, since the SRAM 111 only needs to have a capacity for storing image data of less than one frame, it can contribute to reducing the cost of hardware for presenting an image captured by a flying object that autonomously flies.

<2. Summary>
As described above, according to the embodiment of the present disclosure, when an image captured through a lens is deformed, the image is deformed without storing image data in units of frames, and the deformed image is displayed. An image processing apparatus 100 capable of outputting is provided. For example, when transforming an image captured through a special lens such as a fisheye lens, image processing that can transform the image and output the transformed image without storing image data in units of frames An apparatus 100 is provided.

The image processing apparatus 100 according to the embodiment of the present disclosure reads data from the CIS 101 that can output image data in a block unit including a pixel or a plurality of adjacent pixels and in any order. When reading data from the CIS 101, the image processing apparatus 100 reads data from the CIS 101 by an amount necessary for image processing, for example, distortion correction processing for each line.

Therefore, the image processing apparatus 100 according to the embodiment of the present disclosure does not need to read all data for one frame from the CIS 101 when performing image processing, and thus can perform image processing with little delay. Further, the image processing apparatus 100 according to the embodiment of the present disclosure can reduce the capacity of the SRAM 111 that temporarily stores image data before image processing to an amount less than one frame of image data. Costs can be reduced.

Each step in the processing executed by each device in this specification does not necessarily have to be processed in chronological order in the order described as a sequence diagram or flowchart. For example, each step in the processing executed by each device may be processed in an order different from the order described as the flowchart, or may be processed in parallel.

In addition, it is possible to create a computer program for causing hardware such as CPU, ROM, and RAM incorporated in each device to exhibit functions equivalent to the configuration of each device described above. A storage medium storing the computer program can also be provided. In addition, by configuring each functional block shown in the functional block diagram with hardware or a hardware circuit, a series of processing can be realized with hardware or a hardware circuit.

Also, some or all of the functional blocks shown in the functional block diagram used in the above description may be realized by a server device connected via a network such as the Internet. The configuration of each functional block shown in the functional block diagram used in the above description may be realized by a single device or a system in which a plurality of devices cooperate. A system in which a plurality of devices are linked may include, for example, a combination of a plurality of server devices, a combination of a server device and a terminal device, or the like.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

In addition, the effects described in this specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
Control for determining the data reading order for an image sensor capable of arbitrarily reading data in units of pixels or a pixel group composed of a plurality of adjacent pixels, and reading data from the image sensor based on the determined reading order An image processing apparatus comprising a unit.
(2)
The image processing apparatus according to (1), wherein the control unit performs image processing on an image formed on the imaging element using data read from the imaging element based on the determined reading order.
(3)
The image processing apparatus according to (2), further including a storage unit that stores data read by the control unit.
(4)
The image processing apparatus according to (3), wherein the amount of data stored in the storage unit is an amount of less than one frame required for the unit of the image processing in the control unit.
(5)
The image processing apparatus according to any one of (2) to (4), wherein the control unit performs a deformation process on an image formed on the image sensor using the data.
(6)
The image processing apparatus according to (5), wherein the control unit performs a process of removing distortion of the image as the deformation process.
(7)
The image processing apparatus according to (6), wherein the control unit performs a process of removing distortion of the image formed on the imaging element through a lens that is not a central projection method.
(8)
The image processing apparatus according to (7), wherein the control unit performs a process of removing distortion of the image formed on the imaging element through a fisheye lens as a lens that is not the central projection method.
(9)
The image processing apparatus according to any one of (5) to (8), wherein the control unit performs rotation processing of the image as the deformation processing.
(10)
The image processing apparatus according to any one of (5) to (9), wherein the control unit performs a process of changing an enlargement ratio of the image as the deformation process.
(11)
The image processing apparatus according to any one of (2) to (10), wherein the control unit determines the reading order so as to read from data at a position corresponding to an upper left of the image after the image processing.
(12)
The image processing apparatus according to any one of (1) to (11), wherein the control unit changes the reading order according to operation information from the outside.
(13)
A moving body comprising the image processing device according to any one of (1) to (12).
(14)
The moving body according to (13), wherein the moving body is an automobile.
(15)
The mobile body according to (13), wherein the mobile body is a flying body that autonomously flies.
(16)
A body-mounted electronic device comprising the image processing device according to any one of (1) to (12).
(17)
A data reading order is determined for an image sensor that can arbitrarily read data in units of pixels or a pixel group composed of a plurality of adjacent pixels, and data is read from the image sensor based on the determined reading order. Including an image processing method.
(18)
A data reading order is determined for an image sensor that can arbitrarily read data in units of pixels or a pixel group composed of a plurality of adjacent pixels, and data is read from the image sensor based on the determined reading order. A computer program that causes a computer to execute.

100: Image processing apparatus 102: Image processing unit 103: Input unit 111: SRAM
112: Image transformation unit 113: SRAM
114: Coordinate generating unit 120: First chip 121: Pixel array unit 122a: Pad unit 122b: Pad unit 123: Via 130: Second chip 131: Signal processing unit 132a: Peripheral circuit 132b: Peripheral circuit 133: Output circuit 134a: Column recorder 134b: Column recorder

Claims

Control for determining the data reading order for an image sensor capable of arbitrarily reading data in units of pixels or a pixel group composed of a plurality of adjacent pixels, and reading data from the image sensor based on the determined reading order An image processing apparatus comprising a unit.
The image processing apparatus according to claim 1, wherein the control unit performs image processing on an image formed on the imaging element using data read from the imaging element based on the determined reading order.
The image processing apparatus according to claim 2, further comprising a storage unit that stores data read by the control unit.
4. The image processing apparatus according to claim 3, wherein the amount of data stored in the storage unit is an amount of less than one frame required for the image processing unit in the control unit.
The image processing apparatus according to claim 2, wherein the control unit performs a deformation process on an image formed on the image sensor using the data.
The image processing apparatus according to claim 5, wherein the control unit performs a process of removing distortion of the image as the deformation process.
The image processing apparatus according to claim 6, wherein the control unit performs a process of removing distortion of the image formed on the imaging element through a lens that is not a central projection method.
The image processing apparatus according to claim 7, wherein the control unit performs a process of removing distortion of the image formed on the imaging element through a fish-eye lens as a lens that is not the central projection method.
The image processing apparatus according to claim 5, wherein the control unit performs rotation processing of the image as the deformation processing.
The image processing apparatus according to claim 5, wherein the control unit performs a process of changing an enlargement ratio of the image as the deformation process.
The image processing apparatus according to claim 2, wherein the control unit determines the reading order so as to read from data at a position corresponding to an upper left of the image after the image processing.
The image processing apparatus according to claim 1, wherein the control unit changes the reading order according to operation information from the outside.
A moving body comprising the image processing apparatus according to claim 1.
The moving body according to claim 13, wherein the moving body is an automobile.
The mobile body according to claim 13, wherein the mobile body is an autonomously flying object.
A body-mounted electronic device comprising the image processing device according to claim 1.
A data reading order is determined for an image sensor that can arbitrarily read data in units of pixels or a pixel group composed of a plurality of adjacent pixels, and data is read from the image sensor based on the determined reading order. Including an image processing method.
A data reading order is determined for an image sensor that can arbitrarily read data in units of pixels or a pixel group composed of a plurality of adjacent pixels, and data is read from the image sensor based on the determined reading order. A computer program that causes a computer to execute.