JP2018121160A - Image processing apparatus, image pickup apparatus, and mobile body - Google Patents

Abstract

PROBLEM TO BE SOLVED: To add information convertible and restorable by a simple method in transmission and reception, to an analog image signal.SOLUTION: Image processing equipment 14 comprises an acquisition part 19, an image processing part 20, and an output part 21. The acquisition part 19 acquires a picked-up image. The image processing part 20 adds a positional signal to each of a first area and a second area other than a used area. The used area is included in the picked-up image. The first area extends along a first direction in the picked-up image. The second area extends along a second direction in the picked-up image. The positional signal indicates an arbitrary position within the used area. The output part 21 outputs the picked-up image with the positional signal added thereto.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image processing device, an imaging device, and a moving body.

  For an image signal generated by imaging a subject, there are various status reports corresponding to the image signal. The generated image signal is generally transmitted as an analog signal according to a standard such as NTSC or PAL, and various information is generally transmitted as digital data.

  In order to simplify the configuration and reduce the manufacturing cost, it is considered to add various kinds of information to the inside of the image signal according to the video standard. For example, an imaging apparatus that records parameters for correcting image data in an area of image data corresponding to an area that does not overlap each other in a plurality of images captured by a plurality of imaging apparatuses that perform stereo imaging has been proposed (see Patent Document 1). ).

JP 2008-054056 A

  In the imaging device described in Patent Document 1, it is necessary to convert parameters into an image data embedding mode, and to restore parameters from the image data embedding mode in a detection device that receives image data from the imaging device.

  Accordingly, an object of the present disclosure made in view of the problems of the conventional technology as described above is an image processing apparatus that adds information that can be converted and restored by a simple method at the time of transmission and reception to an analog image signal of a predetermined standard, An imaging device and a moving body are provided.

In order to solve the above-described problems, the image processing apparatus according to the first aspect is
An acquisition unit for acquiring a captured image;
The first region extending along the first direction in the captured image and the second region extending along the second direction in the captured image, other than the use region in the captured image, An image processing unit for adding a position signal indicating an arbitrary position in the use area;
And an output unit that outputs the captured image to which the position signal is added.

An imaging apparatus according to the second aspect is
An imaging unit that generates a captured image by imaging;
The first region extending along the first direction in the captured image and the second region extending along the second direction in the captured image, other than the use region in the captured image, An image processing unit that adds a position signal indicating an arbitrary position in the use area; and an output unit that outputs the captured image to which the position signal is added.

In addition, the mobile object according to the third aspect is
An imaging unit that generates a captured image by imaging, a first region extending along a first direction in the captured image other than a use region in the captured image, and a second direction in the captured image. An image processing unit including an image processing unit that adds a position signal indicating an arbitrary position in the use region to each existing second region, and an output unit that outputs the captured image to which the position signal is added; The imaging device which has is provided.

  According to the image processing device, the imaging device, and the moving body according to the present disclosure configured as described above, information that can be converted and restored by a simple method at the time of transmission / reception can be added to an analog image signal of a predetermined standard .

FIG. 5 is a layout diagram showing mounting positions of a moving body of an imaging apparatus including an image processing apparatus according to the present embodiment. It is a functional block diagram which shows schematic structure of the imaging device of FIG. It is a front view which shows the structure of the light-receiving surface of the image pick-up element of FIG. It is a figure for demonstrating the position of the 1st area | region and 2nd area | region in a captured image. It is a figure for demonstrating the position of the pixel to which a position signal is added within the captured image when expressing arbitrary positions as a point. It is a figure for demonstrating the position of the pixel to which a position signal is added within the captured image when expressing arbitrary positions as a region. It is a figure for demonstrating the position of the pixel to which a position signal is added by the modification in the captured image when expressing arbitrary positions as an area | region. It is a figure for demonstrating the position of the pixel to which a position signal is added within the captured image when expressing several arbitrary positions as a point. It is a flowchart for demonstrating the addition process which the image process part of FIG. 2 performs.

  Embodiments of an image processing apparatus to which the present disclosure is applied will be described below with reference to the drawings.

  As illustrated in FIG. 1, the imaging apparatus 10 including the image processing apparatus according to the first embodiment of the present disclosure is configured so that, for example, the center above the front shield 12 can capture an object in front of the moving body 11. It is installed in the vicinity.

  The mobile body 11 may include, for example, a vehicle, a ship, and an aircraft. The vehicle may include, for example, an automobile, an industrial vehicle, a railway vehicle, a living vehicle, a fixed wing aircraft that runs on a runway, and the like. Automobiles may include, for example, passenger cars, trucks, buses, motorcycles, trolley buses, and the like. Industrial vehicles may include, for example, industrial vehicles for agriculture and construction. Industrial vehicles may include, for example, forklifts and golf carts. Industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combiners, lawn mowers, and the like. Industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, crane trucks, dump trucks, road rollers, and the like. The vehicle may include a vehicle that travels manually. The classification of the vehicle is not limited to the example described above. For example, an automobile may include an industrial vehicle capable of traveling on a road. The same vehicle may be included in multiple classifications. Ships may include, for example, marine jets, boats, tankers, and the like. Aircraft may include, for example, fixed wing aircraft and rotary wing aircraft.

  As illustrated in FIG. 2, the imaging device 10 includes an imaging unit 13 and an image processing device 14. The imaging unit 13 generates a captured image by imaging, and transmits an image signal corresponding to the captured image to the image processing device 14. The image processing device 14 performs predetermined image processing on the image signal received from the imaging unit 13 and transmits the image signal to the external device 15.

  The imaging unit 13 includes an optical system 16, an imaging element 17, and an AFE (Analog Front End) 18.

  The optical system 16 is configured by an optical element such as a lens. The optical system 16 is designed and formed so that optical characteristics such as a field angle and a depth of field have desired values. The optical system 16 forms a subject image.

  The imaging device 17 is, for example, a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The image sensor 17 is disposed at an image forming position of the optical system 16 with respect to a subject at a desired distance. The image sensor 17 captures a subject image formed by the optical system 16. The imaging element 17 generates a captured image as an image signal by imaging.

  As shown in FIG. 3, the light receiving surface ICS of the image sensor 17 is provided with a rectangular light receiving area ICA and a black area OBA that surrounds the light receiving area ICA from four directions. Each pixel arranged in the light receiving area ICA generates a pixel signal corresponding to the amount of received light. Each pixel arranged in the black area OBA has a light receiving surface shielded, and generates a pixel signal corresponding to optical black.

  The AFE 18 performs CDS / SH (Correlated Double Sampling / Sample and Hold) processing, AGC (Auto Gain Control) processing, and AD (Analog to Digital) conversion processing on the captured image acquired from the image sensor 17. In the configuration in which the image sensor 17 is a CMOS image sensor, the AFE 18 may be incorporated in the image sensor 17.

  In FIG. 2, the image processing apparatus 14 includes an acquisition unit 19, an image processing unit 20, and an output unit 21.

  The acquisition unit 19 acquires a captured image that is digital data from the AFE 18.

  The image processing unit 20 includes one or more processors and a memory. The processor may include a general-purpose processor that reads a specific program and executes a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include an FPGA (Field-Programmable Gate Array). The image processing unit 20 may be one of SoC (System-on-a-Chip) and SiP (System In a Package) in which one or more processors cooperate.

  The image processing unit 20 performs predetermined pre-stage image processing such as white balance processing and color interpolation processing. Further, the image processing unit 20 performs subsequent image processing such as γ correction on the captured image that has been subjected to the previous image processing.

  In addition, the image processing unit 20 performs image analysis on the captured image that has been subjected to the pre-stage image processing, and determines whether or not an arbitrary target exists within the use region of the captured image.

  The use area is an area used in the captured image by the external device 15 of the imaging apparatus 10. For example, in a configuration in which the external device 15 is a display device, the use area is an area displayed by the display device in the entire captured image. The optical system 16 and the image sensor 17 are configured so that the area of the captured image corresponding to the light receiving area ICA is larger than the use area.

  An arbitrary object is determined according to the purpose for which the imaging device 10 is used. In the first embodiment, arbitrary objects are, for example, a person and a car. When the image processing unit 20 determines that an arbitrary object exists, the image processing unit 20 recognizes the arbitrary object in the captured image.

  The image processing unit 20 specifies an arbitrary position in the use area of an arbitrary recognized object. The arbitrary position may represent one point such as the position of the center of gravity of an image of an arbitrary recognized object, or may represent an area of an arbitrary width of the image.

  The image processing unit 20 is a position signal that specifies an arbitrary position in the use area of any recognized object in the first area and the second area other than the use area in the captured image subjected to the subsequent image processing. Is added. The addition of the position signal is described in detail below.

  As shown in FIG. 4, the first region A1 and the second region A2 are located in a region cOBA corresponding to the black region OBA of the image sensor 17 in the captured image CI. In the present embodiment, the first region A1 extends in the first direction at both ends sandwiching the captured image CI along the second direction. The second area A2 extends in the second direction at both ends sandwiching the captured image CI in the first direction. In the first embodiment, the first direction and the second direction are the row direction and the column direction, respectively.

  As shown in FIG. 5, the image processing unit 20 applies a predetermined signal to a pixel signal corresponding to the pixel px1 in the first area A1 that is the same position as the position p1 in the first direction of the arbitrary recognized object. A signal value is added as a position signal. The predetermined signal value is a value that can be distinguished from the signal value of the pixel signal corresponding to another pixel in the other first region A1 even after conversion to the analog signal. Since the first area A1 is black, the signal intensity of the pixel signal in the pixels in the first area A1 is zero or a value close to zero. Therefore, the predetermined signal value is distinguishable from zero. In the first embodiment, the predetermined signal value is the maximum value of the gradation of the digital image signal (255 for an 8-bit image signal).

  Note that when the image processing unit 20 represents an area of an arbitrary width of the image, a position signal is added to the captured image CI as described below. As shown in FIG. 6, the image processing unit 20 outputs a predetermined signal value to the pixel signal corresponding to the two pixels px1 in the first area A1 at the same position as the two positions at both ends in the first direction of the image. Is added. Alternatively, as illustrated in FIG. 7, the image processing unit 20 corresponds to all the pixels sandwiched between the two pixels px1 in the first region A1 that are at the same positions as the two positions at both ends in the first direction of the image. A predetermined signal value is added to the pixel signal. In other words, the signal value added to the pixel signal corresponding to all the pixels indicates an arbitrary position having an arbitrary width in the first direction of the image.

  Further, as illustrated in FIG. 5, the image processing unit 20 outputs a predetermined pixel signal corresponding to the pixel px2 in the second area A2 that is the same position as the position of the recognized arbitrary object in the second direction. Are added as position signals. The manner in which the position signal is added in the second region A2 is similar to the manner in which the position signal is added in the first region A1 described above.

  Therefore, the arbitrary position of the arbitrary object includes the straight line sl1 parallel to the second direction passing through the position of the pixel px1 corresponding to the pixel signal to which the position signal is added in the first area A1, and the second area. This is the intersection with the straight line sl2 parallel to the first direction passing through the position of the pixel px2 corresponding to the pixel signal to which the position signal is added in A2.

  In addition, when the image processing unit 20 recognizes a plurality of arbitrary objects, the pixel of the pixel in which the position in the second direction is changed for each arbitrary position of the arbitrary object in the first area A1. A position signal is added to the signal. For example, as illustrated in FIG. 8, when the image processing unit 20 recognizes two arbitrary objects, the image processing unit 20 is in the first region A1 with respect to an arbitrary position (p3, p4) of one object. A position signal is added to the pixel signal of the pixel px3 in the first row, and the pixel signal of the pixel px4 in the second row in the first area A1 is added to any position (p5, p6) of the other object. A position signal is added. In addition, the position in the 1st direction of the pixel of the pixel signal which adds each position signal is as above-mentioned.

  Further, when the image processing unit 20 recognizes a plurality of arbitrary objects, the position signal is a pixel signal of a pixel whose position in the first direction is changed for each arbitrary object in the second area A2. Is added. For example, when the image processing unit 20 recognizes two arbitrary objects, the image processing unit 20 determines the pixel px5 in the first column in the second area A2 with respect to an arbitrary position (p3, p4) of the one object. A position signal is added to the pixel signal, and a position signal is added to the pixel signal of the pixel px6 in the second column in the second area A2 for an arbitrary position (p5, p6) of the other object. In addition, the position in the second direction of the pixel of the pixel signal to which each position signal is added is as described above.

  In such a configuration, by combining an arbitrary position in the second direction in the first area A1 and an arbitrary position in the first direction in the second area A2 in a one-to-one relationship. A plurality of arbitrary positions are indicated by each combination.

  Further, the image processing unit 20 converts the captured image CI to which the position signal specifying an arbitrary position in the use area is added from digital to analog.

  As shown in FIG. 2, the output unit 21 outputs the captured image CI converted to analog to the external device 15.

  The external device 15 extracts a position signal from the acquired captured image CI and uses it to exhibit the functions of the external device 15. For example, the external device 15 is a display device. The display device has a function of superimposing a marker on an arbitrary object when an arbitrary object is imaged while the captured image CI is being displayed. The display device uses a position signal to determine the overlapping position of the marker.

  Next, position signal addition processing executed by the image processing unit 20 in the first embodiment will be described with reference to the flowchart of FIG. The additional process starts when the captured image CI of one frame is acquired from the imaging unit 13.

  In step S100, the image processing unit 20 performs pre-stage image processing on the acquired captured image CI. After performing the pre-stage image processing, the process proceeds to step S101.

  In step S101, the image processing unit 20 performs subsequent image processing on the captured image CI that has been subjected to the previous image processing in step S100. After the post-stage image processing is performed, the process proceeds to step S102.

  In step S102, the image processing unit 20 performs image analysis of the captured image CI that has been subjected to the pre-stage image processing in step S100, and determines whether or not an arbitrary target exists in the use region. If any object is present, the process proceeds to step S103. If there are no arbitrary objects, the process proceeds to step S104.

  In step S103, the image processing unit 20 adds a position signal indicating the position of an arbitrary object whose existence has been confirmed in step S102 to the captured image CI that has been subjected to the subsequent image processing in step S101. After adding the position signal, the process proceeds to step S104.

  In step S <b> 104, the image processing unit 20 sends the captured image CI subjected to the subsequent image processing in step S <b> 101 or the captured image CI added with the position signal in step S <b> 103 to the output unit 21. After the captured image CI is transmitted to the output unit 21, the addition process ends.

  In the image processing apparatus 14 of the first embodiment configured as described above, a position signal indicating an arbitrary position is added to the first area A1 and the second area A2. Since the position of the position signal added to each of the first area A1 and the second area A2 indicates a part of an arbitrary position, it is not necessary to convert and restore the information indicating the position. Therefore, the image processing device 14 can simplify the conversion and restoration of information added to the image signal.

  Further, in the image processing apparatus 14 of the first embodiment, the straight line sl1 parallel to the second direction from the position of the signal added to the first area A1 and the position of the signal added to the second area A2. As shown, an arbitrary position is shown as an intersection with a straight line sl2 parallel to the first direction. With such a configuration, the image processing apparatus 14 can indicate an arbitrary position as a point.

  In the image processing apparatus 14 of the first embodiment, a plurality of position signals indicating arbitrary positions are added in at least one of the first area A1 and the second area A2. With such a configuration, the image processing apparatus 14 can indicate an arbitrary position as a straight line or a two-dimensional surface.

  In the image processing apparatus 14 of the first embodiment, a position signal is added to indicate an arbitrary position having an arbitrary width in at least one of the first area A1 and the second area A2. . With such a configuration, the image processing apparatus 14 can indicate an arbitrary position as a straight line or a two-dimensional surface.

  Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these variations and modifications are included in the scope of the present invention.

  In the present embodiment, the arbitrary position corresponding to the position signal specifies the position of the recognized arbitrary object in the use region, but is not limited to the position of the arbitrary object. For example, an arbitrary position corresponding to the position signal may specify the reference position of the use area in the captured image CI or the position of the use area itself. With such a configuration, image misalignment caused by manufacturing errors and mounting errors of the optical system 16 and the image sensor 17 can be eliminated.

  In this embodiment, when an arbitrary position is represented as a straight line or a two-dimensional surface, position signals are added to positions indicating two places in at least one of the first direction and the second direction. It is not limited. A plurality of position signals indicating a plurality of positions in the region may be added.

DESCRIPTION OF SYMBOLS 10 Image pick-up device 11 Mobile body 12 Front shield 13 Image pick-up part 14 Image processing device 15 External apparatus 16 Optical system 17 Image pick-up element 18 AFE (Analog Front End)
DESCRIPTION OF SYMBOLS 19 Acquisition part 20 Image processing part 21 Output part A1, A2 1st area | region, 2nd area | region CI picked-up image cOBA Area | region equivalent to a black area | region in a picked-up image ICA light-receiving area ICS light-receiving surface OBA black area | region sl1 1st area | region A straight line parallel to the second direction passing through the position of the pixel corresponding to the pixel signal to which the position signal has been added in sl2 The first line passing through the position of the pixel corresponding to the pixel signal to which the position signal has been added in the second region. A straight line parallel to the direction px1 A pixel in the first region that is the same position as the position of the recognized arbitrary object in the first direction px2 It is the same position as the position of the recognized arbitrary object in the second direction Pixels in the second region

Claims (7)

An acquisition unit for acquiring a captured image;
The first region extending along the first direction in the captured image and the second region extending along the second direction in the captured image, other than the use region in the captured image, An image processing unit for adding a position signal indicating an arbitrary position in the use area;
And an output unit that outputs the captured image to which the position signal is added. The image processing apparatus according to claim 1.
The arbitrary position includes a straight line parallel to the second direction passing through the position of the signal added to the first area and the first direction passing through the position of the signal added to the second area. An image processing device that is the intersection of a straight line parallel to The image processing apparatus according to claim 1 or 2,
The image processing device adds a plurality of position signals indicating the arbitrary positions in at least one of the first region and the second region. The image processing apparatus according to claim 1 or 2,
The image processing apparatus adds the position signal to indicate the arbitrary position having an arbitrary width in at least one of the first area and the second area. In the image processing device according to any one of claims 1 to 4,
The image processing unit recognizes an arbitrary object within the use area,
The arbitrary position is a position of the arbitrary object in the use area. An imaging unit that generates a captured image by imaging;
The first region extending along the first direction in the captured image and the second region extending along the second direction in the captured image, other than the use region in the captured image, An image processing apparatus comprising: an image processing unit that adds a position signal indicating an arbitrary position in a use region; and an output unit that outputs the captured image to which the position signal is added. An imaging unit that generates a captured image by imaging, a first region extending along a first direction in the captured image other than a use region in the captured image, and a second direction in the captured image. An image processing unit including an image processing unit that adds a position signal indicating an arbitrary position in the use region to each existing second region, and an output unit that outputs the captured image to which the position signal is added; A moving body comprising an imaging device.

Images