JP2024015890A - Image processing system, image processing method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system that allows a user to easily adjust a cut-out area.

SOLUTION: An image processing system has: image acquisition means that acquires information on a photographed image formed by imaging means that picks up an optical image having a low distortion area and a high distortion area; information acquisition means that acquires cut-out area information indicating the range of a cut-out area in the photographed image information; cut-out boundary information forming means that forms cut-out boundary information indicating the boundary of the cut-out area based on the cut-out area information; and image forming means that forms an image obtained by superimposing distortion boundary information indicating the boundary between the low distortion area and the high distortion area and the cut-out boundary information on the photographed image information.

SELECTED DRAWING: Figure 4

COPYRIGHT: (C)2024,JPO&INPIT

Description

The present invention relates to an image processing system, an image processing method, and a computer program related to cutout regions from images.

In recent years, there has been a desire to replace the room mirrors and side mirrors installed in vehicles with electronic mirrors.

For example, Patent Document 1 discloses that the vehicle has an imaging means for taking an image of the rear outside the vehicle and a display means inside the vehicle, and the image taken by the imaging means is displayed on an electronic rearview mirror inside the vehicle so that the driver can see the rear outside the vehicle. It mentions an electronic rearview mirror system that allows you to check what's going on.

JP2019-166887A

In addition, a special ultra-wide-angle lens (hereinafter referred to simply as the ultra-wide-angle lens) is used to capture images for the electronic rearview mirror and the rear confirmation monitor when reversing (hereinafter referred to as the back monitor) with one camera. A new camera (hereinafter referred to as an ultra-wide-angle camera) is being considered.
However, in such a camera, distortion occurs in the image, and there has not been sufficient consideration as to what kind of UI should be used when operating a partial area of the displayed screen.

SUMMARY OF THE INVENTION In view of the above problems, it is an object of the present invention to provide an image processing system that allows a user to easily adjust a cutout area.

In order to achieve the above object, an image processing system according to one aspect of the present invention includes:
an image acquisition means for acquiring captured image information formed by an imaging means for capturing an optical image having a low distortion region and a high distortion region;
Information acquisition means for acquiring cutout area information indicating a range of the cutout area in the photographed image information;
Cutting boundary information forming means for forming cutting boundary information indicating a boundary of the cutting region based on the cutting region information;
The present invention is characterized by comprising an image forming means for forming an image in which distortion boundary information indicating a boundary between the low distortion area and the high distortion area and the cutting boundary information are superimposed.

According to the present invention, it is possible to realize an image processing system in which a user can easily adjust a cutout area.

1 is a functional block diagram showing the configuration of an image processing system 100 according to a first embodiment. FIG. (A) and (B) are diagrams for explaining the optical characteristics of the ultra-wide-angle lens 11 in Embodiment 1 of the present invention. (A) to (D) are conceptual diagrams illustrating a composite image formed in the first embodiment. (A) to (C) are diagrams showing display examples of the composite image display section 30 during a swipe operation. (A) to (C) are diagrams showing display examples of the composite image display section 30 during a pinch operation. (A) and (B) are diagrams showing a display example of the composite image display section 30 when the composite image distortion correction switching key is pressed. (A) to (C) are diagrams showing examples of displays on the composite image display section 30 when the cut-out image distortion correction switching key is pressed. 5 is a flowchart when forming a composite image in Embodiment 1. FIG. 9 is a flowchart illustrating step S200 in the flow of FIG. 8. 9 is a flowchart explaining step S300 in the flow of FIG. 8. 7 is a flowchart illustrating an example of a processing flow when changing cutout region information in the first embodiment. 7 is a flowchart showing a processing flow when changing the image output mode in the first embodiment. 7 is a flowchart showing a processing flow when forming a cutout image according to the first embodiment. FIG. 2 is a functional block diagram illustrating a configuration example of an image processing system according to a second embodiment. FIG. 3 is a functional block diagram illustrating a configuration example of an image processing system according to a third embodiment. 12 is a diagram illustrating a display example of the composite image display unit 30 during a swipe operation in Embodiment 4. FIG.

Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In each figure, the same reference numerals are given to the same members or elements, and overlapping explanations are omitted or simplified.

<Embodiment 1>
In this embodiment, the user can adjust the cropping area by selecting the intended area in the captured image while checking the range of the cropping area and the degree of distortion in the image captured by the ultra-wide-angle camera. Make it.

FIG. 1 is a functional block diagram showing the configuration of an image processing system 100 according to the first embodiment. The image processing system 100 includes an imaging section 10, an integrated processing section 20, a composite image display section 30, and a cropped image display section 40. Note that although the imaging unit 10 is mounted on a moving body such as a vehicle, the integrated processing unit 20, composite image display unit 30, and cropped image display unit 40 may not be mounted on a moving body such as a vehicle. It may be provided in a remote controller or the like.

Note that some of the functional blocks shown in FIG. 1 are realized by causing a computer (not shown) included in the integrated processing unit 20 or the like to execute a computer program stored in a memory (not shown). However, some or all of them may be realized by hardware. As the hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), etc. can be used.

Furthermore, even if the functional blocks shown in Figure 1 are surrounded by solid lines, they do not have to be built into the same housing, and can be connected to each other via signal paths. It may be configured by separate devices. Note that the above description regarding FIG. 1 also applies to FIGS. 14 and 15.

The imaging unit 10 is, for example, an ultra-wide-angle camera, the integrated processing unit 20 is, for example, a computer mounted on a vehicle as a moving object, and the composite image display unit 30 is, for example, a car navigation monitor or a car navigation monitor equipped with a touch panel and push buttons (switching keys). It is a tablet device. Further, the cutout image display section 40 is, for example, an electronic room mirror equipped with a touch panel and push buttons (switching keys). Note that the electronic rearview mirror is provided, for example, in the upper vicinity of the front side of the driver's seat.

The imaging unit 10 includes an ultra-wide-angle lens 11, an imaging element 12 such as a CMOS image sensor or a CCD image sensor, and a storage unit 13 such as a flash memory. The ultra-wide-angle lens 11 is a lens or a lens group that forms an optical image having a low distortion area and a high distortion area on the light receiving surface of the image sensor 12.

2(A) and 2(B) are diagrams for explaining the optical characteristics of the ultra-wide-angle lens 11 in Embodiment 1 of the present invention, and FIG. 2(A) is a diagram for explaining the optical characteristics of the ultra-wide-angle lens 11 in the present embodiment. , is a diagram showing the image height y at each half angle of view on the light-receiving surface of the image sensor in contour lines.

FIG. 2(B) is a diagram showing the projection characteristic representing the relationship between the image height y and the half angle of view θ of the ultra-wide-angle lens 11 in this embodiment. In FIG. 2(B), the horizontal axis is the half angle of view (the angle between the optical axis and the incident light beam) θ, and the image formation height (image height) y on the sensor surface (image plane) of the image sensor 12 is shown as the vertical axis.

As shown in FIG. 2(B), the ultra-wide-angle lens 11 in this embodiment is configured such that its projection characteristic y(θ) is different between a region less than a predetermined half-field angle θa and a region greater than a half-field angle θa. has been done. Therefore, when the amount of increase in image height y with respect to the half angle of view θ per unit is referred to as resolution, the resolution differs depending on the region.

It can be said that this local resolution is expressed by the differential value dy(θ)/dθ of the projection characteristic y(θ) at the half angle of view θ. That is, it can be said that the larger the slope of the projection characteristic y(θ) in FIG. 2(B), the higher the resolution. It can also be said that the larger the interval between the image heights y in each half angle of view of the contour lines in FIG. 2(A), the higher the resolution.

In this embodiment, the area around the center formed on the light-receiving surface of the image sensor when the half-field angle θ is less than a predetermined half-field angle θa is a high-resolution region 11a, and the half-field angle θ is a predetermined half-field. The area outside the angle θa is called a low resolution area 11b. In this embodiment, the boundary between the high-resolution area 11a and the low-resolution area 11b, that is, the boundary between the low-distortion area and the high-distortion area, is referred to as a distortion boundary.

In this embodiment, the high-resolution area 11a is a low-distortion area with relatively little distortion, and the low-resolution area 11b is a high-distortion area with relatively many distortions. Therefore, in this embodiment, the high-resolution area and the low-resolution area correspond to the low-distortion area and the high-distortion area, respectively, and the high-resolution area and the low-resolution area may be referred to as the low-distortion area and the high-distortion area, respectively. be.

The ultra-wide-angle lens 11 in this embodiment is configured such that its projection characteristic y(θ) is larger than f×θ in the high-resolution area (low-distortion area) 11a (f is the focal length of the ultra-wide-angle lens 11 ). Furthermore, the projection characteristic y(θ) in the high resolution area (low distortion area) is set to be different from the projection characteristic in the low resolution area (high distortion area).

Further, when θmax is the maximum half-field angle that the ultra-wide-angle lens 11 has, the ratio θa/θmax of θa and θmax is desirably equal to or larger than a predetermined lower limit value, and for example, the predetermined lower limit value is 0.15 to 0. .16 is desirable.

Further, the ratio θa/θmax between θa and θmax is desirably less than a predetermined upper limit, for example, preferably 0.25 to 0.35. For example, when θmax is 90°, the predetermined lower limit is 0.15, and the predetermined upper limit is 0.35, it is desirable to determine θa in the range of 13.5 to 31.5°.

Further, the ultra-wide-angle lens 11 is configured such that its projection characteristic y(θ) satisfies the following equation 1.

f is the focal length of the ultra-wide-angle lens 11 as described above, and A is a predetermined constant. By setting the lower limit to 1, the center resolution can be made higher than that of a fisheye lens using the orthogonal projection method (y = f × sin θ), which has the same maximum imaging height, and by setting the upper limit to A, the fisheye lens Good optical performance can be maintained while obtaining the same angle of view. The predetermined constant A may be determined by considering the balance between the resolutions of the high resolution area and the low resolution area, and is preferably set to 1.4 to 1.9.

By configuring the ultra-wide-angle lens 11 as described above, it is possible to obtain a high-definition image at a narrow angle of view around the optical axis, and to obtain a low-resolution captured image at a wide angle of view outside the optical axis. . That is, in the high-resolution area 11a, high resolution can be obtained, while in the low-resolution area 11b, the amount of increase in the image height y with respect to the half angle of view θ per unit is small, and it is possible to capture a wider angle of view. become. Therefore, high resolution can be obtained in the high resolution area 11a while providing an imaging range with a wide angle of view equivalent to that of a fisheye lens.

Furthermore, in this embodiment, in a high resolution region (low distortion region), the central projection method (y=f×tanθ) and the equidistant projection method (y=f×tanθ), which are the projection characteristics of a normal imaging optical system, are used. The projection characteristic is approximated to θ). Therefore, optical distortion is small and it is possible to display finely. Therefore, it is possible to obtain a natural sense of perspective when viewing surrounding vehicles, for example, and to obtain good visibility by suppressing deterioration of image quality.

Note that the ultra-wide-angle lens of this embodiment refers to an optical system having a projection characteristic y(θ) that satisfies the above-mentioned condition of Equation 1, for example. However, the same effect can be obtained as long as the projection characteristic y(θ) satisfies the condition of Equation 1 above, so the characteristics of the ultra-wide-angle lens 11 in the present invention are limited to the projection characteristics shown in FIG. Not done.

Returning to FIG. 1, the image sensor photoelectrically converts the optical image formed by the ultra-wide-angle lens 11 to form an image signal. The image capturing unit 10 has an image processing unit (not shown), and the image processing unit performs image processing on the image signal input from the image sensor 12, and outputs captured image information resulting from RAW development of the image signal to the integrated processing unit 20 as a captured image. It is output to. Specifically, the image processing unit performs debayer processing on the image signal inputted from the image sensor 12 according to the Bayer array, and converts it into RGB raster format photographed image information.

Furthermore, the image processing section performs various correction processes such as white balance adjustment, gain/offset adjustment, gamma processing, color matrix processing, and reversible compression processing. However, irreversible compression processing is not performed, and a so-called RGB image that has been developed in RAW is formed.

The storage unit 13 has a ROM, and stores lens characteristic information 13a specific to the ultra-wide-angle lens 11. The lens characteristic information 13a includes, for example, the optical axis center coordinates of the ultra-wide-angle lens 11 (hereinafter referred to as lens center coordinates L_cp), the coefficient of distortion L_k, the radius L_r of the lens, the radius L_bd of the distortion boundary, and the like.

These pieces of information may be added to the photographed image file as meta information. Note that 1-bit meta information that means a distortion boundary and is formed based on the lens center coordinate L_cp and the radius L_bd of the distortion boundary is called distortion boundary information.

The storage unit 13 has a RAM, and can also store cutout area information 13b, composite image output mode 13c, and cutout image output mode 13d as rewritable information. Note that the information stored in this RAM may be stored in, for example, the RAM 26 of the integrated processing section 20.

The cropping area information CA includes, for example, the centroid coordinate CA_cp of the cropping area on the captured image information selected by the user, a reference size CA_bs representing the number of pixels in the horizontal and vertical directions of the cropping area, and a cropping area for the reference size CA_bs. This includes the magnification CA_m, etc. Note that the cutout area information CA may also be added to the photographed image file as meta information.

Note that the number of pixels in the horizontal and vertical directions calculated by multiplying the reference size CA_bs of the cutout area by the magnification CA_m of the cutout area is called the cutout area size CA_s. Furthermore, 1-bit meta information that is formed based on the barycentric coordinates CA_cp of the cutout area and the cutout area size CA_s and means the boundary between the cutout area and an area other than the cutout area is called cutout boundary information.

The composite image output mode 13c is 1-bit information that is input to the composite image forming unit 23 and determines whether or not to perform distortion correction on the captured image information. For example, if the value of the composite image output mode is 0, it indicates a distortion correction no mode in which the composite image forming unit 23 does not perform distortion correction on the photographed image information, and if the value of the composite image output mode is 1, it indicates a distortion correction mode in which distortion correction is performed on the photographed image information. Indicates correction mode.

The cropped image output mode 13d is 1-bit information that is input to the cropped image forming section 24 and determines whether or not distortion correction is to be performed on the photographed image within the cropped image. For example, if the value of the cropped image output mode is 0, it indicates a distortion correction no mode in which the cropped image forming unit 24 does not perform distortion correction on the captured image within the cropped image, and the value of the cropped image output mode is 1. If so, it indicates a distortion correction mode in which distortion correction is performed on the photographed image within the cropped image.

The integrated processing section 20 has a cutout boundary information calculation section 21, a cutout image output mode control section 22, a composite image formation section 23, and a cutout image formation section 24 as functions of an FPGA (or SoC). It also has a CPU 25 as a computer and a RAM 26 as a main storage medium.

The integrated processing unit 20 also includes a program memory (not shown) that stores computer programs, and a DMAC (Direct Memory Access Controller) that reads and writes various information about the entire image processing system 100.

The cutout boundary information calculation unit 21 forms cutout boundary information to be superimposed on the captured image information. Further, the cropping boundary information calculation unit 21 has a register therein that holds a composite image output mode, a cropped image output mode, lens characteristic information L, cropping area information CA, and cropping boundary information.

Specifically, the cutout boundary information calculation unit 21 determines processing for the cutout area information CA based on the composite image output mode and cutout image output mode held in internal registers. Then, cutting boundary information is formed based on the lens characteristic information L and the cutting area information CA held in the internal register. Further, the cutout boundary information is written into the registers of the cutout image output mode control section 22 and the composite image forming section 23, respectively.

The composite image forming unit 23 forms composite image information by superimposing at least one of distortion boundary information, cut-out boundary information, and message information on the captured image information, and outputs the composite image information to the composite image display unit 30. do. Further, the composite image forming section 23 has a register therein that holds a composite image output mode, lens characteristic information L, distortion boundary information, cutting boundary information, message information, and the like. Note that the message information will be explained together with the explanation of the cutout image output mode control section 22.

The composite image forming unit 23 acquires photographed image information from the image sensor 12 and performs resolution conversion on the photographed image information. Further, the composite image forming unit 23 determines whether or not to perform distortion correction on the captured image information based on the composite image output mode held in an internal register. When the composite image output mode is the distortion correction mode, distortion correction is performed on the photographed image information based on the lens characteristic information L held in the internal register.

In addition, the composite image forming unit 23 superimposes and synthesizes the distortion boundary information and cut-out boundary information held in internal registers, thereby creating the composite images 200a, 200b, 300a, and the like shown in FIGS. 3(A) to 3(D). 300b is formed.

FIGS. 3A to 3D are conceptual diagrams illustrating the composite image formed in the first embodiment, and FIGS. 3A and 3B show that the composite image output mode is the no-distortion correction mode. FIG. 3 is a diagram showing a composite image when . An image in which distortion boundary information 201 and cutout boundary information 202 are superimposed on composite images 200a and 200b, respectively, is shown.

Further, the cropping boundary information 202 is formed by the cropping boundary information calculation unit 21 performing processing according to the composite image output mode and the cropped image output mode, and when the cropped image output mode is the no-distortion correction mode. If there is, it is superimposed as cutting boundary information 202a. Further, when the cutout image output mode is the distortion correction mode, it is superimposed as cutout boundary information 202b. Here, when the shape of the cutout boundary information 202a is a rectangle, the shape of the cutout boundary information 202b is barrel-shaped, and the farther outside the distortion boundary is, the stronger the distortion correction is applied.

On the other hand, the composite images 300a and 300b in FIGS. 3(C) and 3(D) are composite images when the composite image output mode is the distortion correction mode, and are obtained by superimposing distortion boundary information 301 and cutting boundary information 302, respectively. It is an image.

Further, the cropping boundary information 302 is formed by the cropping boundary information calculation unit 21 performing processing according to the values of the composite image output mode and the cropped image output mode, and the cropped image output mode is distortion correction. If there is no mode, it is superimposed as cutting boundary information 302a. On the other hand, when the cutout image output mode is the distortion correction mode, the cutout image is superimposed as cutout boundary information 302b.

Here, when the shape of the cutout boundary information 302b is a rectangle, the shape of the cutout boundary information 302a is a pincushion shape, and the shape becomes more strongly distorted as it moves away from the distortion boundary. Further, the composite image forming section 23 superimposes message information held in an internal register on the composite image information and outputs the superimposed message information to the composite image display section 30.

When the cropped region does not include a highly distorted region, the cropped image output mode control section 22 sets the cropped image output mode so as not to perform distortion correction on the captured image information input to the cropped image forming section 24. Switch to no distortion correction mode. Further, the cutout image output mode control section 22 has internal registers that hold the cutout image output mode, distortion boundary information, cutout boundary information, and message information.

Then, the cropped image output mode control unit 22 determines whether or not the cropped region includes a highly distorted region based on the distortion boundary information and the cropped boundary information held in the internal register. If it is determined that the cutout area does not include a high distortion area, the cutout image output mode is set to distortion correction no mode, and the cutout image output mode in the RAM 26 is rewritten.

On the other hand, if it is determined that the cropped region includes a highly distorted region, the cropped image output mode control unit 22 outputs a 2-bit message according to the cropped image output mode held in the internal register. form the information and write the message information to RAM 26. Here, the message information is, for example, 2-bit information that determines one of the messages 401, 402, and 403 shown in FIGS. 4 to 7.

4(A) to (C) are diagrams showing display examples of the composite image display section 30 during a swipe operation, and FIGS. 5(A) to (C) are diagrams showing display examples of the composite image display section 30 during a pinch operation. FIG. Further, FIGS. 6A and 6B are diagrams showing display examples of the composite image display section 30 when the composite image distortion correction switching key is pressed, and FIGS. 7A to 7C are diagrams showing display examples of the cropped image. 7 is a diagram showing a display example of the composite image display section 30 when a distortion correction switching key is pressed. FIG.

In this embodiment, the cropped image output mode control unit 22 outputs a blank message 401 when the message information is 00, and a blank message 401 when the value is 10. ” message 402. That is, it is possible to display a message indicating whether or not the cutout region includes the low distortion region.

If the value is 11, a message 403 is displayed saying "Mirror distortion is corrected and displayed (on the cropped image)." That is, it is possible to display a message regarding the presence or absence of distortion correction for the image of the cutout area.

Based on the message information formed by the cutout image output mode control section 22, the CPU 25 forms message information of any one of the messages 401, 402, and 403 shown in FIGS. will be superimposed on the screen.

The cutout image forming section 24 forms cutout image information by cutting out the captured image information based on the cutout area information CA, and outputs the cutout image information to the cutout image display section 40 . Further, the cutout image forming section 24 has a register therein that holds a cutout image output mode, lens characteristic information L, cutout area information CA, and cutout image information.

Specifically, the cutout image forming unit 24 acquires photographed image information from the image sensor 12 and performs resolution conversion on the photographed image information. Further, the cropped image forming unit 24 determines whether or not to perform distortion correction on the captured image information based on the cropped image output mode held in an internal register. For example, when the cutout image output mode is the distortion correction mode, distortion correction is performed on the captured image information of the cutout image based on the lens characteristic information L held in an internal register.

Further, the cutout image forming unit 24 forms cutout image information by cutting out the photographed image information based on the cutout area information CA held in the internal register, performs resolution conversion on the cutout image information, The cutout image information is output to the cutout image display section 40.

The CPU 25 performs various controls on the entire image processing system 100 by executing computer programs in the program memory. Further, the CPU 25 communicates between the storage unit 13 and the RAM 26, and reads and writes various information such as lens characteristic information 13a, cutout area information 13b, composite image output mode 13c, and cutout image output mode 13d. . Further, the CPU 25 forms distortion boundary information from the lens center coordinate L_cp and the radius L_bd of the distortion boundary, and writes the distortion boundary information into the RAM 26.

Further, the CPU 25 reads out various information such as lens characteristic information L, cutout area information CA, composite image output mode, cutout image output mode, and distortion boundary information from the RAM 26. Then, these pieces of information are written into the respective registers of the cutout boundary information calculation section 21, cutout image output mode control section 22, composite image formation section 23, and cutout image formation section 24, respectively. Further, the CPU 25 reads message information from the RAM 26, forms message image information according to the message information, and writes the message image information into a register included in the composite image forming section 23.

Further, the CPU 25 detects user operations such as swiping or pinching within the cutout region on the screen based on electrical signals input from at least one of the touch panels of the adjustment value input UI 32 and the adjustment value input UI 42. Then, the cutout area information CA is calculated based on the detected user's operation, and the cutout area information CA in the RAM 26 is rewritten. That is, the adjustment value input UI 32 and the adjustment value input UI 42 function as operation means for the user to change the range of the cutout area.

Further, the CPU 25 detects the pressing of the composite image distortion correction switching key based on the electrical signal input from the push button (switching key) of either the adjustment value input UI 32 or the adjustment value input UI 42, and switches the composite image output mode. Afterwards, the composite image output mode in the RAM 26 is rewritten.

Further, the CPU 25 detects pressing of the cut-out image distortion correction switching key from the input electric signal, switches the cut-out image output mode, and then rewrites the cut-out image output mode in the RAM 26.

The composite image display section 30 and the cutout image display section 40 each have a display section 31 and a display section 41, such as a liquid crystal display or an organic EL display. Further, it has an adjustment value input UI 32 and an adjustment value input UI 42 that are a combination of a projected capacitive type touch panel and a momentary type or termination type push button (switching key), for example.

The electrical signal input to the integrated processing unit 20 by the user operating either the adjustment value input UI 32 or the adjustment value input UI 42 while referring to the composite image displayed on the display unit 31 is converted into a computer program executed by the CPU 25. detected as a user action.

The display unit 31 generates and displays any of the composite images shown in FIG. 3 based on the composite image information input from the composite image forming unit 23. The display unit 41 generates and displays a cutout image based on the cutout image information input from the cutout image forming unit 24.

The adjustment value input UI 32 and the adjustment value input UI 42 each have a touch panel and a push button (switch key). Further, the adjustment value input UI 32 has a synthetic image distortion correction switching key 601 and a cutout image distortion correction switching key 701, and the adjustment value input UI 42 has a distortion correction switch key 701 for the cutout image as a key pressed by the user. It has at least a correction switching key 702.

Here, the composite image distortion correction switching key 601 functions as an operation means for the user to select on/off of distortion correction of the photographed image information. Further, the cropped image distortion correction switching key 701 functions as an operation means for the user to select on/off of cropping boundary information and distortion correction of the image of the cropped area.

Furthermore, when the user presses at least one of the synthesized image distortion correction switching key 601 and the cropped image distortion correction switching keys 701 and 702, a 2-bit electrical signal indicating the depression of the respective key is generated. is formed and outputs an electrical signal to the integrated processing section 20.

The touch panels of the adjustment value input UI 32 and the adjustment value input UI 42 have a plurality of transparent electrode layers arranged on the display section 31 and the display section 41, respectively. In addition, in the touch panel, when a user brings a finger close to the electrode layer, the capacitance between multiple electrodes in the touch panel changes simultaneously, forming an electric signal from each electrode, and outputting the electric signal to the integrated processing unit 20. do.

Next, detection of a swipe operation on the adjustment value input UI 32 and adjustment value input UI 42 will be described. Examples of displays on the composite image display section 30 when a swipe by the user is detected are shown in FIGS. 4(A) to 4(C).

The CPU 25 detects input of an electrical signal from any one point on the touch panel of the adjustment value input UI 32. If the electrical signal is an electrical signal input from within the cutout region superimposed on the composite image on the display unit 31, and it is subsequently detected that the input coordinates of the electrical signal are displaced, the input Detect electrical signals as swipe operations.

On the other hand, when the CPU 25 detects the input of an electrical signal from any one point on the touch panel of the adjustment value input UI 42 and then detects a displacement of the input coordinates of the electrical signal, the CPU 25 performs a swipe operation on the input electrical signal. Detected as.

Further, the CPU 25 converts the coordinates of the electrical signal input from the touch panel of either the adjustment value input UI 32 or the adjustment value input UI 42 into coordinates on the captured image information, and inputs the coordinates from the coordinates before the swipe operation and the coordinates after the swipe operation. The displacement of the coordinates (hereinafter referred to as coordinate displacement) is calculated. By adding the coordinate displacement to the barycenter coordinate CA_cp of the cutout area, the barycenter coordinate CA_cp of the cutout area after the swipe operation is calculated. Thereafter, the value of the barycenter coordinate CA_cp of the cutout area in the RAM 26 is rewritten.

At this time, in response to the input from the adjustment value input UI 32, the coordinate displacement is calculated as a value obtained by subtracting the coordinates before the swipe operation from the coordinates after the swipe operation. Further, for input from the adjustment value input UI 42, the coordinate displacement is calculated as a value obtained by subtracting the coordinates after the swipe operation from the coordinates before the swipe operation. In this embodiment, by calculating as described above, the barycentric coordinate CA_cp of the cropped area is set in the same direction as the coordinate displacement, regardless of whether the user views the composite image and operates the cropped image. Displace.

Next, detection of a pinch operation on the adjustment value input UI 32 and adjustment value input UI 42 will be explained. FIG. 5 shows a display example of the composite image display section 30 when a pinch by the user is detected. If the CPU 25 detects the input of electrical signals from any two points on the touch panel of either the adjustment value input UI 32 or the adjustment value input UI 42, and then detects the displacement of the input coordinates of the two points, the CPU 25 detects the input. Detects the electrical signal as a pinch operation.

Further, the CPU 25 converts the coordinates of each electric signal of two points input from the touch panel of either the adjustment value input UI 32 or the adjustment value input UI 42 into coordinates on the photographed image information, and calculates the pinch point between the two input points. Calculate the distance before the operation and the distance after the pinch operation. Then, the distance displacement between the two input points (hereinafter referred to as distance displacement) is calculated from the distance before the pinch operation and the distance after the pinch operation.

At this time, for the input from the adjustment value input UI 32, the distance displacement is calculated as the value obtained by subtracting the distance before the pinch operation from the distance after the pinch operation, and for the input from the adjustment value input UI 42, the distance displacement is calculated by subtracting the distance before the pinch operation from the distance after the pinch operation. Distance displacement is calculated as the value obtained by subtracting the distance after the pinch operation from the previous distance. In this embodiment, by calculating as described above, whether the user operates by looking at the composite image or by looking at the cutout image, when the distance displacement is positive, the cutout area is enlarged. When it is negative, it can represent a reduction in the cutout area.

Thereafter, the displacement ΔCA_m of the magnification of the cutout region is calculated by multiplying the distance displacement Δx by a constant k (hereinafter referred to as enlargement/contraction sensitivity) that indicates the sensitivity of scaling of the cutout region. Further, by adding the displacement of the magnification to the magnification CA_m of the cropping area, the magnification CA_m_pch of the cropping area after the pinch operation is calculated. Expressing this numerically, it becomes the following formula 2.

CA_m_pch=CA_m+k・Δx...Formula 2

For example, if the magnification CA_m of the cropped area before the pinch operation is 1.0 times and the distance displacement is -10 pixels, and the scaling sensitivity is set to +0.01, the magnification CA_m_pch of the cropped area after the pinch operation is It becomes 0.9. Expressing this in a mathematical formula is the following formula 3.

CA_m_pch=1.0+0.01・(-10)=0.9...Formula 3

Thereafter, the CPU 25 rewrites the cutout area magnification CA_m in the RAM 26 to the cutout area magnification CA_m_pch of the cutout area after the pinch operation.

Next, detection of pressing of a push button (switching key) on the adjustment value input UI 32 and adjustment value input UI 42 will be described. FIG. 6(B) shows a display example when the composite image display unit 30 detects that the user presses the composite image distortion correction switching key 601 in a state where the cropping area is located at the upper left of the captured image. There is.

Further, FIG. 7(B) shows a case where pressing of the distortion correction switching key 701 of the cropped image is detected in the composite image display unit 30 or a pressing of the distortion correction switching key 702 in the cropped image display unit 40 is detected. A display example of the composite image display unit 30 in the case of detection is shown.

Note that FIG. 6B shows a state in which the composite image 200a before distortion correction is switched to the composite image 300a after distortion correction by pressing the composite image distortion correction switching key 601. By displaying the composite image 300a after distortion correction as shown in FIG. 6(B), the user can confirm that the cutout boundary information 302a is more strongly distorted as it moves further outside the distorted boundary information 201.

In addition, in FIG. 7B, when the cropped image distortion correction switching key 701 or the distortion correction switching key 702 is pressed, the cropping boundary information 202b superimposed on the composite image 200a changes to the cropped image 200b. A changed state is shown as shown in area 202c. The user can confirm from the display in FIG. 7B that the further outside the distortion boundary is, the stronger the distortion correction is applied to the cutout region. Note that at this time, the image of the cutout area is displayed on the electronic mirror (display unit 41) shown in FIG. 7(C) in a state in which the distortion has been corrected.

8 to 13 are flowcharts for explaining a series of operations of the integrated processing unit 20 of the first embodiment. In this embodiment, the CPU 25 operates in multithreads for each of the flows shown in FIGS. 8 and 11 to 13. Further, the processing of each step in the flowcharts of FIGS. 8 to 13 is performed by the CPU 25 serving as a computer in the integrated processing unit 20 executing a computer program stored in a memory.

FIG. 8 is a flowchart for forming a composite image in the first embodiment, and is for explaining the flow of forming composite image information and outputting the composite image information to the composite image display section 30.

Further, FIG. 9 is a flowchart for explaining step S200 in the flow of FIG. 8, and is for explaining the operation of the cutout boundary information calculation unit 21. FIG. 10 is a flowchart for explaining step S300 in the flow of FIG. 8, and is for explaining the operation of the cropped image output mode control section 22.

In step S<b>801 , the CPU 25 functions as an image acquisition unit, photographs by the imaging unit 10 , and inputs photographed image information to the composite image forming unit 23 . At this time, step S801 functions as an image acquisition step for acquiring captured image information formed by an imaging means that captures an optical image having a low distortion area and a high distortion area.

In step S802, the CPU 25 functions as an information acquisition unit, acquires the values of the lens characteristic information 13a, cropping area information 13b, composite image output mode 13c, and cropped image output mode 13d from the storage unit 13, and stores them in the RAM 26. Write. At this time, step S802 functions as an information acquisition step for acquiring cutout area information indicating the range of the cutout area in the captured image information.

In step S803, the CPU 25 functions as a cutting boundary information forming means, forms distortion boundary information based on the lens characteristic information L, and writes it into the RAM 26. At this time, step S803 functions as a cutting boundary information forming step of forming cutting boundary information indicating the boundary of the cutting area based on the cutting area information.

In step S804, the CPU 25 reads the values of the lens characteristic information L, distortion boundary information, cutout area information CA, composite image output mode, and cutout image output mode from the RAM 26. Then, the respective values are written into the respective registers of the cutout boundary information calculation section 21, the cutout image output mode control section 22, the composite image formation section 23, and the cutout image formation section 24.

In step S200, the CPU 25 causes the cropping boundary information calculation unit 21 to form cropping boundary information based on the respective values of the lens characteristic information L, the cropping area information CA, the composite image output mode, and the cropped image output mode. . That is, any one of four types of cutout boundary information 202a, 202b, 302a, and 302b in FIG. 3 is formed according to each image output mode. Details of step S200 will be described later using the flowchart of FIG.

Thereafter, the cutout boundary information is passed to the cutout image output mode control section 22, the composite image formation section 23, and the cutout image formation section 24. After that, the process advances to step S300.

In step S300, the CPU 25 causes the cropped image output mode control unit 22 to determine whether or not the cropped region includes a highly distorted region based on the values of the distortion boundary information and the cropped boundary information. . Then, depending on the determination result and the value of the cropped image output mode, three types of messages are generated: a value 00 corresponding to the message 401 in FIGS. 4 to 7, a value 10 corresponding to the message 402, and a value 11 corresponding to the message 403. Form the information and write it to RAM 26.

If it is determined that the cropped area does not include a highly distorted area, the cropped image output mode control section 22 rewrites the cropped image output mode in the RAM 26 to a distortion correction no mode. Details of step S300 will be described later using the flowchart of FIG.

In step S805, the CPU 25 reads the message information value from the RAM 26, forms message information corresponding to any of the three messages 401, 402, and 403 shown in FIGS. 4 to 7, and synthesizes the message information. It is passed to the image forming section 23.

In step S806, the CPU 25 causes the composite image forming unit 23 to determine whether the composite image output mode is the distortion correction mode. If the composite image output mode is a mode with distortion correction, the process advances to step S807, and if it is a mode without distortion correction, the process advances to step S808.

In step S807, the CPU 25 causes the composite image forming unit 23 to perform distortion correction on the photographed image information based on the lens characteristic information L, thereby converting the photographed image information into photographed image information after distortion correction.

In step S808, the CPU 25 functions as an image forming means. Then, the composite image forming unit 23 superimposes the distortion boundary information and the cut-out boundary information on the photographed image information, and synthesizes a composite image having information on at least one of the composite images 200a, 200b, 300a, and 300b shown in FIG. Forming image information. At this time, step S808 functions as an image forming step for forming an image in which the distortion boundary information indicating the boundary between the low distortion area and the high distortion area and the cutting boundary information are superimposed.

Thereafter, the message information is superimposed on the composite image information, and in step S809, the CPU 25 outputs the composite image information to the display unit 31. In step S810, the CPU 25 determines whether there is a termination request from the user. If No, the process advances to step S804, and if Yes, the process advances to step S811. Note that the user's request for termination becomes Yes, for example, when the CPU 25 detects that the engine of the vehicle is turned off.

In step S811, the CPU 25 reads the respective values of the cutout area information CA, the composite image output mode, and the cutout image output mode in the RAM 26. Then, the values of the cutout area information 13b, composite image output mode 13c, and cutout image output mode 13d in the storage section 13 are rewritten, and the operation is then terminated.

In this way, the composite image display section 30 can acquire the composite image information formed by the integration processing section 20 and display the composite image on the display section 31. Here, the display section 31 functions as a display means for displaying an image formed by the image forming means.

FIG. 9 is a flowchart for explaining the process performed by the cutting boundary information calculation unit 21, which is performed in step S200 of the flow of the computer program shown in FIG. 8, as described above.

Note that the cutout boundary information calculation unit 21 has a register for holding various information therein, and performs processing based on the value of the register. It also has, for example, a 1-bit control register for controlling operations, and starts operating when 1 is written to the control register from the CPU 25.

In steps S901 to S903, the CPU 25 determines processing based on each 2-bit image output mode, with the upper bit being the composite image output mode and the lower bit being the cutout image output mode. When both the composite image output mode and the cropped image output mode are the no-distortion correction mode, 202a is generated as cropping boundary information as shown in FIG. 3(A).

Further, when the composite image output mode and cropped image output mode are respectively a mode without distortion correction and a mode with distortion correction, 202b is generated as cropping boundary information as shown in FIG. 3(A). Further, when the composite image output mode and the cropped image output mode are respectively the distortion correction mode and the distortion correction non-mode, the posture 302a is set as the cropping boundary information as shown in FIG. 3(C). Further, when both the composite image output mode and the cropped image output mode are the distortion correction mode, 302b is generated as cropping boundary information as shown in FIG. 3(D).

That is, if both the composite image output mode and the cropped image output mode are No in step S901 and step S902, that is, the distortion correction no mode (value 00), the process advances to step S906. If the result in step S902 is Yes, that is, if the distortion correction mode is not present or the distortion correction mode is present (value 01), the process advances to step S904.

If No in step S903, that is, in the mode with distortion correction or the mode without distortion correction (value 10), the process advances to step S908. If Yes in step S903, that is, in the distortion correction mode/distortion correction mode (value 11), the process advances to step S905.

In each of steps S904 and S905, the CPU 25 performs the same transformation as the distortion correction on the barycentric coordinate CA_cp of the cutout area based on the lens characteristic information L and the cutout area information CA, so that the The barycenter coordinates CA_cp of the cutout region are formed. After the process in step S904, the process advances to step S907, and after the process in step S905, the process advances to step S909.

In each of steps S906 and S908, the CPU 25 forms a cutout area size CA_s based on the cutout area information CA, and forms cutout boundary information based on the barycenter coordinate CA_cp of the cutout area and the cutout area size CA_s. . After the processing in step S908, the process advances to step S911.

In each of steps S907 and S909, the CPU 25 forms a cutting area size CA_s based on the cutting area information CA, and forms a cutting area boundary based on the barycentric coordinates CA_cp of the cutting area after distortion correction and the cutting area size CA_s. Form information. After the processing in step S907, the process advances to step S910. In step S910, the CPU 25 performs a transformation inverse to distortion correction on each coordinate of the cutout boundary information based on the lens characteristic information L and the cutout boundary information.

On the other hand, in step S911, the CPU 25 performs the same transformation as distortion correction on each coordinate of the cutout boundary information based on the lens characteristic information L and the cutout boundary information. Further, after any one of step S906, step S910, step S911, and step S909, the CPU 25 causes each section of the cutout image output mode control section 22 and the composite image forming section 23 to hold the formed cutout boundary information. Write to register. Thereafter, the control register is set to 0 and the operation is completed.

As described above, FIG. 10 is a flowchart for explaining step S300 in the flow of FIG. 8, and is for explaining the operation of the cropped image output mode control section 22.

Note that the cutout image output mode control section 22 has a register that holds various information therein, and performs processing based on the value of the register. It also has, for example, a 1-bit control register for controlling operations, and starts operating when 1 is written to the control register from the CPU 25.

In step S1001, the CPU 25 determines whether the cutout region includes a highly distorted region based on the distortion boundary information and cutout boundary information. If the cutout area includes a high distortion area, the detection result is set to 1, and if the cutout area does not include a high distortion area, the detection result is set to 0 and held in the register.

In steps S1002 and S1003, the CPU 25 determines processing based on 2-bit mode information in which the upper bit is the detection result in step S1001 and the lower bit is the cutout image output mode. With this mode information, if the cutout area does not include a high distortion area (value 00 or value 01), the process advances to step S1004. If the cutout area includes a high distortion area and the cutout image output mode is the distortion correction no mode (value 10), the process advances to step S1007. If the cutout area includes a high distortion area and the cutout image output mode is the distortion correction mode (value 11), the process advances to step S1006.

In step S1004, the CPU 25 sets the cutout image output mode to a distortion correction no mode. After that, the process advances to step S1005.

In each of steps S1005, S1006, and S1007, the CPU 25 forms 2-bit message information according to the value of the 2-bit mode information. If the mode information does not include a high distortion area in the cutout area (value 00 or 01), in step S1005, the CPU 25 sets message 401 (no message) in FIG. 4(A) as message information. Write the corresponding value 00.

If the cropped region includes a highly distorted region and the cropped image output mode is the no-distortion correction mode (value 10), in step S1007, the CPU 25 outputs the message information shown in FIG. A value 10 corresponding to the message 402 in FIGS. 6(A) and 7(A) is written.

If the cropped region includes a highly distorted region and the cropped image output mode is the distortion correction mode (value 11), in step S1006, the CPU 25 responds to the message 403 in FIG. 7(B). Write the corresponding value 11.

After step S1005, step S1006, or step S1007, the CPU 25 writes the message information and the values of the cropped image output mode into the RAM 26, sets the control register to 0, and ends the operation.

In this way, the CPU 25 can form message information in step S105 of FIG. 8 based on the message information formed by the cutout image output mode control section 22.

FIG. 11 is a flowchart showing an example of a processing flow when changing the cutout area information in the first embodiment, in which a user's operation is detected from the touch panel, and the cutout area information CA in the RAM 26 is changed according to the user's operation. This is to explain the rewriting flow.

In step S1101, the CPU 25 determines whether an electrical signal has been input from the touch panel of the adjustment value input UI 32 or the adjustment value input UI 42. If an electrical signal has been input, the process advances to step S1102; otherwise, the process advances to step S1109.

In step S1102, the CPU 25 detects the user's operation according to the electrical signal input to either the adjustment value input UI 32 or the adjustment value input UI 42. Here, any one of a swipe operation, a pinch operation, and other electrical signals is detected.

In step S1103, the CPU 25 determines whether the input from the user is a swipe operation. If it is a swipe operation, the process advances to step S1104, and if it is not a swipe operation, the process advances to step S1106.

In step S1104, the CPU 25 calculates the coordinate displacement based on the input electric signal, adds the coordinate displacement to the barycenter coordinate CA_cp of the cutout area before the swipe operation, and adds the coordinate displacement to the barycenter coordinate CA_cp of the cutout area after the swipe operation. Calculate CA_cp. In step S1105, the CPU 25 writes the barycentric coordinate CA_cp of the cut-out region into the RAM 26. After that, the process advances to step S1109.

In step S1106, the CPU 25 determines whether the input from the user is a pinch operation. If it is a pinch operation, the process advances to step S1107, and if it is not a pinch operation, the process advances to step S1109.

In step S1107, the CPU 25 calculates the displacement of the magnification of the cutout area based on the input electric signal, adds the displacement of the magnification to the magnification CA_m of the cutout area before the pinch operation, and adds the displacement of the magnification to the magnification CA_m of the cutout area before the pinch operation. The magnification CA_m of the output area is calculated. In step S1108, the CPU 25 rewrites the magnification CA_m of the cutout area in the RAM 26. After that, the process advances to step S1109.

In step S1109, it is determined whether there is a termination request from the user. If No, the process advances to step S1101, and if Yes, the operation ends.

In this way, each of the composite image display section 30 and cutout image display section 40 inputs the user's operation on the touch panel to the integrated processing section 20, and the CPU 25 stores the cutout area information CA in the RAM 26 in accordance with the user's operation. Can be rewritten. Therefore, the cutting area can be adjusted.

FIG. 12 is a flowchart showing the processing flow when changing the image output mode in the first embodiment, in which the press of a key by the user is detected and the cutout area information CA in the RAM 26 is updated according to the key pressed by the user. This is to explain the rewriting flow.

In step S1201, the CPU 25 determines whether an electrical signal has been input from the various push buttons of the adjustment value input UI 32 or the adjustment value input UI 42. If there is an input of an electrical signal, the process advances to step S1202; otherwise, the process advances to step S1207.

In step S1202, the CPU 25 controls the rise of the electrical signal from the composite image distortion correction switching key and the cropped image distortion correction switching key according to the electrical signal input to either the adjustment value input UI 32 or the adjustment value input UI 42. Detect.

In step S1203, the CPU 25 determines whether the composite image distortion correction switching key has been pressed, based on the detection results of the rising edges of the electrical signals from the various keys. If the button has been pressed, the process advances to step S1204; if the button has not been pressed, the process advances to step S1205.

In step S1204, the CPU 25 switches the composite image output mode and writes it into the RAM 26. In step S1205, the CPU 25 determines whether the cut-out image distortion correction switching key has been pressed, based on the detection results of the rising edges of the electrical signals from the various keys. If the button has been pressed, the process advances to step S1206; if the button has not been pressed, the process advances to step S1207.

In step S1206, the CPU 25 switches the cutout image output mode and writes it into the RAM 26. In step S1207, the CPU 25 determines whether there is a termination request from the user. If No, the process returns to step S1201, and if Yes, the operation ends.

Therefore, when each of the composite image display section 30 and the cropped image display section 40 inputs a push button from the user to the integrated processing section 20, the CPU 25 selects a composite image output mode and a cropped image output mode corresponding to various keys. Each mode can be switched. In other words, by pressing various keys, it is possible to switch between the presence or absence of distortion correction for the composite image and the presence or absence of distortion correction for the cropped image.

FIG. 13 is a flowchart showing a processing flow when forming a cutout image according to the first embodiment, and describes a flow of forming cutout image information and outputting the cutout image information to the cutout image display section 40. It is something.

In step S<b>1301 , the CPU 25 uses the imaging unit 10 to take a picture, and inputs the captured image information to the cropped image forming unit 24 . In step S1302, the CPU 25 reads the values of the lens characteristic information L, the cutout area information CA, and the cutout image output mode from the RAM 26, and writes them into the register included in the cutout image forming unit 24.

In step S1303, the CPU 25 causes the cutout image forming unit 24 to determine whether the cutout image output mode is the distortion correction mode. If the cropped image output mode is the mode with distortion correction, the process advances to step S1304, and if the mode does not have distortion correction, the process advances to step S1305.

In step S1304, the CPU 25 causes the cropped image forming unit 24 to perform distortion correction on the photographed image information based on the value of the lens characteristic information L, thereby converting the photographed image information into photographed image information after distortion correction. . In step S1305, the CPU 25 functions as a cutout image forming unit, and the cutout image forming unit 24 forms cutout image information by cutting out a part of the image information from the captured image information based on the cutout area information CA. do.

Thereafter, in step S1306, the CPU 25 causes the display section 41 of the cut-out image display section 40 to output the cut-out image information. In step S1307, the CPU 25 determines whether there is a termination request from the user. If No, the process advances to step S1302, and if Yes, the operation ends. Through such a flow, the cropped image display section 40 can acquire the cropped image information formed by the integration processing section 20 and display the cropped image on the display section 41.

As described above, according to the present embodiment, by displaying a composite image in which distortion boundary information and cropping boundary information are superimposed on a photographed image, the user can can be easily adjusted. Distortion correction of the image in the cutout area can then be selectively performed as appropriate.

According to the present embodiment, the coordinate displacement is calculated by reversing the correlation between the coordinates after the swipe operation and the coordinates before the swipe operation in the adjustment value input UI 32 and the adjustment value input UI 42. Further, the adjustment value input UI 32 and the adjustment value input UI 42 calculate the displacement of the magnification of the cutout region by inverting the correlation between the distance displacement after the pinch operation and the distance displacement before the pinch operation.

By calculating as described above, when the user operates the adjustment value input UI 32 while viewing the composite image, the user can operate the adjustment value input UI 32 as if the user had a cutout area. On the other hand, when operating the adjustment value input UI 42 while looking at the cropped image, the user can operate the adjustment value input UI 42 as if he were holding a photographed image.

Further, according to the present embodiment, if the cropped region does not include a highly distorted region and there is no need to perform distortion correction on the cropped image, distortion correction is automatically performed by switching not to perform distortion correction. It is possible to save processing time and display cutout images with low delay.

<Embodiment 2>
In the first embodiment, the integrated processing unit 20 has two adjustment value input UIs, ie, adjustment value input UIs 32 and 42, respectively, through a user interface for inputting user operations to a computer mounted on a vehicle, for example. However, the number of adjustment value input UIs is not limited to two, and it is sufficient to have at least one adjustment value input UI.

Further, as in the second embodiment described below, an adjustment value input UI 50 is connected to the integrated processing section separately from the adjustment value input UI 32 of the composite image display section 30 and the adjustment value input UI 42 of the cropped image display section 40. It's good as well.

FIG. 14 is a functional block diagram showing a configuration example of an image processing system according to the second embodiment, and the adjustment value input UI 50 is, for example, a keyboard and has a plurality of keys. Specifically, the adjustment value input UI 50 uses, for example, a right direction key, a left direction key, an up direction key, a down direction key, a cutout region enlargement key, a cutout region reduction key, a composite image distortion correction switch key, and a switch key. Each has a distortion correction switching key for output images.

When the CPU 25 detects an input from the left/right/up/down keys, the CPU 25 changes the value of a constant coordinate displacement corresponding to each direction key by, for example, +1 pixel with respect to the barycentric coordinate CA_cp of the cutout region, for example, from when each direction key is pressed until it is returned. are added at regular intervals, for example, at 0.1 second intervals. Thereafter, the barycenter coordinate CA_cp of the cutout area in the RAM 26 is rewritten.

Further, when the CPU 25 detects an input from the cutout area enlargement key or the cutout area reduction key, the CPU 25 sets a fixed value to the cutout area magnification CA_m at regular intervals, for example, from the time each key is pressed until the key is pressed again. (for example, at 0.1 second intervals). Thereafter, the magnification CA_m of the cutout area in the RAM 26 is rewritten. Note that the above-mentioned constant value may be set to, for example, +0.05 for enlargement, -0.05 for reduction, etc.

As described above, according to the configuration of the second embodiment shown in FIG. 14, it is possible to realize an image processing system that detects user operations and adjusts the cutout area without configuring a touch panel in the adjustment value input UI.

<Embodiment 3>
Note that in the second embodiment, the reference size CA_bs included in the cutout area information 13b is not rewritten in the image processing system, but it may be rewritten according to the display resolution of the cutout image display section 40. . Embodiment 3 is shown in FIG. 15, in which the cropped image display section 40 has a storage section 43.

FIG. 15 is a functional block diagram showing a configuration example of an image processing system according to the third embodiment. In FIG. has. The CPU 25 reads display resolution information from the storage unit 43 and rewrites the reference size CA_bs in the RAM 26 based on the display resolution.

Further, the CPU 25 reads the reference size CA_bs from the RAM 26 and rewrites the reference size CA_bs included in the cutout area information 13b of the storage unit 13. According to the configuration of the third embodiment shown in FIG. 15, it is possible to realize an image processing system that adjusts the reference size CA_bs of the cutout area using the display resolution specific to the electronic room mirror.

In the third embodiment, the CPU 25 operates in multi-threads for each of the flows shown in FIGS. 8 and 11 to 13. However, the imaging section 10, the integrated processing section 20, the composite image display section 30, and the cropped image display section 40 each have a CPU, and each CPU executes each flow of FIGS. 8 and 11 to 13. Distributed processing may be performed by executing a computer program. In this way, by performing distributed processing in each section, the load on the CPU 25 of the integrated processing section 20 can be reduced.

Note that in the first and third embodiments described above, each of the adjustment value input UIs 32 and 42 has a push button. However, at least one area may be provided on the touch panel of either of the adjustment value input UIs 32 and 42 to represent various keys such as a push-button composite image distortion correction switch key and a cropped image distortion correction switch key. . Then, the CPU 25 may be configured to detect an electrical signal generated by touching that area as a key press.

Further, it may be configured such that both the composite image distortion correction switching key and the cropped image distortion correction switching key of the push buttons of the adjustment value input UIs 32 and 42 are included only in the composite image display section 30, and the cropped image It may also be configured to be included only in the display section 40. With such a configuration, the adjustment value input UIs 32 and 42 can be realized using only a touch panel, and the cost for push buttons can be reduced.

In the first and third embodiments described above, each of the adjustment value input UIs 32 and 42 is configured with a touch panel capable of multi-touch, and the cutout area is enlarged or reduced by detecting a pinch operation. However, the adjustment value input UI 32, 42, etc. may be configured to have various keys such as a cutout area enlargement key, a cutout area reduction key, a composite image distortion correction switch key, and a cutout image distortion correction switch key. .

Then, the CPU 25 detects the press of either the cutout area enlargement key or the cutout area reduction key, and adds a constant value having a sign corresponding to enlargement or reduction to the cutout area magnification CA_m. It's okay. With such a configuration, the adjustment value input UI 32, 42 can be configured to have a touch panel that supports only single touch and push buttons equipped with various keys, and can be used as a touch panel that is not a projected capacitive type. can also be accommodated.

<Embodiment 4>
In the first to third embodiments, in the flow shown in FIG. 10, after determining the process in step S1002 and step S1003, the process proceeds to any one of step S1004, step S1006, and step S1007. However, in the fourth embodiment, if it is determined in step S1002 that the cropped area includes a highly distorted area, the process proceeds to the step of setting the cropped image output mode to the distortion correction mode without performing step S1003. FIG. 16 shows the operation of the image processing system when this flow is adopted.

FIGS. 16A and 16B are diagrams showing display examples of the composite image display section 30 during a swipe operation in the fourth embodiment. When the cropping area indicated by the range of the cropping boundary information 202b is swiped outside the range of the low distortion area indicated by the distortion boundary information 201, the composite image 200a is automatically switched to the composite image 200b, and the cropping is performed. Automatically performs distortion correction on the image of the region. That is, when the cutout area includes a low distortion area, distortion correction of the image of the cutout area is automatically performed.

As shown in FIG. 6(B), even if you swipe the cutout area in the distortion-corrected composite image 300a and the cutout area goes outside the range of the low distortion area indicated by the distortion boundary information 301, the cutout area Distortion correction may be automatically performed on the image.

In this way, by automatically switching the presence or absence of distortion correction on the cropped image at the distortion boundary, distortion correction can be performed on the cropped image without making the user feel that the display of the cropped image changes depending on whether or not distortion correction is performed. can be switched.

Note that in this embodiment, the register for the upper bits of message information and the register for the detection result of whether or not a high distortion area is included in the cutout area are configured separately; may be configured with the same register.

Furthermore, in this embodiment, the register for the lower bits of the message information and the register for the cropped image output mode are configured separately, but the lower bits of the message information and the cropped image output mode may also be configured in the same register. good. With such a configuration, it is possible to omit the steps of forming message information according to the detection result and cutout image output mode, which correspond to steps S1004 to S1007.

In this embodiment, each of the cutout boundary information calculation unit 21, cutout image output mode control unit 22, composite image formation unit 23, and cutout image formation unit 24 as functions of the FPGA holds various information. Each register is separate. However, the same information may be configured in the same register, and each part may be configured to perform processing based on the register. With such a configuration, the circuit resources of registers within the FPGA can be reduced.

In addition, in the above embodiment, an example was explained in which the distorted boundary and the cropping boundary are always displayed superimposed on the image, but an operation switch etc. for selectively turning on/off at least one of the superimposed display of the distorted boundary and the cropping boundary is also provided. may be provided.

Further, in the above embodiment, an example was explained in which the distortion boundary or the cutout boundary is displayed superimposed on the photographed image, but instead of the photographed image, it may be superimposed on a background image consisting of CG or a grid as shown in FIG. It may also be displayed.

Incidentally, in the above-described embodiments, an example of an automobile was explained as the moving object. However, the mobile object may be any mobile device that moves, such as a car, train, ship, airplane, robot, drone, or autonomous mobile object. Note that the autonomous mobile body may be, for example, an AGV (Automatic Guided Vehicle) or an AMR (Autonomous Mobile Robot).

Furthermore, even if the vehicle does not move completely autonomously, it may be used as a driving support. Moreover, a part of the image processing system of the embodiment may or may not be mounted on those moving bodies. Furthermore, the image processing system of the present invention can be applied to remote control of a moving body.

Although the present invention has been described above in detail based on its preferred embodiments, the present invention is not limited to these specific embodiments, and the present invention may take various forms without departing from the gist of the present invention. included. Some of the embodiments described above may be combined as appropriate. Note that this embodiment includes the following combinations.

(Configuration 1) Image acquisition means for acquiring photographed image information formed by an imaging means for photographing an optical image having a low distortion region and a high distortion region, and a cutout region indicating the range of the cutout region in the photographed image information. information acquisition means for acquiring information; cutout boundary information forming means for forming cutout boundary information indicating a boundary of the cutout region based on the cutout region information; and information on the low distortion region and the high distortion region. 1. An image processing system comprising: an image forming means for forming an image in which distorted boundary information indicating a boundary of a target area and the cut-out boundary information are superimposed.

(Configuration 2) The image processing system according to Configuration 1, further comprising a display unit that displays the image formed by the image forming unit.

(Structure 3) According to structure 1 or 2, the image forming means has a cutout image forming means for forming the image by cutting out a part of the photographed image information based on the cutout area information. image processing system.

(Structure 4) The image processing system according to any one of Structures 1 to 3, further comprising an operation means for a user to change the range of the cutout area.

(Structure 5) The image processing system according to Structure 4, wherein the operation means includes a touch panel.

(Structure 6) The image processing system according to Structure 5, wherein at least one of a swipe operation and a pinch operation can be performed using the touch panel.

(Configuration 7) The image processing system according to any one of configurations 1 to 6, further comprising an operation means for a user to select distortion correction of the photographed image information.

(Structure 8) The image processing system according to any one of Structures 1 to 7, further comprising an operation means for a user to select distortion correction of the cutting boundary information.

(Structure 9) The image processing system according to any one of Structures 1 to 8, further comprising an operation means for a user to select distortion correction for the image of the cutout area.

(Structure 10) The image processing system according to any one of Structures 1 to 9, wherein the image forming means displays a message regarding whether or not the cutout area includes the low distortion area.

(Structure 11) The image processing system according to any one of Structures 1 to 10, wherein the image forming means displays a message regarding whether or not distortion correction is to be performed on the image of the cutout area.

(Structure 12) According to any one of Structures 1 to 11, the image forming means performs distortion correction on the image of the cutout area when the cutout area includes the low distortion area. image processing system.

(Structure 13) The focal length of the optical system that forms the optical image is f, the half angle of view is θ, the image height on the image plane is y, and the projection characteristic representing the relationship between the image height y and the half angle of view θ is y. (θ), y(θ) in the low distortion region is larger than f×θ and is different from the projection characteristic in the high distortion region, according to any one of configurations 1 to 12. image processing system.

(Structure 14) The low distortion area is configured so that the projection characteristic approximates a central projection method (y=f×tanθ) or an equidistant projection method (y=f×θ). The image processing system according to configuration 13.

を満足するように構成されていることを特徴とする構成１３に記載の画像処理システム。 (Configuration 15) When θmax is the maximum half angle of view that the optical system has, and A is a predetermined constant,

The image processing system according to configuration 13, wherein the image processing system is configured to satisfy the following.

(Method) An image acquisition step of acquiring captured image information formed by an imaging means that captures an optical image having a low distortion area and a high distortion area, and cutout area information indicating the range of the cutout area in the captured image information. an information acquisition step of acquiring information about the extraction region; a cutting boundary information forming step of forming cutting boundary information indicating a boundary of the cutting region based on the cutting region information; An image processing method comprising an image forming step of forming an image in which distorted boundary information indicating a boundary and the cut-out boundary information are superimposed.

(Program) A computer program for controlling each means of the image processing system according to any one of configurations 1 to 15 by a computer.

<Other embodiments>
The present invention also applies when a software program that implements the functions of the above-described embodiments is supplied directly from a recording medium or using wired/wireless communication to a system or device having a computer capable of executing the program, and the program is executed. Included in invention.

Therefore, in order to realize the functional processing of the present invention on a computer, the program code itself that is supplied and installed in the computer also realizes the present invention. In other words, the present invention also includes a computer program itself for realizing the functional processing of the present invention.

In this case, the form of the program does not matter, such as an object code, a program executed by an interpreter, or script data supplied to the OS, as long as it has the function of a program.

The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical/magnetic optical storage medium, or a nonvolatile semiconductor memory.

Further, as a method for supplying the program, a method may be considered in which a computer program forming the present invention is stored in a server on a computer network, and a connected client computer downloads and programs the computer program.

10: Imaging unit 11: Ultra-wide-angle lens 12: Imaging element 20: Integrated processing unit 21: Cutout boundary information calculation unit 22: Cutout image output mode control unit 23: Composite image formation unit 24: Cutout image formation unit

Claims (17)

an image acquisition means for acquiring captured image information formed by an imaging means for capturing an optical image having a low distortion region and a high distortion region;
Information acquisition means for acquiring cutout area information indicating a range of the cutout area in the photographed image information;
Cutting boundary information forming means for forming cutting boundary information indicating a boundary of the cutting region based on the cutting region information;
An image processing system comprising: an image forming means for forming an image in which distortion boundary information indicating a boundary between the low distortion area and the high distortion area and the cutting boundary information are superimposed. The image processing system according to claim 1, further comprising display means for displaying the image formed by the image forming means. 2. The image processing system according to claim 1, wherein the image forming means includes a cut-out image forming means for forming the image by cutting out a part of the photographed image information based on the cut-out area information. The image processing system according to claim 1, further comprising operation means for a user to change the range of the cutout area. The image processing system according to claim 4, wherein the operation means includes a touch panel. The image processing system according to claim 5, wherein the touch panel allows at least one of a swipe operation and a pinch operation. 2. The image processing system according to claim 1, further comprising operation means for a user to select distortion correction of the photographed image information. 2. The image processing system according to claim 1, further comprising operation means for a user to select distortion correction of the cutout boundary information. 2. The image processing system according to claim 1, further comprising operation means for a user to select distortion correction for the image of the cutout area. The image processing system according to claim 1, wherein the image forming means displays a message regarding whether or not the cutout area includes the low distortion area. 2. The image processing system according to claim 1, wherein the image forming means displays a message regarding whether distortion correction is to be performed on the image of the cutout area. 2. The image processing system according to claim 1, wherein the image forming means performs distortion correction on the image of the cut-out area when the cut-out area includes the low distortion area. The focal length of the optical system that forms the optical image is f, the half angle of view is θ, the image height on the image plane is y, and the projection characteristic representing the relationship between the image height y and the half angle of view θ is y(θ). and when,
2. The image processing system according to claim 1, wherein y(θ) in the low distortion region is larger than f×θ and is different from the projection characteristic in the high distortion region. 13. The low distortion area is configured so that the projection characteristic approximates a central projection method (y=f×tanθ) or an equidistant projection method (y=f×θ). The image processing system described in . When θmax is the maximum half angle of view that the optical system has, and A is a predetermined constant,

The image processing system according to claim 13, wherein the image processing system is configured to satisfy the following. an image acquisition step of acquiring captured image information formed by an imaging means that captures an optical image having a low distortion area and a high distortion area;
an information acquisition step of acquiring cutout area information indicating the range of the cutout area in the photographed image information;
a cutting boundary information forming step of forming cutting boundary information indicating a boundary of the cutting area based on the cutting area information;
An image processing method comprising: forming an image in which distortion boundary information indicating a boundary between the low distortion area and the high distortion area and the cutting boundary information are superimposed. A computer program for controlling each means of the image processing system according to any one of claims 1 to 15 by a computer.

Images