JP2013172316A - Image processor and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display a video subjected to appropriate image processing according to a user instruction of display magnification, or the like, on a display means during or immediately after imaging.SOLUTION: The image processor includes an input means for inputting a plurality of image data acquired by a plurality of imaging sections, a determination means that determines the image processing for generating output image data from at least one of the plurality of image data, based on the size ratio of the display size of an output image determined based on at least one of the plurality of image data and the display size of at least one of the plurality of image data, and a generation means for generating the output image data based on at least one of the plurality of image data, by using the processing determined by the determination means.

Description

  The present invention relates to an image processing apparatus and an image processing method for processing image data input from a plurality of imaging units.

  Currently, as a method for confirming a captured image at the time of capturing or after capturing in an image capturing apparatus, a method of displaying and confirming an image using a finder such as a so-called liquid crystal display is known.

  However, the display means used as a finder usually has a lower resolution than the resolution of the image sensor. Therefore, it is difficult to confirm an image having the same image quality as the captured image by the confirmation using the finder.

  In order to solve this problem, various techniques have been proposed. For example, by limiting the display area of the display target image, in other words, by increasing the display magnification of the display target image, displaying a part of the image and performing a display with an increased resolution for the part of the image (For example, refer to Patent Document 1).

  In addition, there is a technique for generating an image with improved image quality in the process of combining a plurality of frames of images and generating one image, and displaying the generated image (see, for example, Patent Document 2).

  Furthermore, there is a technique for changing the display resolution on the viewfinder between the imaging mode and the playback mode and increasing the resolution instead of lowering the frame rate in the playback mode (see, for example, Patent Document 3).

JP 2009-17517 A JP 2007-288235 A JP 2010-273035 A

  However, any of the above-described techniques causes a problem when an image having the same image quality as that of the captured image is confirmed on the display unit during or after imaging. For example, in the technique according to Patent Document 1, display up to the resolution of the image sensor is limited, and a high-resolution image higher than the resolution of the image sensor, such as a captured image obtained by combining a plurality of captured images, is displayed. It was impossible to do.

  In the technique according to Patent Document 2, display with a resolution higher than the resolution of the image sensor is possible. However, when applied to a moving subject, the image quality deteriorates. In addition, it is necessary to accurately measure the camera position shift between frames, and there is a problem in accuracy in super-resolution processing.

  Furthermore, with the technique according to Patent Document 3, it is possible to increase the resolution in the playback mode, but the resizing process takes time. Therefore, it has been impossible to display a high-resolution image in real time in the imaging mode.

  An image processing apparatus according to the present invention includes an input unit that inputs a plurality of image data acquired by a plurality of imaging units, and a display size of an output image that is determined based on at least one image data of the plurality of image data Determining means for determining image processing for generating output image data from at least one image data of the plurality of image data, based on a size ratio between the display size of at least one image data of the plurality of image data, and Generating means for generating the output image data based on at least one of the plurality of image data using the process determined by the determining means.

  At the time of imaging and immediately after imaging, it is possible to display on the display means video that has been subjected to appropriate image processing in accordance with a user instruction such as display magnification. As a result, it becomes possible for the user to easily confirm an image having the same image quality as the image finally obtained after imaging, and the usability is improved.

It is a figure which shows an example of the hardware block diagram of the whole apparatus in Example 1 of this invention. It is the figure which showed the example of arrangement | positioning of each imaging part in Example 1 of this invention, and the example of an angle of view of each imaging part. It is a figure which shows an example of the main operation | movement flowchart in the imaging mode of Example 1 of this invention. It is a figure which shows an example of the detailed operation | movement flowchart of the low-resolution display step in the imaging mode of Example 1 of this invention. It is a figure which shows an example of the detailed operation | movement flowchart of the through display step in the imaging mode of Example 1 of this invention. It is a figure which shows an example of the detailed operation | movement flowchart of the super-resolution display step in the imaging mode of Example 1 of this invention. It is a figure which shows an example of the detailed operation | movement flowchart of the stitch display step in the imaging mode of Example 1 of this invention. It is a figure explaining the example of determination of the stitch process application of Example 1 of this invention. It is a figure which shows an example of the main operation | movement flowchart in the reproduction | regeneration mode of Example 1 of this invention. It is a figure which shows an example of the detailed operation | movement flowchart of the low resolution display step in the reproduction | regeneration mode of Example 1 of this invention. It is a figure which shows an example of the detailed operation | movement flowchart of the through display step in the reproduction | regeneration mode of Example 1 of this invention. It is a figure which shows an example of the detailed operation | movement flowchart of the super-resolution display step in the reproduction | regeneration mode of Example 1 of this invention. It is a figure which shows an example of the detailed operation | movement flowchart of the stitch display step in the reproduction | regeneration mode of Example 1 of this invention. It is a figure which shows an example of the main operation | movement flowchart in the imaging mode of Example 2 of this invention. It is a figure which shows an example of the hardware block diagram of the whole apparatus in Example 3 of this invention. It is a figure which shows an example of the main operation | movement flowchart in the imaging mode of Example 3 of this invention.

[Example 1]
Hereinafter, Example 1 of the present invention will be described in detail with reference to the drawings. First, the operation mode used in the present embodiment will be described. In this embodiment, an example in which two modes of an imaging mode and a reproduction mode are used as operation modes will be described. Here, the imaging mode is a mode in which an image in an imaging range (angle of view) that the user intends to image using the imaging device is continuously displayed on the display unit of the imaging device. The reproduction mode is a mode in which an image immediately before being captured in the imaging mode is temporarily displayed on the display unit of the imaging device.

  FIG. 1 is a block diagram illustrating an example of the entire hardware configuration of the imaging apparatus 10 according to the present embodiment. The imaging unit includes an optical system (not shown), a sensor that is an imaging device, a sensor control unit, and the like, and generates and outputs an image data signal of the captured image. In the present embodiment, the imaging unit group 100 including the imaging units 100a and 100b captures substantially the same imaging range (view angle), and the imaging unit group 100 is collectively referred to as a standard imaging unit group. The individual imaging units 100a and 100b of the group are referred to as standard imaging units. The imaging unit 101a and the imaging unit 101b have a part of the angle of view of the standard imaging unit as an imaging range. The imaging unit group 101 is collectively referred to as a telephoto imaging unit group, and individual imaging units 101a of the telephoto imaging unit group. And 101b are referred to as a telephoto imaging unit.

  Although two image pickup units included in the two types of image pickup unit groups are described in FIG. 1, this is only an example shown for the sake of simplification of description, and a larger number (for example, 100c, 100d, ..., 101c, 101d, ...) can also be provided. Each imaging unit is assigned an ID and can be managed by an ID number. The number and configuration example of each imaging unit assumed in this embodiment will be described later with reference to FIG.

  The video capture unit 102 selects a part of the image data group acquired at the same time from the memory transfer control unit 103 that transfers the image data group acquired from each imaging unit to the input data temporary storage buffer memory 105 and the like. And a camera selection unit 104 that outputs to That is, the image data group input from each imaging unit is transferred to the input data temporary storage buffer memory 105, and a part of the image data group is selected by the camera selection unit 104 and output to the subsequent stage. The image data group transferred to the input data temporary holding buffer memory 105 is used in the reproduction mode. The image data group transferred to the input data temporary holding buffer memory 105 is transferred to the storage unit 115 and stored when it is captured by the user.

  The camera control unit 106 controls the individual imaging units of the imaging unit groups 100 and 101 in accordance with instructions from the overall control unit (CPU) 108.

  The input selection unit 107 selects whether to input the data of the input data temporary storage buffer memory 105 or the data of the camera selection unit 104 to a plurality of image processing units connected in the subsequent stage. The input is selected depending on whether the operation mode is the imaging mode or the playback mode. The input selection unit 107 includes a plurality of selection units 107a, 107b, and 107c corresponding to the plurality of image processing units connected to the subsequent stage.

  In addition to controlling the camera control unit 106 and the input selection unit 107, the overall control unit (CPU) 108 controls the entire apparatus such as the video capture unit 102, the image composition / generation units 110 to 112, the display control unit 118, and the like. Do.

  The data processing unit 109 adds tag data to the captured image data.

  The super-resolution image composition unit 110 inputs a plurality of image data, performs so-called super-resolution image processing, and generates and outputs image data having a resolution higher than the resolution of each input image data. Details of the super-resolution processing will be described later.

  The stitch image generation unit 111 inputs a plurality of image data, performs alignment, and generates and outputs one piece of image data. Details of the stitch image processing will be described later.

  The low-resolution image generation unit 112 inputs a single image, performs thinning processing, low-pass filter processing, and the like, and generates and outputs image data having a lower resolution than the input image data.

  The RAM 113 is used as a program work area. The ROM 114 stores a program and also stores position (arrangement) information and performance information of each imaging unit.

  The storage unit 115 stores captured image data groups. The communication unit 116 performs data transfer within the apparatus, network connection outside the apparatus, and data transfer with the storage device 30 existing on the network 20.

  The display buffer memory 117 stores display output image data to be displayed on the display unit 119. The display control unit 118 controls the display buffer memory 117 and the display unit 119. The display unit 119 is a display unit such as a so-called finder, such as a liquid crystal display, and is used for confirming image data during or after imaging. Usually, the resolution of the display unit 119 is lower than the images generated by the imaging units 100 and 101.

  The operation input unit 120 receives an input of an operation of the apparatus by the user by a shutter button, an operation dial, or the like. The network 20 is connected to the apparatus by a communication unit. The storage device 30 exists on the network and is used for storing captured image data.

  FIG. 2 is a diagram schematically illustrating an arrangement example of each imaging unit of the imaging apparatus in the present embodiment, and a diagram illustrating an example of an imaging range (view angle) of each imaging unit. FIG. 2A shows an example in which 25 imaging units are arranged. Four imaging units 101a to 101d are telephoto imaging units, and the other 21 imaging units are standard imaging units. FIG. 2B shows an imaging range (view angle) covered by each standard imaging unit in FIG. If the distance is a long distance, the standard imaging unit covers a substantially matching angle of view. FIG. 2 is a view of the imaging range (view angle) covered by the two telephoto imaging units 101a and 101b in FIG. Although there are imaging ranges that partially overlap at a far distance, many regions have different imaging ranges.

  FIG. 2D is a diagram illustrating the overlapping of the imaging ranges (view angles) as seen on a certain plane. An area 200 indicates an angle of view covered by the standard imaging unit 100. On the other hand, regions 201a to 201d indicate the angles of view covered by the telephoto imaging units 101a to 101d, respectively. As can be seen from FIG. 2 (d), in each telephoto imaging unit of the angle of view 200, the lower left approximately 1/4 field angle, the lower right approximately 1/4 field angle, and the upper right approximately 1/4. Covers the angle of view, approximately 1/4 of the angle of view at the upper left. Further, the four angles of view by the telephoto imaging unit are arranged so as to overlap each other little by little.

  Hereinafter, the operation of this embodiment will be described in detail. Since the processing differs depending on the image display mode before the image capture (image capture mode) and the image display mode after the image capture (reproduction mode), each mode will be described.

<1. Image display mode before imaging (imaging mode)>
FIG. 3 shows a main operation flowchart in the imaging mode. In step S301, when the image pickup apparatus is turned on in the image pickup mode or switched from another operation mode to the image pickup mode, the process proceeds to step S302.

  In step S302, display on the finder is performed using the image data generated by the low-resolution image generation unit 112. Since the resolution of the finder is generally lower than the resolution of the image sensor of the standard imaging unit, the image data having the entire imaging range covered by any standard imaging unit in the standard imaging unit group cannot be displayed as it is on the finder. Therefore, by reducing the resolution via the low-resolution image generation unit 112, image data having the entire imaging range covered by any standard imaging unit in the standard imaging unit group can be displayed on the finder. For example, consider a case where image data having a full HD resolution of 1920 × 1080 is input from the standard imaging unit when the finder has a resolution of VGA of 640 × 480. In this case, the entire image data with full HD resolution cannot be displayed on the viewfinder, and therefore, low resolution processing is performed to reduce the resolution to a resolution that can be displayed with VGA resolution. The display size of the displayed image subjected to the low resolution processing is smaller than the display size of the image with the full HD resolution. Note that the image data generated by the low-resolution image generation unit 112 is not limited to the entire imaging range covered by one imaging unit, and is a part of image data captured by one imaging unit (that is, An enlarged predetermined display area) can be generated. Details of the low resolution processing will be described later.

  Next, in step S303, when the imaging mode is ended or when the finder display is ended, the process proceeds to step S341. If not, the process proceeds to step S304 to determine whether there is an instruction to change the display area (enlarged / reduced display). The display area change instruction may be performed, for example, by turning an operation dial. If there is no change instruction, the process returns to step S302. If there is a change instruction, the process proceeds to step S305.

  In step S305, it is determined whether the instruction is an enlarged display or a reduced display. If the instruction is an enlarged display, the process proceeds to step S306, and the size (resolution) of the designated display area is determined in advance. It is determined whether or not the number is exceeded. This determination processing is performed based on, for example, a size ratio between the display size of image data output as a result of enlargement and the display size of image data captured by any standard imaging unit in the standard imaging unit group. . This size ratio can be said to be a display magnification of image data. More specifically, when the display size of the image data output as a result of enlargement indicated by the size ratio is equal to the display size that can be displayed by the finder, it is determined that the threshold value has been exceeded. That is, the threshold is set to a value that is determined to exceed the threshold when the resolution of the image data output as a result of enlargement substantially matches the resolution of the finder. This threshold value (first threshold value) can be set to have a predetermined width.

  In step S306, if the resolution of the designated display area exceeds (exceeds) a predetermined threshold value, the process proceeds to step S311. If not, the process returns to step S302.

  On the other hand, if a reduction display is instructed in step S305, the process returns to step S302.

  In step S <b> 311, the image data output from the camera selection unit 104 is used without being subjected to resolution conversion and displayed on the viewfinder. This display process is referred to as through display. Details will be described later. Note that the image data output from the camera selection unit 104 is partially extracted from the image data captured by the imaging unit in order to display an area for the specified enlarged display size (this extraction process is also image processing). Can be regarded as a kind of data).

  In step S312, when the imaging mode is ended or when the finder display is ended, the process proceeds to step S341. If not, the process proceeds to step S313 to determine whether there is an instruction to change the display area (enlarged / reduced display). If there is no change instruction, the process returns to step S311. If there is a change instruction, the process proceeds to step S314.

  In step S314, it is determined whether the instruction is an enlarged display or a reduced display. If the instruction is an enlarged display, the process proceeds to step S321. If the display is reduced, the process returns to step S302 to perform low resolution display.

  In step S321, display on the finder is performed using an image using the super-resolution image composition unit 110. In other words, when the display size of the image data output as a result of enlargement is smaller than the display size that can be displayed by the viewfinder, in other words, when the resolution of the image data output as a result of enlargement is smaller than the resolution of the viewfinder Then, super-resolution synthesis processing is performed. By using a plurality of image data obtained from a plurality of imaging units as in this embodiment, image data obtained by performing various image processing after imaging is an image obtained from image data obtained by a single imaging unit. It is assumed that the resolution is higher than the data. According to the present embodiment, it is possible to confirm how the image data with such a high resolution is captured in the imaging mode with the finder. Details of the super-resolution composition processing will be described later.

  In step S322, when the imaging mode is ended or when the finder display is ended, the process proceeds to step S341. If not, the process proceeds to step S323, where it is determined whether there is an instruction to change the display area (enlargement / reduction display). If there is no change instruction, the process returns to step S321. If there is a change instruction, the process proceeds to step S324.

  In step S324, it is determined whether the instruction is an enlarged display or a reduced display. If the instruction is an enlarged display, the process proceeds to step S325. If the instruction is a reduced display, the process proceeds to step S326.

  In step S325, whether or not the resolution of the display area according to the instructed magnification display magnification matches the resolution of image data generated from any telephoto imaging unit of the telephoto imaging unit group 101 including, for example, a telephoto lens. Determine. This determination processing is performed based on, for example, a size ratio between the display size of image data output as a result of enlargement and the display size of image data captured by any imaging unit in the standard imaging unit group. Or you may perform based on the size ratio of the display size of the image data output as a result of enlarging, and the display size of the image data imaged by the arbitrary imaging parts of the telephoto imaging part group. In any case, this size ratio can be said to be a display magnification of image data. More specifically, when the display size of the image data output as a result of enlargement indicated by the size ratio is substantially equal to the display size of the image data imaged by any imaging unit in the telephoto imaging unit group, In step S325, it is determined that they match. It is assumed that the resolution of the image data captured from the imaging unit of the telephoto imaging unit is higher than the resolution of the viewfinder. For this reason, when it corresponds in step S325, it is assumed that the imaging range (field angle) of the image data imaged from the imaging unit of the telephoto imaging unit is not the entire imaging range but a part thereof. Although “match” is described in step S325, it may not be completely matched. Further, for example, it is possible to determine that the resolutions to be matched match in step S325 when the resolutions to be judged have a matching width and the resolutions within the certain range are matched. That is, a resolution having a certain width can be set as the second threshold value.

  If it is determined that they match, the image data from the telephoto imaging unit group 101 including the telephoto lens can be used as image data to be displayed on the viewfinder. If they match, the process proceeds to step S331, and if they do not match, the process returns to step S321.

  In step S326, as in step S325, whether or not the resolution of the display area according to the specified reduction display magnification matches the resolution of a part of the image data generated from the telephoto imaging unit 101 including the telephoto lens. Determine. If they match, the process proceeds to step S331, and if they do not match, the process proceeds to step S327. That is, in the process of step S326, in the case where super-resolution display is performed by enlarging the display magnification that can be described later, the stitching process can be performed again by reducing the display magnification. Judging.

  In step S327, when the resolution of the display area according to the instructed reduced display magnification exceeds (below) a predetermined threshold value, the process returns to step S311 to perform through display. Otherwise, the process returns to step S321.

  In step S331, the image is displayed on the viewfinder using either the image using the stitch image generation unit 111 or the image output from the camera selection unit 104. In this step S331, image data obtained from the image pickup unit of the image pickup unit group 101 including the telephoto lens is used. Details will be described later.

  In step S332, when the imaging mode is ended or when the finder display is ended, the process proceeds to step S341. If not, the process proceeds to step S333 to determine whether or not there is an instruction to change the display area (enlargement / reduction display). If there is no change instruction, the process returns to step S331. If there is a change instruction, the process returns to step S321.

  In step S341, an imaging mode end process or a finder display end process is performed.

  Next, details of the operations of the low resolution display (step S302), the through display (S311), the super-resolution display (S321), and the stitch display (S331) will be described with reference to FIGS.

<Description of Low Resolution Display (Step S302)>
The low resolution display (step S321) shown in FIG. 3 will be described using the operation flowchart shown in FIG.

  In step S401, the overall control unit 108 acquires information on an area to be displayed. The area information is information in a format such as the coordinate value of the upper left corner of the quadrangular area and the lengths of the vertical and horizontal directions. That is, the area information is given by the position information and size information of the area to be displayed. The initial value when the power is turned on is given by the entire imaging range in an arbitrary imaging unit of the standard imaging unit group, and is stored in the RAM 113.

  The position information is changed, for example, by an operation such as pressing a direction button (not shown) provided in the up / down / left / right direction included in the operation input unit 120, and the information in the RAM 113 is changed. The size information is continuously changed by, for example, an operation such as turning a dial included in the operation input unit 120 or tilting the shuttle switch in either direction, and the information in the RAM 113 is changed. In step S401, the overall control unit 108 acquires area information stored in the RAM 113 in this way.

  In step S402, the overall control unit 108 generates instruction information for the camera selection unit 104, the input selection unit 107, and the low-resolution image generation unit 112 based on the area information acquired in step S401. More specifically, the overall control unit 108 provides the camera selection unit 104 with one camera ID number of the standard imaging unit, for example, the ID of the standard imaging unit 100a, and the entire area as the area information of the standard imaging unit. An instruction to set the output destination as the low-resolution image generation unit 112 is generated as instruction information. The overall control unit 108 generates information for selecting a signal from the camera selection unit 104 as instruction information for the input selection unit 107 as a signal to be output to the subsequent stage. The overall control unit 108 instructs the low-resolution image generation unit 112, for example, the ratio information and display start position coordinates of the display area with respect to the entire imaging range based on the position information and the size information stored in the RAM 113. Generate as information.

  In step S403, the overall control unit 108 transmits the instruction information to the camera selection unit 104, the input selection unit 107, and the low resolution image generation unit 112, respectively.

  In step S <b> 404, the camera selection unit 104 that has received the instruction information transfers the image of the standard imaging unit 100 a to the low resolution image generation unit 112 via the input selection unit 107.

  In step S405, the low resolution image generation unit 112 generates a low resolution image from the input image based on the instruction information. Various methods for generating a low-resolution image are known and will not be described here.

  In step S406, the generated low-resolution image is transferred to the display buffer memory 117 via the display control unit 118.

  Finally, in step S407, the display control unit 118 reads out the low resolution image from the display buffer memory 117 at an appropriate timing and transfers the low resolution image to the display unit 119 for display.

  By repeating the above operation, for example, every 1/30 seconds, a moving image is displayed on the display unit 119.

  When the position information of the display area is changed, the position of the display area is changed, and when the size information is changed, the size of the display area is changed.

<Description of Through Display (Step S311)>
Next, the through display (step S311) processing of FIG. 3 will be described using the operation flowchart shown in FIG.

  In step S501, overall control unit 108 acquires area information of an area to be displayed. This is the same operation as step S401.

  In step S502, the overall control unit 108 generates instruction information for the camera selection unit 104 based on the acquired area information. More specifically, the overall control unit 108 displays the camera ID number of one imaging unit in the standard imaging unit group, for example, the ID of the standard imaging unit 100a, and the display buffer memory 117 that outputs the output destination via the display control unit 118. Is generated as instruction information. This instruction information includes an instruction for designating an area equivalent to the resolution of the display unit based on the area information acquired in step S501 as the area information of one imaging unit in the standard imaging unit group.

  In step S <b> 503, overall control unit 108 transmits the instruction information to camera selection unit 104.

  In step S504, the camera selection unit 104 that has received the instruction information transfers the image data of the standard imaging unit 100a to the display buffer memory 117 via the display control unit 118. The image data transferred in step S504 is data obtained by partially extracting an area corresponding to the acquired area information from the image data.

  Finally, in step S505, the display control unit 118 reads out the low resolution image from the display buffer memory 117 at an appropriate timing and transfers the low resolution image to the display unit 119 for display.

  Similar to the low-resolution display, the moving image is displayed by repeating the above operation and the display is changed by changing the region information.

<Description of Super-Resolution Display (Step S321)>
First, the super-resolution image composition processing will be briefly described. The process of generating an output image with a resolution higher than the resolution of the input image is generally called up-conversion or super-resolution image processing. Several methods are known for super-resolution image processing. Roughly speaking, in-frame processing that performs processing such as contour correction and dot noise elimination on one piece of image data (frame), This is divided into inter-frame processing for processing a plurality of image data (frames).

  In the inter-frame processing, there are a case where a plurality of frames taken in the time axis direction are used and a case where a plurality of frames taken simultaneously with a plurality of cameras are used. This embodiment assumes the case where a plurality of cameras are used.

  Next, the super-resolution display (step S321) of FIG. 3 will be described using the operation flowchart shown in FIG.

  In step S601, overall control unit 108 acquires area information of an area to be displayed. This is the same operation as step S401.

  In step S <b> 602, the overall control unit 108 acquires information on the number of cameras selected from the instruction information to the camera selection unit 104 based on the acquired area information. As described above, the inter-frame processing assumed in this embodiment can obtain higher image quality than in-frame processing, but requires a large amount of computation and a large amount of memory. Become. In order to realize such processing in real time, a great amount of computing power is required. On the other hand, the smaller the display area, in other words, the higher the display magnification, the smaller the number of pixels required for super-resolution image composition processing, so the number of frames that can be processed within a certain time (1/30 second) increases. To go. From this, in step S602, it is possible to acquire the determined number of selected cameras based on the display area size information among the area information acquired in step S601.

  In step S <b> 603, the overall control unit 108 acquires arrangement information and performance information of each imaging unit from the ROM 114.

  Next, in step S604, the overall control unit 108 determines which imaging unit to use the imaging data captured from the information obtained in steps S602 and S603.

  In step S605, the overall control unit 108 generates instruction information for the camera selection unit 104, the input selection unit 107, and the super-resolution image composition unit 110.

  More specifically, for the camera selection unit 104, a plurality of camera ID numbers selected from the standard imaging units, region information for the plurality of standard imaging units necessary for generating an output image, and output An instruction to set the destination as the super-resolution image composition unit 110 is generated as instruction information. That is, the overall control unit 108 generates an instruction as to which image pickup unit to use image data from to the camera selection unit 104. Then, an area corresponding to the area information in the image data is designated. As a result, only the data portion necessary for display on the finder is transmitted to the super-resolution synthesis unit, and real-time super-resolution processing is performed without applying a load.

  The overall control unit 108 generates information for selecting a signal from the camera selection unit 104 as instruction information for the input selection unit 107 as a signal to be output to the subsequent stage.

  For the super-resolution image composition unit 110, the overall control unit 108 generates, for example, the arrangement information of the selected standard imaging unit and the size information of the image output from the camera selection unit 104 as instruction information.

  In step S606, overall control unit 108 transmits the instruction information to camera selection unit 104, input selection unit 107, and super-resolution image synthesis unit 110, respectively.

  In step S <b> 607, the camera selection unit 104 that has received the instruction information passes through the input selection unit 107 data corresponding to the designated area from among the image data captured by the plurality of standard imaging units, and the super-resolution image. Transfer to the combining unit 110.

  In step S608, the super-resolution image composition unit 110 generates super-resolution image data from the input image data based on the instruction information. As described above, in the present embodiment, the super-resolution processing is performed using a plurality of images taken at the same time. The super-resolution image composition process using a plurality of images assumed in the present embodiment includes the following two processes. That is, alignment processing and image composition processing.

  The alignment process is a process of selecting one reference image from a plurality of images and calculating the relative movement amount between the other image and the selected image. In the image composition process, an initial high-resolution image is generated from one image selected from the plurality of images (preferably the reference image) by an interpolation process or the like. Then, the initial high-resolution image is updated from the other images not selected and the relative movement amount calculated by the alignment process, thereby obtaining a high-resolution image. The above is an example of the super-resolution processing, but the super-resolution processing may be performed using other various methods.

  In step S 609, the generated super-resolution image is transferred to the display buffer memory 117 via the display control unit 118.

  Finally, in step S610, the display control unit 118 reads out the super-resolution image from the display buffer memory 117 at an appropriate timing and transfers it to the display unit 119, thereby displaying.

  Similar to the low-resolution display, the moving image is displayed by repeating the above operation and the display is changed by changing the region information.

  In the above processing, the example in which the camera selection unit extracts image data necessary for the super-resolution processing has been described. However, image data from a plurality of imaging units may be transferred as they are from the camera selection unit to the super-resolution image generation unit, and a necessary area may be extracted in the super-resolution generation unit.

<Description of Stitch Display (Step S331)>
Next, the process of the stitch display (step S331) in FIG. 3 will be described using the operation flowchart shown in FIG.

  In step S701, the overall control unit 108 acquires area information of an area to be displayed. This is the same operation as step S401.

  In step S <b> 702, the overall control unit 108 determines whether to apply stitch processing based on the acquired area information. A more specific determination method is shown in FIG.

  FIG. 8 is a diagram illustrating an outline of a determination example of stitch processing. When a region to be displayed such as the region 801 is covered (overlapped) by a plurality of imaging units of the telephoto imaging unit group, stitch processing is applied, and a single telephoto imaging unit group such as the region 802 is applied. It is determined that the stitch process is not applied. If it is determined in step S702 that the stitch process is to be applied, the process proceeds to step S703.

  In step S703, the overall control unit 108 generates instruction information for the camera selection unit 104, the input selection unit 107, and the stitch image generation unit 111 based on the acquired area information. More specifically, for the camera selection unit 104, the camera ID number of the telephoto imaging unit that covers the area to be displayed, the region information for each telephoto imaging unit, and the output destination is the stitch image generation unit 111. An instruction is generated as instruction information. Here, the instruction information for the camera selection unit includes an instruction for designating a region equivalent to the resolution of the display unit based on the region information acquired in step S701 as the region information of the telephoto imaging unit.

  The overall control unit 108 uses the information for selecting the signal from the camera selection unit 104 as the instruction information for the input selection unit 107 as a signal to be output to the subsequent stage.

  For the stitch image generation unit 111, the overall control unit 108 uses, for example, the arrangement information of the selected telephoto imaging unit and the size information for each of a plurality of pieces of image data output from the camera selection unit 104 as instruction information.

  In step S704, the overall control unit 108 transmits the instruction information generated in step S703 to the camera selection unit 104, the input selection unit 107, and the stitch image generation unit 111, respectively.

  In step S <b> 705, the camera selection unit 104 that has received the instruction transfers the image data captured by the plurality of imaging units in the telephoto imaging unit group to the stitch image generation unit 111 via the input selection unit 107. Note that the image data transferred from the camera selection unit 104 may be data obtained by extracting a necessary area, as described in the super-resolution display section, or all of the images captured by a plurality of imaging units. It may be data.

  In step S706, the stitch image generation unit 111 performs stitch processing from the input image data based on the instruction information to generate stitch image data. Here, the stitching process is a process of generating a single image by performing a positioning process and a synthesis process from a plurality of images captured by shifting positions. Here, the alignment process is a process of calculating a relative movement amount for matching corresponding pixels in the overlap region in a plurality of images having the overlap region. Further, the composition processing is processing for generating a single image by joining a plurality of images including non-overlapping regions based on the alignment processing result.

  In step S 707, the generated stitch image data is transferred to the display buffer memory 117 via the display control unit 118.

  Finally, in step S708, the display control unit 118 reads the stitch image data from the display buffer memory 117 at an appropriate timing and transfers the stitch image data to the display unit 119 for display.

  On the other hand, if it is determined in step S702 that the stitch process is not applied, the process proceeds to step S709.

  In step S709, the overall control unit 108 calculates instruction information to the camera selection unit 104 based on the area information acquired in step S701.

  More specifically, the camera ID number of the single imaging unit in the telephoto imaging unit group that covers the area to be displayed, the necessary area information, and the output buffer memory via the display control unit 118 An instruction 117 is generated as instruction information. This instruction information includes an instruction to designate an area equivalent to the resolution of the display unit based on the area information acquired in step S701 as the area information of one imaging unit of the telephoto imaging unit group.

  In step S710, the overall control unit 108 transmits the instruction information to the camera selection unit 104, and proceeds to step S707. The operations after step S707 are as described above. Similar to the low-resolution display, the moving image is displayed by repeating the above operation and the display is changed by changing the region information.

  This completes the description of the image display mode (imaging mode) before shooting. Next, an image display mode (reproduction mode) after imaging will be described.

<2. Image display mode after shooting (playback mode)>
FIG. 9 shows a main operation flowchart in the reproduction mode. Note that description of parts common to the imaging mode of FIG. 3 will be omitted, and description will be given below centering on parts different from the imaging mode.

  In the imaging mode processing described with reference to FIG. 3, the image output from the camera selection unit 104 is used to generate display image data to be stored in the display buffer memory 117 of FIG. 1. On the other hand, in the reproduction mode, in order to generate display image data to be stored in the display buffer memory 117, an image transferred from the input data temporary storage buffer memory 105 is selected and input to each image processing unit. Alternatively, the overall control unit 108 controls to transfer the image transferred from the input data temporary holding buffer memory 105 directly to the display buffer memory 117. Note that the image data from each imaging unit stored in the input data temporary storage buffer memory 105 can be stored in association with each other.

  In step S901, if the power is turned on in the playback mode or the mode is switched from the other mode to the playback mode, the process proceeds to step S902.

  In step S902, the image transferred from the input data temporary storage buffer memory 105 is processed using the low-resolution image generation unit 112 and displayed on the viewfinder. Details of the processing in step S902 will be described later.

  If the playback mode is to be ended in step S903, the process proceeds to step S941 to perform a playback mode end process. If not, the process proceeds to step S304.

  The determination process in step S304 is the same as that in FIG. If there is no instruction to change the display area, the process proceeds to step S904. If there is a change instruction, the process proceeds to step S305.

  In step S904, if there is an instruction to switch the image to be reproduced and displayed, the process proceeds to step S905. If there is no instruction to switch, the process returns to step S902. In step S905, an image switching process is performed. The operations in step S305 and subsequent steps are the same as those in the imaging mode, and will be omitted.

  In step S911, the image transferred from the input data temporary holding buffer memory 105 is used as it is and displayed on the viewfinder. Details of the processing in step S911 will be described later.

  If the playback mode is to be ended in step S912, the process proceeds to step S941, and a playback mode end process is performed. If not, the process proceeds to step S313. Since the following operations are the same as those in the imaging mode, the description is omitted.

  Next, in step S921, display on the finder is performed using an image using the super-resolution image composition unit 110. Details of the processing in step S921 will be described later. If the playback mode is to be ended in step S922, the process proceeds to step S941, and a playback mode end process is performed. If not, the process proceeds to step S323. Since the following operations are the same as those in the imaging mode, the description is omitted.

  In step S931, the image is displayed on the finder using either the image using the stitch image generation unit 111 or the image transferred from the input data temporary storage buffer memory 105. Details of the processing in step S931 will be described later. If the playback mode is to be ended in step S932, the process proceeds to step S941, and the playback mode end processing is performed. If not, the process proceeds to step S333. Since the following operations are the same as those in the imaging mode, the description is omitted.

  Next, details of operations in the low-resolution display (step S902), the through display (S911), the super-resolution display (S921), and the stitch display (S931) will be described with reference to FIGS. In the description of FIGS. 10 to 13, the description of the parts common to the imaging mode is omitted, and the description is focused on the different parts.

<Description of Low Resolution Display (Step S902)>
The low resolution display (step S902) in FIG. 9 will be described with reference to the operation flowchart shown in FIG.

  In step S1001, the overall control unit 108 generates instruction information to the memory transfer control unit 103 and the low resolution image generation unit 112 based on the area information acquired in step S401. That is, it is different from the imaging mode in that instruction information for the memory transfer control unit 103 instead of the camera selection unit 104 is generated. More specifically, the following instruction information is generated for the memory transfer control unit 103. In other words, the first address of the memory address where one image data of the image pickup unit of the standard image pickup unit group in the last imaged image data is stored, and the transfer amount and transfer destination for transferring the entire image data are reduced. An instruction for the resolution image generation unit 112 is generated as instruction information.

  For the input selection unit 107, the overall control unit 108 generates information for selecting a signal from the memory transfer control unit 103 as instruction information as a signal to be output to the subsequent stage.

  The instruction to the low resolution image generation unit 112 is the same as in the imaging mode.

  In step S1002, the overall control unit 108 transmits the instruction information to the memory transfer control unit 103, the input selection unit 107, and the low resolution image generation unit 112.

  In step S <b> 1003, the memory transfer control unit 103 that has received the instruction information from the overall control unit 108 reads out data from the input data temporary storage buffer memory 105, and passes through the input selection unit 107 to the low resolution image generation unit 112. Forward to.

  The following operations are the same as those in FIG.

<Description of Through Display (Step S911)>
The through display (step S911) of FIG. 9 will be described using the operation flowchart shown in FIG.

  In step S1101, the overall control unit 108 generates instruction information for the memory transfer control unit 103 based on the area information acquired in step S501.

  More specifically, it is calculated from the transfer amount for transferring an area equivalent to the resolution of the display unit in the image data displayed in step S902, and the transfer amount and the current transfer memory address start address. A new transfer memory address head address is generated as instruction information. Further, the instruction information includes instructing the display buffer memory 117 via the display control unit 118 as a transfer destination.

  In step S 1102, the overall control unit 108 transmits the instruction information to the memory transfer control unit 103.

  The following operations are the same as those in FIG.

<Description of Super-Resolution Display (Step S921)>
The super-resolution display (step S921) in FIG. 9 will be described using the operation flowchart shown in FIG.

  Since the operations from S601 to S604 are the same as those in FIG. In step S1201, the overall control unit 108 generates instruction information to the memory transfer control unit 103, the input selection unit 107, and the super-resolution image composition unit 110 based on the area information acquired up to step S604.

  More specifically, the memory transfer control unit 103 is instructed to set the transfer memory address head address, transfer amount, and transfer destination corresponding to the selected standard imaging unit region to the super-resolution image combining unit 110. Is generated as instruction information.

  For the input selection unit 107, information for selecting a signal from the memory transfer control unit 103 is generated as instruction information as a signal to be output to the subsequent stage.

  For the super-resolution image composition unit 110, for example, the arrangement information of the selected standard imaging unit and the size information of the image output from the memory transfer control unit 103 are generated as instruction information.

  In step S <b> 1202, overall control unit 108 transmits the instruction information to memory transfer control unit 103, input selection unit 107, and super-resolution image composition unit 110.

The operations after S607 are the same as those in FIG.
<Description of Stitch Display (Step S931)>
The stitch display (step S931) in FIG. 9 will be described with reference to the operation flowchart shown in FIG.

  The operations in S701 and S702 are the same as those in FIG. In step S1301, the overall control unit 108 generates instruction information to the memory transfer control unit 103, the input selection unit 107, and the stitch image generation unit 111 based on the information acquired in step S701.

  More specifically, the memory transfer control unit 103 is instructed to set the transfer memory address start address, transfer amount, and transfer destination to the stitch image generation unit 111 corresponding to the selected areas of the plurality of telephoto imaging units. Generated as instruction information.

  For the input selection unit 107, information for selecting a signal from the memory transfer control unit 103 is generated as instruction information as a signal to be output to the subsequent stage.

  For the stitch image generation unit 111, for example, the arrangement information of the selected telephoto imaging unit and the size information for each of a plurality of images transferred from the memory transfer control unit 103 are generated as instruction information.

  In step S1302, the instruction information is transmitted to the memory transfer control unit 103, the input selection unit 107, and the stitch image generation unit 111, respectively.

  In step S1303, the overall control unit 108 generates instruction information for the memory transfer control unit 103 based on the information acquired in step S701.

  More specifically, an instruction to set the start address of the transfer memory address corresponding to the area of the selected telephoto imaging unit, the transfer amount, and the transfer destination to the display buffer memory 117 via the display control unit 118 is generated as instruction information. .

  In step S 1304, the instruction information is transmitted to the memory transfer control unit 103. The operations after S705 are the same as those in FIG.

  With the operation described above, it is possible to display an image subjected to appropriate image processing in accordance with a user instruction on the display unit at the time of imaging and immediately after imaging. As a result, it becomes possible for the user to easily check an image having the same image quality as the image finally obtained after imaging on the display unit of the imaging apparatus, and usability is improved. Further, by using images picked up almost simultaneously from a plurality of image pickup units, the accuracy in the super-resolution processing can be maintained without being deteriorated in image quality when applied to a moving subject. When super-resolution processing is performed, a super-resolution image can be displayed in real time by using only necessary image data.

  In addition, about the number and arrangement | positioning of a some imaging part, it is not limited to a present Example, Various examples can be assumed.

  Furthermore, in this embodiment, when a plurality of image processing units (110, 111, 112) are used, one of them is used. For example, the output of the stitch image generation unit 111 is output as a low-resolution image generation unit. It is also possible to use an image that is input to 112 and generated. For example, by inputting the image data obtained by the imaging unit of the telephoto imaging unit group to the low resolution image generating unit 112, the image of the entire imaging range (view angle) of the imaging unit of the telephoto imaging unit group is displayed with low resolution. It is also possible to display on the viewfinder.

[Example 2]
In the first embodiment, the size information of the area information is assumed to be continuously changed by an operation such as turning a dial or depressing a shuttle switch. It is also possible to change to

  Further, in the first embodiment, it is assumed that the position information is continuously changed by the operation of the direction button. However, this is a display unit including a touch panel, for example, and the position information is discrete by touch operation on the touch panel. It is also possible to change to

  Hereinafter, an example in which the size information and the position information included in the region information are changed discretely will be described. In addition, description is abbreviate | omitted about the matter which is common in Example 1, and it demonstrates centering on a different location from Example 1. FIG.

  FIG. 14 shows a main operation flowchart of the imaging mode in the present embodiment.

  In step S1402, it is determined whether there is an instruction for enlarged display. As described above, in the present embodiment, the instruction is given by pressing the enlargement display button. If there is an enlarged display instruction, the process proceeds to step S311; otherwise, the process returns to step S1401. Details of steps S1401 and S1411 will be described later.

In step S1412, it is determined whether there is an instruction for enlarged display or reduced display.
If it is an enlarged display instruction, the process proceeds to step S1421. If it is a reduction display instruction, the process returns to step S1401. Details of step S1421 will be described later.

  In step S1422, it is determined whether there is an instruction for enlargement display or reduction display. If it is an enlarged display instruction, the process proceeds to step S1431. If it is a reduction display instruction, the process returns to step S1411. Details of step S1431 will be described later.

  In step S1432, it is determined whether there is a reduction display instruction. If it is a reduction display instruction, the process returns to step S1421. If there is no instruction, the process returns to step S1431. Details of step S1431 will be described later.

  Next, the low resolution display (step S1401), the through display (S1411), the super resolution display (S1421), and the stitch display (S1431) will be described. Note that various flowcharts similar to those in FIGS. 4 to 7 can be used, and are not shown.

  The difference from the first embodiment is a step of acquiring region information. As described in the first embodiment, the area information is given in the form of the coordinate value of the upper left corner of the quadrangular area and the length of each of the vertical and horizontal directions, that is, position information and size information.

  In the second embodiment, the position information is changed by pointing one point on the touch panel included in the operation input unit 120 to the center of a desired area with a finger or a touch pen, and the touched coordinates are used as the center position information. It is changed and stored in the RAM 113.

  In the second embodiment, the size information is given in advance by low resolution display, through display, super-resolution display, and stitch display.

  The operation in the reproduction mode is also changed from the above-described embodiment in accordance with the above, but the description is not repeated here because the same description is repeated.

  By the above operation, it is possible to quickly display an image whose display magnification has been changed by pressing a button on the display unit. Therefore, an image having the same image quality as the image finally obtained for the user after imaging is obtained from the above embodiment. Can be easily confirmed, and has the effect of improving usability.

  In this embodiment, when the enlarged display is instructed by pressing the button, the low resolution display, the through display, the super resolution display, and the stitch display are sequentially switched.

  However, the present invention is not limited to this, and other configurations such as switching only low-resolution display and super-resolution display or switching only low-resolution display and stitch display are also conceivable.

[Example 3]
In the first and second embodiments, only one display unit is provided, but a configuration including a plurality of display units is also conceivable.

  FIG. 15 shows an example of a hardware configuration block diagram in the present embodiment.

  15 differs from the configuration of FIG. 1 in the configuration of display buffer memories 117a and 117b, display control units 118a and 118b, and display units 119a and 119b.

  In this embodiment, 119a is a main display section (main finder), and 119b is a sub display section (sub finder).

  The main operation flowchart for the main finder is equivalent to FIG. 3 in the first embodiment or FIG. 14 in the second embodiment.

  FIG. 20 shows a main operation flowchart for the sub-finder in the imaging mode in this embodiment. In the display on the sub finder, when there is a display instruction on the sub finder in step S2001, sub finder control is performed in step S2002. The operation in step S2002 is the same as that in FIG. 3 in the first embodiment or FIG. 14 in the second embodiment, like the main finder. That is, the control of the main / sub-finder operates simultaneously. The same applies to the playback mode.

  The above operation makes it possible to check multiple resolution images at a time. For example, while checking the angle of view with a low-resolution image, it is possible to check the image quality with a super-resolution image. In any case, it has the effect of further improving usability.

  In the present embodiment, an example of a configuration including a plurality of display units has been described. However, a configuration in which a plurality of screen displays are performed, such as a parent-child screen display for one display unit, is also conceivable.

<Other embodiments>
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (11)

Input means for inputting a plurality of image data acquired by a plurality of imaging units;
Based on the size ratio between the display size of the output image determined based on at least one image data of the plurality of image data and the display size of at least one image data of the plurality of image data, the plurality of image data Determining means for determining image processing for generating output image data from at least one image data;
Generating means for generating the output image data based on at least one image data of the plurality of image data using the processing determined by the determining means;
An image processing apparatus.   The image processing apparatus according to claim 1, wherein the determining unit selects one image process from an image process including an alignment process and an image process not including the alignment process.   The determining means determines image processing for generating the output image data based on position information of the output image determined based on at least one image data of the plurality of image data. The image processing apparatus according to claim 2.   The determining means determines image processing including the alignment processing when the size ratio indicates image processing for generating image data having a resolution higher than the resolution indicated by the at least one image data. The image processing apparatus according to claim 2 or 3. The determining means includes
When the size ratio does not exceed the first threshold value, low resolution processing that generates image data having a resolution lower than the resolution indicated by the at least one image data is image processing that does not include the alignment processing. Decide
When the size ratio is equal to the first threshold value, a process for generating image data having a resolution indicated by the at least one image data is determined as an image process not including the alignment process,
When the size ratio exceeds the first threshold value, super-resolution that generates image data having a resolution higher than the resolution indicated by the at least one image data using at least two of the plurality of image data The process is determined as an image process including the alignment process,
When the size ratio is equal to a second threshold value that is greater than the first threshold value, an image different from the image pickup unit used for imaging image data determined to be equal to the first threshold value. The image processing apparatus according to claim 3, wherein a process for generating image data using image data acquired by an imaging unit having a corner is determined as the image process. When the size ratio is equal to the second threshold value, the image processing apparatus further includes a determination unit that determines, based on the position information, whether the output image overlaps an area of image data input from a plurality of imaging units. ,
The determination unit is acquired by a plurality of imaging units having a different angle of view from the imaging unit used for imaging the image data determined to be equal to the first threshold value when it is determined by the determination unit to overlap. 6. The image processing apparatus according to claim 5, wherein a process of generating image data using the processed image data is determined as an image process including the alignment process.   The image processing apparatus according to claim 5, wherein the determining unit determines the number of imaging units used for the super-resolution processing based on the size ratio.   The input unit selects whether to input the image data from the plurality of imaging units or to input from a holding unit that temporarily holds the image data from the plurality of imaging units according to an operation mode. An image processing apparatus according to claim 1, wherein: The determining means determines at least two image processes for generating output image data from at least one image data of the plurality of image data;
The generation unit generates at least two output image data based on at least one image data of the plurality of image data using at least two image processes determined by the determination unit. The image processing apparatus according to claim 1. An input step of inputting a plurality of image data acquired by a plurality of imaging units;
Based on the size ratio between the display size of the output image determined based on at least one image data of the plurality of image data and the display size of at least one image data of the plurality of image data, the plurality of image data A determining step for determining image processing for generating output image data from at least one image data;
A generation step of generating the output image data based on at least one image data of the plurality of image data using the processing determined in the determination step;
An image processing method.   A program for causing a computer to function as the image processing apparatus according to claim 1.

Images