JP2014003441A - Imaging device, imaging method, and imaging program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily perform photographing at the same viewpoint and in the same direction as the previous photographing.SOLUTION: An imaging unit photographs a subject and creates a photographed image. A correction processing unit corrects a recording image stored in advance in a storage device or the photographed image. A feature point automatic adjustment unit determines a correction parameter used by the correction processing unit during correction. The feature point automatic adjustment unit determines the correction parameter such that a viewpoint, a direction, and an angle of view corrected by the correction processing unit coincide substantially with a viewpoint, a direction, and an angle of view not corrected by the correction processing unit in the recording image and the photographed image.

Description

  The present invention relates to an imaging device, a photographing method, and a photographing program.

There is an image pickup apparatus that displays grid lines so that shooting can be performed in the same viewpoint and in the same direction as when the previous shooting was performed. In addition, there are some which extract feature points from previously taken images and display the feature points so that the user can easily align the feature points. In addition, there is a function that makes it easy to align the image by displaying the image taken before and the image to be taken at the time of photographing. A feature point is a point where brightness values such as straight lines, corners (corners), and points in an image are greatly different from other values. In addition, a specific curve and a virtual curve connecting a plurality of points or points may be used as feature points.
In addition, there is an invention of an imaging apparatus including means for bringing photographing conditions such as a photographing position and a current position, ISO (International Organization for Standardization) sensitivity, and shutter speed close to the same conditions (see Patent Document 1).

JP 2005-26872 A

  However, in the image pickup apparatus described in Patent Document 1, it may take time depending on the ability of the user and the shooting environment when performing positioning so that shooting is performed in the same viewpoint and in the same direction as before. There was a problem. For example, if a long time such as one month or one year has passed since the previous shooting, or if the user is different, the shooting location may not be known accurately, so it takes time to align. End up.

  The present invention has been made in view of the above points, and provides an imaging apparatus, an imaging method, and an imaging program that can easily perform imaging in the same viewpoint and in the same direction as when previously captured.

  (1) The present invention has been made to solve the above-described problems, and one aspect of the present invention is an imaging device, which captures an object and generates a captured image, and a storage device in advance. A correction processing unit that corrects the recorded image or the captured image stored in the memory, and a feature point automatic adjustment unit that determines a correction parameter used when the correction processing unit corrects the feature image, and the feature point automatic adjustment The viewpoint, direction, and angle of view of the image corrected by the correction processing section substantially coincide with the viewpoint, direction, and angle of view of the recorded image and the captured image that are not corrected by the correction processing section. The correction parameter is determined as follows.

  (2) Moreover, one aspect of the present invention is the imaging device according to (1), in which a first feature point extraction unit that extracts a first feature point of an image to be corrected by the correction processing unit; A second feature point extracting unit that extracts a second feature point of the image that is not corrected by the correction processing unit, wherein the parameter determining unit is corrected by the correction processing unit among the information on the second feature point The correction parameter is determined so that the number of pieces of information relating to the first feature point in the image that coincide with the information relating to the first feature point corresponding to the second feature point is maximized. .

  (3) According to another aspect of the present invention, the image pickup apparatus according to (2) can record an image based on the photographed image when the number of the matching items is equal to or greater than a preset threshold value. An imaging determination unit is provided.

  (4) According to another aspect of the present invention, the image pickup apparatus according to (2) can record an image based on the photographed image when the number of the matching items is equal to or greater than a preset threshold value. An imaging determination unit is provided.

  (5) Moreover, one aspect of the present invention is the imaging apparatus according to (1), in which the correction parameters are an angle of view, a rotation angle, and up / down / left / right positions. A distance between the feature point and another feature point, a slope of a straight line passing through the feature point and the other feature point, a coordinate of the feature point, and the first feature point corresponding to the second feature point The number of objects having the same distance, the number of objects having the same slope of the straight line passing through the first feature point corresponding to the second feature point, the coordinates of the first feature point corresponding to the coordinates of the second feature point, and The correction parameter is determined such that the number of the two that coincide with each other is maximized.

  (6) One embodiment of the present invention is an imaging method, the first step of capturing a subject and generating a captured image, and correcting a recorded image or the captured image stored in advance in a storage device A second step of determining a correction parameter to be used for correction, and a viewpoint, direction, and angle of view of the corrected image of the image that is not corrected among the recorded image and the captured image. And a fourth step of determining the correction parameter so as to substantially match the viewpoint, direction, and angle of view.

  (7) One embodiment of the present invention is an imaging program that includes a first unit that captures a subject and generates a captured image, and corrects a recorded image or the captured image previously stored in a storage device. Second means for determining, third means for determining a correction parameter to be used for correction, and the viewpoint, direction, and angle of view of the corrected image of the recorded image and the photographed image that are not corrected. A fourth means for determining the correction parameter so as to substantially coincide with the viewpoint, direction, and angle of view is executed.

  According to the present invention, it is possible to easily perform photographing in the same viewpoint and in the same direction as when photographing previously.

1 is a schematic block diagram illustrating a configuration of an imaging apparatus according to a first embodiment of the present invention. It is a schematic block diagram which shows the structure of the control part which concerns on this embodiment. It is a schematic block diagram which shows the structure of the feature point automatic adjustment part which concerns on this embodiment. It is a conceptual diagram which shows an example of the wide angle imaging | photography of the imaging device which concerns on this embodiment. It is a conceptual diagram which shows an example of the method of the correction process of the picked-up image which concerns on this embodiment. It is a conceptual image which shows an example which extracted from the recorded image which concerns on this embodiment, and displayed the coordinate of the feature point. It is a conceptual image which shows an example of the picked-up image when a user aligns and image | photographs based on the coordinate of the feature point extracted from the recorded image which concerns on this embodiment. It is a conceptual image which shows an example which compared the coordinate of the feature point extracted from the recording image which concerns on this embodiment, and the coordinate of the feature point of the picked-up image image | photographed and registered by the user based on it. It is a flowchart which shows an example of the operation | movement in the feature point automatic adjustment process of the picked-up image which concerns on this embodiment. It is a flowchart which shows an example of operation | movement of the feature point automatic adjustment part which concerns on this embodiment. It is a flowchart which shows an example of operation | movement of the imaging determination part in the feature point automatic adjustment process which concerns on this embodiment.

(First embodiment)
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a schematic block diagram showing the configuration of the imaging device D1 according to the first embodiment of the present invention.
The imaging device D1 includes an imaging unit 1, a signal conversion unit 2, a signal processing unit 3, a dimming sensor unit 4, an orientation sensor unit 5, a control unit C1, a flash light emitting unit 6, a display device 7, a DRAM 8, a flash memory 9, and an operation The unit 10 and the position sensor unit 11 are configured. In addition, the imaging device D1 has other generally known functions of the imaging device.

  The imaging unit 1 includes an imaging element such as a photographing lens 1a and a CCD (Charge Coupled Device), an automatic aperture mechanism, and the like. The light flux of the subject condensed through the photographing lens 1a is reflected on the imaging element (CCD). To form an image. The sensitivity of the CCD is determined by the ISO sensitivity setting value. The CCD converts an image formed through the taking lens 1a into an electric signal. The CCD outputs the converted electrical signal to the signal converter 2.

  The signal conversion unit 2 includes a signal conversion circuit such as an A / D converter. The signal conversion unit 2 converts the electrical signal into digital data based on the electrical signal input from the imaging unit 1. The signal conversion unit 2 outputs the converted digital data to the signal processing unit 3.

  The signal processing unit 3 temporarily holds the digital data input from the signal conversion unit 2 in the form of Bayer data corresponding to the pixels of the CCD, and complements the Bayer data to create RGB (Red Green Blue) data. The signal processing unit 3 obtains signal components such as luminance and color difference multiplexed signals (Y, Cb, Cr data) from the RGB data. The signal processing unit 3 outputs the Y, Cb, Cr data (hereinafter referred to as image data) to a DRAM (Dynamic Random Access Memory) 8 via the control unit C1, and temporarily records the data.

  The light control sensor unit 4 measures the amount of light around the subject and reflects the measurement result on the automatic aperture mechanism of the imaging unit 1. The azimuth sensor unit 5 measures which direction (azimuth) the photographic lens 1a is facing, and outputs azimuth information indicating the measured azimuth to the control unit C1.

The control unit C1 includes a CPU, a RAM (Random Access Memory), a program storage ROM (Read Only Memory), and a microprocessor having a timing mechanism such as a timer.
The control unit C1 includes the image data input from the signal processing unit 3, the dimming measurement result input from the dimming sensor unit 4, the azimuth information input from the azimuth sensor unit 5, and the operation signal input from the operation unit 10. Further, processing such as generation of Exif (Exchangeable Image File Format) data to be associated with the image data based on the position information input from the position sensor unit 11 and image correction is performed. Here, the Exif data includes, for example, shooting date / time, shooting device manufacturer, shooting device model name, overall image resolution, horizontal / vertical resolution, shooting direction, shutter speed, aperture (F value), ISO sensitivity, dimming A mode, presence / absence of flash, exposure correction step value, focal length, color space (color space), position information (longitude, latitude, altitude), and thumbnail image. Here, the image correction is to read the image recorded in the flash memory 9, extract the feature point coordinates from the read recorded image, and match the feature points of the extracted recorded image with the coordinates of the feature points of the live view. The angle of view, the rotation angle, and the vertical and horizontal positions are corrected so as to maximize the number of feature point coordinates. Details will be described later. Here, the correction processing is, for example, the live view so that the number of feature point coordinate distances that coincide with the coordinate distance of each feature point of the recorded image calculated by the feature point related information generation unit C109 is maximized. This is a process of correcting the angle of view of the live view based on the coordinate distance of the feature point. The control unit C1 performs known data processing, signal processing, expansion / compression, and the like, but description thereof is omitted.

  When the flash light emission unit 6 is set to the flash light emission mode by the operation signal input from the operation unit 10 via the control unit C1, the flash light emission unit 6 charges the light emission amount according to the ambient light amount. The flash light emitting unit 6 emits light based on a (shutter) signal instructed to be recorded input from the operation unit 10 to compensate for the peripheral light amount. The display unit 7 includes, for example, a display device such as a liquid crystal monitor screen (LCD: Liquid Crystal Display), and displays a recorded image, a thumbnail image, and a live view in the shooting mode. The display unit 7 displays the corrected live view input from the control unit C1.

  The DRAM 8 temporarily records image data input from the signal processing unit 3 via the control unit C1. In addition, the azimuth information input from the azimuth sensor unit 5 to the control unit C1, the position information input from the position sensor unit 11 to the control unit C1, the dimming information input from the dimming sensor unit 4, and the like are temporarily stored. To do. The flash memory 9 saves and records a JPEG (Joint Photographic Experts Group) -compressed captured image and Exif data associated therewith.

The operation unit 10 includes, for example, buttons and switches (icons) such as a shutter key, a shooting mode switching switch, an image data storage / deletion switch, an angle of view adjustment key, a menu key, a cursor key, and a flash emission setting key. . When these keys, buttons, or switches are operated, the operation unit 10 outputs a status signal and a set value to the control unit C1.
The position sensor unit 11 includes a module that acquires position information, for example, a GPS (Global Positioning System) receiver, and acquires position information of the imaging device D1.

FIG. 2 is a schematic block diagram illustrating the configuration of the control unit C1 according to the present embodiment.
The control unit C1 includes a recorded image photographing information processing unit C101, a recorded image feature point extracting unit C102, a position adjusting unit C103, a position / orientation information determining unit C104, an imaging processing unit C105, a live view feature point extracting unit C106, a reference feature point. Determination unit C107, live view feature point storage unit C108, feature point related information generation unit C109, count unit C110, imaging determination unit C111, image composition unit C112, feature point automatic adjustment unit A1, threshold storage unit C113, correction processing unit C114 And a writing unit C115. Each of these units is realized by the CPU of the control unit C1 reading and executing the program from the program storage ROM.

  The recorded image photographing information processing unit C101 reads a recorded image recorded in the flash memory 9 and Exif data associated with the recorded image. The recorded image photographing information processing unit C101 outputs the recorded image read from the flash memory 9 to the recorded image feature point extracting unit C102. The recorded image photographing information processing unit C101 outputs the Exif data read from the flash memory 9 to the position adjustment unit C103.

  The recorded image feature point extracting unit (second feature point extracting unit) C102 extracts the coordinates of the feature points of the recorded image input from the recorded image photographing information processing unit C101. Here, for “extraction of feature point coordinates”, for example, a known feature point detection method such as corner detection such as Harris method or principal curvature method or line detection such as Canny method or zero crossing method is used. The recorded image feature point extraction unit C102 outputs the coordinates of the extracted feature points to the recorded image photographing information processing unit C101. Here, the coordinates of feature points are straight lines composed of coordinates of points whose luminance values (lightness) are significantly different from those of surroundings, coordinates of representative points of corners, and the like, and specific colors and colors, or points and points. May be the coordinates of the representative point of a virtual curve connecting a plurality of. The recorded image feature point extraction unit C102 provides priority in extracting the coordinates of the feature points, and selects the feature points with, for example, a straight line. In addition, the priority in extracting the coordinates of the feature points may be set according to the type, such as making the corner higher than the straight line, the area of the figure connecting the coordinates of several feature points, You may set according to the number of points. The priority is related to the feature of the feature point detection accuracy. The recorded image feature point extraction unit C102 outputs the coordinates of the extracted feature points of the recorded image to the reference feature point determination unit C107. The recorded image feature point extraction unit C102 outputs the coordinates of the extracted feature points of the recorded image to the feature point related information generation unit C109. The recorded image feature point extraction unit C102 outputs the coordinates of the extracted feature points of the recorded image to the image composition unit C112.

  The position adjustment unit C103 includes the position information input from the position sensor unit 11, the direction information input from the direction sensor unit 5, and the position information in the recording image Exif data input from the recording image photographing information processing unit C101. , The orientation information in the Exif data is notified to the user. Here, “notify the user” is, for example, to notify the display unit 7 of numerical values such as latitude, longitude, altitude, and the like indicating the position information in the case of position information. You may notify a positional information and direction information by displaying a map. The user adjusts the shooting position and shooting direction based on the notified position information and direction information. The position adjustment unit C103 displays the position information and the direction information acquired from the position sensor unit 11 and the direction sensor unit 5, the position information of the recorded image, and the direction information based on the position adjusted by the user and the direction. It outputs to the direction information determination part C104.

  The position / orientation information determination unit C104 determines whether or not the difference between the position information and orientation information acquired from the sensor unit input from the position adjustment unit C103 and the position information and orientation information of the recorded image is within a range that can be automatically corrected. That is, it is determined whether or not the difference is equal to or less than a threshold value. Specifically, it is determined whether or not the position information of the imaging device D1 and the position information of the recorded image can be corrected in image data acquired in advance at a wide angle (for example, a wide angle of about several tens of pixels in the vertical and horizontal directions). It is determined whether or not the azimuth information of the imaging device D1 and the azimuth information of the recorded image can be corrected in image data acquired in advance at a wide angle. The position / orientation information determination unit C104 notifies the determination result to the live view feature point extraction unit C106 when it is within the range in which automatic correction is possible. On the other hand, when it is determined that it is not within the range in which automatic correction is possible, the position / orientation information determination unit C104 outputs a determination result to the position adjustment unit C103. The position adjustment unit C103 notifies the user of the position information and the direction information based on the determination result input from the position / orientation information determination unit C104. The user readjusts the position and direction based on the notified position information and direction information.

  The imaging processing unit C105 outputs the image input from the signal processing unit 3 to the correction processing unit C114 as a live view. If no correction parameter is set in the correction processing unit C114, the imaging processing unit C105 outputs the image input from the signal processing unit 3 to the live view feature point extraction unit C106 as a live view. In addition, when the imaging processing unit C105 receives a signal indicating that the shutter button has been pressed from the imaging determination unit C111, the imaging processing unit C105 instructs the signal processing unit 3 to adjust the white balance, and the imaging unit 1 is instructed. To instruct the exposure adjustment and to execute auto focus control. When an image reflecting these adjustments is input from the signal processing unit 3, the imaging processing unit C105 outputs the image to the correction processing unit C114 as an image for photographing.

  The live view feature point extraction unit (first feature point extraction unit) C106 starts the feature point extraction process when the determination result notified from the position / orientation information determination unit C104 indicates that it is within the automatic correction range. . The feature point extraction processing is processing for extracting the coordinates of feature points from the live view input by the imaging processing unit C105 or the corrected live view input by the correction processing unit C114. The live view feature point extraction unit C106 stores the coordinates of the extracted feature points in the live view feature point storage unit C108 as the coordinates of the feature points of the live view.

  The reference feature point determination unit C107 refers to the coordinates of the feature points of the recorded image input from the recorded image feature point extraction unit C102 and the coordinates of the feature points of the live view read from the live view feature point storage unit C108. The coordinates of the feature points of the live view corresponding to the coordinates of the feature points of the recorded image are searched. The reference feature point determination unit C107 selects a feature point coordinate serving as a reference in the recorded image from among the feature point coordinates of the recorded image in which the corresponding feature point coordinates have been detected by the search. The reference feature point determination unit C107 selects the coordinate of the feature point of the live view corresponding to the coordinate of the feature point serving as the reference in the recorded image as the reference feature point in the live view. The reference feature point determination unit C107 outputs the coordinates of these reference feature points to the feature point related information generation unit C109. The reference feature point determination unit C107 refers to the coordinates of the feature points of the live view read from the live view feature point storage unit C108 and the coordinates of the feature points of the recorded image input from the recorded image feature point extraction unit C102. Thus, the coordinates of the feature points of the live view corresponding to the coordinates of the feature points of the recorded image are used as the coordinates of the reference feature points.

The live view feature point storage unit C108 stores the coordinates of the feature points of the live view input from the live view feature point extraction unit C106 in the order of extraction.
The feature point related information generation unit C109 includes the coordinates of the feature points of the recorded image input from the recorded image feature point extraction unit C102, the coordinates of the feature points of the live view read from the live view feature point storage unit C108, and the reference feature. Based on the coordinates of the reference feature point input from the point determination unit C107, information on the feature point is generated. Here, the information on the feature points includes, for example, the coordinates of the feature points serving as the reference in the recorded image, the coordinates of each feature point, the coordinates of the feature point serving as the reference in the live view, and the coordinates of each feature point. , The inclination of a straight line connecting the coordinates of the feature point serving as the reference and the coordinates of the feature point of each image, and the position coordinates of each feature point. The feature point related information generation unit C109 outputs information on the feature points to the feature point automatic adjustment unit A1.

The count unit C110 is an increment counter, and calculates the total number of matching coordinate distances, the number of matching line inclinations, and the number of matching position coordinates input from the feature point automatic adjustment unit A1. The count unit C110 outputs the calculated sum to the imaging determination unit C111.
The imaging determination unit C111 determines whether the viewpoint, direction, and angle of view of the recorded image and the corrected live view are substantially the same. Specifically, the imaging determination unit C111 compares the sum calculated by the count unit C110 with the threshold value stored in the threshold value storage unit C113, and determines that the sum is approximately equal when the sum exceeds the threshold value. To do. When the imaging determination unit C111 determines that they are substantially coincident, the shutter key of the operation unit 10 can be operated. When the operation unit 10 detects that the shutter key has been pressed, the imaging determination unit C111 notifies the imaging processing unit C105 and the correction processing unit C114 of a signal indicating that the shutter key has been pressed.

  The image composition unit C112 synthesizes an image to be displayed on the display unit 7. The image to be displayed is, for example, an image obtained by combining the recorded image and the live view, an image obtained by combining the coordinates of the feature points of the recorded image and the live view, or an image obtained by combining the coordinates of the feature points of the recorded image and the live view. This is an image in which the coordinates of the feature points of the recorded image and the coordinates of the feature points of the live view are synthesized.

  The correction processing unit C114 performs correction processing on the live view input from the imaging processing unit C105 according to the correction parameters determined by the feature point automatic adjustment unit A1. The correction processing unit C114 reads the angle-of-view correction parameter determined by the feature point automatic adjustment unit A1 from the correction parameter storage unit A18. The correction processing unit C114 enlarges / reduces the live view by α times based on the read angle-of-view correction parameter α.

The correction processing unit C114 reads the rotation angle correction parameter determined by the feature point automatic adjustment unit A1 from the correction parameter storage unit A18. The correction processing unit C114 rotates the enlarged / reduced live view by β based on the read rotation angle correction parameter β. The correction processing unit C114 reads the vertical / horizontal position correction parameters determined by the feature point automatic adjustment unit A1 from the correction parameter storage unit A18. The correction processing unit C114 cuts out from the rotated live view an image having a predetermined size centered at a position shifted from the center of the rotated live view by the movement amount indicated by the vertical / horizontal position correction parameter γ. The correction processing unit C114 sets the cut-out image as a corrected live view.

  Note that the correction processing unit C114 normally outputs the corrected live view to the live view feature point extraction unit C106 and the image composition unit C112. However, when a signal indicating that the shutter button has been pressed is notified from the imaging determination unit C111, the correction processing unit C114 writes the image for shooting input from the imaging processing unit C105 as a captured image to the writing unit C115. Output to.

  The writing unit C115 converts the captured image input from the correction processing unit C114 into an image format such as a JPEG format and writes the converted image in the flash memory 9. The writing unit C115 associates the position information acquired from the position sensor unit 11 and the direction information acquired from the direction sensor unit 5 with the captured image when writing the converted captured image into the flash memory 9. Write.

  The feature point automatic adjustment unit A1 determines whether the angle of view, the rotation angle, and the vertical and horizontal positions of the recorded image and the corrected live view match. Here, the term “match” means, for example, that the number of pieces of information related to the feature points of the live view that matches the information related to the feature points of the recorded image is equal to or greater than a threshold. When any of the angle of view, the rotation angle, and the vertical and horizontal positions does not match, the correction parameter corresponding to the mismatched item is updated. Specifically, the feature point automatic adjustment unit A1 uses the coordinate distance input from the feature point related information generation unit C109 to determine whether or not the angle of view matches. When the feature point automatic adjustment unit A1 determines that they match, the feature point automatic adjustment unit A1 outputs the number of coordinate distances that match between the recorded image and the corrected live view to the count unit C110.

  The feature point automatic adjustment unit A1 uses the slope of the straight line input from the feature point related information generation unit C109 to determine whether the rotation angle matches. If the feature point automatic adjustment unit A1 determines that they match, the feature point automatic adjustment unit A1 outputs the number of straight line slopes that match between the recorded image and the corrected live view to the count unit C110. Further, the feature point automatic adjustment unit A1 determines whether the vertical and horizontal positions match using the position coordinates input from the feature point related information generation unit C109. When the feature point automatic adjustment unit A1 determines that they match, the feature point automatic adjustment unit A1 outputs the number of position coordinates that match between the recorded image and the corrected live view to the count unit C110. The detailed operation of the feature point automatic adjustment unit A1 will be described later.

FIG. 3 is a schematic block diagram illustrating an example of the configuration of the feature point automatic adjustment unit A1 according to the present embodiment.
The feature point automatic adjustment unit A1 includes a threshold storage unit A11, an angle of view determination unit A12, an angle of view parameter determination unit A13, a rotation angle determination unit A14, a rotation angle parameter determination unit A15, an up / down / left / right position determination unit A16, and an up / down / left / right position parameter. A determination unit A17 and a correction parameter storage unit A18 are included.

  The threshold storage unit A11 stores a threshold for determining that the angle of view, the rotation angle, and the vertical, horizontal, and horizontal positions match each other. The angle-of-view determination unit A12 reads a threshold value related to the angle of view stored in the threshold value storage unit A11. The angle-of-view determination unit A12 determines whether the angles of view of these images match based on the coordinate distance of each feature point of each image input from the feature point related information generation unit C109. Specifically, the angle-of-view determination unit A12 determines whether or not the number of coordinate distances of the feature points of the live view that matches the coordinate distances of the feature points of the recorded image is greater than or equal to a threshold value. Judge that you are doing.

  When it is determined that they match, the angle-of-view determination unit A12 determines whether or not each of the coordinates of the feature points of the live view having the matching coordinate distance is stored in the live view feature point storage unit C108. The angle-of-view determination unit A12 stores the coordinates of feature points that are not stored in the live view feature point storage unit C108 and outputs these numbers to the count unit C110. On the other hand, when it is determined that they do not match, the view angle determination unit A12 instructs the view angle parameter determination unit A13 to determine the view angle correction parameter.

  The view angle parameter determination unit A13 reads the view angle correction parameter stored in the correction parameter storage unit A18. The angle-of-view parameter determination unit A13, based on the coordinate distance of the feature point of each image input from the feature point related information generation unit C109, matches the coordinate distance of the feature point of the recorded image and the coordinate of the feature point of the matching live view. The magnification for adjusting the angle of view is determined so that the number of distances is maximized. The view angle parameter determination unit A13 calculates a difference between the magnification for adjusting the determined view angle and the previous view angle correction parameter read from the correction parameter storage unit A18. The view angle parameter determination unit A13 sets the calculated difference as the view angle correction parameter α. The view angle parameter determination unit A13 stores the view angle correction parameter in the correction parameter storage unit A18.

  The rotation angle determination unit A14 reads a threshold value related to the rotation angle stored in the threshold value storage unit A11. The rotation angle determination unit A14 determines whether or not the rotation angles of these images match based on the slope of the straight line of each feature point of each image input from the feature point related information generation unit C109. . Specifically, the rotation angle determination unit A14 determines whether or not the number of inclinations of the straight line of the feature point of the live view that matches the inclination of the straight line of the feature point of the recorded image is equal to or greater than the threshold value. Is determined to match.

  When it is determined that they match, the rotation angle determination unit A14 determines whether or not each of the coordinates of the feature points of the live view having the same straight line inclination is stored in the live view feature point storage unit C108. . The rotation angle determination unit A14 stores the coordinates of the feature points that are not stored in the live view feature point storage unit C108, and outputs these numbers to the count unit C110. On the other hand, when it is determined that they do not match, the rotation angle determination unit A14 instructs the rotation angle parameter determination unit A15 to determine the rotation angle correction parameter.

  The rotation angle parameter determination unit A15 reads the rotation angle correction parameter stored in the correction parameter storage unit A18. The rotation angle parameter determination unit A15, based on the slope of the straight line of the feature point of each image input from the feature point related information generation unit C109, matches the slope of the straight line of the feature point of the recorded image. The angle at which the rotation angle is adjusted is determined so that the number of inclinations of the straight line becomes the maximum. The rotation angle parameter determination unit A15 calculates the sum of the angle for adjusting the determined rotation angle and the previous rotation angle correction parameter read from the correction parameter storage unit A18. The rotation angle parameter determination unit A15 sets the calculated sum as the rotation angle correction parameter β. The rotation angle parameter determination unit A15 stores the rotation angle correction parameter in the correction parameter storage unit A18.

  The up / down / left / right position determination unit A16 reads a threshold value related to the up / down / left / right position stored in the threshold value storage unit A11. The vertical / horizontal position determination unit A16 determines whether the vertical / horizontal positions of the images match based on the position coordinates of the feature points of the images input from the feature point related information generation unit C109. To do. Specifically, the up / down / left / right position determination unit A16 determines whether or not the number of position coordinates of the feature point of the live view that matches the position coordinates of the feature point of the recorded image is greater than or equal to a threshold value. It is determined that they match.

  When it is determined that they match, the vertical / horizontal position determination unit A16 determines whether or not each of the coordinates of the feature points of the live view whose position coordinates match is stored in the live view feature point storage unit C108. . The up / down / left / right position determination unit A16 stores the coordinates of the feature points not stored in the live view feature point storage unit C108, and outputs these numbers to the count unit C110. On the other hand, when it is determined that they do not match, the vertical / horizontal position determination unit A16 instructs the vertical / horizontal position parameter determination unit A17 to determine the vertical / horizontal position correction parameter.

  The vertical / horizontal position parameter determination unit A17 reads the vertical / horizontal position correction parameter stored in the correction parameter storage unit A18. The up / down / left / right position parameter determination unit A17, based on the slope of the straight line of the feature point of each image input from the feature point related information generation unit C109, matches the slope of the straight line of the feature point of the recorded image. The amount of movement for adjusting the vertical and horizontal positions is determined so that the number of slopes of the straight line of the point is maximized. The up / down / left / right position parameter determination unit A17 calculates a difference between the determined movement amount for adjusting the up / down / left / right position and the previous vertical / horizontal position correction parameter read from the correction parameter storage unit A18. The vertical / horizontal position parameter determination unit A17 sets the calculated difference as the vertical / horizontal position correction parameter γ. The up / down / left / right position parameter determination unit A17 stores the determined up / down / left / right position correction parameter in the correction parameter storage unit A18.

FIG. 4 is a conceptual diagram illustrating an example of wide-angle shooting of the imaging device D1 according to the present embodiment.
The figure on the left is a recorded image shot and recorded by the imaging device D1. On the other hand, the right figure is a photographed image taken at a wide angle of about several tens of pixels. The imaging device D1 determines correction parameters for the angle of view, the rotation angle, and the vertical, horizontal, and horizontal positions, and the correction processing unit C114 performs correction processing using the correction parameters.

FIG. 5 is a conceptual diagram showing an example of a method for correcting a captured image according to the present embodiment.
The feature point automatic adjustment unit A1 adjusts the angle α for adjusting the angle of view, the angle β for adjusting the rotation angle, and the vertical and horizontal positions using the coordinate distance, the inclination of the straight line, and the position coordinates of the feature points of each image. Correction processing is performed in the order of the movement amount γ. Since the live view performs correction processing based on image data acquired in advance at a wide angle, no image is lost due to the correction processing. In order to prevent image deterioration due to the correction process, the correction process is always performed on the live view before the correction process. For example, an image that is enlarged 1.2 times from the image from the live view and further enlarged 1.68 times from the live view before the correction processing is created rather than an image that is further magnified 1.4 times.
In the present embodiment, the correction process is performed on the live view, but the correction process may be performed on the recorded image.

FIG. 6 is a conceptual image showing an example in which the coordinates of feature points are extracted from the recorded image according to the present embodiment.
The coordinates of the feature points of the recorded image extracted by the recorded image feature point extraction unit C102 include, for example, straight lines and corners formed by connecting the coordinates of several feature points with straight lines. In the display of the feature point coordinates, the straight line portion and the corner portion are represented by lines. However, the feature point coordinates may be represented by points without connecting the feature points by straight lines.

FIG. 7 is a conceptual image showing an example of a captured image when the user performs a position alignment based on the coordinates of the feature points extracted from the recorded image according to the present embodiment.
FIG. 7 shows an example in which a live view is superimposed and displayed on the image composition unit C112 based on the recorded image, and there is a difference in the coordinates of feature points between the recorded image and the live view. For example, when the coordinates of the feature points of the recorded image are transparently displayed on the display unit 7 and the user adjusts the position based on the coordinates, it is difficult to match the feature points of the recorded image and the live view. Yes, it depends heavily on the ability of the user. Also, adjustment takes time.

FIG. 8 is a concept showing an example in which the coordinates of the feature points extracted from the recorded image according to the present embodiment are compared with the coordinates of the feature points of the captured image taken by the user based on the coordinates. It is an image.
8 shows the coordinates of the feature points of the recorded image extracted by the recorded image feature point extraction unit C102, the coordinates of the feature points of the live view extracted by the live view feature point extraction unit C106, the recorded image, and the live view. Based on the above, the recorded image, the live view, and the coordinates of the feature points of each image are overlapped and displayed by the image composition unit C112. There is a difference between the coordinates of the feature points of the highlighted recorded image and the coordinates of the feature points of the live view.

FIG. 9 is a flowchart showing an example of the operation in the feature point automatic adjustment processing of the captured image according to the present embodiment.
The recorded image photographing information processing unit C101 reads out the recorded image recorded in the flash memory 9 and displays it on the display unit 7 (S101). The recorded image photographing information processing unit C101 reads the Exif data associated with the recorded image and acquires the position information of the recorded image (S102).
The recorded image feature point extraction unit C102 extracts feature point coordinates from the recorded image read by the recorded image photographing information processing unit C101 (S103).
The count unit C110 initializes the counter with a zero value (S104).
The position adjustment unit C103 notifies the user to adjust the position based on the recorded image displayed on the display unit 7 and the position information associated with the recorded image in step S102 (S105).
The position sensor unit 11 acquires the current position information (S106), compares the position information of the recorded image notified in step S105 with the current position information, and determines whether or not the current position information is within a range that can be automatically corrected. Determination is made (S107).
When step S107 is within a range in which automatic correction is possible (S107-YES), the recorded image photographing information processing unit C101 changes the recorded image being displayed on the display unit 7 to a transparent display (through image) and displays it. (S108). On the other hand, when step S107 is not within the range where automatic correction is possible (S107-NO), the processing from step S105 to step S107 is repeated.
The recorded image photographing information processing unit C101 acquires azimuth information from the Exif data read out in step S102 (S109).
Count unit C110 clears the counter with a zero value (S110).
The imaging processing unit C105 captures a live view from the imaging unit 1 (S111).
The correction processing unit C114 performs correction processing on the captured live view (S112).
The image composition unit C112 superimposes the live view and the transparently displayed recorded image on the display unit 7 (S113).
The position adjustment unit C103 notifies the user to align the orientation while viewing the image in step S113 (S115).
For the direction adjusted in step S115, the direction sensor unit 5 acquires the current direction information (S116).
It is determined whether the position information of the recorded image and the current position information, and the azimuth information of the recorded image and the current azimuth information are within a range that can be automatically corrected (S117).
When step S117 is within the range in which automatic correction is possible (S117-YES), the live view feature point extraction unit C106 records the coordinates of the feature points extracted from the live view in the order of extraction (S118). On the other hand, if step S117 is not within the range where automatic correction is possible (S117-NO), if the shutter icon is lit, it is turned off (S114).
The reference feature point determination unit C107 searches for the feature point coordinates corresponding to the feature point coordinates from the coordinates of the feature points of the recorded image and the feature point coordinates of the live view. The coordinates of each feature point serving as a reference are selected from (S119).
Based on the coordinates of the reference feature points of each image selected in S119, the feature point related information generation unit C109 coordinates the coordinates of each of the feature points serving as a reference and the coordinates of each feature point of each image. Calculate the distance. The feature point related information generation unit C109 calculates the inclination of a straight line connecting the coordinates of each feature point serving as a reference and the coordinates of each feature point of each image.
The feature point automatic adjustment unit A1 determines whether or not the number of feature point coordinates that match the coordinate of the feature point of the recorded image and the coordinate of the feature point of the live view is equal to or greater than a threshold (S120).
In step S120, when the number of coordinates of the matching feature points is equal to or greater than the threshold (S120-YES), a shutter icon is displayed (S301). On the other hand, in step S120, if the number of matching feature point coordinates is less than the threshold (S120-NO), it is determined whether or not all the extracted feature point coordinates of the live view have been compared (S121). Specifically, the reference feature point determination unit C107 searches for the coordinates of the feature points of the live view corresponding to the coordinates of the feature points of the recorded image, and the feature points of the live view corresponding to the coordinates of the feature points of the recorded image. Each is determined as the coordinates of the reference feature point. In each of the determined coordinates of the reference feature point, the angle of view, the rotation angle, and the vertical and horizontal positions are adjusted and added to the count unit C110. If all the coordinates of the extracted feature points of the live view have not been determined (S121—NO), the shutter icon is turned off (S122).
It is determined whether or not the angle of view matches (S123).
When step S123 matches (S123-YES), the feature point automatic adjustment unit A1 determines whether or not the coordinates of the feature points are not yet recorded in the table (S126). On the other hand, if step S123 does not match, the view angle parameter determination unit A13 adjusts the view angle so that the number of coordinate distances of the matching feature points is maximized (S124). Specifically, if the live view is smaller than the recorded image so that the number of coordinate distances of the matching feature points in the coordinate distance of the feature points of the recorded image and the coordinate distance of the feature points of the live view is maximized, the zoom is performed. If the live view is larger than the recorded image, the image is zoomed down and the angle of view is roughly adjusted (S124). Further, the angle-of-view parameter determination unit A13 records the live view so that the number of coordinate distances of the matching feature points in the coordinate distance of the feature points of the recorded image and the coordinate distance of the feature points of the live view is maximized. If it is smaller than the image, the live view is enlarged, and if the live view is larger than the recorded image, the live view is reduced (S125).
In step S126, when it is the coordinates of the feature point of the image data recorded in the table (S126-NO), the rotation angle determination unit A14 determines whether or not the rotation angle is matched (S201). On the other hand, when the coordinates of the feature points of the image data are not stored in the table in step S126 (S126-YES), the count unit C110 calculates the total number of coordinate distances of the matching feature points (S127). The coordinates of the feature points of the image data are recorded in the table (S128).

FIG. 10 is a flowchart showing an example of the operation of the feature point automatic adjustment unit A1 according to the present embodiment.
Specifically, the step S201 includes the inclination of the straight line connecting the coordinates of the reference feature points of the recorded image input from the feature point related information generation unit C109 and the coordinates of each feature point, and the reference of the live view. Based on the inclination of the straight line connecting the coordinates of the feature points and the coordinates of each of the feature points, it is determined whether or not the rotation angles match.
If step S201 matches (S201—YES), the feature point automatic adjustment unit A1 determines whether or not the coordinates of the feature points are not yet recorded in the table (S203). On the other hand, if step S201 does not match, the rotation angle parameter determination unit A15 adjusts the rotation angle so that the number of straight line inclinations of the matching feature points is maximized (S202). Specifically, the number of straight line inclinations of the coordinate of feature points in the recorded image and the straight line inclination of the coordinate of feature points in the live view are maximized. The live view is rotated clockwise and counterclockwise, and the rotation angle is adjusted (S202).
In step S203, if the coordinates of the feature point of the live view recorded in the table (S203-NO), the vertical / horizontal position determination unit A16 determines whether the vertical / horizontal position matches (S206). On the other hand, when the coordinates of the feature points of the live view are not stored in the table in step S203 (S203—YES), the count unit C110 calculates the total number of linear inclinations of the coordinates of the matching feature points ( S204), the coordinates of the feature points of the live view are recorded in the table (S205).
Specifically, in step S206, based on the coordinates of each feature point of the recorded image input from the feature point related information generation unit C109 and the coordinates of each feature point of the live view, the vertical and horizontal positions are determined. It is determined whether or not they match.
When step S206 matches (S206—YES), the feature point automatic adjustment unit A1 determines whether the coordinates of the feature points are not yet recorded in the table (S208). On the other hand, if step S206 does not match, the up / down / left / right position parameter determination unit A17 adjusts the up / down / left / right position so that the number of position coordinates of the matching feature points is maximized (S207). Specifically, the live view is right-justified / left-justified and up-aligned so that the number of matching feature point position coordinates in the position coordinates of the feature points of the recorded image and the position coordinates of the feature points of the live view is maximized. / Lowering.
In step S208, if the coordinates of the feature points of the live view recorded in the table (S208-NO), the respective correction parameters adjusted by the feature point automatic adjustment unit A1 are output to the correction processing unit C114 to be matched. In S120, the number of feature point coordinates is determined as a threshold value. On the other hand, when the coordinates of the feature points of the live view are not stored in the table in step S208 (S208-YES), the count unit C110 calculates the total number of position coordinates of the matching feature points (S209). The coordinates of the feature points of the live view are recorded in the table (S210).

FIG. 11 is a flowchart illustrating an example of the operation of the imaging determination unit C111 in the automatic correction process according to the present embodiment.
If the number of coordinates of the matching feature points in step S120 is greater than or equal to the threshold, a shutter icon is displayed (S301).
If the shutter key is pressed based on step 301 (S302-YES), the captured image is taken into the DRAM 9 (S303), and the shutter icon is turned off (S304).
The captured image captured in step S303 is displayed on the display unit 7 (S305).
When the captured image displayed on the display unit 7 in step S305 is recorded and saved (S306-YES), it is recorded in the flash memory 9 (S307).
Exif data associated with the photographed image is stored in the flash memory 9 (S308).

  Thus, according to the present embodiment, the imaging unit 1 captures a subject and generates a captured image. The correction processing unit C114 corrects the recorded image or the captured image stored in advance in the storage device (flash memory 9). The feature point automatic adjustment unit A1 determines a correction parameter used when the correction processing unit C114 performs correction. The feature point automatic adjustment unit A1 has the viewpoint, direction, and angle of view of the image corrected by the correction processing unit C114 as the viewpoint, direction, and angle of view of the recorded image and the captured image that are not corrected by the correction processing unit C114. The correction parameters are determined so as to substantially match. As a result, it is possible to easily perform photographing from the same viewpoint and in the same direction as when photographing previously.

  In the present embodiment, correction parameters are determined so that the number of feature point coordinates that match the coordinates of the feature points of the recorded image and the coordinates of the feature points of the live view is maximized, and the live view correction process is performed. However, the correction processing of the recorded image may be performed using the correction parameter.

  In addition, you may make it implement | achieve part or all of the imaging device D1 in embodiment mentioned above with a computer. In that case, the program for realizing the control function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by a computer system and executed.

  Here, the “computer system” is a computer system built in the imaging apparatus D1, and includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” is a medium that dynamically holds a program for a short time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, In such a case, a volatile memory inside a computer system serving as a server or a client may be included and a program that holds a program for a certain period of time. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  In addition, a part or all of the imaging device D1 in the above-described embodiment may be realized as an integrated circuit such as an LSI (Large Scale Integration). Each functional block of the imaging device D1 may be individually made into a processor, or a part or all of them may be integrated into a processor. Further, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Further, in the case where an integrated circuit technology that replaces LSI appears due to progress in semiconductor technology, an integrated circuit based on the technology may be used.

  As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to

D1 ... Imaging device 1 ... Imaging unit 1a ... Shooting lens 2 ... Signal conversion unit 3 ... Signal processing unit 4 ... Light control sensor unit 5 ... Direction sensor unit 6 ... Flash light emitting unit 7 ... display unit 8 ... DRAM
DESCRIPTION OF SYMBOLS 9 ... Flash memory 10 ... Operation part 11 ... Position sensor part C1 ... Control part C101 ... Record image imaging information processing part C102 ... Record image feature point extraction part C103 ... Position Adjustment unit C104 ... Position / orientation information determination unit C105 ... Imaging processing unit C106 ... Live view feature point extraction unit C107 ... Reference feature point determination unit C108 ... Live view feature point storage unit C109 .. Feature point related information generation unit C110 ... count unit C111 ... imaging determination unit C112 ... image synthesis unit C113 ... correction threshold storage unit C114 ... correction processing unit C115 ... writing unit A1 ... Feature point automatic adjustment unit A11 ... Threshold storage unit A12 ... View angle determination unit A13 ... View angle parameter determination unit A14 ... Rotation angle determination unit A15 ... Rotation angle parameter Meter determining unit A16 · · · vertical and lateral position determination unit A17 · · · vertical and lateral position parameter determining unit A18 · · · correction parameter storage unit

Claims (7)

An imaging unit that shoots a subject and generates a captured image;
A correction processing unit for correcting a recorded image or the captured image stored in advance in a storage device;
The correction processing unit comprises a feature point automatic adjustment unit that determines a correction parameter used when correcting, and
The feature point automatic adjustment unit includes a viewpoint, a direction, and an image of an image that is corrected by the correction processing unit, the viewpoint, the direction, and the angle of view of the recorded image and the captured image that are not corrected by the correction processing unit. The correction parameter is determined so as to substantially coincide with a corner. A first feature point extraction unit for extracting a first feature point of an image to be corrected by the correction processing unit;
A second feature point extraction unit that extracts a second feature point of an image that is not corrected by the correction processing unit;
The feature point automatic adjustment unit is information on the first feature point in the image corrected by the correction processing unit among the information on the second feature point, and the first feature point corresponding to the second feature point. The imaging apparatus according to claim 1, wherein the correction parameter is determined so that the number of items that coincide with the information regarding the point is maximized.   The imaging apparatus according to claim 2, further comprising an imaging determination unit configured to record an image based on the captured image when the number of matching items is equal to or greater than a preset threshold value. The correction parameters are an angle of view, a rotation angle, an up / down / left / right position,
The information on the feature point includes a distance between the feature point and another feature point, a slope of a straight line passing through the feature point and the other feature point, and coordinates of the feature point, and corresponds to the second feature point. Corresponding to the first feature point, the number of matching straight lines passing through the first feature point corresponding to the second feature point, and the coordinates of the second feature point. The imaging apparatus according to claim 2, wherein the correction parameter is determined so that the number of objects that coincide with the coordinates of the first feature point is maximized. An image synthesis unit that synthesizes the image related to the corrected image and the image related to the uncorrected image;
The imaging device according to claim 1, further comprising: a display unit that displays the synthesized image. A first step of shooting a subject and generating a shot image;
A second step of correcting a recorded image or the captured image stored in advance in a storage device;
A third step of determining a correction parameter to be used for correction;
Fourth step of determining the correction parameter so that the viewpoint, direction, and angle of view of the corrected image substantially coincide with the viewpoint, direction, and angle of view of the image that is not corrected among the recorded image and the captured image. An imaging method comprising: A first means for photographing a subject and generating a photographed image;
A second means for correcting a recorded image or the captured image stored in advance in a storage device;
A third means for determining a correction parameter used for correction;
Fourth means for determining the correction parameter so that the viewpoint, direction, and angle of view of the corrected image substantially coincide with the viewpoint, direction, and angle of view of the uncorrected image of the recorded image and the captured image. An imaging program characterized by executing and.

Images