JP5925007B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program, and more particularly, to a technique suitable for use in arranging content in a layout.

  2. Description of the Related Art Conventionally, a technique for automatically laying out an electronic data document in consideration of metadata such as position information attached to content is known. As a specific example, a technique for displaying a still image or a moving image at a corresponding point on three-dimensional stereoscopic data based on the shooting position, orientation, and angle acquired at the time of shooting is known (for example, patents). Reference 1). As another example, based on the information on the position and orientation of the pointing tool held by the observer, the object providing the information is arranged in a suitable position and orientation when viewed from the observer's viewpoint. A technique is known (for example, refer to Patent Document 2). In addition, referring to data indicating the positional relationship in the real world for a plurality of elements to be captured included in the captured image, the orientation of the image is determined so that the layout best reflects the hierarchical relationship. A technique is also known (see, for example, Patent Document 3). Furthermore, a technique for detecting an overlapping portion between two images and synthesizing the images is also known (see, for example, Patent Document 4).

JP-A-11-122638 JP 2006-252468 A Japanese Patent No. 4064186 Japanese Patent No. 3832894 JP 2010-102501 A

C. Harris and M.J. Stephens, "A combined corner and edge detector," In Alvey Vision Conference, pages 147-152, 1988. David G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints," International Journal of Computer Vision, 60, 2 (2004), pp.91-110. M.A. Fischler and R.C. Bolles, "Random sample consensus: A paradigm formodel fitting with applications to image analysis and automated cartography," Commun. ACM, no.24, vol.6, pp.381-395, June 1981.

  Here, as an important point in creating the layout, there is no discomfort to the viewer. For example, if the layout is created in such a way that the positional relationship between the subjects in each photo is opposite to that of the real world, it will give a strange feeling to those who know the real world. Moreover, those who do not know the real world cannot fully grasp the positional relationship of the subject.

  According to the method described in Patent Document 1, when arranging on a two-dimensional plane such as a paper album, three-dimensional solid data is projected. In that case, there is a possibility that the contents overlap each other. In the case of the method described in Patent Document 2, an indicator is required, and the origin is limited to the viewpoint. Therefore, the method cannot be applied when creating an album format layout of images taken with other cameras.

  Also, with the method described in Patent Document 3, an appropriate layout cannot be created if data indicating the positional relationship of the subject is not held. Further, since the technique described in Patent Document 4 is not a technique for creating a layout, it cannot be applied when there is a limit on the size of an image.

  The present invention has been made in view of the above-described problems, and it is an object of the present invention to make it possible to easily determine the position of content that can be easily viewed by a viewer without using a special device or data.

An image processing apparatus according to the present invention includes an extraction unit that extracts features from a plurality of contents in which a series of subjects are partially imaged, and a real world when the subject is imaged from the features extracted by the extraction units. An estimation unit that estimates a positional relationship; and a determination unit that determines a layout position of the plurality of contents that maintains a positional relationship in the real world based on a result of estimation by the estimation unit. The linear component of content is extracted as the feature, and the estimation means determines the position of the plurality of content in the real world by the continuity of the linear component when there is no common part among the plurality of content. The relationship is estimated .

  According to the present invention, it is possible to easily determine the position of the content that is easy for the viewer to see without requiring a special device or data.

It is a figure explaining an example of the layout produced in 1st Embodiment. It is a block diagram which shows the hardware structural example of the layout processing apparatus which concerns on embodiment. It is a block diagram which shows the function structural example of the layout processing apparatus which concerns on embodiment. 6 is a flowchart illustrating an example of a processing procedure for creating a layout in the first embodiment. 5 is a flowchart illustrating an example of a processing procedure for determining whether or not to estimate a relative position of a subject in the first embodiment. It is a figure explaining an example of the layout produced in 2nd Embodiment. It is a figure explaining an example of the layout produced in 3rd Embodiment. It is a figure explaining an example of the layout produced in 4th Embodiment. It is a figure explaining an example of the layout produced in 5th Embodiment. 19 is a flowchart illustrating an example of a processing procedure for determining whether or not to estimate a relative position of a subject in the sixth embodiment. It is a figure explaining an example of the layout produced in 7th Embodiment. It is a figure explaining an example of the layout produced in 8th Embodiment.

(First embodiment)
Hereinafter, a first embodiment of the present invention will be described. Here, for example, as shown in FIG. 1A, in the real world 101, a scene in which a prism is placed on the left side of a patterned paper object 108 and a cylinder is placed on the right side is taken. To do. FIG. 1B shows a photograph 102 in which a prism is photographed and a photograph 103 in which a cylinder is photographed. At this time, if the layout is created when the order of taking is the order of the photograph 103 and the photograph 102, generally, the layout 104 shown in FIG. 1C is created. As described above, when arranged in a time-sequential order in the left-open album, a layout 104 in which photos are arranged in order from left to right is obtained. If such a shape is formed, the photos are arranged in the opposite direction to the actual arrangement of the subject, giving the viewer a sense of incongruity. Therefore, in the present embodiment, the positional relationship between the photographs is estimated using the feature data of the image, and the layout 105 shown in FIG. 1C is created. With this layout 105, it can be said that the photos are arranged in the same direction as the arrangement of the subject in the real world and are easy to see.

FIG. 2 is a block diagram illustrating a hardware configuration example of the layout processing apparatus 200 according to the present embodiment.
As illustrated in FIG. 2, the layout processing apparatus 200 that is an image processing apparatus includes a CPU (Central Processing Unit) 201, an input device 202, an output device 203, and a storage device 204. Further, a RAM (Random Access Memory) 205, a ROM (Read Only Memory) 206, and a BUS 207 are provided.

  The CPU 201 performs logical operations and determinations for various data processing, and controls each component connected by the BUS 207. The input device 202 is connected to a keyboard having various function keys such as alphabet keys, hiragana keys, katakana keys, character symbol input keys such as punctuation marks, cursor movement keys for instructing cursor movement, and the like. Yes. In addition, an image pickup device such as a camera that inputs an image to be laid out or a storage such as an HDD that holds and inputs an image recorded by the image pickup device can be cited as another connection. Furthermore, a pointing device such as a mouse or a stick pointer that indicates a controllable position of the screen on a GUI (Graphical User Interface) and performs a function selection instruction or the like is also included.

  The output device 203 is various display devices such as a liquid crystal panel. The storage device 204 stores various information such as input / output data and processing programs. As a storage medium for storing these data and programs, a hard disk, CD-ROM, DVD-ROM, flash memory, or the like can be used. The RAM 205 is used when temporarily storing various data from each component. The ROM 206 stores a control program such as a processing program executed in the present embodiment. All these components are connected by BUS207.

FIG. 3 is a block diagram illustrating a functional configuration example of the layout processing apparatus 200 according to the present embodiment.
The layout processing apparatus 200 includes a content feature information acquisition unit 301, a content correspondence acquisition unit 302, a position / orientation estimation unit 303, and a content arrangement position determination unit 304. Hereinafter, content means a photograph. Hereinafter, these functions will be described together with FIG. 4 showing the overall flow.

FIG. 4 is a flowchart illustrating an example of a processing procedure for creating a layout by the layout processing apparatus 200 according to the present embodiment.
First, in step S 401, the content feature information acquisition unit 301 reads content from the storage device 204, performs predetermined processing on the read content, acquires feature data, and sets it in the RAM 205. Note that the data acquired at this time only needs to include data used in future processing, and other data may or may not be included. Details of the feature data acquisition process will be described later.

  Next, in step S <b> 402, the content feature information acquisition unit 301 selects content to be laid out. Here, as a selection method, selection may be made using information attached to the content or the like, or selection may be made according to information input from the input device 202 by a user operation.

  As information and selection method used when selecting using information attached to content, for example, selecting a location within a predetermined distance using a shooting location, or using a shooting time Can be selected within a predetermined interval. Alternatively, a method may be used in which content with a similar composition is selected based on the position of an object having a similar face position or shape.

  In addition, when selecting according to information input from the input device 202 by the user's operation, in addition to simply selecting the content to be used by clicking or the like, there may be a means for performing selection using a switch button or the like. . Note that when manually selecting content to be used, the processing for acquiring content feature data (step S401) may be performed after this.

  Next, in step S403, the content correspondence acquisition unit 302 compares the acquired feature data, estimates a common part or a common object among a plurality of contents, and acquires information on the correspondence. In step S404, the position / orientation estimation unit 303 estimates the position / orientation with respect to the imaging device in the real world between the subjects in each content.

  Next, in step S405, the content arrangement position determination unit 304 determines the layout position on the album of each content based on the estimated position and orientation. In step S406, each content is arranged at the layout position determined in step S405. Thereby, a layout that reproduces the position and orientation in the real world between subjects in each content such as the layout 105 can be created on the album.

  Various types of feature data acquired in step S401 and their acquisition methods can be considered. In this embodiment, a method is used in which a local feature amount (local feature amount) of an image is acquired as feature data, and a similar image is searched using the local feature amount. In this method, first, characteristic points (local feature points) are extracted from an image. As a method for extracting local feature points, for example, the method disclosed in Non-Patent Document 1 is used. Then, based on the local feature point and surrounding image information, a feature amount (local feature amount) corresponding to the local feature point is calculated. As a method for calculating the local feature amount, for example, a method disclosed in Non-Patent Document 2 is used. The image search is performed by matching local feature amounts.

  In the method using the local feature amount, the local feature amount is defined as information including a plurality of elements that are rotation invariant and enlargement / reduction invariant. Thereby, even when the image is rotated, enlarged or reduced, the search can be performed. Further, the local feature amount is generally expressed as a vector. However, it is a theoretical story that the local feature is invariant to rotation and enlargement / reduction. In an actual digital image, the local feature before image rotation and enlargement / reduction processing corresponds to that after processing. Some variation occurs between the local feature amount. In order to calculate the rotation-invariant local feature amount, for example, in the method described in Non-Patent Document 2, the main direction is calculated from the pixel pattern of the local region around the local feature point, and the main direction is calculated when the local feature amount is calculated. Rotate the local region with reference to normalize the direction. In addition, in order to calculate the local feature amount that does not change in size, the local feature amount is calculated by internally generating images of different scales and extracting local feature points from the images of the respective scales. Here, a set of images of different scales generated internally is generally called a scale space. The feature amount of each feature point also stores a rotation angle and a scale space in which the feature amount is calculated when normalizing the direction.

Details of step S403 will be described with reference to FIG. FIG. 5 is a flowchart illustrating an example of a detailed processing procedure of step S403 of FIG. 4 in the present embodiment.
First, in step S501, it is determined whether the target content is taken at the same position or at the same time. As a result of the determination, if the target content is taken at the same position or at the same time, the process proceeds to the next step S502. This determination is a process necessary for reproducing the relative position and orientation of each subject observed from a specific place, that is, when the photographer takes a picture. For this determination, for example, latitude / longitude information provided by GPS may be used, or information on a place manually input may be used.

  In addition, when there is no significant difference in shooting time among a plurality of contents, it can be estimated that the photographer has not changed the shooting position. Therefore, it is also possible to use the shooting time given to the contents. . However, since it is considered necessary to take into account the measurement error of the sensor and the imaging interval, the same position is considered up to a predetermined range. If the result of determination in step S501 is not that the images were taken at the same position or at the same time, the reordering in this embodiment is not applied, and each content is arranged according to other rules, such as being arranged in chronological order. Let's create a layout. For this reason, the process proceeds to step S504, the flag for creating a layout based on the relative position is turned OFF, and the process ends.

  Next, in step S502, the feature data (local feature amount) acquired in step S401 is compared. There are various methods for comparing images by comparing local feature points or local feature amounts. In this embodiment, RANSAC disclosed in Non-Patent Document 3 is used. Patent Document 5 discloses an outline of the search process.

  As a specific comparison procedure, first, for each local feature amount of the search source image, a local feature amount of the comparison destination image that has the smallest feature distance is described in pairs. Next, three local feature quantities are randomly selected from the search source image, and an affine transformation matrix is calculated from the correspondence of the coordinates with the local feature quantity group of the comparison destination image in which the distance between the features is minimum. Ask. Using this affine transformation matrix, the coordinates of the remaining local features in the search source image are converted to the coordinates of the comparison destination image, and whether there is a pair of local features that minimize the distance between the features in the vicinity. If there is, vote one vote, and if not, do not vote.

  Finally, when the number of votes reaches a predetermined value, it is determined that there is a partially matching area between the search source image and the comparison target image, and the greater the number of votes, the larger the matching area. I think. On the other hand, if the number of votes does not reach a predetermined value, the processing is performed again from the processing of newly selecting three local feature quantities from the search source image and obtaining the affine transformation matrix. Repeat within the specified number of repetition counts. If the number of votes does not exceed a predetermined value even when the number of repetition counts is reached, it is determined that there is no partially matching area, and the comparison process is terminated.

  Then, when there is a partially matching region, a feature amount corresponding to the focused local feature amount in the search source image can be obtained using a pair of the affine transformation matrix and the feature distance that are found to be minimum. it can. At this time, the relationship between the rotation angle between the search source image and the comparison target image can be obtained from the rotation angle at the time of normalizing the direction when obtaining the local feature amount. Furthermore, the scaling relationship between the search source image and the comparison destination image can also be obtained using the scale space for obtaining the local feature amount.

  Next, in step S503, it is determined whether or not correspondence between a plurality of contents can be achieved by the processing in step S502. That is, it is determined whether or not there are a predetermined number or more of local feature amounts of pairs having the smallest feature distance. As a result of this determination, if correspondence can be achieved between a plurality of contents, in step S505, a flag for creating a layout based on the relative position is set to ON, and the process ends. On the other hand, if the result of determination in step S503 is that correspondence cannot be obtained, processing proceeds to step S504, where the flag for creating a layout based on the relative position is turned OFF, and the processing is terminated.

  Here, an example of creating a layout of the photos 102 and 103 shown in FIG. 1B will be described as a specific example. First, in step S401, a local feature amount in an image is acquired as feature data. Next, in step S402, photographs 102 and 103 are selected as contents.

  In step S403, the acquired information is compared for the selected photos 102 and 103. As details of this processing, first, in step S501, the photographing time information is compared. For the two photographs 102 and 103, it is determined that the shooting interval is shorter than the predetermined interval, and both were taken from the same place. Next, in step S502, local feature amounts are compared. As a result, as shown in FIG. 1B, it can be seen that the local area 106 of the photograph 102 and the local area 107 of the photograph 103 are the same part and can be matched. Therefore, in step S503, it is determined that a plurality of contents can be handled, and in step S505, a flag for creating a layout based on the relative position is turned ON.

  Next, in step S404, the position and orientation with respect to the subject are estimated based on the local areas 106 and 107 determined to be the same part. Specifically, since the local area 106 is present at the right end of the photograph 102 and the local area 107 is present at the left end of the photograph 103, the photograph 102 is a photograph of the left part as viewed from the photographer. It can be presumed that the photograph 103 is a photograph of the right part.

  In step S405, based on the estimation result, it is determined that the photograph 102 is arranged on the left side of the page and the photograph 103 is arranged on the right side. In step S406, a layout is created with the arrangement determined in step S405. As a result, the layout 105 shown in FIG. 1C is created. In the present embodiment, the positional relationship in the horizontal direction is estimated. However, the positional relationship in the vertical direction can be estimated as long as there is an effective corresponding portion. In this case, the content is arranged vertically.

  As described above, according to the present embodiment, it is possible to reduce a viewer from being given a contradiction in the positional relationship between subjects in each content, a sense of incongruity caused by a lack of natural line of sight.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the first embodiment, the layout is created by estimating the positional relationship in the horizontal and vertical directions. In the present embodiment, an example in which the layout is created by estimating the positional relationship in the depth direction (front-rear direction) will be described. Note that the configuration and processing procedure of the layout processing apparatus according to this embodiment are basically the same as those in the first embodiment, and a description thereof will be omitted. In the present embodiment, differences from the first embodiment will be described.

  For example, as shown in FIG. 6A, in the real world 601, a scene in which a prism is placed in front of a patterned paper and a cylinder is placed in the back is photographed. FIG. 6B shows a photograph 602 in which a prism is photographed and a photograph 603 in which a cylinder is photographed. Here, when the layout is created when the order of photographing is the order of the photograph 603 and the photograph 602, a layout 604 shown in FIG. 6C is generally created. When arranged in time-sequential order in the left-open album in this way, the size is changed according to the depth, and a layout 604 is formed in which photos are arranged in order from top to bottom. If such a shape is formed, the photos are arranged in the opposite direction to the actual arrangement of the subject, giving the viewer a sense of incongruity. Therefore, in the present embodiment, the position and orientation of each photograph with respect to the imaging apparatus is estimated using the feature data of the image, and the positional relationship between each photograph is determined therefrom, and the layout 605 shown in FIG. 6C is created. With this layout 605, it can be said that the photos are arranged in the same direction as the arrangement of the subject in the real world and are easy to see.

  In this embodiment, the method for estimating the relative position in step S404 is different. As a specific example, the size of the areas determined as the same part is first compared. Since there is a case where an image is magnified as the object is in the back, the size of the same part may be larger in the content in which the object in the back is imaged than in the content in which the object in the foreground is captured. The local feature amount described in the first embodiment is invariant to enlargement / reduction as described above. Therefore, since the comparison is possible even if the size is changed, it can be estimated that the subject of the content in which the common part is greatly reflected is in the back.

  Even when this property does not hold, it is possible to estimate the position and orientation of the subject by performing plane estimation from the feature points of the common part. Since the plane can be obtained if there are at least three points, it is possible to estimate the position and orientation of the subject by estimating the plane from three or more common feature points and determining where the subject is in the plane. It is. Alternatively, since the scale information of the matching feature points is proportional to the size of the corresponding part, the depth can be estimated based on this property.

  A layout 605 shown in FIG. 6C is a result of arrangement in step S406 based on the result estimated in this way. In this embodiment, a two-dimensional plane such as a paper album is targeted for layout, but it is also possible to lay out a three-dimensional display device such as a three-dimensional display in a manner that reproduces the relative positional relationship.

  As described above, according to the present embodiment, not only in the horizontal and vertical directions but also in the depth direction, the viewer cannot contradict the positional relationship between subjects in each content and cannot naturally carry his gaze. Giving a sense of incongruity can be reduced.

(Third embodiment)
Hereinafter, a third embodiment of the present invention will be described. In the present embodiment, an example will be described in which the layout is created by estimating the positional relationship between the horizontal direction and the vertical direction when the same portion does not exist. Note that the configuration and processing procedure of the layout processing apparatus according to this embodiment are basically the same as those in the first embodiment, and a description thereof will be omitted. In the present embodiment, differences from the first embodiment will be described.

  For example, it is assumed that photographs 701 and 702 shown in FIG. 7A are taken in the real world 101 shown in FIG. If the layout is created when the photographed order is the order of the photograph 702 and the photograph 701, the layout is arranged in chronological order, and thus a layout 703 shown in FIG. 7B is generally created. If such a shape is formed, the photos are arranged in the opposite direction to the actual arrangement of the subject, giving the viewer a sense of incongruity. Further, in the method shown in the first embodiment, since the same portion does not exist between the photographs 701 and 702, they are arranged in time series. Therefore, in this embodiment, an example of creating the layout 704 in FIG. 7B by estimating the positional relationship between photographs using image feature data even in such a case will be described.

  The specific processing flow is the same as in the first embodiment, but in this case, the same portion does not exist between the photograph 701 and the photograph 702. Therefore, a photograph 705 including both of the two subjects (a prism and a cylinder) shown in FIG. 7C and including all of the rectangular object 108 is prepared. In step S403, the photograph 705 and The correspondence with the photos 701 and 702 will be taken. Note that the rectangular object 108 need not be entirely included, and a partially included photograph may be used in some cases.

  As a means for obtaining a photograph 705 shown in FIG. 7C, a photograph taken by an arbitrary person may be used, or a photograph may be taken in advance using a super multi-pixel camera. Also, when the camera is turned on, it is always possible to take a picture and combine the pictures, or to take a moving picture and extract several frames from it and combine them. Further, how to take the correspondence and estimate the position and orientation are performed in the same manner as in the first embodiment. As a result, it can be seen that the photograph 701 corresponds to the area 706 in the photograph 705, and the photograph 702 corresponds to the area 707 in the photograph 705. Therefore, the photograph 701 is on the left side and the photograph 702 is on the right side as viewed from the photographer. It is presumed that the part that was in was taken.

  Based on this estimation result, in step S405, it is determined that the photograph 701 is arranged on the left side of the page and the photograph 702 is arranged on the right side. In step S406, a layout is actually created. As a result, a layout 704 shown in FIG. 7B is created. In the example shown in FIG. 7, the positional relationship in the horizontal direction is estimated. However, as long as there is an effective corresponding portion, the positional relationship in the vertical direction can be estimated as in the first embodiment. In this case, the content is arranged vertically.

  As described above, according to the present embodiment, by using content that is not subject to layout, the viewer is given a contradiction in the position and orientation between subjects in each content, a sense of incongruity caused by the fact that they cannot naturally carry their line of sight, etc. Can reduce that.

(Fourth embodiment)
The fourth embodiment of the present invention will be described below. In the present embodiment, an example of creating a layout by estimating the positional relationship in the depth direction when the same portion does not exist will be described. Note that the configuration and processing procedure of the layout processing apparatus according to the present embodiment are basically the same as those of the second embodiment, and a description thereof will be omitted. In the present embodiment, differences from the second embodiment will be described.

  For example, it is assumed that photographs 801 and 802 shown in FIG. 8A are taken in the real world 601 shown in FIG. Then, when the layout is created when the photographed order is the order of the photograph 802 and the photograph 801, they are arranged in chronological order, and generally a layout 803 shown in FIG. 8B is created. If such a shape is formed, the photos are arranged in the opposite direction to the actual arrangement of the subject, giving the viewer a sense of incongruity. Further, in the method shown in the second embodiment, since the same portion does not exist between the photos 801 and 802, they are arranged in chronological order. Therefore, in this embodiment, even in such a case, an example will be described in which the positional relationship between each photograph is estimated using image feature data, and the layout 804 in FIG. 8B is created.

  The specific processing flow is the same as in the second embodiment, but the same portion does not exist between the photograph 801 and the photograph 802. Therefore, as in the third embodiment, a photograph 805 is prepared which includes two subjects (a prism and a cylinder) as shown in FIG. 8C. In step S403, the photograph 805 and photographs 801 and 802 are displayed. We will take action. The means for obtaining this photograph 805 is the same as in the third embodiment.

  Further, how to take the correspondence and estimate the position and orientation are performed in the same manner as in the second embodiment. As a result, since the photograph 801 corresponds to the area 806 in the photograph 805 and the photograph 802 corresponds to the area 807 in the photograph 805, the photograph 801 is on the front and the photograph 802 is on the back as viewed by the photographer. It is estimated that there was.

  Based on the result estimated in this way, the layout 804 shown in FIG. 8B is created as a result of the arrangement in step S406. In the present embodiment, as in the second embodiment, the relative positional relationship is reproduced not only on a two-dimensional plane such as a paper album but also on a three-dimensional display device such as a three-dimensional display. Can be laid out.

  As described above, according to the present embodiment, content that is not subject to layout even in the depth direction is used, and the viewer is inconsistent in the position and orientation between subjects in the content, or feels uncomfortable due to the inability to carry the line of sight naturally. Can be reduced.

(Fifth embodiment)
The fifth embodiment of the present invention will be described below. In the present embodiment, as in the third and fourth embodiments, an example will be described in which a layout is created by estimating the positional relationship between contents when there is no identical portion between two photos. In the present embodiment, the layout is created by using the point that the same object is divided and shown in two photos. Further, the configuration and processing procedure of the layout processing apparatus according to the present embodiment are basically the same as those in the first embodiment, and thus the description thereof is omitted. In the present embodiment, differences from the first embodiment will be described.

  For example, it is assumed that photographs 901 and 902 shown in FIG. 9A are taken in the scene of the real world 101 shown in FIG. Here, the object 907 in the photograph 901 and the object 908 in the photograph 902 are both rectangular objects 108 existing in the real world 101. Therefore, if it can be determined that the objects 907 and 908 are the same object, it is possible to estimate the position and orientation of the subject with respect to the imaging apparatus using this as a clue.

  The specific processing flow is the same as that of the first embodiment, but in this embodiment, the feature data acquired in step S401 and the matching method between a plurality of contents in step S403 are different. In the present embodiment, each side of the object 108 is extracted as a straight line, and a response is taken by comparing the components as features.

  Specifically, first, in step S401, a linear component in the image is extracted. In this process, a known method may be used. For example, a straight line included in the image can be extracted by performing Hough transform on the image. For example, straight lines 903 and 904 are extracted from the photograph 901, and straight lines 905 and 906 are extracted from the photograph 902, respectively.

  In step S403, the linear components extracted from the images are compared. As described above, since there is a restriction that the difference in photographing time or place is within a predetermined range for the image to be used, if there is a set of straight lines having almost the same parameters between the two images, they are continuous. It can be determined that this is a straight line. In the example shown in FIG. 9A, it is determined that the straight line 903 and the straight line 905 are straight lines indicating continuity, and the straight line 904 and the straight line 906 are straight lines indicating continuity. If this is utilized, the object which has a continuous straight line as an edge | side can be considered that it is the same object. That is, it is determined that the object 907 and the object 908 are the same object.

  By using this correspondence, the position and orientation of the subject can be estimated as in the first to fourth embodiments, and the arrangement positions of the photos in the vertical and horizontal directions and in the depth direction can be determined. In the photo 901, an object 907 that is regarded as identical between the two photos exists on the right side of the photo, and in the photo 902, the object 908 also exists on the left side. It is estimated that this is the part that

  Based on the estimation result, in step S405, it is determined that the photograph 901 is arranged on the left side of the page and the photograph 902 is arranged on the right side. In step S406, a layout is actually created. As a result, a layout 909 shown in FIG. 9B is generated. Note that the feature used to acquire the correspondence relationship may not be a linear component. For example, an object shape or color may be used. For example, when an object having no linear component such as a circle is shown in both photographs, it is effective to use color as a feature.

  As described above, according to the present embodiment, even if there is no common part between the contents, the viewer can feel the inconsistency in the position and orientation between the subjects in the contents and the sense of incongruity caused by the fact that the line of sight cannot be carried naturally. Can reduce giving.

(Sixth embodiment)
The sixth embodiment of the present invention will be described below. In the present embodiment, an example of creating a layout by estimating the positional relationship between contents when the correspondence between images cannot be estimated using image feature data will be described. Note that the configuration and processing procedure of the layout processing apparatus according to this embodiment are basically the same as those in the first embodiment, and a description thereof will be omitted. In the present embodiment, differences from the first embodiment will be described.

  In the first to fifth embodiments, local feature amounts, straight lines, colors, and the like have been cited as image feature data in order to estimate the correspondence between images. However, there is a possibility that no correspondence can be obtained by using any feature amount obtained from an image including these. In that case, features other than those that can be extracted from the image are used as an alternative method. For example, sensor information such as an electronic compass may be used as the feature to be used. It is also possible to manually specify the correspondence relationship. Even in consideration of this situation, the basic processing flow is the same as in the first embodiment, but the processing in step S403 is different in this embodiment.

  FIG. 10 is a flowchart illustrating an example of a detailed processing procedure of step S403 of FIG. 4 in the present embodiment. Hereinafter, differences from FIG. 5 will be described.

  If the result of determination in step S503 is that no correspondence can be obtained using image feature data, it is determined in step S1001 whether or not the above-described alternative method can be used. As a result of the determination, if the alternative method cannot be used, the process proceeds to step S504. On the other hand, if the alternative method can be used, the flag using the alternative method is turned ON in step S1002. Thereby, in step S404 of FIG. 4, the positional relationship between each content is estimated by an alternative method.

  As described above, according to the present embodiment, even when the correspondence is not obtained even if the feature amount is used, the viewer is not able to carry the line of sight between the subjects in each content or the line of sight naturally. Giving a feeling of strangeness can be reduced.

(Seventh embodiment)
The seventh embodiment of the present invention will be described below. In the present embodiment, an example of creating a layout of three or more subjects (photos) will be described. Note that the configuration and processing procedure of the layout processing apparatus according to this embodiment are basically the same as those in the first embodiment, and a description thereof will be omitted. In the present embodiment, differences from the first embodiment will be described.

  For example, as shown in FIG. 11A, in a real space 1101, a scene in which a prism is placed on the left side, a cylinder on the center, and a cube on the right side is photographed in the vicinity of the patterned paper. FIG. 11B shows a photograph 1102 in which a prism is photographed, a photograph 1103 in which a cylinder is photographed, and a photograph 1104 in which a cube is photographed.

  As a specific processing flow, first, the position and orientation of each sheet are estimated by the procedure shown in FIG. 4, and the position and orientation are repeated in a combination that can estimate the positional relationship. Then, the entire positional relationship is estimated using a combination of photographs with estimated positional relationships as a clue.

  For example, in the case of photographs 1102, 1103, and 1104, first, the positions and orientations of the photographs 1102 and 1103 are estimated by the same method as in the fifth embodiment, and the photograph 1102 is on the left and the photograph 1103 is on the right as viewed by the photographer. It is presumed that the part was shot. Similarly, for the photos 1103 and 1104, estimation is performed in the same manner as in the first embodiment, and it is estimated that the photograph 1103 is taken on the left side and the photo 1104 is on the right side as viewed from the photographer. . From these two estimation results, it can be estimated that the photos 1102, 1103, and 1104 were taken from the scenes in the order of 1102, 1103, and 1104 from the left as viewed from the photographer.

  Based on this estimation result, in step S405, it is determined that the photograph 1102 is placed on the left side of the page, the photograph 1103 is placed in the center of the page, and the photograph 1104 is placed on the right side of the page. In step S406, a layout is actually created. As a result, a layout 1105 shown in FIG. 11C is generated. In the above example, the method shown in the first and fifth embodiments is used to estimate the positional relationship between two photographs, but of course, the method shown in other embodiments may be used. It is also possible to estimate the positional relationship in the horizontal direction for one set and to estimate the positional relationship in the vertical direction for another set and combine them.

  As described above, according to the present embodiment, even if there are three or more target contents, the viewer can feel the inconsistency in the position and orientation between subjects in each content, the uncomfortable feeling caused by the fact that the line of sight cannot be carried naturally. Can reduce giving.

(Eighth embodiment)
The eighth embodiment of the present invention will be described below. In the present embodiment, an example will be described in which the layout is created by estimating the positional relationship between the contents when the subject is photographed with an inclination. Note that the configuration and processing procedure of the layout processing apparatus according to this embodiment are basically the same as those in the first embodiment, and a description thereof will be omitted. In the present embodiment, differences from the first embodiment will be described.

  For example, assume that a scene of the real world 101 shown in FIG. 1A is photographed, and photographs 1201 and 1202 shown in FIG. 12A are obtained. Here, the photograph 1201 is inclined as compared with the real world 101. For this reason, if the layout is created as it is according to the first to seventh embodiments, it is difficult for the viewer to see. Therefore, in the present embodiment, an example will be described in which the layout is created by using the invariance of the local feature amount to rotation and correcting the inclination so as not to make it difficult to see.

  The basic processing flow is the same as in the first embodiment, but in this embodiment, image rotation is taken into consideration when matching content in step S403. The photos 1201 and 1202 correspond to local regions 1203 and 1204. The local feature amounts in the two local regions are rotated with reference to the main direction as described in the first embodiment, and the direction is normalized. Further, as described above, when the local region is obtained, the relationship of the rotation angle can also be obtained. Therefore, if the angle rotated at this time is stored, the inclination can be corrected.

  Here, the reference selection method for correcting the inclination is to manually select the reference photo, compare the linear components in the photo, and select the one with more horizontal straight lines. There is a method of using a photograph in which all subjects are shown as in the embodiment. Then, after estimating the inclination by these processes, the inclination of the photograph is corrected, and a part of the rectangle inscribed in the subject is cut out to generate a photograph 1205 shown in FIG.

  Based on this correction and the estimation of the position and orientation similar to those of the first embodiment, in step S405, it is determined that the photograph 1205 is arranged on the left side of the page and the photograph 1202 is arranged on the right side of the page. In step S406, a layout is actually created. As a result, a layout 1206 shown in FIG. 12C is generated. In the above example, estimation in the real world similar to that of the first embodiment is shown, but it is of course applicable to the examples shown in other embodiments.

  As described above, according to the present embodiment, even when the target content is tilted, the viewer is given a discrepancy in the position and orientation between subjects in the content, a sense of incongruity caused by the fact that the line of sight cannot be carried naturally. Can reduce that.

(Other embodiments)
In the first to eighth embodiments described above, an example in which a relative position in one direction between images is estimated based on one feature data basically for a photograph has been described. On the other hand, even if it is a moving image, it can be used as a layout contrast if the feature does not change significantly in the captured image. Further, the relative position may be estimated by combining two or more feature data, or may be estimated by combining the relative positions in two or more directions.

  For example, consider the relationship between the relative positions of a prism and a cylinder in the real world 601 shown in FIG. In the second embodiment, only the local feature amount is used only for the relationship between the relative positions of the depths of the two subjects. However, the positional relationship between the two subjects is oblique. Therefore, in order to reproduce this oblique relationship, the positional relationship in the horizontal direction as well as the depth direction is estimated. Further, in this case, not only the local feature amount but also the shape of the object is used as the feature. Thereby, the positional relationship in the oblique direction can be reproduced with the layout. Moreover, the estimation precision of a positional relationship can be improved by using several information. Furthermore, a moving image that does not change the above-described characteristics can be used instead of a still image. As described above, it is possible to more accurately reduce the viewer's inconsistency in the position and orientation between subjects in each content, and the uncomfortable feeling caused by not being able to carry the line of sight naturally.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

301 content feature information acquisition unit 302 content correspondence acquisition unit 303 position and orientation estimation unit 304 content arrangement position determination unit

Claims (12)

Extraction means for extracting features from a plurality of contents in which a series of subjects are partially imaged;
Estimating means for estimating a positional relationship in the real world when the subject is imaged from the features extracted by the extracting means;
Determining means for determining the arrangement position of the plurality of contents based on the result of estimation by the estimation means, and maintaining the positional relationship in the real world ;
The extraction unit extracts a linear component of the content as the feature, and the estimation unit determines, based on the continuity of the linear component, if there is no common part between the plurality of contents. An image processing apparatus characterized by estimating a positional relationship in the real world .   The image processing apparatus according to claim 1, wherein the feature is information on a portion common to the plurality of contents.   The image processing apparatus according to claim 2, wherein the extraction unit calculates a local feature amount from the plurality of contents and extracts information of a portion common to the plurality of contents.   The image processing apparatus according to claim 1, wherein the characteristic is information on a portion corresponding to the plurality of contents.   5. The image processing apparatus according to claim 4, wherein information on a portion corresponding to the plurality of contents is at least one of a linear component, an object color, and an object shape. 6. The image according to claim 4 or 5 , wherein the estimating means estimates a positional relationship by estimating continuity of the subject using information of corresponding portions between the plurality of contents. Processing equipment.   The estimation means further estimates a positional relationship based on content in which all or a part of the series of subjects is imaged and includes a portion common to the plurality of contents. Item 8. The image processing apparatus according to Item 1.   The said estimation means estimates the positional relationship based on the combination of two contents among the said three or more contents, when the said some content is three or more contents. The image processing apparatus according to 1. The relationship between the position estimated by the estimating means, front and rear, right and left, an image processing apparatus according to any one of claims 1-8, characterized in that the vertical relationship. The content, the image processing apparatus according to any one of claim 1 to 9, characterized in that a still image or a moving image. An extraction step of extracting features from a plurality of contents in which a series of subjects are partially imaged,
An estimation step for estimating a positional relationship in the real world when the subject is imaged from the feature extracted in the extraction step;
A determination step for determining an arrangement position of the plurality of contents based on a result of the estimation in the estimation step, and maintaining a positional relationship in the real world ;
In the extraction step, a linear component of content is extracted as the feature, and in the estimation step, when there is no common part between the plurality of contents, the continuity of the linear components causes the plurality of content to be extracted. An image processing method characterized by estimating a positional relationship in the real world . An extraction step of extracting features from a plurality of contents in which a series of subjects are partially imaged,
An estimation step for estimating a positional relationship in the real world when the subject is imaged from the feature extracted in the extraction step;
Based on the estimation result in the estimation step, the computer executes a determination step for determining the arrangement position of the plurality of contents, maintaining a positional relationship in the real world ,
In the extraction step, a linear component of content is extracted as the feature, and in the estimation step, when there is no common part between the plurality of contents, the continuity of the linear components causes the plurality of content to be extracted. A program characterized by estimating positional relationships in the real world .

Images