JP4064186B2 - Method and apparatus for determining the top and bottom of an image - Google Patents

Description

を算出し、
【数２】

により求められる。空と木に対応する画像領域の関係の評価の場合、図１２の参照データ第１行目によれば、木が下にある確率が７５％であって、空が下にある確率５％よりも高いので、上記のΔｘおよびΔｙは、
【数３】

となる。現段階で評価しようとしている画像の向きは図１１（ａ）の向きであり、この向きではｘ_空＞ｘ_木、ｙ_空＝ｙ_木となっているので、Δｘ＞０、Δｙ＝０となり、重み係数はｗ_ｘ＝１、ｗ_ｙ＝０となる。
【００５４】
続いて、現段階の画像方向における当該２つの画像領域の相対的な位置的上下関係の評価値ｓ_ＡＢを、
【数４】

により算出する。ここで、ｐ_{ＡＢ横並び}は参照データによる撮影対象要素ＡとＢが横に並ぶ確率、ｍａｘ（ｐ_{Ａ下、Ｂ上}，ｐ_{Ａ上、Ｂ下}）はｐ_{Ａ下、Ｂ上}とｐ_{Ａ上、Ｂ下}のうち大きい方、ｍｉｎ（ｐ_{Ａ下、Ｂ上}，ｐ_{Ａ上、Ｂ下}）はｐ_{Ａ下、Ｂ上}とｐ_{Ａ上、Ｂ下}のうち小さい方を指す。図１１（ａ）の向きにおける空と木に対応する画像領域の相対的な位置的上下関係については、上述のようにΔｙ＝０であるので、評価値は、
【数５】

となる。
【００５５】
同様に、空と建物、空と地面の相対的な位置的上下関係についても、評価値ｓ_空建物およびｓ_空地面を算出する。これで、空に対応する画像領域を注目領域としたステップ７４の工程が終了する。
【００５６】
次に、ステップ７６に移り、次の注目領域を探索する。ここでは、上述したように本実施例における探索順序は空→山→木→建物→地面であるので、２番目の「山」に対応する画像領域が評価対象の画像中において探索される。しかし、「山」に対応する画像領域はないので、次に「木」に対応する画像領域が探索され、注目領域として特定される。
【００５７】
すると、次の注目領域が特定されたのでステップ７８で「Ｎｏ」となり、ステップ７４に戻って、上記と同様の手法により、「木」に対応する注目領域とその他の画像領域との相対的な位置的上下関係が評価される。ここで、空と木に対応する画像領域の相対的な位置的上下関係については既に評価が行われているので、評価対象から除かれ、木と建物および木と地面について、評価値ｓ_木建物およびｓ_木地面が算出される。
【００５８】
以上のステップ７４、７６および７８からなる繰返しの工程が全ての注目領域について終了すると、ステップ８０へと進む。この段階で、図１３に示したように６組の撮影対象要素に対応する画像領域の図１１（ａ）の画像方向における相対的な位置的上下関係について、評価値ｓ_ＡＢが算出されていることになる。ステップ８０では、それら６つの評価値の合計値Ｓが算出される。この合計値は、現在評価を行っている画像方向が上下が正しく配置された画像方向である可能性を表すパラメータとしての意味を持ち、本実施形態では、この合計値Ｓが大きいほど当該画像方向が正しい可能性が高い。図１３に示すとおり、図１１（ａ）の画像方向が正しい可能性を示す合計値Ｓは、ここでは１５９となる。
【００５９】
次に、ステップ８２を経てステップ８４へ進み、画像の方向が仮想的に９０°回転され、図１１（ｂ）に示す方向とされる。その上でステップ７０から８０の一連の工程が繰り返され、図１１（ｂ）に示す画像方向についても、当該画像方向が正しい可能性を表す合計値Ｓが算出される。同様の工程が、図１１（ｃ）に示す画像方向および図１１（ｄ）に示す画像方向についても、順次行われる。これらの各画像方向についての評価の結果を、図１４−１６に示す。
【００６０】
以上、４方向全ての画像方向の評価が終了すると、ステップ８６へと進み、合計値Ｓの値が最も大きくなった画像方向を、正しい画像方向として特定する。本例では、図１１（ａ）−（ｄ）の各画像方向に対する合計値Ｓがそれぞれ１５９、２、１５９および３６８となっているので、図１１（ｄ）の画像方向が、正しい画像方向として特定される。
【００６１】
以上の各ステップにより、図５の画像の上下が正しく決定され、当該画像は、図１１（ｄ）に示した向きで表示または記録されることになる。
【００６２】
なお、上記の実施形態では、画像全体中における各画像領域の位置を代表する指標として、各画像領域の重心座標を用いたが、これに限らず他の指標を用いてもよい。たとえば、各画像領域をなす画素のうち最も高い上下位置にある画素の座標等を用いてもよい。
【００６３】
また、上記の実施形態のステップ８６では、単純に評価値の合計値Ｓが最大となった方向を正しい画像方向として特定しているが、ある閾値を設定して、４方向に対する合計値Ｓのうち最大のものがその閾値を越えた場合に限り、その最大合計値に対応する方向を正しい画像方向として特定してもよい。あるいは、最大合計値の、該最大合計値と２番目に大きい合計値の和に対する比が、設定された閾値（たとえば０．６）を超える場合に限り、その最大合計値に対応する方向を正しい画像方向として特定してもよい。これらの場合、その設定閾値では画像の上下を判別できなかった画像については、サービス事業者等が目視で画像方向を確認し修正することになる。
【００６４】
また、上記の実施形態においては、各画像方向における全ての評価値の合計値に基づいていずれの画像方向が正しいかを決定しているが、各画像方向における評価値ｓ_ＡＢの最大値等に基づいて決定してもよい。
【００６５】
また、上記の実施形態においては、予め注目領域とその探索順序が規定されていたが、評価対象である画像中に存在する画像領域を全て順次探索することとし、それらの画像領域の全ての組合せについて相対的な位置的上下関係を評価するようにしてもよい。
【００６６】
以上説明した各実施形態によれば、画像の上下は、空、木、地面等の複数の撮影対象要素の実世界における相対的な位置的上下関係に関する参照データに基づいて決定されるので、これらの撮影対象要素のうち少なくとも２つに対応する画像領域を含む画像であれば、上下を決定することができる。第２の実施形態において閾値を設ける場合等には、一部の画像は目視確認および手動操作による向きの修正を必要とすることになるが、それでも本発明では単数ではなく複数の撮影対象要素の相対的な位置的上下関係に基づいて画像の上下を決定するため、かかる目視確認および手動操作が必要とされる画像の割合は、従来に比べて大幅に減少する。
【００６７】
また、上記の各実施形態によれば、複雑なプログラムの変更等を行わなくても、参照データを修正したり新たな撮影対象要素に関するデータを追加することにより、画像の上下を決定する基準を容易に変更することができる。
【００６８】
以上、本発明の２つの実施形態について詳細に述べたが、これら２つの実施形態は例示的なものに過ぎず、本発明の技術的範囲は、本明細書中の特許請求の範囲のみによって定められるべきものである。
【図面の簡単な説明】
【図１】本発明による画像の上下決定工程を含む一連の画像処理工程を示したフローチャート
【図２】画像領域への分割手法の例を示した工程図
【図３】画像領域と撮影対象要素の対応づけの手法の例を示したフローチャート
【図４】図３に示した手法の各工程を示す工程図
【図５】本発明による上下の決定の対象となる画像を示した図
【図６】本発明の第１の実施形態による画像の上下決定方法を示すフローチャート
【図７】本発明の第１の実施形態において使用される参照データの例を示した表
【図８】本発明の第１の実施形態において、画像に上下配置レベルが割り当てられた状態を示す図
【図９】本発明の第１の実施形態における、ｘ方向およびｙ方向に関する上下配置レベルの変化傾向を表す差分値の導出方法を示す図
【図１０】本発明の第２の実施形態による画像の上下決定方法を示すフローチャート
【図１１】本発明の第２の実施形態において評価される画像の４つの方向を示した図
【図１２】本発明の第２の実施形態において使用される参照データの例を示した表
【図１３】本発明の第２の実施形態による、図１１の（ａ）の方向についての評価値を示した表
【図１４】本発明の第２の実施形態による、図１１の（ｂ）の方向についての評価値を示した表
【図１５】本発明の第２の実施形態による、図１１の（ｃ）の方向についての評価値を示した表
【図１６】本発明の第２の実施形態による、図１１の（ｄ）の方向についての評価値を示した表[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a method and apparatus for determining the top and bottom of an image, and more particularly, to a method and apparatus for automatically determining the top and bottom when displaying a photographic image represented by digital data or recording it on a medium.
[0002]
[Prior art]
When photographing with a digital camera, the photographer freely decides the direction of the camera each time. Therefore, when a series of captured images are displayed on a digital camera monitor or personal computer screen, they are displayed in irregular directions, so the orientation is adjusted manually one by one while visually checking each displayed image. There is a need. A similar problem also occurs when a photographic image taken on a negative film is read by a film scanner and a read image represented by digital data is displayed.
[0003]
The burden of the work of adjusting the orientation of the image is not only when the photographer personally displays and records the image, but as a commercial service, the medium on which the image taken with the digital camera is recorded as it is, There is also a problem when a photographed negative film is deposited from a customer, and a series of photographed images are aligned and recorded on a CD-R and provided to the customer, or displayed on a homepage. In particular, when such a commercial service processes a large amount of images, such work is a factor that greatly reduces the efficiency of the service. Therefore, the automation of the work of determining the top and bottom of the image represented by digital data and adjusting the orientation is strongly desired for the business operator who provides the commercial service as described above.
[0004]
As a method that has already been developed for automating such work, the sky area in the captured image is identified, and the image is oriented by determining the top and bottom of the image based on the color gradient of the sky area. There exists a method (for example, refer patent document 1). In other words, this method focuses on the feature that the color of the sky in the real world is bluer at the zenith side and becomes whiter as it approaches the horizon, and prevents a blue region that is not empty from being erroneously detected as a sky region. Thus, the sky region is specified with high reliability, and the orientation of the image is determined from the color gradient.
[0005]
However, the actual photographic image does not always include the sky, and in that case, the top and bottom of the image cannot be determined by the above method. Even if the sky is reflected, the above method cannot be applied in a cloudy state where almost the entire surface of the sky is covered with clouds. Therefore, it is considered that there are a considerable number of images that have not been correctly determined by the above method among the actually captured images. After all, the service providers etc. visually confirm the orientation of each image. It must be trimmed.
[0006]
[Patent Document 1]
JP 2001-202525 A
[0007]
[Problems to be solved by the invention]
As described above, the method of determining the top and bottom of an image based on the color gradient or the like by focusing on only one shooting target element such as “sky” among the shooting target elements that can be included in the shot image is as follows. The percentage of applicable images is low. For this reason, it has not been sufficient to meet the above-mentioned demand for improving the work efficiency in commercial services and the like by reducing the burden of visually checking images one by one and adjusting the orientation by manual operation.
[0008]
In view of such circumstances, the present invention provides an image up / down determination method and apparatus applicable to most of photographic images represented by digital data, and finally requires visual confirmation and correction of orientation by manual operation. It is an object of the present invention to drastically reduce the ratio of images taken and greatly improve the work efficiency in the above-mentioned commercial services.
[0009]
[Means for Solving the Problems]
That is, the method for determining the vertical direction of an image according to the present invention includes a step of creating reference data regarding a positional vertical relationship in the real world of a plurality of imaging target elements, and at least corresponding to at least two of the plurality of imaging target elements. With respect to an image including two image areas, with reference to the reference data, the upper and lower sides of the image are arranged so that the arrangement of the at least two image areas best reflects the positional relationship in the real world. The method includes the step of determining.
[0010]
In the method, the above-described step of determining the top and bottom of the image is performed with reference to the reference data, using the data in which one vertical arrangement level is assigned to each of the plurality of imaging target elements as the reference data. Assigning a vertical arrangement level of the imaging target element corresponding to the image area to each of the at least two image areas, and a change tendency of the assigned vertical arrangement level in the first direction of the entire image. Deriving a first parameter to represent, and deriving a second parameter representing a change tendency of the assigned vertical arrangement level over a second direction orthogonal to the first direction of the entire image. And a step including a step of determining the upper and lower sides of the image based on the first parameter and the second parameter.
[0011]
Alternatively, in the image up / down determination method according to the present invention, for at least a part of the combination of two of the plurality of imaging target elements, the positional vertical relationship in the real world of the two imaging target elements forming the combination. The above-mentioned step of determining the top and bottom of the image using the data representing the probability of the above as the reference data, the step of deriving an index representative of the position in the image of the at least two image regions, Assuming that one of the four directions differing by 90 ° of the image is the upward direction, the positional vertical relationship in the real world of the imaging target elements corresponding to the at least two image regions in the reference data is as follows. Based on the probabilistic data portion and the above-mentioned index representing the position of the at least two image regions in the image, the possibility that the assumption is correct is shown. The process of deriving the parameters, the process of deriving the parameters for the remaining three directions in the four directions of the image, and the most of the four parameters derived from the four directions of the image It is good also as a process including the process of specifying the direction corresponding to the parameter showing high possibility as the up direction of the image concerned.
[0012]
Also, the image up / down determination apparatus according to the present invention includes a storage unit that stores reference data related to a positional relationship in the real world of a plurality of imaging target elements, and the plurality of imaging targets represented by input image data. For an image including at least two image regions corresponding to at least two of the elements, the arrangement of the at least two image regions best reflects the positional vertical relationship in the real world with reference to the reference data. It is an apparatus characterized by including a determining means for determining the top and bottom of the image so as to be arranged.
[0013]
In the apparatus, data in which one upper and lower arrangement level is assigned to each of the plurality of imaging target elements is used as the reference data, and the determination unit refers to the reference data, and the at least two Means for assigning to each of the image areas an upper and lower arrangement level of the imaging target element corresponding to the image area, and a first parameter representing a change tendency of the assigned upper and lower arrangement level in the first direction of the entire image And means for deriving a second parameter representing a change tendency of the assigned vertical arrangement level in a second direction orthogonal to the first direction of the entire image, and Means including means for determining the top and bottom of the image based on the first parameter and the second parameter may be used.
[0014]
Alternatively, in the image up / down determination apparatus according to the present invention, for at least a part of the combination of two of the plurality of imaging target elements, the positional vertical relationship in the real world of the two imaging target elements forming the combination. Is used as the reference data, and the determining means includes means for deriving an index representing the position of the at least two image regions in the image, and 90 ° of the image. Assuming that each of the four different directions is an upward direction, the positional relationship in the real world of the imaging target elements corresponding to the at least two image regions in the reference data is stochastic. The possibility that the assumption is correct is based on the data part expressed in (1) and the above-mentioned index representing the position of the at least two image areas in the image. And means for deriving a parameter corresponding to the parameter representing the highest possibility among the four parameters derived from the four directions of the image as the upward direction of the image. Also good.
[0015]
In the present invention, the “photographing element” refers to an individual element that exists in the real world and can be a subject of photography, and includes, for example, the sky, clouds, mountains, trees, buildings, water, and the ground. . In the present invention, an “image region” is an individual significant region divided in an image handled by the method and apparatus according to the present invention, and each corresponds to one of the above “photographing target elements”. An area that can be identified as a thing.
[0016]
【The invention's effect】
Since the image up / down determination method and apparatus according to the present invention determines the image up / down based on the reference data regarding the relative positional relationship in the real world of a plurality of imaging target elements such as sky, trees, ground, etc. If the image includes an image region corresponding to at least two of these photographing target elements, it is possible to determine the upper and lower sides. Therefore, the ratio of images that can be determined up and down by the method and apparatus according to the present invention is much higher than in the case of the conventional method and apparatus, and finally it is necessary to visually confirm and correct the orientation by manual operation. The ratio of images can be greatly reduced. This greatly improves the efficiency of commercial services in which a large number of images represented by digital data are organized and recorded on a CD-R or displayed on a homepage in response to customer demand.
[0017]
Further, in the image up / down determination method and apparatus according to the present invention, the reference for determining the up / down of the image can be easily changed by correcting / adding the reference data without changing a complicated program or the like. There is also an advantage of being able to.
[0018]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.
[0019]
FIG. 1 is a flowchart showing a flow of a series of image processing steps including an image up / down determination step according to the present invention. This series of image processing steps makes it possible to display or record an original image recorded in an irregular direction so as to be aligned in the correct direction. First, in step 10, digital image data representing an original image is read. Next, in step 12, the original image is divided into a plurality of image areas, and in step 14, each image area is associated with an imaging target element existing in the real world. Subsequently, in step 16, the upper and lower sides of the image are determined with reference to the reference data relating to the positional relationship in the real world of the imaging target element. Finally, in step 18, the original image is rotated and displayed or recorded in the correct orientation according to the top and bottom of the image determined in step 16.
[0020]
The present invention relates to a method and apparatus for realizing step 16 in the series of image processing steps shown in FIG. 1, but prior to the description of the embodiment of the present invention, to the image region in step 12. An example of this dividing method and an example of a method for associating each image region with the imaging target element in step 14 will be briefly described below.
[0021]
An example of the method of dividing into image areas in step 12 will be described with reference to FIG.
[0022]
First, FIG. 2A shows an original image to be processed, and each pixel constituting the original image has each color and brightness. Therefore, the colors and brightness of adjacent pixels are compared, and when the degree of similarity exceeds a predetermined standard, these pixels are integrated. This comparison and integration is sequentially repeated until no further integration occurs according to the predetermined criteria, and the area of pixels having similar colors and lightness is expanded. Assume that the state after the integration of the pixels is the state shown in FIG. 2B, for example.
[0023]
Here, among the areas constituting the image shown in FIG. 2 (b), an area whose peripheral length is shorter than a predetermined length is called a “micro area”, and an area whose peripheral length is equal to or greater than the predetermined length. It shall be called “non-micro area”. In FIG. 2B, the areas 20, 22, 24, 26 and 28 are non-micro areas, and the areas 30 and 32 are micro areas.
[0024]
Next, each area constituting the image of FIG. 2 (b) is further compared with the adjacent area to further integrate what can be integrated. Varies by area. For a micro area, a micro area that is completely contained in one non-micro area (eg, a micro area 30 that is fully contained in a non-micro area 24) is integrated into that one non-micro area. To do. Further, a micro area that borders two or more non-micro areas (for example, micro area 32 that borders non-micro areas 20 and 24) is a non-micro area having a longer border length (in the above case, non-micro area). It is assumed that they are integrated into the micro area 24). According to this standard, the state after the integration of the minute areas is as shown in FIG.
[0025]
For non-micro areas, the average color and lightness of the pixels forming the non-micro area are compared with the average color and lightness of the pixels forming each adjacent non-micro area, and the degree of similarity exceeds a predetermined standard. Integration is performed if there are non-micro areas. For example, for the average color and lightness of the non-micro area 22 in FIG. 2 (c), the degree of similarity to the average color and lightness of the non-micro area 26 exceeds the predetermined criteria described above, but each other adjacent non-area If the degree of similarity of the average color and lightness of the micro-areas 20, 24 and 28 is below the predetermined criteria, the non-micro-area 22 is integrated with the non-micro-area 26 and the non-micro-area 20, 24 and 28 are not integrated. The final result of the integration of the non-micro area according to such a predetermined standard is, for example, as shown in FIG. Each area constituting the final image is an “image area” in the present invention.
[0026]
The example of the method for dividing the image area in step 12 in FIG. 1 has been described with reference to FIG. 2. Needless to say, the division into the image area in step 12 may be performed by any other known method. Yes.
[0027]
Next, an example of a method for associating each image region with the imaging target element in step 14 in FIG. 1 will be described with reference to FIGS. 3 and 4. In this example, each image region is associated with an imaging target element using a self-organizing map created in advance.
[0028]
First, in step 40 of FIG. 3, an image that has been divided into image regions by the method already described with reference to FIG. 2 is divided into a plurality of blocks as shown in FIG. Here, for example, a square block made up of 32 × 32 pixels is used.
[0029]
Next, in step 42, a block included in one image area to be associated with the imaging target element is specified. Here, first, assuming that the image region 28 in FIG. 4A is associated with the imaging target element, the image region 28 includes a plurality of blocks shown by shading in FIG. Identified as a block.
[0030]
Subsequently, in step 44, a feature vector is extracted for one of the blocks identified in step 42. This "feature vector" is a vector having a plurality of feature quantities for the block as components, and in addition to the components indicating color features and lightness features, these components are images such as depth information. Ingredients that exhibit these characteristics may also be included.
[0031]
Next, in step 46, the feature vector extracted in step 44 is mapped onto the self-organizing map, and an imaging target element corresponding to the block is specified. The “self-organizing map” is a map in which points corresponding to a plurality of reference feature vectors are two-dimensionally arranged, and points corresponding to reference feature vectors of images having similar features are arranged at positions close to each other. Has been. This self-organizing map is created in advance by letting a computer learn the characteristics of a large number of images that are known to be “sky” and “ground” images. An imaging target element that is most likely to correspond to each point is known in advance. The feature vector extracted in step 44 is mapped to a point corresponding to the most similar reference feature vector on this self-organizing map. Here, the similarity between the extracted feature vector and each reference feature vector is evaluated using the Euclidean distance between the vectors as an index. By mapping onto the self-organizing map, it is possible to identify the imaging target element that is most likely to correspond to the current block.
[0032]
Subsequently, the processing of FIG. 3 returns to step 44 through step 48, and the corresponding photographing target element is similarly specified for the following blocks among the shaded blocks in FIG. 4B.
[0033]
When the imaging target elements corresponding to all the blocks are identified, the process in FIG. At this time, as shown in FIG. 4C, most of the blocks in the image area 28 are identified as corresponding to “empty”. However, some of the image features of the block are “empty”. For example, it is closer to the feature of “water” than “,” and those specified as corresponding to “water” are mixed. However, in step 50, the largest number among the elements to be photographed corresponding to the respective blocks in the image area 28 is identified as the element to be photographed corresponding to the image area 28 by a so-called majority method. Therefore, as shown in FIG. 4D, the image area 28 corresponds to “empty”.
[0034]
Thereafter, the processing of FIG. 3 returns to step 42 through step 52, and steps 42 to 52 are repeated until the corresponding imaging target elements are specified for all image regions included in the image.
[0035]
The example of the method for associating each image region with the imaging target element in step 14 in FIG. 1 has been described with reference to FIGS. 3 and 4. The association in this step 14 is based on any other known method. It goes without saying that things can be used.
[0036]
Now, based on the premise that the image to be determined up and down is divided into image areas by the above-described method and the like, and the imaging target area corresponding to each image area is specified, the image upper and lower according to the present invention will be described below. Two embodiments of the determination method (ie, step 16 of FIG. 1) will be described.
[0037]
First, the 1st Embodiment of this invention is described using FIGS. 5-9. In this embodiment, the upper and lower sides of an image are determined using data in which one vertical arrangement level is assigned to each of a plurality of imaging target elements as reference data. Here, it is considered to determine the upper and lower sides of the image shown in FIG. 5 that is divided into a plurality of image regions by using the method described above, and in which each image region is associated with the imaging target element existing in the real world. . The image is composed of M × N pixels. At this stage, the meaning of “sky”, “tree”, “building” or “ground” is assigned to each pixel in each image area. The x and y coordinates of the image are as shown in the figure. Note that the image in FIG. 5 is simplified from the images in FIGS. 2 and 4, but this is merely for convenience of the following description.
[0038]
Image up / down determination according to the present embodiment is performed according to the procedure shown in the flowchart of FIG.
[0039]
First, in step 60, an upper and lower arrangement level corresponding to a corresponding photographing target element is assigned to each image region of the image of FIG. Strictly speaking, a corresponding vertical arrangement level is assigned to each pixel belonging to each image area. This assignment is performed with reference to tabular reference data as shown in FIG. This reference data is created in advance based on the relative positional relationship between the elements to be imaged in the real world and is stored in the storage means, and is likely to be relatively high in the real world. The higher the vertical arrangement level is given to the target element. According to this reference data, in this example, the upper and lower arrangement levels are assigned as shown in FIG.
[0040]
Next, in steps 62 and 62 ′, the image is divided into two equal parts as shown in FIG. 9 for each of the x direction and the y direction, and the average value of the upper and lower arrangement levels assigned to the pixels in each of the bisected areas. Is calculated. In the example of FIG. 9, the average value of the upper and lower arrangement levels in each bisected area in the x direction is 0.9 and 1.8, and the average value of the upper and lower arrangement levels in each bisector area in the y direction is Let 1.2 and 1.3.
[0041]
Subsequently, in steps 64 and 64 ′, the average of the upper and lower arrangement levels in each of the bisected regions in each of the x direction and the y direction is used as a parameter representing the change tendency of the upper and lower arrangement levels in the x direction and the y direction. The difference value of the value is obtained. Here, with respect to the x direction, in FIG. 9, the average value in the left bisection region is subtracted from the average value in the right bisection region, and in the y direction, in the lower bisection region. The average value in the upper bisection region is subtracted from the average value. That is, in the example of FIG. 9, the difference value in the x direction is 1.8−0.9 = + 0.9, and the difference value in the y direction is 1.3−1.2 = + 0.1.
[0042]
Next, in step 66, the difference value in the x direction is compared with the difference value in the y direction, and the axis corresponding to the larger absolute value is specified as the vertical axis of the image. In the example of FIG. 9, as described above, the difference value in the x direction is +0.9, the difference value in the y direction is +0.1, and the absolute value of the difference value in the x direction is larger. Identified as an axis.
[0043]
Subsequently, in step 68, from the sign of the above difference value with respect to the specified vertical axis, if the sign is positive, the + x or + y direction is positive, if it is negative, the -x or -y direction is the upward direction of the image. Identified as Here, in step 66, the x axis is specified as the vertical axis, and the difference value in the x direction is a positive value of +0.9, so the + x direction is specified as the upward direction.
[0044]
Through the above steps, the upward direction of the image shown in FIG. 5 is determined to be the + x direction. From this result, the image in FIG. 5 is rotated 90 ° to the left and displayed or recorded.
[0045]
In the above-described embodiment, the image is divided into two equal parts in each of the x direction and the y direction, and the upper and lower parts of the image are determined using the difference value of the average value of the upper and lower arrangement levels in each of the bisection areas as a parameter. However, any other parameter may be used as long as it appropriately represents the change tendency of the vertical arrangement level in the x and y directions of the entire image. For example, as one modified example of the above-described embodiment, the average value of the inclination for each row and column of the vertical arrangement level assigned to each pixel is obtained in the x direction and the y direction, and the average value is calculated as x An example of using it as a parameter representing the change tendency of the vertical arrangement level in the direction and the y direction can be considered.
[0046]
Next, a second embodiment of the present invention will be described with reference to FIGS. This embodiment uses, as reference data, data that stochastically represents the positional relationship in the real world of two imaging target elements forming the combination for at least a part of a combination of two of the plurality of imaging target elements. To determine the top and bottom of the image. Here, the specific procedure will be described using the image shown in FIG. 5 as in the description of the first embodiment.
[0047]
Image up / down determination according to the present embodiment is performed according to the procedure shown in the flowchart of FIG.
[0048]
First, in step 70, the center-of-gravity coordinates of each image region are specified as shown in FIG. 11A for the image of FIG. 5 that has already been divided into image regions and associated with each image region and the imaging target element. . This barycentric coordinate is used as an index representing the position of each image region in the entire image in the subsequent steps.
[0049]
Next, in step 72, a search for “region of interest” is performed. The “attention area” refers to an image area corresponding to an image area included in an image that is designated as an imaging target element to be particularly noted for determining the upper and lower sides of the image. That is, the top and bottom of the image in the present embodiment is determined based on the relative positional top and bottom relationship between this attention area and another image area. For example, the image in FIG. 11A includes image areas corresponding to four shooting target areas of the sky, a tree, a building, and the ground. Here, only the image area corresponding to the sky is designated as the attention area. If so, the top and bottom of the image is determined based on the relative positional relationship between the sky and the tree, the sky and the building, and the sky and the ground. When image areas corresponding to five imaging target elements of sky, mountain, tree, building, and ground are designated as attention areas, the image of FIG. 11A includes the sky, tree, and building. And the image area corresponding to four of the ground, sky and tree, sky and building, sky and ground, tree and building, tree and ground, and relative positional top and bottom relationship between building and ground Thus, the top and bottom of the image are determined. Which image area corresponding to which imaging target element is to be used as the attention area, and if there are a plurality of attention areas, which of the attention areas to examine the relative positional relationship with other image areas (that is, search) The order is assumed to be specified in advance. In the following, image areas corresponding to five shooting target elements of sky, mountain, tree, building, and ground are designated as attention areas, and the search order is empty → mountain → tree → building → ground. A case will be described as an example. Therefore, in step 72, first, an image area corresponding to the sky is searched for as an attention area. Since the image of FIG. 11A includes an image area corresponding to the sky, the process proceeds to the next step 74 using this as an attention area.
[0050]
In step 74, the relative positional relationship between the attention area corresponding to the sky and all other image areas (that is, image areas corresponding to trees, buildings, and the ground) is evaluated.
[0051]
In the present embodiment, this evaluation is performed with reference to the tabular reference data shown in FIG. This reference data is a probabilistic representation of the positional relationship in the real world for all or part of the two possible combinations of the imaging target elements. For example, the data part in the first row of FIG. 12 shows the probability that the sky is above the tree, the probability that the sky and the tree are arranged side by side, and the sky is more than the tree in the real world landscape. It represents the probability below. This reference data is created in advance and stored in the storage means. Note that FIG. 12 shows only the data portion necessary for the description of the present embodiment, but in actuality, this reference data defines relationships among many other imaging target elements.
[0052]
Evaluation of the relationship between two image regions, one of which is a region of interest, starts with reading out a data portion that defines the relationship between two imaging target elements corresponding to the two image regions in the reference data of FIG. In this embodiment, first, in order to evaluate the relationship between the attention area corresponding to the sky and the image area corresponding to the tree, the data portion of the first row in FIG. 12 is read.
[0053]
Next, based on the read data portion information and the barycentric coordinates specified in step 70, weighting factors (w _x , W _y ) Is calculated. Where the weighting factor (w _x , W _y ) Represents the barycentric coordinates of the image areas corresponding to the two shooting target areas A and B, respectively (x _A , Y _A ) And (x _B , Y _B ) As a probability p that the object A to be photographed is at the bottom and B is at the top according to the read data portion _{Below A, above B} , And the probability p that the subject A is above and B is below _{Above A, Below B} On the basis of the,
[Expression 1]

To calculate
[Expression 2]

Is required. In the evaluation of the relationship between the sky and the image area corresponding to the tree, according to the first row of reference data in FIG. 12, the probability that the tree is below is 75% and the probability that the sky is below is 5%. Since the above Δx and Δy are
[Equation 3]

It becomes. The orientation of the image to be evaluated at this stage is the orientation of FIG. 11A, and in this orientation, x _Sky > X _wood , Y _Sky = Y _wood Therefore, Δx> 0, Δy = 0, and the weight coefficient is w _x = 1, w _y = 0.
[0054]
Subsequently, the evaluation value s of the relative positional relationship between the two image areas in the current image direction. _AB The
[Expression 4]

Calculated by Where p _{AB side by side} Is the probability that the imaging target elements A and B are arranged side by side with reference data, max (p _{Below A, above B} , P _{Above A, Below B} ) Is p _{Below A, above B} And p _{Above A, Below B} Whichever is larger, min (p _{Below A, above B} , P _{Above A, Below B} ) Is p _{Below A, above B} And p _{Above A, Below B} The smaller of the two points. Since the relative positional relationship between the image area corresponding to the sky and the tree in the direction of FIG. 11A is Δy = 0 as described above, the evaluation value is
[Equation 5]

It becomes.
[0055]
Similarly, for the relative positional relationship between the sky and the building and between the sky and the ground, the evaluation value s _{Sky building} And s _{Empty ground} Is calculated. This completes the process of step 74 in which the image area corresponding to the sky is the attention area.
[0056]
Next, the process moves to step 76, and the next attention area is searched. Here, as described above, the search order in this embodiment is sky → mountain → tree → building → ground, so the image region corresponding to the second “mountain” is searched in the image to be evaluated. However, since there is no image area corresponding to “mountain”, the image area corresponding to “tree” is searched next and specified as the attention area.
[0057]
Then, since the next region of interest has been identified, “No” is determined in step 78, and the process returns to step 74, and the region of interest corresponding to “tree” is compared with other image regions by the same method as described above. The positional relationship is evaluated. Here, since the relative positional relationship between the image areas corresponding to the sky and the tree has already been evaluated, it is excluded from the evaluation target, and the evaluation value s for the tree and the building and the tree and the ground is excluded. _{Wooden building} And s _{Wood ground} Is calculated.
[0058]
When the repetitive process composed of steps 74, 76 and 78 is completed for all the attention areas, the process proceeds to step 80. At this stage, as shown in FIG. 13, the evaluation value s regarding the relative positional relationship in the image direction of FIG. 11A of the image regions corresponding to the six sets of imaging target elements. _AB Is calculated. In step 80, the total value S of these six evaluation values is calculated. This total value has a meaning as a parameter indicating the possibility that the image direction currently being evaluated is an image direction in which the top and bottom are correctly arranged. In this embodiment, the larger the total value S, the more the image direction. Is likely to be correct. As shown in FIG. 13, the total value S indicating the possibility that the image direction in FIG.
[0059]
Next, the process proceeds to step 84 through step 82, and the direction of the image is virtually rotated by 90 ° to be the direction shown in FIG. Then, a series of steps 70 to 80 are repeated, and a total value S representing the possibility that the image direction is correct is calculated for the image direction shown in FIG. Similar steps are sequentially performed for the image direction shown in FIG. 11C and the image direction shown in FIG. The evaluation results for each of these image directions are shown in FIGS. 14-16.
[0060]
When the evaluation of all the image directions in the four directions is completed, the process proceeds to step 86, and the image direction in which the total value S is the largest is specified as the correct image direction. In this example, the total values S for the respective image directions in FIGS. 11A to 11D are 159, 2, 159, and 368, respectively. Therefore, the image direction in FIG. Identified.
[0061]
Through the above steps, the top and bottom of the image in FIG. 5 are correctly determined, and the image is displayed or recorded in the orientation shown in FIG.
[0062]
In the above embodiment, the barycentric coordinates of each image area are used as an index representing the position of each image area in the entire image. However, the present invention is not limited to this, and other indices may be used. For example, the coordinates of the pixel at the highest vertical position among the pixels forming each image area may be used.
[0063]
In step 86 of the above-described embodiment, the direction in which the total value S of the evaluation values is maximum is simply specified as the correct image direction. Only when the maximum value exceeds the threshold value, the direction corresponding to the maximum total value may be specified as the correct image direction. Alternatively, the direction corresponding to the maximum total value is correct only when the ratio of the maximum total value to the sum of the maximum total value and the second highest total value exceeds a set threshold (for example, 0.6). The image direction may be specified. In these cases, with respect to an image for which the upper and lower sides of the image cannot be determined with the set threshold value, the service provider or the like visually confirms and corrects the image direction.
[0064]
In the above embodiment, which image direction is correct is determined based on the total value of all evaluation values in each image direction. However, the evaluation value s in each image direction is determined. _AB It may be determined based on the maximum value of.
[0065]
Further, in the above embodiment, the attention area and the search order thereof are defined in advance. However, all the image areas existing in the image to be evaluated are sequentially searched, and all combinations of these image areas are performed. You may make it evaluate the relative positional relationship between about.
[0066]
According to each of the embodiments described above, the top and bottom of the image are determined based on the reference data regarding the relative positional top and bottom relationships in the real world of a plurality of shooting target elements such as the sky, trees, and the ground. If the image includes an image area corresponding to at least two of the imaging target elements, the upper and lower sides can be determined. When a threshold value is provided in the second embodiment, some images require visual confirmation and manual orientation correction. However, in the present invention, a plurality of imaging target elements are not used in the present invention. Since the top and bottom of the image is determined based on the relative positional top and bottom relationship, the proportion of images that require such visual confirmation and manual operation is greatly reduced compared to the conventional case.
[0067]
In addition, according to each of the above embodiments, the reference for determining the top and bottom of the image can be determined by correcting the reference data or adding data related to a new photographing target element without changing a complicated program. It can be easily changed.
[0068]
The two embodiments of the present invention have been described in detail above. However, these two embodiments are merely illustrative, and the technical scope of the present invention is defined only by the claims in this specification. It should be done.
[Brief description of the drawings]
FIG. 1 is a flowchart showing a series of image processing steps including an image up / down determination step according to the present invention.
FIG. 2 is a process diagram illustrating an example of a method for dividing an image area.
FIG. 3 is a flowchart showing an example of a method for associating an image area with an imaging target element.
4 is a process diagram showing each process of the method shown in FIG. 3;
FIG. 5 is a diagram showing an image to be determined up and down according to the present invention.
FIG. 6 is a flowchart showing an image up / down determination method according to the first embodiment of the present invention;
FIG. 7 is a table showing an example of reference data used in the first embodiment of the present invention.
FIG. 8 is a diagram showing a state in which an upper and lower arrangement level is assigned to an image in the first embodiment of the present invention.
FIG. 9 is a diagram showing a method for deriving a difference value representing a change tendency of the vertical arrangement level with respect to the x direction and the y direction in the first embodiment of the present invention.
FIG. 10 is a flowchart showing an image up / down determination method according to the second embodiment of the present invention;
FIG. 11 is a diagram showing four directions of an image evaluated in the second embodiment of the present invention.
FIG. 12 is a table showing an example of reference data used in the second embodiment of the present invention.
13 is a table showing evaluation values in the direction of FIG. 11A according to the second embodiment of the present invention.
14 is a table showing evaluation values in the direction of FIG. 11B according to the second embodiment of the present invention.
FIG. 15 is a table showing evaluation values in the direction of (c) of FIG. 11 according to the second embodiment of the present invention.
FIG. 16 is a table showing evaluation values in the direction of FIG. 11 (d) according to the second embodiment of the present invention.

Claims (4)

Creating reference data that assigns one vertical arrangement level to each of the plurality of imaging target elements, with respect to the positional vertical relationship in the real world of the plurality of imaging target elements;
With respect to an image including at least two image areas corresponding to at least two of the plurality of imaging target elements, the imaging corresponding to the image area is referred to in each of the at least two image areas with reference to the reference data. Assigning the upper and lower arrangement levels of the target element;
Deriving a first parameter representing a change tendency of the assigned top and bottom arrangement level across a first direction of the entire image;
Deriving a second parameter representing a change tendency of the assigned top and bottom arrangement level over a second direction orthogonal to the first direction of the entire image;
Based on the first parameter and the second parameter, the upper and lower sides of the image are determined so that the arrangement of the at least two image areas best reflects the positional vertical relationship in the real world. An image up / down determination method comprising a step. Regarding at least a part of a combination of two of the plurality of imaging target elements with respect to a positional vertical relationship in the real world of the plurality of imaging target elements, the positional relationship in the real world of the two imaging target elements forming the combination Creating reference data that stochastically represents the hierarchical relationship;
Deriving an index representative of the position of the at least two image areas in the image for an image including at least two image areas corresponding to at least two of the plurality of imaging target elements ;
Assuming that one of the four directions that differ by 90 ° of the image is the upward direction, the position in the real world of the imaging target element corresponding to the at least two image regions in the reference data Deriving a parameter representing the likelihood that the assumption is correct based on a data portion that stochastically represents a hierarchical relationship and the index representative of the position of the at least two image regions in the image;
Repeating the step of deriving the parameters for the remaining three directions of the four directions of the image;
Represents the highest possibility of the four parameters derived for the four directions of the image so that the arrangement of the at least two image regions best reflects the positional top-bottom relationship in the real world. A method for determining the up / down direction of an image, comprising the step of specifying a direction corresponding to a parameter as the upward direction of the image . Storage means for storing reference data in which one vertical arrangement level is assigned to each of the plurality of imaging target elements, with respect to the positional vertical relationship in the real world of the plurality of imaging target elements;
With respect to an image including at least two image areas corresponding to at least two of the plurality of imaging target elements, the imaging corresponding to the image area is referred to in each of the at least two image areas with reference to the reference data. Means for assigning the upper and lower arrangement levels of the target element;
Means for deriving a first parameter representing a change tendency of the assigned top and bottom arrangement level across a first direction of the entire image;
Means for deriving a second parameter representing a change tendency of the assigned top and bottom arrangement level over a second direction orthogonal to the first direction of the entire image;
Based on the first parameter and the second parameter, the upper and lower sides of the image are determined so that the arrangement of the at least two image areas best reflects the positional vertical relationship in the real world. An apparatus for determining the up and down of an image, characterized in that it includes means. Regarding at least a part of a combination of two of the plurality of imaging target elements with respect to a positional vertical relationship in the real world of the plurality of imaging target elements, positions of the two imaging target elements forming the combination in the real world Storage means in which reference data that stochastically represents the hierarchical relationship is stored ,
Means for deriving an index representative of the position of the at least two image areas in the image for an image including at least two image areas corresponding to at least two of the plurality of imaging target elements ;
The real world of the imaging target element corresponding to the at least two image regions in the reference data, assuming that the direction is an upward direction for each of four directions that differ by 90 ° in the image. A parameter representing the likelihood that the assumption is correct is derived based on a data portion that stochastically represents the positional relationship in FIG. 5 and the index representative of the position of the at least two image regions in the image. Means,
Represents the highest possibility of the four parameters derived for the four directions of the image so that the arrangement of the at least two image regions best reflects the positional top-bottom relationship in the real world. An apparatus for determining an up / down direction of an image, comprising means for specifying a direction corresponding to a parameter as the upward direction of the image .

Images