KR20110074442A - Image processing apparatus, image processing method and recording medium - Google Patents

Abstract

PURPOSE: An image processor, an image processing method, and a recording medium are provided to display an image which reflects information about a light source. CONSTITUTION: A light source number calculator(120) produces the number of light sources. A light source information calculator(140) produces parameters for the light source in the real space as parameters in a 3D space on the basis of produced light source coordinate information. A 3D image preparing unit(150) prepares a 3D image on the basis of 3D model information and the produced light source information. A display unit(160) displays the prepared 3D image.

Description

Image processing apparatus, image processing method and recording medium

The present invention relates to an image processing apparatus, an image processing method and a recording medium.

In recent years, various kinds of image processing techniques are performed on images captured by a camera. In particular, techniques have been developed to improve the inaccurate reproduction of color information of each part of an object according to a light source or an illumination.

An embodiment of the present invention is to provide an image processing apparatus, an image processing method, and a recording medium capable of acquiring information on a light source existing in a real space and displaying an image reflecting information on the light source.

In order to achieve the above technical problem, an image processing apparatus according to an embodiment of the present invention includes a photographing unit having a fisheye lens and photographing an image through the fisheye lens; A storage unit for storing three-dimensional model information defining a three-dimensional space; A light source number calculating unit for calculating the number of light sources captured by the image capturing unit and calculating light source coordinate information indicating a position in the image of the light source corresponding to the calculated number; A light source information calculation unit calculating a parameter in a real space with respect to the light source based on the calculated light source coordinate information as light source information that is a parameter in the three-dimensional space; A three-dimensional image creating unit which creates a three-dimensional image based on the stored three-dimensional model information and the calculated light source information; And a display unit for displaying the created three-dimensional image.

The light source information calculating unit may calculate, as the light source information, azimuth information indicating an orientation in a real space of the light source based on the position of the photographing unit by projection conversion based on the calculated light source coordinate information.

The light source information calculating unit calculates position information indicating the position of the light source in the real space based on the orientation information and a predetermined value as the light source information.

The photographing unit includes at least two fisheye lenses, photographing each image through the at least two fisheye lenses,

The light source number calculator calculates a light source number and light source information for each image photographed by the photographing unit,

The light source information calculating unit calculates orientation information for each image photographed by the photographing unit, and calculates, as the light source information, position information indicating a position in the real space of the light source based on the calculated orientation information. Characterized in that.

The light source information calculating unit may calculate at least one of the intensity and the color of the light generated by the light source based on the calculated light source coordinate information as the light source information.

The light source number calculating unit detects a light source region corresponding to the calculated number from the image, calculates a position in the image of the detected light source region as light source region coordinate information,

The light source information calculating unit may calculate, as the light source information, the spread of light generated by the light source based on the light source region coordinate information calculated by the light source number calculating unit.

The light source information calculating unit approximates the light source region to two vectors based on the light source region coordinate information calculated by the light source number calculating unit, and calculates the spread of the light as the light source information by the two vectors. It is done.

The light source information calculating unit calculates at least one of the intensity and color of the environmental light in the real space as the environmental light information as a parameter in the three-dimensional space based on the light source coordinate information calculated by the light source number calculating unit,

The three-dimensional image creating unit specifies an object color in the three-dimensional space based on the ambient light information calculated by the light source information calculating unit, and generates a three-dimensional image having the specific object color. .

The three-dimensional image creating unit calculates the direction and number of shadows generated in the three-dimensional space based on the light source information calculated by the light source information calculating unit and the three-dimensional model information stored in the storage unit, and calculates the calculated number. A shadow area is specified based on the direction and number of shadows, and a three-dimensional image having a shadow in the specific shadow area is created.

The three-dimensional image creating unit specifies a half shadow area generated in the three-dimensional space based on the positional information calculated as the light source information by the light source information calculating unit and the three-dimensional model information stored in the storage unit, and specifies the specific region. And a three-dimensional image having a half shadow in the half shadow area.

And the light source information calculating unit compares with the previous light source information which is the light source information previously used for creating the three-dimensional image. Determine whether the current light source information has changed,

The three-dimensional image creating unit re-creates the three-dimensional image when it is determined by the light source information calculating unit that the current light source information has been changed from the previous light source information,

And the display unit displays the recreated three-dimensional image again.

The photographing unit may receive an image from the outside of the image processing apparatus through the fisheye lens to capture an image.

In accordance with another aspect of the present invention, an image processing method includes: photographing an image through a fisheye lens; Calculating the number of light sources imprinted on the photographed image, and calculating light source coordinate information indicating a position in the image of the light source corresponding to the calculated number; Calculating a parameter in real space with respect to the light source based on the calculated light source coordinate information as light source information that is a parameter in the three-dimensional space; Creating a three-dimensional image based on the three-dimensional model information defining the three-dimensional space stored in advance and the calculated light source information; And displaying the created three-dimensional image.

In order to achieve the another technical problem, a recording medium including a program for executing the image processing method on a computer according to another embodiment of the present invention is recorded.

According to an embodiment of the present invention, information about a light source existing in a real space may be acquired to display an image reflecting information about the light source.

1 is a schematic diagram of an image processing apparatus 100 according to an embodiment of the present invention.
FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus 100 shown in FIG. 1.
3 is a view for explaining the function of the light source number calculation unit 120 shown in FIG.
4 is a diagram for describing a function of the light source information calculating unit 140 shown in FIG. 2.
5A is a diagram illustrating a display example of an image processed by the image processing apparatus 100 shown in FIG. 1.
5B is a diagram showing a display example of an image displayed by a general image display apparatus.
FIG. 6A is a diagram illustrating a display example of an image processed by the image processing apparatus 100 shown in FIG. 1.
FIG. 6B is a diagram illustrating a display example of an image processed by the image processing apparatus 100 shown in FIG. 1.
6C is a diagram illustrating a display example of an image processed by the image processing apparatus 100 shown in FIG. 1.
FIG. 7 is a diagram for explaining that the light source information calculating unit 140 shown in FIG. 2 calculates a direction of a light source as light source information.
FIG. 8 is a diagram for explaining elevation angles and azimuth angles used to calculate the orientation of a light source.
FIG. 9 is a view for explaining that the light source information calculating unit 140 shown in FIG. 2 calculates the position of the light source as the light source information.

As the inventive concept allows for various changes and numerous embodiments, particular embodiments will be illustrated in the drawings and described in detail in the written description. However, this is not intended to limit the present invention to specific embodiments, it should be understood to include all transformations, equivalents, and substitutes included in the spirit and scope of the present invention. In the following description of the present invention, if it is determined that the detailed description of the related known technology may obscure the gist of the present invention, the detailed description thereof will be omitted.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from another.

The terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "comprise" or "have" are intended to indicate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, and one or more other features. It is to be understood that the present invention does not exclude the possibility of the presence or the addition of numbers, steps, operations, components, components, or a combination thereof.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and in the following description with reference to the accompanying drawings, the same or corresponding components will be given the same reference numerals and redundant description thereof will be omitted. do.

1 is a schematic diagram of an image processing apparatus 100 according to an embodiment of the present invention.

As shown in FIG. 1, the image processing apparatus 100 according to the embodiment of the present invention includes at least a display unit 160, and the display unit 160 displays an image processed by the image processing apparatus 100. . For example, as shown in FIG. 1, the display unit 160 displays an object 161 for a user to input operation information to the image processing apparatus 100.

For example, the image processing apparatus 100 is provided with a light source L1 and a light source L2 around it, and is displayed by the display unit 160 by the light generated by the light source L1 and the light source L2. The object 161 is irradiated. In this case, to indicate that the object 161 is three-dimensionally configured, it is preferable that a shadow is displayed on the lower right side of the object 161 by irradiating the light generated by the light source L1 to the object 161 (object). (161a)). The lower right in this case is a direction opposite to the direction in which the light source L1 exists based on the object 161. Therefore, the user can see the real image through the image processing apparatus 100.

Similarly, it is preferable that the light generated by the light source L2 is irradiated to the object 161 so that a shadow is displayed at the lower left of the object 161 (see object 161b). The lower left side in this case becomes a direction opposite to the direction in which the light source L2 exists based on the object 161. In addition, it is preferable that the light generated by both the light source L1 and the light source L2 is irradiated to the object 161 so that a shadow is displayed on the lower right and lower left of the object 161 (see object 161c). In particular, the area corresponding to the bottom of the object 161 is preferably displayed as a relatively dark shadow because both shadows overlap.

In order to realize this, it is assumed that the image processing apparatus 100 according to the embodiment of the present invention includes the photographing unit 110. The imaging unit 110 has at least a fisheye lens 111 and an imaging device 112, and the imaging device 112 receives an image from the outside of the image processing apparatus 100 through the fisheye lens 111 to capture an image. Has the function. In the embodiment of the present invention, the fisheye lens 111 is used as a lens used for capturing an image as described above, instead of a flat lens (hereinafter referred to as a normal lens). Since the angle of view of the lens is usually narrower than 180 degrees, there is a possibility that the light source for generating the light irradiating the object 161 does not converge within the shooting range. However, since the angle of view of the fisheye lens 111 is 180 degrees, the light source for generating light irradiating the object 161 may converge within the photographing range.

For example, a technique for acquiring information about environmental light irradiating the image processing apparatus 100 using a photometric device including an integrating sphere may acquire information about environmental light, No information could be obtained. Therefore, it is not possible to determine the direction or the like to display the shadow as described above. In the embodiment of the present invention, since the image processing apparatus 100 can photograph using the fisheye lens, it is possible to acquire information about the light source and to grasp in which direction the shadow should be displayed. Moreover, since the image processing apparatus 100 which concerns on embodiment of this invention can acquire the information about a light source, it can also acquire the information about environmental light separately from the information about a light source.

Embodiment of this invention acquires the information about the light source which exists in real space by processing the image picked up by the imaging | photography part 110 which has the fisheye lens 111, and displays the image which reflected the information about the light source. As a result, a more realistic image can be displayed to the user.

With reference to FIG. 2, the functional structure of the image processing apparatus which concerns on embodiment of this invention is demonstrated. FIG. 2 is a diagram illustrating a functional configuration of the image processing apparatus 100 shown in FIG. 1.

As shown in FIG. 2, the image processing apparatus 100 according to the embodiment of the present invention includes a photographing unit 110, a light source number calculating unit 120, a storage unit 130, a light source information calculating unit 140, and 3. The dimension CG generator 150 and the display unit 160 are included.

The photographing unit 110 includes a fisheye lens 111 and functions to receive an image from the outside of the image processing apparatus 100 through the fisheye lens 111. The photographing unit 110 may include one or more fisheye lenses 111. In addition to the fisheye lens 111, the photographing unit 110 includes an imaging device 112 that converts light incident from the outside of the image processing apparatus 100 through the fisheye lens 111 into an electrical signal.

The storage unit 130 has a function of storing three-dimensional computer graphics (CG) model information 131 that defines a three-dimensional space. The three-dimensional space is a virtual space displayed by the display unit 160 and is defined by an object having, for example, a shape, a size, a position, a direction, and a color. The object will be described later. The three-dimensional CG model information 131 is information that is an example of three-dimensional model information. The storage unit 130 stores data, programs, and the like used by the components of the other image processing apparatus 100.

The light source number calculating unit 120 performs light source number calculation processing for calculating the number of light sources captured on the image photographed by the photographing unit 110, and the light source coordinates indicating the position in the image of the light source corresponding to the number calculated. Calculate the information. Details of the function of the light source number calculator 120 will be described later with reference to FIG. 3.

The light source information calculating unit 140 calculates a parameter in the real space related to the light source as light source information that is a parameter in the three-dimensional space based on the light source coordinate information calculated by the light source number calculating unit 120. Details of the function of the light source number calculator 120 will be described later with reference to FIG. 4.

The three-dimensional CG creation unit 150 creates a three-dimensional image based on the three-dimensional CG model information 131 stored in the storage unit 130 and the light source information calculated by the light source information calculating unit 140. . The three-dimensional CG creation unit 150 functions as an example of the three-dimensional image creation unit.

The display unit 160 displays the three-dimensional image created by the three-dimensional CG creating unit 150. The display unit 160 may display other information for viewing by the image processing apparatus 100 as necessary. The display unit 160 may be configured by, for example, a display device.

Here, each of the components, for example, the light source number calculating unit 120, the light source information calculating unit 140, and the three-dimensional CG generating unit 150 are composed of a CPU and a RAM, and the CPU is stored in the storage unit 130. The function can be realized by developing the program stored in the RAM and executing the program deployed in the RAM.

With reference to FIG. 3, the function of the light source number calculation part of the image processing apparatus which concerns on embodiment of this invention is demonstrated. 3 is a view for explaining the function of the light source number calculation unit 120 shown in FIG.

As illustrated in FIG. 3, the photographing unit 110 may acquire the entire image IMA as a result of photographing, for example. The whole image IMA includes the intra-lens image IMB and the out-lens image IMC. The intra-lens image IMB is an image captured by the photographing unit 110 through the fisheye lens 111 from the outside of the image processing apparatus 100, and the out-of-lens image IMC is obtained by the photographing unit 110. It is an image received and photographed without passing through the lens 111. In the embodiment of the present invention, the intra-lens image IMB is mainly used instead of the out-lens image IMC. In addition, the light source position P1 corresponds to the part where the light source L1 was imaged, and the light source position P2 corresponds to the part where the light source L2 was imaged.

As described above, the light source number calculator 120 performs a light source number calculation process for calculating the number of light sources captured in the in-lens image IMB photographed by the photographing unit 110. For example, the light source number calculator 120 uses a histogram of the in-lens image IMB to make a portion having a luminance equal to or greater than a predetermined value as the light source candidate region. The predetermined value in this case can be stored, for example, in the storage unit 130.

In the example shown in FIG. 3, the two elliptic regions shown in the intra-lens image IMB are the light source candidate regions. In this case, since the light source L1 and the light source L2 are point light sources, the light source candidate region was photographed as an elliptic region. The light source candidate region is not necessarily elliptical, for example, a rectangular shape when the light source is an elongated fluorescent lamp. It can also be. As such, the shape of the light source candidate region may be changed by the shape of the light source.

Subsequently, the light source number calculator 120 detects an envelope of the light source candidate area, and if the area enclosed by the envelope has an area equal to or greater than a predetermined value, the light source candidate area is used as a light source area. To be considered. The predetermined value in this case can also be stored in the storage unit 130, for example. The light source number calculation unit 120 detects, for example, a center position in which the luminance of the light source region is measured as the light source position.

The envelope may be any line that encloses the light source candidate region, but may be a line defining a region as small as possible to include the light source candidate region. The reason why one light source exists in an area determined to have an area equal to or greater than a predetermined value is to prevent a mistake in the light source being present in an area photographed by simple reflection of light.

In the example shown in FIG. 3, the light source number calculation part 120 detects rectangular area IM1 and rectangular area IM2 as the area | region enclosed by an envelope, for example. The light source number calculator 120 determines that both the rectangular area IM1 and the rectangular area IM2 have an area equal to or greater than a predetermined value, and regards each of the two elliptic areas as the light source area. The light source number calculation unit 120 calculates the center of the two light source regions as the light source coordinate information as the light source position P1 and the light source position P2.

As described above, the light source number calculating unit 120 first calculates the number of light sources imprinted on the intra-lens image IMB photographed by the photographing unit 110, and calculates the number of light sources corresponding to the calculated number. Calculate the light source coordinate information indicating the position of. Calculating the number of light sources before calculating the light source coordinates thus regards a region including two light sources as one light source region, and misidentifies a center position based on the luminance of the light source region as a light source position. This is to prevent.

With reference to FIG. 4, the function of the light source information calculation part of the image processing apparatus which concerns on embodiment of this invention is demonstrated. 4 is a diagram for describing a function of the light source information calculating unit 140 shown in FIG. 2.

As described above, the light source information calculator 140 calculates a parameter in a real space related to a light source as light source information that is a parameter in a three-dimensional space based on the light source coordinate information calculated by the light source number calculator 120. Perform information calculation processing. In the example shown in FIG. 3, the light source number calculation unit 120 calculated the light source position P1 and the light source position P2 as light source coordinates in the in-lens image IMB. In the example shown in FIG. 4, the light source information calculating unit 140 is a thread related to the light source L1 and the light source L2 based on the light source position P1 and the light source position P2 in the in-lens image IMB. The parameter in space is calculated as light source information which is a parameter in three-dimensional space. Although there are various parameters in the real space with respect to the light source, for example, the orientation, position of the light source, intensity of light generated by the light source, color, spread, and the like. The light source information calculation unit 140 calculates these values for each rectangular area (rectangular area IM1 or rectangular area IM2).

The light source information calculating unit 140 is, for example, based on the light source coordinate information calculated by the light source number calculating unit 120 in the real space of the light source based on the position of the photographing unit 110 by projection conversion. The orientation information indicating the orientation is calculated as the light source information. The calculation method of the orientation information will be described later with reference to FIGS. 7 and 8. In addition, the light source information calculator 140 may calculate positional information indicating the position of the light source in the real space as the light source information. There are various methods of calculating the positional information of the light source, for example, a method of calculating based on azimuth information and a predetermined value, a method of calculating using several fisheye lenses, and the like. The former will be described later with reference to FIGS. 7 and 8. The latter will be described later with reference to FIG. 9.

When the positions of the plurality of light sources are calculated by the light source information calculating unit 140, the 3D CG preparing unit 150 may generate a 3D image including half shadows based on the positions of the plurality of light sources. A half shadow is a shadow that occurs because light generated by a light source of some of the light sources does not reach. More specifically, the three-dimensional CG creation unit 150 is three-dimensional based on the position information calculated as the light source information by the light source information calculating unit 140 and the three-dimensional model information stored by the storage unit 130. The half shadow area | region which arises in space is specified, and the three-dimensional image which has a half shadow in a specific half shadow area is created.

The light source information calculating unit 140 may add and calculate at least one of the intensity and the color of the light generated by the light source based on the light source coordinate information calculated by the light source number calculating unit 120 as the light source information and add it to the orientation information. . The light source information calculating unit 140 calculates, for example, the intensity of the light source based on the pixel value of the light source position P1 or the light source position P2 and the exposure value of the photographing unit 110. The pixel value of a light source position is computed by the calculation using three RGB of a light source position. The exposure value of the photographing unit 110 is a value representing the exposure degree determined by the aperture value and the exposure time or the shutter speed. The light source information calculating unit 140 calculates, for example, based on the RGB of the light source position P1 or the light source position P2 as the color of the light source.

The light source information calculator 140 may calculate the spread of light generated by the light source based on the light source region coordinate information calculated by the light source number calculator 120 as the light source information and adds the orientation information. More specifically, the light source information calculating unit 140 approximates the light source region to two vectors based on the light source region coordinate information calculated by the light source number calculating unit 120, for example. It can be used as a spread.

In the example shown in FIG. 4, the light source information calculating unit 140 approximates the short axis of the light source region existing in the rectangular region IM1 with the vector V1a and the long axis with the vector V1b, and to the rectangular region IM2. The case where the long axis of the existing light source area is approximated by the vector V2a and the short axis by the vector V2b is shown. However, the method of approximating the light source region to two vectors is not particularly limited. For example, when the light source region is a rectangle, the light source information calculator 140 may approximate the vertical and horizontal sides of the rectangle as two vectors. It is also possible to approximate not only the case where the long axis, the short axis, and the vertical and horizontal directions are accurately approximated by a vector, but also the vector multiplied by a predetermined integer.

The light source information calculating unit 140 uses at least one of the intensity and color of the environmental light in the real space as the environmental light information as a parameter in the three-dimensional space based on the light source coordinate information calculated by the light source number calculating unit 120. In addition to the information can also be calculated. The light source information calculator 140 calculates the intensity of the ambient light in the real space based on, for example, the pixel value and the exposure value of a region that does not belong to the rectangular region IM1 or the rectangular region IM2 in the in-lens image IMB. Can be calculated.

In addition, the light source information calculation unit 140 calculates the color of the ambient light in the real space based on, for example, RGB of an area of the intra-lens image IMB that does not belong to the rectangular area IM1 or the rectangular area IM2. Can be. The three-dimensional CG creating unit 150 specifies the color of the object in the three-dimensional space based on the ambient light information calculated by the light source information calculating unit 140, and creates a three-dimensional image having a specific object color. The ambient light information is a factor that contributes particularly significantly to the brightness of the part where light from the light source does not reach. The part which light does not reach from a light source means the part which the shadow which appears in the following description occurs.

As described above, in the embodiment of the present invention, since the image processing apparatus 100 can acquire the information about the light source from the image photographed using the fisheye lens, the image processing device 100 can acquire the information about the ambient light distinct from the information about the light source. Can be. For example, when information on light entering the integrating sphere is acquired using an integrating sphere other than the fisheye lens, the information on the light source and the information on the ambient light cannot be divided from the acquired information.

The case where one light source exists as an example of display of the image processed by the image processing apparatus which concerns on embodiment of this invention with reference to FIG. 5A is demonstrated. 5A is a diagram illustrating a display example of an image processed by the image processing apparatus 100 shown in FIG. 1.

As shown in FIG. 5A, the object 220a is included in the screen 210a displayed by the image processing apparatus 200 as an example of the image processing apparatus 100. This object 220a is defined by the three-dimensional CG model information 131 and displayed on the display unit 160 as a three-dimensional image. Here, as illustrated in FIG. 5A, the object 220a includes an object 231 and an object 232 having a height in a vertical direction with respect to the screen 210a. In such a case, the information regarding the light source which generate | occur | produces the light which irradiates the image processing apparatus 200 is acquired by the function which the light source information calculating part 140 of the image processing apparatus 200 has. In addition, the shadow 241 and the shadow 242 based on the information on the light source are displayed on the screen 210a by the function of the three-dimensional CG creating unit 150 of the image processing apparatus 200. As a result, when the image processing apparatus 200 is installed indoors, the screen 210a having increased affinity with the environment around the image processing apparatus 200 can be displayed.

More specifically, the three-dimensional CG creation unit 150 of the image processing apparatus 200 includes light source information calculated by the light source information calculating unit 140 and three-dimensional CG model information stored by the storage unit 130. Based on 131, the direction and number of shadows generated in the three-dimensional space are calculated. The 3D CG creation unit 150 specifies a shadow area based on the calculated direction and number of shadows, and creates a 3D image having shadows in the specific shadow area. The display unit 160 displays the three-dimensional image generated by the three-dimensional CG creating unit 150.

Recently, portable displays such as digital photo frames, smart phones, and tablet PCs have been widely distributed. Such a display is often useful, because the environment used is often changed. In addition, when displaying a sample of a product to be sold in three dimensions, an image with a shade adapted to a light source is displayed so that the difference between an impression when the purchaser actually obtains the product and an impression when the sample is viewed is displayed. Can be reduced.

In addition, the image processing apparatus 200 may change the installed light source as soon as the installed environment changes, and change the image of the display object into a three-dimensional image adapted immediately after the change. In more detail, the light source information calculating unit 140 determines whether the current light source information has changed by comparing with the previous light source information, which is the light source information used last time for creating the 3D image, and the 3D CG creating unit 150. ) Re-creates the 3D image when it is determined by the light source information calculating unit 140 that the current light source information has been changed from the previous light source information, and the display unit 160 is executed by the 3D CG creating unit 150. The recreated three-dimensional image is displayed again.

A case in which one light source exists as an example of display of an image displayed by a general image processing apparatus will be described with reference to FIG. 5B. 5B is a diagram showing a display example of an image displayed by a general image display apparatus.

As shown in FIG. 5B, a general image processing apparatus displays a screen 210b including an object 220b. The object 220b has an object 231 and an object 232, but the shadow generated by the presence of the object 231 and the object 232 is not particularly shown in the object 220b.

With reference to FIG. 6A, the case where one point light source exists as an example of display of the image processed by the image processing apparatus which concerns on embodiment of this invention is demonstrated. FIG. 6A is a diagram illustrating a display example of an image processed by the image processing apparatus 100 shown in FIG. 1.

As shown in FIG. 6A, the image processing apparatus 300 as an example of the image processing apparatus 100 displays the screen 310a, and the screen 310a includes the object 321, the object 322, and the object 323. ), Etc. are included. The information about the light source is obtained by the function of the light source information calculating unit 140 of the image processing apparatus 300, and the object 321 is provided by the function of the three-dimensional CG creating unit 150 of the image processing apparatus 300. And the object 322 as a starting point, a three-dimensional image having a shadow in the direction S1 is generated. The display unit 160 of the image processing apparatus 300 displays a three-dimensional image. Here, the case where one point light source exists is shown.

With reference to FIG. 6B, the case where two point light sources exist as a display example of the image processed by the image processing apparatus which concerns on embodiment of this invention is demonstrated. FIG. 6B is a diagram illustrating a display example of an image processed by the image processing apparatus 100 shown in FIG. 1.

As shown in FIG. 6B, the image processing apparatus 300 as an example of the image processing apparatus 100 displays a screen 310b, and the screen 310b includes an object 321, an object 322, and an object 323. ), Etc. are included. The information about the light source is obtained by the function of the light source information calculating unit 140 of the image processing apparatus 300, and the object 321 is provided by the function of the three-dimensional CG creating unit 150 of the image processing apparatus 300. And the object 322 as a starting point, a three-dimensional image having a shadow in the direction S2 and the direction S3 is generated. The display unit 160 of the image processing apparatus 300 displays a three-dimensional image. Here, the case where two point light sources exist is shown.

With reference to FIG. 6C, the case where one point light source and one surface light source exist as a display example of the image processed by the image processing apparatus which concerns on embodiment of this invention is demonstrated. 6C is a diagram illustrating a display example of an image processed by the image processing apparatus 100 shown in FIG. 1.

As shown in FIG. 6C, the image processing apparatus 300 as an example of the image processing apparatus 100 displays the screen 310c, and the screen 310b includes the object 321, the object 322, and the object 323. ), Etc. are included. The information about the light source is obtained by the function of the light source information calculating unit 140 of the image processing apparatus 300, and the object 321 is provided by the function of the three-dimensional CG creating unit 150 of the image processing apparatus 300. And the object 322 as a starting point, a three-dimensional image having a shadow in the direction S2 and the direction S3 is generated. The display unit 160 of the image processing apparatus 300 displays a three-dimensional image. Here, the case where one point light source and one surface light source exist is shown. Since light is irradiated from the surface light source, the shadow generated in the direction S2 has a smooth border. The information indicating the spread of the light source is acquired as the light source information by the light source information calculating unit 140 of the image processing apparatus 300 so that the three-dimensional CG creating unit 150 of the image processing apparatus 300 considers the enlargement of the light source. This is because a three-dimensional image is created.

With reference to FIG. 7, the method which calculates the orientation of a light source as light source information is demonstrated with the light source information calculation part of the image processing apparatus which concerns on embodiment of this invention. FIG. 7 is a diagram for explaining that the light source information calculating unit 140 shown in FIG. 2 calculates a direction of a light source as light source information. Here, one fisheye lens 111 used by the image processing apparatus 100 is assumed.

The light source coordinates i, j in the intra-lens image IMB of the light source are obtained as two-dimensional coordinates, and the fisheye lens 111 is an intra-lens image IMB with a stationary cylindrical projection, which is one of equidistant projections. The light is projected onto and the length that becomes 90 degrees in the image is called W. At this time, the relationship between the light source coordinates (i, j) and the orientation (azimuth, elevation) of the light source = (θ, φ) is expressed by the following equation (1). Definitions of elevation and azimuth are shown in FIG. 8.

When the 3D CG preparing unit 150 does not draw the anti-shadow in 3D CG, the shadow may be created even if the position of the light source is not known using the direction of the light source. When the three-dimensional CG preparing unit 150 draws the half shadow, an appropriate radius R is set and the light source coordinates (x, y, z) obtained by the following equation (2) are used. The radius R can be stored in the storage unit 130, for example, and can be appropriately changed by a manager or the like who manages the image processing apparatus 100.

An elevation angle and an azimuth angle used to calculate an orientation of a light source will be described with reference to FIG. 8. 8 is a diagram for explaining elevation angles and azimuth angles used to calculate the orientation of the light source.

The elevation angle and azimuth angle used for calculating the direction of the light source by the image processing apparatus 100 are defined, for example, as shown in FIG. 8.

With reference to FIG. 9, the method to calculate the position of a light source as a light source information in the light source information calculation part of the image processing apparatus which concerns on embodiment of this invention is demonstrated. FIG. 9 is a view for explaining that the light source information calculating unit 140 shown in FIG. 2 calculates the position of the light source as the light source information. It is assumed here that there are several fisheye lenses 111 used by the image processing apparatus 100.

If there are several fisheye lenses 111, the light source coordinates can be obtained without assuming a radius R. Simply there are two fisheye lenses 111 and the coordinates (x1, y1, z1), (x2, y2, z2) in each real space are known in advance in the image processing apparatus 100 and each fisheye lens At 111, the orientations θ1, φ1, and θ2, φ2 of the light source are calculated by the image processing apparatus 100. At this time, the direction vectors (v1x, v1y, v1z) and (v2x, v2y, v2z) of the light source are represented by the following equation (3).

Using Equation 3, the coordinates (x, y, z) of the light source are represented by the following Equation 4.

Where s, t is a parameter and is defined by Equation 5 († denotes Moore-Penrose's general inverse matrix): Here, it is effective even if the number of light sources is n (n is 3 or more) instead of two.

As described above, the technique according to the present embodiment can acquire information about the light source existing in the real space and display an image reflecting the information about the light source. Therefore, the user can see the real image through the image processing apparatus 100.

An apparatus according to the present invention may include a processor, a memory for storing and executing program data, a permanent storage such as a disk drive, a communication port for communicating with an external device, a user interface such as a touch panel, a key, Devices, and the like. Methods implemented by software modules or algorithms may be stored on a computer readable recording medium as computer readable codes or program instructions executable on the processor. Here, the computer-readable recording medium may be a magnetic storage medium such as a read-only memory (ROM), a random-access memory (RAM), a floppy disk, a hard disk, ), And a DVD (Digital Versatile Disc). The computer-readable recording medium may be distributed over networked computer systems so that computer readable code can be stored and executed in a distributed manner. The medium is readable by a computer, stored in a memory, and executable on a processor.

All documents including publications, patent applications, patents, etc. cited in the present invention can be incorporated into the present invention in the same manner as each cited document individually and concretely, .

In order to facilitate understanding of the present invention, reference will be made to the preferred embodiments shown in the drawings, and specific terminology is used to describe the embodiments of the present invention. However, the present invention is not limited to the specific terminology, Lt; / RTI > may include all elements commonly conceivable by those skilled in the art.

The invention can be represented by functional block configurations and various processing steps. Such functional blocks may be implemented in various numbers of hardware or / and software configurations that perform particular functions. For example, the present invention is an integrated circuit configuration such as memory, processing, logic, look-up table, etc., capable of executing various functions by the control of one or more microprocessors or other control devices. You can employ them. Similar to the components of the present invention that may be implemented with software programming or software components, the present invention may be implemented as a combination of C, C ++, and C ++, including various algorithms implemented with data structures, processes, routines, , Java (Java), assembler, and the like. The functional aspects may be implemented with an algorithm running on one or more processors. In addition, the present invention may employ the prior art for electronic environment setting, signal processing, and / or data processing. Terms such as "mechanism", "element", "means" and "configuration" can be used widely and are not limited to mechanical and physical configurations. The term may include the meaning of a series of routines of software in conjunction with a processor or the like.

The specific acts described in the present invention are, by way of example, not intended to limit the scope of the invention in any way. For brevity of description, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of the systems may be omitted. Also, the connections or connecting members of the lines between the components shown in the figures are illustrative of functional connections and / or physical or circuit connections, which may be replaced or additionally provided by a variety of functional connections, physical Connection, or circuit connections. Also, unless explicitly mentioned, such as " essential ", " importantly ", etc., it may not be a necessary component for application of the present invention.

The use of the terms " above " and similar indication words in the specification of the present invention (particularly in the claims) may refer to both singular and plural. In addition, in the present invention, when a range is described, it includes the invention to which the individual values belonging to the above range are applied (unless there is contradiction thereto), and each individual value constituting the above range is described in the detailed description of the invention The same. Finally, the steps may be performed in any suitable order, unless explicitly stated or contrary to the description of the steps constituting the method according to the invention. The present invention is not necessarily limited to the order of description of the above steps. The use of all examples or exemplary language (e.g., etc.) in this invention is for the purpose of describing the invention in detail and is not to be construed as a limitation on the scope of the invention, It is not. In addition, one of ordinary skill in the art appreciates that various modifications, combinations and changes can be made depending on design conditions and factors within the scope of the appended claims or equivalents thereof.

100 image processing unit
110 Shooting Department
111 Fisheye Lens
112 image pickup device
120 Number of Light Sources
130 memory
131 3D CG Model Information (3D Model Information)
1140 light source information calculation unit
150 3D CG Creator (3D Image Creator)
160 display

Claims (14)

A photographing unit having a fisheye lens and photographing an image through the fisheye lens;
A storage unit for storing three-dimensional model information defining a three-dimensional space;
A light source number calculating unit for calculating the number of light sources captured by the image capturing unit and calculating light source coordinate information indicating a position in the image of the light source corresponding to the calculated number;
A light source information calculation unit calculating a parameter in a real space with respect to the light source based on the calculated light source coordinate information as light source information that is a parameter in the three-dimensional space;
A three-dimensional image creating unit which creates a three-dimensional image based on the stored three-dimensional model information and the calculated light source information; And
An image processing apparatus comprising a display unit for displaying the created three-dimensional image. The method of claim 1,
The light source information calculation unit,
And the orientation information indicating the orientation in the real space of the light source with respect to the position of the photographing unit based on the calculated light source coordinate information as the light source information. The method of claim 2,
The light source information calculation unit,
And position information indicating a position in a real space of the light source based on the orientation information and a predetermined value as the light source information. The method of claim 2,
Wherein,
Including at least two fisheye lenses, photographing each image through the at least two fisheye lenses,
The light source number calculation unit,
Calculating light source number and light source information for each image photographed by the photographing unit;
The light source information calculation unit,
Azimuth information is calculated for each image photographed by the photographing unit, and based on the calculated azimuth information, positional information indicating a position in the real space of the light source is calculated as the light source information. Image processing apparatus. The method according to any one of claims 2 to 4,
The light source information calculation unit,
And at least one of an intensity and a color of light generated by the light source based on the calculated light source coordinate information as the light source information. The method according to any one of claims 2 to 4,
The light source number calculation unit,
Detecting a light source region corresponding to the calculated number from the image, calculating a position in the image of the detected light source region as light source region coordinate information,
The light source information calculation unit,
And the spread of light generated by the light source as the light source information based on the light source region coordinate information calculated by the light source number calculation unit. The method according to claim 6,
The light source information calculation unit,
An image processing apparatus characterized by approximating the light source region to two vectors based on the light source region coordinate information calculated by the light source number calculating unit, and calculating the spread of the light as the light source information by the two vectors. . The method of claim 1,
The light source information calculation unit,
Calculating at least one of the intensity and the color of the environmental light in the real space based on the light source coordinate information calculated by the light source number calculating unit as the environmental light information as a parameter in the three-dimensional space,
The three-dimensional image creation unit,
An object color in the three-dimensional space is specified based on the ambient light information calculated by the light source information calculating unit, and a three-dimensional image having the specific object color is created. The method of claim 1,
The three-dimensional image creation unit,
The direction and number of shadows generated in the three-dimensional space are calculated based on the light source information calculated by the light source information calculating unit and the three-dimensional model information stored in the storage unit. Specifying a shadow area on the basis of the shadow area, and creating a three-dimensional image having a shadow in the specific shadow area. The method according to claim 3 or 4,
The three-dimensional image creation unit,
Based on the positional information calculated as the light source information by the light source information calculating unit and the three-dimensional model information stored in the storage unit, a half shadow region generated in the three-dimensional space is specified, and a half shadow is formed in the specific half shadow region. An image processing apparatus characterized by creating a three-dimensional image. The method of claim 1,
The light source information calculation unit,
Compare with previous light source information which is light source information previously used for the creation of the three-dimensional image. Determine whether the current light source information has changed,
The three-dimensional image creation unit,
Reconstruct the three-dimensional image when it is determined by the light source information calculating unit that the current light source information has been changed from the previous light source information,
The display unit,
And re-display the recreated three-dimensional image. The method of claim 1,
Wherein,
And an image is received by the outside of the image processing apparatus through the fisheye lens to capture an image. Photographing an image through a fisheye lens;
Calculating the number of light sources imprinted on the photographed image, and calculating light source coordinate information indicating a position in the image of the light source corresponding to the calculated number;
Calculating a parameter in real space with respect to the light source based on the calculated light source coordinate information as light source information that is a parameter in the three-dimensional space;
Creating a three-dimensional image based on the three-dimensional model information defining the three-dimensional space stored in advance and the calculated light source information; And
An image processing method comprising the step of displaying the created three-dimensional image. A recording medium having recorded thereon a program for executing the method according to claim 13 on a computer.

Images