WO2013136602A1

WO2013136602A1 - Imaging device with projector and imaging control method therefor

Info

Publication number: WO2013136602A1
Application number: PCT/JP2012/080614
Authority: WO
Inventors: 林　大輔; 近藤　茂; 貴嗣青木; 和紀井上; 宗之大島; 三沢　岳志; 三沢　充史
Original assignee: 富士フイルム株式会社
Priority date: 2012-03-16
Filing date: 2012-11-27
Publication date: 2013-09-19

Abstract

This imaging control method for an imaging device with a projector involves capturing a captured image formed by projecting each of a plurality of marker patterns on different planar regions of a subject with a projector, and calculating the slope of each planar region on which the marker patterns are projected from the degree of deformation of each marker pattern. The planar region having the marker pattern with the smallest degree of deformation according to the slope values is selected, and a first image, which is to be projected on the subject, is oppositely deformed according to the value of the slope of the selected planar region. The subject is photographed by an imaging unit in a state in which the deformed first image is projected on the selected planar region of the subject by the projector.

Description

Imaging device with projector and imaging control method thereof

The present invention relates to a photographing apparatus with a projector that increases the utility value of a projector function and a photographing control method thereof.

In recent years, a projector equipped with a photographing device such as a digital camera has begun to spread. When a digital camera has a projector, the captured image is not displayed on the small display on the back of the camera and checked. It is convenient because it can be confirmed and viewed with magnified projection.

However, the projector mounted on the digital camera is limited to a small one because it can be mounted only in a narrow empty space in the digital camera housing. For this reason, compared with the case where the photographed image is displayed on a large television receiver at home or is projected by a dedicated large projector, the image quality and fineness of the displayed image are inferior.

If only low-quality images can be displayed in this way, digital camera users cannot find utility value in the projector, and they are reluctant to purchase a digital camera with a projector that would increase the cost by installing the projector. End up.

Therefore, in addition to improving the image quality of the projected image, it is necessary to increase the utility value of the projector function so that the camera user is interested in the digital camera with the projector. For example, the digital cameras with a projector disclosed in Patent Documents 1 and 2 are configured to project an image to the front side of the camera so that a catch light can be put in the subject's eyes or the color of the face image of the subject can be changed. It has become.

Japanese Unexamined Patent Publication No. 2008-249956 Japanese Unexamined Patent Publication No. 2008-148089

Projecting a catchlight image or a face decoration image on the front side of the camera can capture a beautiful subject, increasing the utility value of the projector function. However, the techniques disclosed in Patent Documents 1 and 2 are not sufficient to satisfy the user's willingness to purchase a digital camera with a projector, and further increase the utility value of the projector function and improve the usability of the projector function.

An object of the present invention is to provide a photographing apparatus with a projector and a photographing control method thereof that can enhance the utility value of the projector function and improve the usability thereof.

An imaging device with a projector according to the present invention includes an imaging unit that captures an image of a subject, a first image that is an image of a collective pattern obtained by combining characters, figures, symbols, or any of these, and a marker pattern having a preset shape. Each of the projectors projecting onto the subject, and the projector captures captured images in a state where a plurality of the marker patterns are respectively projected onto different plane regions of the subject, and the degree of deformation of each marker pattern in the captured image An inclination calculation unit that calculates the inclination of each of the planar areas on which the marker pattern is projected, a plane selection unit that selects a planar area where the degree of deformation of the marker pattern is the smallest from the calculated inclination values, and the plane According to the value of the inclination with respect to the planar area selected by the selection unit, the first image projected onto the subject A first image deformation unit that deforms an image by reversing the enlargement / reduction direction of the deformation, and the first image obtained by deforming the image by the first image deformation unit by the projector using the selected plane area of the subject. And an imaging control unit that images the subject by the imaging unit in a state of being projected onto the subject.

An imaging control method for an imaging apparatus with a projector according to the present invention includes an imaging unit that images a subject, a first image that is an image of a collective pattern obtained by combining characters, figures, symbols, or any of these, and a preset shape And a projector for projecting each of the marker patterns onto the subject, and capturing an image obtained by projecting a plurality of the marker patterns onto the different planar regions of the subject by the projector. Calculating the inclination of each of the planar areas onto which the marker pattern is projected from the degree of deformation of each marker pattern in the captured image, and the degree of deformation of the marker pattern from the calculated inclination value. Selecting the smallest planar area, and for the selected planar area The step of deforming the first image to be projected onto the subject according to the inclination value by reversing the deformation direction of the deformation, and the deformed first image by the projector using the projector. And imaging the subject with the imaging unit in a state where the subject is projected onto the selected planar region.

According to the present invention, since an arbitrary character information image can be embedded in a subject image in a natural state, the utility value and usability of the projector function are improved.

1 is an external perspective view of a photographing apparatus according to a first embodiment of the present invention. It is a functional block block diagram of the imaging device shown in FIG. It is a flowchart which shows the control procedure of the imaging device which concerns on the 1st Embodiment of this invention. It is a figure which shows the state which projected the marker pattern which concerns on the 1st Embodiment of this invention on the to-be-photographed object. It is a figure which shows the state which projected the character information etc. which concern on the 1st Embodiment of this invention on the to-be-photographed object. It is explanatory drawing of the normal vector which concerns on the 1st Embodiment of this invention. It is modification explanatory drawing of the character information etc. which concern on the 1st Embodiment of this invention. It is a functional block block diagram of the imaging device which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows the control procedure of the imaging device which concerns on the 2nd Embodiment of this invention. It is explanatory drawing of the multiple marker pattern projection by the 2nd Embodiment of this invention. It is explanatory drawing of the projection character information etc. by the modification of the 2nd Embodiment of this invention. It is explanatory drawing of the projection character information etc. by the further modification of the 2nd Embodiment of this invention. It is a functional block block diagram of the imaging device which concerns on the 3rd Embodiment of this invention. It is a flowchart which shows the control procedure of the imaging device which concerns on the 3rd Embodiment of this invention. It is explanatory drawing of the projection marker pattern by the 3rd Embodiment of this invention. It is a functional block block diagram of the imaging device which concerns on the 4th Embodiment of this invention. It is a flowchart which shows the control procedure of the imaging device which concerns on the 4th Embodiment of this invention. It is explanatory drawing of the projection character information etc. by the 4th Embodiment of this invention. It is a functional block block diagram of the imaging device which concerns on the 5th Embodiment of this invention. It is a flowchart which shows the control procedure of the imaging device which concerns on the 5th Embodiment of this invention. It is explanatory drawing of the projection marker pattern by the 5th Embodiment of this invention. It is an external appearance perspective view of the imaging device which concerns on another embodiment of this invention. It is a functional block block diagram of the imaging device shown in FIG.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

FIG. 1 is an external perspective view according to a first embodiment of a photographing apparatus with a projector (digital camera) 10. The digital camera 10 includes a photographic lens 12 at the front of a rectangular camera casing 11. The taking lens 12 is housed in a retractable lens barrel 13. A shutter release button 14 is provided on the left shoulder portion of the camera casing 11. A liquid crystal display unit (LCD 16 in FIG. 2) for displaying captured images, through images (live view images), camera menu images, and the like is provided on the rear surface of the camera housing 11. A flash light emitting unit 44 is provided at the front of the right shoulder.

A front projection type projector (video projection unit) 17 is provided inside the upper side of the camera housing 11 and projects a display image of a built-in small liquid crystal display unit to the front of the camera through a projection window 18 at the front. It is configured.

In addition, a live view image is displayed on the small liquid crystal display unit built in the projector 17 and the user can look into the live view image through a finder window provided on the rear side of the camera, so that the electronic view finder device can also be used. good. In the illustrated example, the projector 17 is built in, but may be provided outside the camera housing as described in, for example, Japanese Patent Application Laid-Open No. 2006-80875.

FIG. 2 is a functional block configuration diagram of the digital camera shown in FIG. The digital camera 10 includes an image sensor (a CMOS image sensor in this embodiment) 21 provided on the back of the photographing lens 12 and a control unit (CPU) 22 that performs overall control of the digital camera 10.

The CPU 22 is connected to the bus 23, and the bus interface 23 stores a frame memory 24, a signal processing unit 25 that performs image processing, and a card interface (stored in the external memory card 26) in the external memory card 26. I / F) 27, a display control unit 28 that performs display control of an LCD (liquid crystal display unit) 16 on the back of the camera, an OSD signal generation unit 29 that generates OSD signals such as character information to be displayed on the LCD 16, and a projector 17 Is connected to the video projection control unit 30 for controlling the image projection.

The bus 23 is further connected to a projected character deformation unit 31, a plane normal vector calculation unit 32, and a marker detection unit 33. These can be constituted by individual electronic components, or can be realized as one of the processing functions of the CPU 22. The same applies to the following embodiments.

The projected character deforming unit 31 is an image (first character image) of characters, figures, symbols, or a set pattern (hereinafter referred to as character information) projected from the projector 17 of FIG. 1 image). The marker detection unit 33 detects an image of a marker pattern projected from the projector 17 through the projection window 18 onto a subject in front of the camera. The plane normal vector calculation unit (tilt calculation unit) 32 calculates the normal vector of the marker pattern detected by the marker detection unit 33, thereby calculating the degree of plane tilt.

The CPU 22 is connected to a ROM 22a and a RAM 22b that store control programs and the like, and an operation unit 40 including a shutter release button 14 is connected. The digital camera 10 also drives the lens drive unit 41 that controls the focus lens position and the like of the photographing lens 12, the timing generator (TG) 42 that generates the drive timing signal of the image sensor 21, and the image sensor 21. A driver 43 and a flash control circuit 45 that performs light emission control of the flash light emitting unit 44 are provided, and these are instructed to be controlled by the CPU 22. The CPU 22 functions as an imaging control unit.

FIG. 3 is a flowchart showing the processing procedure of the control program executed when the user images the subject using the digital camera 10. This control program is stored in the ROM 22a, read out to the RAM 22b at the time of execution, and executed by the CPU 22.

In the digital camera 10 of the present embodiment, for example, as shown in FIG. 5, a character image “January 1, 2012” representing the shooting date and time is projected onto the plane portion of the subject, and the subject of the subject on which this character image is projected is projected. An image is taken through the taking lens 12.

First, in step S1, a marker pattern is projected onto the subject. An example in which a marker pattern is projected is shown in FIG. In this example, the marker pattern is a square frame 51 and a character 52 of “copy” inside the square frame 51. The marker pattern may be only the square frame 51. However, since there are many square and rectangular subjects in the world, the character 52 of "copy" is also used as a marker pattern so that it can be easily distinguished from a normal square subject. It is included. Of course, the marker pattern may be a colored pattern such as red or green.

In the next step S2, the live view image of the subject on which the marker pattern is projected is captured, and in step S3, the marker pattern region in the subject image is detected. The front surface of the pedestal in the subject image shown in FIG. 4 is perpendicular to the camera optical axis (= the optical axis of the projector 17). For this reason, the square frame (marker pattern) 51 in the subject image on which the square marker is projected is maintained in a square shape.

On the other hand, as shown in FIG. 6A, when a square marker pattern is projected onto a plane 54 that is oblique to the camera optical axis, the marker pattern 51 in the subject image is deeper than the near side dimension. It will deform | transform into the trapezoid image with which the dimension of the side extended. The marker pattern may not be detected in the live view image depending on the shape of the subject, the reflectance, the irregular reflectance, and the like.

Therefore, in step S4, it is determined whether or not the marker pattern is detected in the live view image. If the marker pattern 51 is not detected, the process returns to step S1, and if the marker pattern 51 is detected, the process proceeds to step S5.

In step S5, a normal vector of the plane on which the marker pattern is projected is calculated. In step S6, character information such as characters to be projected is deformed according to the normal vector.

For example, when “January 1, 2012” in which all characters are the same size as shown in FIG. 7A is projected onto the oblique plane 54 in FIG. 6, as shown in FIG. 7B. In the projected image, the smaller the character in the foreground, the larger the character in the back. Therefore, as shown in FIG. 7C, the size of the character to be projected is reversed according to the inclination of the plane 54 so that the character projected to the front is larger and the character projected to the back is smaller. Deform. That is, the projection image projected on the oblique plane 54 is deformed by reversing the expansion / contraction direction of the deformation of the marker pattern 51 detected in the live view image. That is, a portion where the marker pattern is greatly deformed (enlarged deformation) is deformed small, and a portion where the marker pattern is deformed (reduced deformation) is largely deformed in units of pixels or several pixels. Then, this character is displayed on the small liquid crystal display unit in the projector 17 so as to be projected, and is projected toward the plane 54 of the subject (step S7). As a result, as shown in FIG. 6B, the shooting date information can be projected in the subject image so that all the characters have the same size.

The extent to which the character size is deformed depends on how oblique the plane 54 is to the camera optical axis. That is, the character to be projected may be affine transformed according to the direction of the plane normal vector.

In step S7, the deformed character is projected toward the object plane, and while maintaining this projection state, an image of the object is captured in step S8, and this process ends.

When writing a photo, it has been conventional to write the date of shooting into the image by post-processing. However, since this conventional writing is written at a position that appears to float from the subject image regardless of the depth position of the photographed subject image, the character information has an unnatural feeling.

On the other hand, as in this embodiment, by capturing an image of the shooting date projected onto the subject by the projector together with the subject image, natural shooting date can be superimposed.

In the embodiment, the marker pattern is the square frame 51. However, since the shooting date is horizontally long, it may be a horizontally long rectangular frame. Alternatively, when a vertically written shooting date is desired to be projected, a vertically long rectangular frame may be used. Further, although the embodiment has been described by taking the shooting date as an example, arbitrary character information may be projected. For example, “* Hakone trip *” can be projected vertically. Furthermore, it is possible to project character information extending over 2 rows, 3 rows, 2 columns, and 3 columns.

FIG. 8 is a functional block configuration diagram of a digital camera according to the second embodiment of the present invention. 2 is different from the embodiment of FIG. 2 in that the bus 23 is connected to a plane selection unit 34 that minimizes the angle between the normal vector of the marker pattern and the optical axis.

FIG. 9 is a flowchart showing a control procedure according to the second embodiment. This embodiment differs from the flowchart of FIG. 3 in the following three points. First, step S1a is executed instead of step S1, and a plurality of marker patterns are projected onto the subject. The second point is that step S4a is executed instead of step S4 to determine whether one or more marker patterns have been detected. Third, step S11 is provided between step S5 and step S6.

In the present embodiment, a plurality of marker patterns are projected from the projector 17. For example, the shooting screen is divided into 16 × 4 × 4, and one marker pattern is projected on each divided screen. Then, a plane on which a projection image that is most easily viewable when a character or character is projected is selected.

FIG. 10 is an explanatory diagram of the second embodiment. In this embodiment, a plurality of marker patterns are projected onto the subject in step S1a, and it is determined whether or not one or more markers are detected in step S4a. Then, in step S5, each normal vector is calculated for each detected marker pattern, and in the next step 11, a plane that minimizes the angle formed by the normal vector and the camera optical axis is selected. This is because the projection image onto the plane is the least deformed and easy to see.

Then, the character information projected on the plane is affine transformed (step S6), and the subject image is captured with the affine transformed character information projected (step S7) (step S8).

According to this embodiment, when the oblique plane 54 and the directly-facing plane 55 shown in FIG. 10 are detected, the directly-facing plane 55 is selected as the projection target plane, and easy-to-see character information and the like are projected. .

When projecting long text information on the plane 55, for example, “January 1, 2012” may not fit within the horizontal length of the plane 55. When detecting such a case, the CPU 22 causes the projected character deforming unit (first image deforming unit) 31 to fold the projected character information and, as shown in FIG. 11, “2012” and “January 1”. The image is changed to a folded image divided into two lines and projected onto the plane 55. Alternatively, as shown in FIG. 12, a divided image of projection character information divided into “2012” and “January 1” is generated, and “2012” is projected on the plane 55, and “January 1” is generated. ”May be projected onto the plane 54. Of course, the character information is affine transformed according to the direction of each normal vector.

In the second embodiment, the screen is equally divided and a plurality of marker patterns are projected. However, when character information is projected onto an important part of the subject, it may be bothersome and disturbing. In such a case, for example, the user designates an area in which character information is not projected in advance while viewing a live view image displayed on the liquid crystal display unit 16 on the back of the camera. And it is good to project a some marker pattern to an area | region other than that, and to look for the plane which writes in character information.

Alternatively, before projecting the marker pattern onto the subject, for example, a fine mesh image is projected onto the subject side with infrared rays, the captured image (live view image) is captured, and the position on the screen is determined from the degree of deformation of the mesh image. When the CPU 22 checks in advance how large the plane on which character information can be projected is, the plane candidates are displayed on the liquid crystal display unit 16 to be selected by the user, and the user selects a plurality of candidates. The second embodiment may be executed.

FIG. 13 is a functional block configuration diagram of a digital camera according to the third embodiment of the present invention. 2 differs from the embodiment of FIG. 2 in that a marker color setting unit 35, a complementary color calculation unit 36, and a color determination unit 37 on the projection surface are connected to the bus 23. The color determination unit for the projection surface determines the color of the subject area (projection surface) on which the character information is projected by the projector 17. The complementary color calculation unit 36 calculates and obtains a complementary color of the color of the projection surface, and the marker color setting unit 35 sets the marker color as the complementary color.

FIG. 14 is a flowchart showing a control procedure of the third embodiment. 3 is different from the flowchart of FIG. 3 only in that steps S12 and S13 are provided before the step S1.

In this embodiment, first, in step S12, a live view image is captured, and the color information of the plane 55 serving as the projection surface shown in FIG. Then, a complementary color of the color of the plane 55 is obtained, and in step S13, the color of the marker pattern is set to this complementary color. Thereafter, the process proceeds to step S1 to step S8, and a marker pattern 51 which is a complementary color to the color of the plane 55 is projected. Since the marker pattern 51 has a complementary color relationship with the color of the plane 55, the marker detection accuracy on the camera side is improved. Needless to say, the color may be different from the color of the plane 55 even if it is not a complementary color.

FIG. 16 is a functional block configuration diagram of a digital camera according to the fourth embodiment of the present invention. Compared with the third embodiment of FIG. 13, the difference is that a projection character color setting unit 38 is connected to the bus 23.

FIG. 17 is a flowchart showing a control procedure of the fourth embodiment. This embodiment is different from the flowchart of FIG. 14 in that step S14 is provided between step S6 and step S7, and the complementary color of the plane color is set as the projection color.

According to the present embodiment, as shown in FIG. 18, the character string 56 of the shooting date is displayed by the projected character color setting unit (color setting unit) 38 in a color that is complementary to the color of the plane 55 that is the projection surface. Is projected. Thereby, easy-to-read text information and the like are projected onto the subject and imaged. Needless to say, the color may be different from the color of the projection surface even if it is not a complementary color.

FIG. 19 is a functional block configuration diagram of a digital camera according to the fifth embodiment of the present invention.
Compared with the embodiment of FIG. 2, the projection marker expansion / contraction unit 39 is connected to the bus 23.

FIG. 20 is a flowchart showing a control procedure of the fifth embodiment. Compared with the embodiment of FIG. 2, step S6a is executed instead of step 6, and when a marker pattern is not detected in step S4, step S15 is executed and then the process returns to step S1.

In the embodiment described with reference to FIG. 11, when the plane to be projected is narrow, the character information to be projected is turned back, and in FIG. 12, an example in which the projection is divided into two planes has been described. However, there are cases where it is absolutely necessary to project all character information and the like in one plane area. In the fifth embodiment, this is realized.

In this embodiment, similarly to the first embodiment, steps S1 to S4 in FIG. 20 are executed, and it is determined whether or not a marker pattern is detected in step S4. When the marker pattern is not detected, it can be determined that the marker pattern has not been detected because the marker pattern image larger than the plane having a predetermined width has been projected. For this reason, in step S15, the projection marker enlargement / reduction unit (marker size changing unit) 39 reduces the marker pattern by a predetermined rate. For example, it is multiplied by 0.9 and the process returns to step S1. In this way, as shown in FIG. 21, steps S1 to S4 are repeated until the marker pattern is detected while reducing the marker pattern.

After the marker pattern is detected in step S4, the direction of the plane normal vector is calculated in step S5. Then, in the next step S6a, as in step S6 of the first embodiment, the projected character deforming unit (first image size changing unit) 31 converts the affine transformation according to the direction of the normal vector into the projection target character information or the like. Apply. In step S6a, when performing the affine transformation, the size of the projection target character information and the like is reduced and deformed by the reduction rate of the marker pattern.

This makes it possible to project all projection target character information and the like in one plane.

According to the embodiment described above, since the subject is imaged in a state where arbitrary character information or the like is projected onto the plane area of the subject, the character information or the like is embedded in the captured image in a natural form. Since the projector function can be used in this way, the utility value of the projector function is improved and the usability of the projector function is also improved.

In each of the above-described embodiments, the planar area of the subject is detected with a marker pattern. However, when photographing a natural object as a subject, a complete “plane” often does not exist. The user may set a threshold value for how many planes are used as the “plane” for character information projection. Alternatively, although not a complete plane, the CPU 22 may automatically determine that it is a “plane” on which character information can be projected.

For example, when a straight line in a marker pattern is projected onto an uneven surface, it becomes a projected image of a zigzag line instead of a straight line image. Alternatively, the surface of a tree or the like is close to the surface of a cylinder, and the straight projection image is an arc line image. When these lines can be regarded as straight lines within a certain threshold range, they can be regarded as planes on which character information can be projected.

Also, if the processing capability of the CPU 22 is high, it is possible to handle uneven surfaces and curved surfaces as a plurality of small plane aggregates, obtain normal vectors of individual planes, and transform character information projected on the individual planes. become. Thereby, for example, when photographing a subject with a small bird on a branch, it is possible to capture a subject image in which character information indicating the photographing date and photographing location is projected on the image of the branch.

At this time, if the CPU 22 can recognize that the elongated flat surface capable of projecting character information extends obliquely, the character information is drawn along the oblique plane by drawing the character information obliquely on the small liquid crystal display unit in the projector 17. Can be projected.

In addition, although it demonstrated as another embodiment from 1st Embodiment to 5th Embodiment and its modification, respectively, it is also possible to implement combining some arbitrary embodiment.

As described above, the digital camera has been described as the embodiment of the photographing apparatus of the present invention, but the configuration of the photographing apparatus is not limited to this. As another imaging device of the present invention, for example, a built-in type or an external type PC camera or a portable terminal device having a shooting function as described below can be used.

Examples of the portable terminal device that is an embodiment of the photographing apparatus of the present invention include a mobile phone, a smartphone, a PDA (Personal Digital Assistant), and a portable game machine. Hereinafter, a smartphone will be described as an example, and will be described in detail with reference to the drawings.

FIG. 22 shows the appearance of a smartphone 200 that is an embodiment of the photographing apparatus of the present invention. A smartphone 200 illustrated in FIG. 22 includes a flat housing 201, and a display input in which a display panel 202 as a display unit and an operation panel 203 as an input unit are integrated on one surface of the housing 201. Part 204 is provided. Such a housing 201 includes a speaker 205, a microphone 206, an operation unit 207, and a camera unit 208. Note that the configuration of the housing 201 is not limited thereto, and for example, a configuration in which the display unit and the input unit are independent can be employed, or a configuration having a folding structure and a slide mechanism can be employed. Although not shown in FIG. 22, the projector 17 (see FIG. 23) that projects an image of character information or the like on the subject is provided as in FIG. 1.

FIG. 23 is a block diagram showing a configuration of the smartphone 200 shown in FIG. As shown in FIG. 23, the main components of the smartphone include a wireless communication unit 210, a display input unit 204, a call unit 211, an operation unit 207, a camera unit 208, a storage unit 212, and an external input / output unit. 213, a GPS (Global Positioning System) receiving unit 214, a motion sensor unit 215, a power supply unit 216, and a main control unit 220. As a main function of the smartphone 200, a wireless communication function for performing mobile wireless communication via a base station device BS (not shown) and a mobile communication network NW (not shown) is provided.

The wireless communication unit 210 performs wireless communication with the base station apparatus BS accommodated in the mobile communication network NW according to an instruction from the main control unit 220. Using this wireless communication, transmission and reception of various file data such as audio data and image data, e-mail data, and reception of Web data and streaming data are performed.

The display input unit 204 displays images (still images and moving images), character information, and the like, visually transmits information to the user under the control of the main control unit 220, and detects user operations on the displayed information. A so-called touch panel, which includes a display panel 202 and an operation panel 203.

The display panel 202 uses an LCD (Liquid Crystal Display), an OELD (Organic Electro-Luminescence Display), or the like as a display device. The operation panel 203 is a device on which an image displayed on the display surface of the display panel 202 is placed so as to be visible, and detects one or a plurality of coordinates operated by a user's finger or stylus. When this device is operated with a user's finger or stylus, a detection signal generated due to the operation is output to the main control unit 220. Next, the main control unit 220 detects an operation position (coordinates) on the display panel 202 based on the received detection signal. The projector 17 is connected to the main control unit 220.

As shown in FIG. 22, the display panel 202 and the operation panel 203 of the smartphone 200 exemplified as an embodiment of the photographing apparatus of the present invention integrally constitute a display input unit 204. The arrangement 203 covers the display panel 202 completely. When such an arrangement is adopted, the operation panel 203 may have a function of detecting a user operation even in an area outside the display panel 202. In other words, the operation panel 203 includes a detection area (hereinafter referred to as a display area) for an overlapping portion that overlaps the display panel 202 and a detection area (hereinafter, a non-display area) for an outer edge portion that does not overlap the other display panel 202. May be included).

Although the size of the display area and the size of the display panel 202 may be completely matched, it is not always necessary to match the two. In addition, the operation panel 203 may include two sensitive areas of the outer edge portion and the other inner portion. Further, the width of the outer edge portion is appropriately designed according to the size of the housing 201 and the like. Furthermore, examples of the position detection method employed in the operation panel 203 include a matrix switch method, a resistance film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, a capacitance method, and the like. You can also

The call unit 211 includes a speaker 205 and a microphone 206, converts user's voice input through the microphone 206 into voice data that can be processed by the main control unit 220, and outputs the voice data to the main control unit 220. 210 or the audio data received by the external input / output unit 213 is decoded and output from the speaker 205. As shown in FIG. 22, for example, the speaker 205 can be mounted on the same surface as the display input unit 204, and the microphone 206 can be mounted on the side surface of the housing 201.

The operation unit 207 is a hardware key using a key switch or the like, and receives an instruction from the user. For example, as illustrated in FIG. 22, the operation unit 207 is mounted on the side surface of the housing 201 of the smartphone 200 and is turned on when pressed with a finger or the like, and turned off when the finger is released with a restoring force such as a spring. It is a push button type switch.

The storage unit 212 includes a control program and control data of the main control unit 220, application software, address data that associates the name and telephone number of a communication partner, transmitted / received e-mail data, Web data downloaded by Web browsing, The downloaded content data is stored, and streaming data and the like are temporarily stored. The storage unit 212 includes an internal storage unit 217 built in the smartphone and an external storage unit 218 having a removable external memory slot. Each of the internal storage unit 217 and external storage unit 218 constituting the storage unit 212 includes a flash memory type (flash memory type), a hard disk type (hard disk type), a multimedia card micro type (multimedia card micro type), It is realized using a storage medium such as a card type memory (for example, MicroSD (registered trademark) memory), a RAM (Random Access Memory), a ROM (Read Only Memory) or the like.

The external input / output unit 213 serves as an interface with all external devices connected to the smartphone 200, and communicates with other external devices (for example, universal serial bus (USB), IEEE 1394, etc.) or a network. (For example, the Internet, wireless LAN, Bluetooth (registered trademark), RFID (Radio Frequency Identification), infrared communication (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), ZigBee ( ZigBee) (registered trademark, etc.) for direct or indirect connection.

Examples of external devices connected to the smartphone 200 include a memory card connected via a wired / wireless headset, wired / wireless external charger, wired / wireless data port, card socket, and SIM (Subscriber). Identity Module Card) / UIM User Identity Module Card, external audio / video equipment connected via audio / video I / O (Input / Output) terminal, external audio / video equipment connected wirelessly, yes / no There are a wirelessly connected smartphone, a wired / wireless personal computer, a wired / wireless PDA, a wired / wireless personal computer, an earphone, and the like. The external input / output unit 213 transmits data received from such an external device to each component inside the smartphone 200, or allows the data inside the smartphone 200 to be transmitted to the external device. Can do.

The GPS receiving unit 214 receives GPS signals transmitted from the GPS satellites ST1 to STn in accordance with instructions from the main control unit 220, executes positioning calculation processing based on the received GPS signals, and calculates the latitude of the smartphone 200. Detect the position consisting of longitude and altitude. When the GPS reception unit 214 can acquire position information from the wireless communication unit 210 or the external input / output unit 213 (for example, a wireless LAN), the GPS reception unit 214 can also detect the position using the position information.

The motion sensor unit 215 includes, for example, a three-axis acceleration sensor, and detects the physical movement of the smartphone 200 in accordance with an instruction from the main control unit 220. By detecting the physical movement of the smartphone 200, the moving direction and acceleration of the smartphone 200 are detected. The detection result is output to the main control unit 220.

The power supply unit 216 supplies power stored in a battery (not shown) to each unit of the smartphone 200 in accordance with an instruction from the main control unit 220.

The main control unit 220 includes a microprocessor, operates according to a control program and control data stored in the storage unit 212, and controls each unit of the smartphone 200 in an integrated manner. In addition, the main control unit 220 includes a mobile communication control function that controls each unit of the communication system and an application processing function in order to perform voice communication and data communication through the wireless communication unit 210.

The application processing function is realized by the main control unit 220 operating according to the application software stored in the storage unit 212. Application processing functions include, for example, an infrared communication function that controls the external input / output unit 213 to perform data communication with the opposite device, an e-mail function that transmits and receives e-mails, and a web browsing function that browses web pages. .

Also, the main control unit 220 has an image processing function such as displaying video on the display input unit 204 based on image data (still image or moving image data) such as received data or downloaded streaming data. The image processing function is a function in which the main control unit 220 decodes the image data, performs image processing on the decoding result, and displays an image on the display input unit 204.

Further, the main control unit 220 executes display control for the display panel 202 and operation detection control for detecting a user operation through the operation unit 207 and the operation panel 203. By executing the display control, the main control unit 220 displays an icon for starting application software, a software key such as a scroll bar, or displays a window for creating an e-mail. Note that the scroll bar refers to a software key for accepting an instruction to move the display portion of a large image that does not fit in the display area of the display panel 202.

In addition, by executing the operation detection control, the main control unit 220 detects a user operation through the operation unit 207 or accepts an operation on the icon or an input of a character string in the input field of the window through the operation panel 203. Or a display image scroll request through a scroll bar.

Further, by executing the operation detection control, the main control unit 220 causes the operation position with respect to the operation panel 203 to overlap with the display panel 202 (display area) or other outer edge part (non-display area) that does not overlap with the display panel 202. And a touch panel control function for controlling the sensitive area of the operation panel 203 and the display position of the software key.

The main control unit 220 can also detect a gesture operation on the operation panel 203 and execute a preset function in accordance with the detected gesture operation. Gesture operation is not a conventional simple touch operation, but an operation of drawing a trajectory with at least one position from a plurality of positions by drawing a trajectory with a finger or the like, or simultaneously specifying a plurality of positions. means.

The camera unit 208 is a digital camera that performs electronic photography using an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge-Coupled Device). In addition, the camera unit 208 converts image data obtained by imaging into compressed image data such as JPEG (Joint Photographic coding Experts Group) under the control of the main control unit 220 and records the data in the storage unit 212 or externally. The data can be output through the input / output unit 213 and the wireless communication unit 210. In the smartphone 200 shown in FIG. 22, the camera unit 208 is mounted on the same surface as the display input unit 204, but the mounting position of the camera unit 208 is not limited to this, and the camera unit 208 may be mounted on the back surface of the display input unit 204. Alternatively, a plurality of camera units 208 may be mounted. When a plurality of camera units 208 are installed, the camera unit 208 used for shooting can be switched for shooting alone, or a plurality of camera units 208 can be used for shooting simultaneously. it can.

In addition, the camera unit 208 can be used for various functions of the smartphone 200. For example, an image acquired by the camera unit 208 can be displayed on the display panel 202, or the image of the camera unit 208 can be used as one of operation inputs of the operation panel 203. Further, when the GPS receiving unit 214 detects a position, the position can be detected with reference to an image from the camera unit 208. Furthermore, referring to the image from the camera unit 208, the optical axis direction of the camera unit 208 of the smartphone 200 is determined without using the triaxial acceleration sensor or in combination with the triaxial acceleration sensor. It is also possible to determine the current usage environment. Of course, the image from the camera unit 208 can also be used in the application software.

A camera unit 208 shown in FIG. 22 is a camera for self-portrait when making a videophone call. Is provided.

In addition, the position information acquired by the GPS receiver 214 to the image data of the still image or the moving image, the voice information acquired by the microphone 206 (the text information may be converted into voice information by the main control unit or the like), Posture information and the like acquired by the motion sensor unit 215 can be added and stored in the storage unit 212, or can be output through the external input / output unit 213 or the wireless communication unit 210.

As described above, the imaging device with a projector according to the present embodiment includes an imaging unit that images a subject, a first image that is an image of a set pattern in which characters, figures, symbols, or any of these are combined, and a preset image. A projector that projects a marker pattern of a shape formed on the subject, and a captured image in a state in which the plurality of marker patterns are respectively projected onto different planar areas of the subject by the projector, and each of the captured images An inclination calculation unit that calculates the inclination of each of the planar areas on which the marker pattern is projected from the degree of deformation of the marker pattern, and a planar area that has the smallest degree of deformation of the marker pattern is selected from the calculated inclination values. Depending on the plane selection part and the value of the inclination relative to the plane area selected by the plane selection part, A first image deforming unit that deforms the first image projected on the subject by reversing the expansion / contraction direction of the deformation, and the first image deformed by the first image deforming unit is An imaging control unit that images the subject with the imaging unit in a state of being projected onto the selected planar area.

In addition, the photographing apparatus with a projector according to the embodiment captures the live view image of the subject, determines the color of the selected plane area of the subject, and the marker having a color different from the determined color A marker color setting unit for projecting a pattern by the projector is provided.

In addition, the photographing apparatus with a projector according to the embodiment captures the live view image of the subject, determines the color of the selected plane area of the subject, and determines the color of the subject that is different from the determined color. A color setting unit for projecting one image by the projector is provided.

The imaging apparatus with a projector according to the embodiment includes a marker size changing unit that changes the size of the marker pattern when the marker pattern projected from the projector onto the subject is undetectable, and the changed marker pattern And a first image size changing unit that deforms the first image projected from the projector according to the inclination of the planar area detected by the inclination calculating unit.

In addition, the first image deformation unit of the photographing apparatus with a projector according to the embodiment changes the first image to a folded image that is folded halfway when the first image before the change does not fit in the planar area. It is characterized by doing.

Further, the first image deformation unit of the photographing apparatus with a projector according to the embodiment has a second plane area adjacent to the first plane area when the first image does not fit in the first plane area. The first image of a portion of the first image that is out of the first plane area is deformed according to the inclination of the second plane area, and the first image is transformed into the first plane area and An image for generating a projection divided into the second plane region is generated.

In addition, the imaging control method of the imaging apparatus with a projector according to the embodiment includes an imaging unit that images a subject, a first image that is an image of a collective pattern obtained by combining characters, figures, symbols, or any of these, and a preset image And a projector for projecting a marker pattern of a different shape onto the subject, and a captured image obtained by projecting a plurality of the marker patterns onto different planar areas of the subject by the projector. A step of calculating the inclination of each of the planar areas on which the marker pattern is projected from the degree of deformation of each marker pattern in the captured image, and the deformation of the marker pattern from the calculated inclination value A step of selecting the plane area having the smallest degree and the plane area selected above The step of deforming the first image projected onto the subject with the inclination direction of the deformation reversed according to the value of the inclination with respect to the subject, and the deformed first image of the subject by the projector. Imaging the subject with the imaging unit in a state of being projected onto the selected plane area.

According to the embodiment described above, arbitrary character information is projected onto an arbitrary plane area of the subject, and the subject image is imaged together with the projected character information image, so that the usability and utility value of the projector function can be improved. It becomes possible.

The imaging apparatus with a projector according to the present invention can project character information and the like onto a subject and image the subject, so that the character information and the like can be embedded in the captured image in a natural state, and the projector function can be used. This is useful for spreading high-value imaging devices.

This application is based on a Japanese patent application filed on March 16, 2012 (Japanese Patent Application No. 2012-60380), the contents of which are incorporated herein by reference.

DESCRIPTION OF SYMBOLS 10,200 Image | photographing apparatus 12 with a projector Image | photographing lens 17 Front projection type projector 21 Image sensor 22,220 Control part (CPU)
31 Projected character deformation section (character information deformation section)
32 Planar Normal Vector Calculation Unit 33 Marker Detection Unit 34 Plane Selection Unit 35 Marker Color Setting Unit 36 Complementary Color Calculation Unit 37 Projection Surface Color Discrimination Unit 38 Projection Character Color Setting Unit 39 Projection Marker Enlargement / Reduction Unit 51 Marker Pattern

Claims

An imaging unit for imaging a subject;
A first image that is an image of a character, figure, symbol, or a set pattern obtained by combining any of these, and a projector that projects a marker pattern of a preset shape onto the subject,
The plane on which a plurality of marker patterns are projected by the projector in a state where each of the marker patterns is projected onto different plane areas of the subject, and the marker pattern is projected from the degree of deformation of each marker pattern in the captured image. An inclination calculator for calculating the inclination of each area;
A plane selection unit that selects a plane region having the smallest deformation degree of the marker pattern from the calculated inclination value;
A first image deforming unit configured to deform the first image projected onto the subject in a reverse direction of the deformation according to the value of the inclination with respect to the plane region selected by the plane selecting unit;
An imaging control unit that images the subject by the imaging unit in a state in which the first image deformed by the first image transformation unit is projected by the projector onto the selected plane area of the subject;
An imaging device with a projector.
The photographing apparatus with a projector according to claim 1,
Marker color setting unit that captures the live view image of the subject, determines the color of the selected plane area of the subject, and causes the projector to project the marker pattern having a color different from the determined color An imaging device with a projector.
The imaging device with a projector according to claim 1 or 2,
A color setting unit that captures the live view image of the subject, determines the color of the selected plane area of the subject, and causes the projector to project the first image having a color different from the determined color An imaging device with a projector.
A photographing apparatus with a projector according to any one of claims 1 to 3,
A marker size changing unit that changes the size of the marker pattern when the marker pattern projected onto the subject from the projector is a size that cannot be detected;
A first image size changing unit that deforms the first image projected from the projector according to the changed size of the marker pattern and the inclination of the planar area detected by the inclination calculating unit;
An imaging device with a projector.
A photographing device with a projector according to any one of claims 1 to 4,
The imaging device with a projector, wherein the first image deforming unit changes the first image to a folded image that is folded halfway when the first image before the change does not fit in the planar area.
A photographing device with a projector according to any one of claims 1 to 5,
The first image deforming unit may include the first image of the first image in a second plane area adjacent to the first plane area when the first image does not fit in the first plane area. The first image of the portion outside the plane area is deformed according to the inclination of the second plane area, and the first image is divided into the first plane area and the second plane area. An imaging device with a projector that generates an image for projection.
An imaging unit that images a subject, a first image that is an image of a set pattern in which characters, figures, symbols, or any of these are combined, and a projector that projects a marker pattern having a preset shape onto the subject, respectively. A photographing control method for a photographing apparatus with a projector, comprising:
Capturing captured images obtained by projecting a plurality of the marker patterns on the different planar regions of the subject by the projector;
Calculating the inclination of each of the planar areas on which the marker pattern is projected from the degree of deformation of each marker pattern in the captured image;
Selecting a plane area having the smallest deformation degree of the marker pattern from the calculated inclination value;
Transforming the first image projected onto the subject in a reverse direction of the deformation according to the value of the inclination with respect to the selected plane region;
Imaging the subject with the imaging unit in a state in which the deformed first image is projected onto the selected plane area of the subject by the projector;
Control method for a photographing apparatus with a projector including: