JP5906745B2 - Image display device, image display method, and program - Google Patents

Description

  The present invention relates to an image display device, an image display method, and a program.

  Conventionally, a dedicated marker having a high contrast, such as a two-dimensional barcode, is detected from an image captured using a mobile terminal or the like, and a virtual object (for example, spatially aligned with the detected marker position (for example, Augmented reality (AR) technology for expanding a user's real environment by additionally displaying a three-dimensional model or the like is known (see, for example, Patent Document 1).

JP 2011-76589 A

  However, in the above technique, it is necessary to provide a marker at a position where a virtual object is to be additionally displayed. Therefore, when a virtual object is to be displayed on a part of the subject or on the whole, a marker that is completely unrelated to the subject is displayed on the part of the subject. Or it has to be arranged in the whole, and there is a problem that the quality of the subject itself is lowered. Further, although the displayed virtual object is related to the imaged object, there is a possibility that an uncomfortable feeling far from the content of the image (particularly the main subject) may be displayed.

  The present invention has been made in view of such problems, and an object of the present invention is to display an image having a high affinity for the contents of the subject without deteriorating the quality of the subject to be imaged. An image display apparatus, an image display method, and a program that can be used.

In order to solve the above-described problem, one aspect of the present invention provides an extraction unit that extracts a specific image area in a quadrilateral frame shape of a predetermined image including a specific subject, and a predetermined image with respect to the predetermined image. An image processing means for creating a processed image that has been subjected to the conversion process, and a projective conversion process is performed on the processed image so as to correspond to the inclination of the specific image area of the quadrilateral frame shape, and the four sides A projection conversion processing means for generating a shape image; a generation means for generating a composite image obtained by synthesizing a projection conversion image of the processed image in the specific image area; the predetermined image; and the processed image. Among the plurality of images acquired by the first acquisition unit, the first acquisition unit that sequentially acquires a plurality of images in which the specific subject is continuously captured, Feature points from one image A plurality of feature point extracting means and a processed image corresponding to the predetermined image are acquired from the recording means and associated with each feature point similar to the plurality of feature points extracted by the feature point extracting means. A second acquisition unit that acquires the processed image from the recording unit, and the generation unit generates an image area corresponding to a plurality of feature points extracted by the feature point extraction unit in the one image. On the other hand, the image display device is characterized in that a composite image is generated by combining the processed images acquired by the second acquisition means .

  According to the present invention, it is possible to display an image having high affinity for the content of the subject without degrading the quality of the subject to be imaged.

It is a block diagram which shows schematic structure of the portable terminal of Embodiment 1 to which this invention is applied. It is a figure which shows an example of the 1st processed image table registered into the portable terminal of FIG. It is a figure which shows typically an example of the feature point information extracted by the portable terminal of FIG. 3 is a flowchart illustrating an example of an operation related to an image registration process by the mobile terminal of FIG. 1. It is a figure which shows typically an example of the image which concerns on the image registration process of FIG. 3 is a flowchart illustrating an example of an operation related to image display processing by the mobile terminal of FIG. 1. It is a figure for demonstrating the image display process of FIG. It is a figure which shows typically an example of the image which concerns on the image display process of FIG. It is a figure which shows typically an example of the image which concerns on the image display process of FIG. It is a figure for demonstrating the image display process of FIG. It is a block diagram which shows schematic structure of the portable terminal of Embodiment 2 to which this invention is applied. 12 is a flowchart illustrating an example of an operation related to an image registration process performed by the mobile terminal in FIG. 11. 13 is a flowchart illustrating an example of an operation related to image display processing by the mobile terminal of FIG. It is a figure for demonstrating the image display process of FIG. It is a figure for demonstrating the image display process of FIG. 1 is a diagram showing a schematic configuration of an image display system according to the present invention.

  Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.

[Embodiment 1]
FIG. 1 is a block diagram showing a schematic configuration of a mobile terminal 100 according to the first embodiment to which the present invention is applied.
As shown in FIG. 1, the mobile terminal 100 includes a central control unit 1, a memory 2, an imaging unit 3, an imaging control unit 4, an image data generation unit 5, a recording unit 6, an image processing unit 7, a display control unit 8, and a display. Unit 9, transmission / reception unit 10, communication control unit 11, operation input unit 12 and the like.
The mobile terminal 100 includes, for example, an imaging device having a communication function, a mobile station used in a mobile communication network such as a mobile phone or a PHS (Personal Handy-phone System), a PDA (Personal Data Assistants), and the like. Has been.

  The central control unit 1 controls each unit of the mobile terminal 100. Specifically, the central control unit 1 includes a CPU (not shown) that controls each part of the mobile terminal 100, and performs various control operations according to various processing programs (not shown).

  The memory 2 is composed of, for example, a DRAM (Dynamic Random Access Memory) or the like, and is a buffer memory that temporarily records image information, a working memory such as the central control unit 1, and various programs related to the functions of the mobile terminal 100 And a program memory (not shown) in which data is stored.

The imaging unit 3 captures a specific subject S and generates an image signal of the frame image F.
Here, the specific subject S may be a two-dimensional image or a three-dimensional solid. For example, when the specific subject S is a two-dimensional image, a photograph placed on the wall surface can be cited (see FIG. 7).

The imaging unit 3 includes a lens unit 3a and an electronic imaging unit 3b.
The lens unit 3a includes a plurality of lenses such as a zoom lens and a focus lens.
The electronic imaging unit 3b is composed of, for example, an image sensor such as a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS), and converts an optical image that has passed through various lenses of the lens unit 3a into a two-dimensional image signal. To do.
In addition, although illustration is abbreviate | omitted, the imaging part 3 may be provided with the aperture_diaphragm | restriction which adjusts the quantity of the light which passes the lens part 3a.

The imaging control unit 4 controls the imaging of the subject by the imaging unit 3. That is, the imaging control unit 4 includes a timing generator, a driver, and the like, although not illustrated. Then, the imaging control unit 4 scans and drives the electronic imaging unit 3b with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit 3b at a predetermined period, and the electronic imaging unit 3b. The frame image F is read out from the imaging area for each screen and is output to the image data generation unit 5.
In addition, the imaging control unit 4 performs adjustment control of imaging conditions of the subject such as AF (automatic focusing process), AE (automatic exposure process), AWB (automatic white balance), and the like.

The image data generation unit 5 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the frame image F transferred from the electronic image pickup unit 3b, and then samples and holds it by a sample hold circuit (not shown). The digital data is converted into digital data by an A / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb, Cr (YUV data) is generated.
Further, the image data generation unit 5 reduces the YUV data of the generated frame image F at a predetermined magnification in both horizontal and vertical directions, and generates low-resolution (for example, VGA, QVGA size, etc.) image data for live view display. Is generated. Specifically, the image data generation unit 5 has low resolution image data for live view display from the YUV data of the frame image F at a predetermined timing corresponding to a predetermined display frame rate of the live view image by the display unit 9. Is generated.

  Then, the image processing unit 7 sequentially outputs the generated YUV data of each frame image F to the memory 2 and stores it in the memory 2.

  The recording unit 6 is composed of, for example, a nonvolatile memory (flash memory) or the like, and performs writing and reading of various data. Specifically, the recording unit 6 records, for example, image information as an Exif format image file, and encodes it in a predetermined compression format (eg, JPEG format) by an encoding unit (not shown) of the image processing unit 7. At least one piece of recorded image information for recording is recorded.

The recording unit 6 records the first processed image table T1 (see FIG. 2).
The first processed image table T1 is a table for recording information registered by the image registration process. Specifically, as shown in FIG. 2, the first processed image table T1 includes an “image ID” (for example, “A” to “C”,...) For identifying an image to be registered. And “feature information” (for example, “feature information 1” to “feature information 3”,...) Related to the feature information of the image, and a processed image P2 obtained by performing a predetermined image conversion process on the image. Are associated with “processed images” (for example, “first processed image” to “third processed image”,...).

Here, the image ID identifies a predetermined image P1 including a specific subject S. In other words, the image ID is associated with a predetermined image P1 captured using an object that is substantially equal to the specific subject S captured as a live view image by the imaging unit 3 as an imaging target. For example, in the case of a framed photo (see FIG. 7), the image ID is registered in association with the image data of the original image of the photo recorded in the recording unit 6.
The “image ID” is schematically represented as “A” to “C”, but is only an example and is not limited thereto, and can be arbitrarily changed as appropriate.

The feature information includes various feature amounts such as shapes and colors related to feature points extracted by a feature amount calculation unit 7e of the image processing unit 7 described later. For example, when feature information is expressed using SIFT (Scale-Invariant Features Transform) feature amounts, it is defined by the coordinate position of each feature point in the XY plane space and an m-dimensional feature vector (see FIG. 3). .
Further, the feature information is extracted from the first square image P1a (see FIG. 5B) generated by performing the square conversion process on the image data of the predetermined image P1 to be registered. Information on a plurality of feature points. Here, the square conversion process is specifically a process of relatively enlarging the short side of the predetermined image P1 so as to match the long side, or a relative so that the long side is matched with the short side. Including a process of automatically reducing.
Note that the feature information may be expressed using a SURF (Speeded Up Robust Features) feature quantity, or a brightness value (0 to 255) expressing the brightness level of the feature point in 8 bits (256 levels). Alternatively, RGB values (0 to 255) expressing each color component of RGB in 8 bits (256 levels) may be used.
Further, since SIFT feature values and SURF feature values are known techniques, detailed description thereof is omitted here.
The “feature information” is schematically represented as “feature information 1” to “feature information 3”. However, the “feature information” is merely an example, and is not limited thereto, and can be arbitrarily changed as appropriate.

The types of image conversion processing include, for example, various art conversion processing (for example, “color pencil style”, “oil painting style”, “watercolor style”, etc .; details will be described later) by an image conversion processing section 7a, which will be described later, Processing, gradation correction processing, binarization processing, and the like.
The processed image P2 may be a single still image or a moving image composed of a plurality of frame images. In addition, whether each of the frame images F constituting the moving image has been subjected to image conversion processing can be arbitrarily changed.
In addition, although “processed image” is schematically represented as “first processed image” to “third processed image”, it is an example, and the present invention is not limited to this, and can be arbitrarily changed as appropriate.

As described above, the recording unit 6 performs a predetermined image conversion process on the predetermined image P1 in association with a plurality of feature points extracted from the predetermined image P1 obtained by capturing a specific subject S in advance. The processed image P2 is recorded.
Note that the recording unit 6 may be configured to include a recording medium such as a nonvolatile memory (flash memory), and in this case, may include a mechanism for attaching and detaching the recording medium.

  The image processing unit 7 includes an image conversion processing unit 7a, a first acquisition unit 7b, an image region specifying unit 7c, a square image generation unit 7d, a feature amount calculation unit 7e, a second acquisition unit 7f, a quadrilateral image generation unit 7g, a composition An image generation unit 7h is provided.

The image conversion processing unit 7a performs an image conversion process on a predetermined image P1 that is a registration process target.
Image conversion processing is processing for converting the visual effect of an image, and includes processing for changing the display mode of a still image, processing for generating a moving image composed of a plurality of frame images F,. .

Specifically, the image conversion processing unit 7a performs an art conversion process for processing a predetermined image P1 that is an object of registration processing into an image having various visual effects.
Here, the art conversion process is an image process for changing the visual effect of the predetermined image P1 to be processed, that is, the display mode in the state displayed on the display unit 9. As art conversion processing, for example, “color pencil style” (see FIG. 5C) that is processed into an image having a visual effect written with a colored pencil, or an image having a visual effect drawn with an oil paint. "Oil painting style" to be processed into "Watercolor painting style" that is processed into an image having a visual effect as drawn with watercolor paints, etc., but is not limited to these, It can be changed.
The image conversion processing unit 7a performs an art conversion process (for example, “color pencil style”) having a predetermined processing content designated based on a predetermined operation of the operation input unit 12 by the user on the predetermined image P1. Specifically, the image conversion processing unit 7a, among the images recorded in the recording unit 6, image data (YUV data) of a predetermined image P1 designated based on a predetermined operation of the operation input unit 12 by the user. Is obtained as a target of registration processing (see FIG. 5A). After that, the image conversion processing unit 7a performs an art conversion process (for example, “color pencil style”) of predetermined processing contents designated based on a predetermined operation of the operation input unit 12 by the user on the image data. Thus, a processed image P2 (see FIG. 5C) processed into a picture-like image is generated.

  Note that the technology for processing an image into an image having various visual effects is realized by, for example, substantially the same processing as processing using known image processing related software, and hue, saturation, and brightness in the HSV color space. However, since this is a known technique, detailed description thereof is omitted. “OO style” refers to a visual effect that has been subjected to art conversion processing that can be realized by known image processing-related software.

  Note that the image conversion process is not limited to the art conversion process for processing the predetermined image P1 into a picture-like image. can do. The image conversion processing unit 7a may convert the predetermined image P1 into a moving image composed of a plurality of frame images.

The first acquisition unit 7b sequentially acquires a plurality of frame images F,.
That is, the first acquisition unit 7b sequentially acquires a plurality of images in which a specific subject S is continuously captured. Specifically, a live view display of a plurality of frame images F,..., In which a specific subject S (for example, a framed photograph or the like) is continuously captured by the imaging unit 3 and sequentially generated by the image data generation unit 5. The low-resolution image data for use is sequentially acquired from the memory 2.

The image area specifying unit 7c specifies a specific image area A.
That is, the image area specifying unit 7c specifies a specific image area A in one image among the plurality of images acquired by the first acquiring unit 7b. Specifically, the image area specifying unit 7c is included in the low-resolution image data for live view display of one frame image F among the plurality of frame images F acquired by the first acquiring unit 7b. A specific image area A having a quadrangular frame shape is specified (see FIG. 8A). The image area specifying unit 7c calculates the contrast of low-resolution image data for live view display of one frame image F, for example, and changes the contrast with respect to the color (for example, white) of the adjacent area. A region composed of a large color (for example, black) is specified as a specific image region A.
Specifically, for example, when taking a framed photo, the image area specifying unit 7c has a rectangular frame G having a predetermined color in one frame image F generated by taking the photo. A specific image area A having a quadrangular frame shape corresponding to is specified.
The specific image area A is exemplified by a quadrilateral frame shape having a predetermined color. However, the specific image area A is not limited to this, and the specific image area A is included in one frame image F. The region can be arbitrarily changed as long as the region has a color and shape that can be distinguished from other regions.

The square image generation unit 7d generates a square image.
That is, the square image generation unit 7d performs a square conversion process on the image of the quadrilateral area A1 inside the specific image area A having the quadrilateral frame shape specified by the image area specifying unit 7c, thereby performing the second square image. P3 is generated. Specifically, for example, the square image generation unit 7d specifies the coordinate positions of the corners inside the four vertices of the specific image region A having a quadrilateral frame shape, and is surrounded by the four inner corners. In this area, a straight line detection process (for example, a Hough transform process) is performed to specify a quadrilateral area A1 inside a specific image area A having a quadrilateral frame shape (see FIG. 8B). Thereafter, the square image generation unit 7d acquires image data of the image of the quadrangular region A1, performs a square conversion process on the image data, and relatively enlarges the short side to match the long side. Alternatively, the second square image P3 (see FIG. 8C) is generated by relatively reducing the long side to match the short side.
Note that the square image generation unit 7d may specify a region surrounded by the corners inside the four vertices of the specific image region A having a quadrilateral frame shape as the quadrilateral region A1. For example, in the case of a framed photo, if the inner edge of the frame G overlaps the photo, there is no gap between the frame G and the photo in plan view, and therefore the square image generating unit 7d Specifies a region surrounded by four sides connecting two adjacent corners among the corners inside the four vertices of the specific image region A having a quadrangular frame shape as the quadrangular region A1.

Further, the square image generation unit 7d obtains a copy of the predetermined image P1 to be registered, that is, the image data of the image before the art conversion processing is performed by the image conversion processing unit 7a. A square conversion process is performed on the data to generate a first square image P1a (see FIG. 5B).
Here, the number of pixels in each of the horizontal and vertical directions of the first square image P1a and the second square image P3 is preferably equal to each other. In this case, the first square image P1a and the second square image P3 are the farthest pixel in the horizontal direction and the farthest pixel in the vertical direction (xn, ym) with the position of the origin (0, 0) being matched. The positions of are matched.

The feature amount calculation unit 7e performs a feature extraction process of extracting effective feature points from one frame image F among the plurality of frame images F,.
That is, the feature amount calculation unit 7e extracts a plurality of effective feature points from one frame image F among the plurality of frame images F acquired by the first acquisition unit 7b. Specifically, the feature amount calculation unit 7e has a large amount of shading information in a region around the pixel such as a corner constituted by a contrast boundary from within the specific image region A specified by the image region specifying unit 7c. Multiple effective feature points are extracted.
For example, the feature amount calculation unit 7e performs a predetermined feature extraction process on the image data of the second square image P3 generated by the square image generation unit 7d out of one frame image F for live view display. A plurality of feature points are extracted, and SIFT feature amounts (coordinate positions of each pixel in the XY plane space, m-dimensional feature vectors, etc.) at the effective feature points are calculated. Further, the feature amount calculation unit 7e associates various pieces of information related to the extracted feature points and temporarily records them in the memory 2 as feature point information D (see FIG. 3).
As shown in FIG. 3, the SIFT feature amount includes “feature point numbers” (for example, “1” to “3”,...) Obtained by numbering feature points in a predetermined order, and an XY plane space of each feature point. “X-coordinate” and “y-coordinate” (for example, “36, 120” to “612, 398”,... (For example, “127, 25, 68, 495, 54, 2, 866,...” To “68, 5, 47, 2, 5, 9, 87, 56, 475,...”) Are associated with each other.
The “feature point No.” is schematically represented as “1” to “3”, and the “x coordinate” and the “y coordinate” are schematically represented as “36, 120” to “612, 398”. In addition, “feature information” is schematically represented as “127, 25, 68, 495, 54, 2, 866,...” To “68, 5, 47, 2, 5, 9, 87, 56, 475,. Thus, the present invention is not limited to these and can be arbitrarily changed as appropriate.

  In addition, the feature amount calculation unit 7e performs predetermined feature extraction processing on the image data of the first square image P1a generated by the square image generation unit 7d in the same manner as described above, so that the SIFT feature at the effective feature point is obtained. The amount (coordinate position of each pixel in the XY plane space, m-dimensional feature vector, etc.) is calculated.

The second acquisition unit 7f acquires the processed image P2 from the recording unit 6.
Specifically, the second acquisition unit 7f records the plurality of feature points extracted from the specific image area A specified by the image area specifying unit 7c and the first processed image table T1 of the recording unit 6. A dissimilarity calculating unit f1 that calculates the dissimilarity with the plurality of feature points that have been processed, and based on the dissimilarity calculated by the dissimilarity calculating unit f1, processed images are processed from the first processed image table T1. P2 is specified.

That is, the difference calculation unit f1 calculates the difference between the plurality of feature points extracted by the feature amount calculation unit 7e and the plurality of feature points recorded in the recording unit 6. Specifically, the dissimilarity calculation unit f1 reads out and obtains feature information (feature point information D) of each of a plurality of feature points extracted from the second square image P3 from the memory 2 and processes each processed image. Feature information of each of a plurality of feature points extracted in advance from the first square image P1a corresponding to P2 is read from the first processed image table T1 and obtained. Then, the dissimilarity calculation unit f1 associates the feature information of each of the plurality of feature points extracted from the second square image P3 with the feature information of each of the plurality of feature points previously extracted from the first square image P1a. The degree of difference between images is calculated according to a predetermined arithmetic expression.
The second acquisition unit 7f determines whether or not the difference calculated by the difference calculation unit f1 is equal to or less than a predetermined determination value. As a result of this determination, when it is determined that the degree of difference is equal to or less than a predetermined determination value, the second acquisition unit 7f associates the plurality of feature points of the first square image P1a used for calculating the difference. The image data of the processed image P2 thus obtained is acquired from the recording unit 6.

As described above, the second acquisition unit 7f acquires the processed image P2 associated with each feature point similar to the plurality of feature points extracted by the feature amount calculation unit 7e from the recording unit 6.
Note that the configuration of the second acquisition unit 7f is an example and is not limited thereto, and can be arbitrarily changed as appropriate. For example, whether or not the difference degree calculation unit f1 is provided can be arbitrarily changed as appropriate. It is. The second acquisition unit 7f records the processed image P2 corresponding to the extracted predetermined image when the feature amount calculation unit 7e extracts an image region corresponding to the predetermined image from one image. You may make it acquire from the part 6. FIG.

The quadrilateral image generation unit 7g generates a quadrilateral image P2a from the processed image P2.
That is, the quadrilateral image generation unit 7g processes the processed image P2 acquired by the second acquisition unit 7f so as to correspond to the inclination of the specific image region A having the quadrangular frame shape specified by the image region specifying unit 7c. The image data of the quadrilateral image P2a is generated by performing projective transformation processing on the image data (see FIG. 9). Specifically, the quadrilateral image generation unit 7g performs, for example, a straight line detection process (for example, a Hough transform process) on a specific image area A having a quadrilateral frame shape or a quadrilateral area A1 inside the image area. To identify the slopes of the four line segments that make up the quadrilateral. Then, the quadrilateral image generation unit 7g correlates the inclinations of the four sides (upper, lower, left and right sides) of the processed image P2 acquired by the second acquisition unit 7f with the inclinations of the four line segments. Projective transformation processing is performed on the image data of the processed image P2, and image data of a quadrilateral image P2a having a shape substantially equal to the quadrilateral region A1 inside the specific image region A is generated.

The composite image generation unit 7h generates a composite image Fa obtained by combining the processed image P2 with the one frame image F among the plurality of frame images F,.
That is, the composite image generation unit 7h applies to an image area (for example, a quadrilateral area A1) corresponding to a plurality of feature points extracted from the one frame image F among the image areas of the one frame image F. Then, a combined image Fa is generated by combining the processed image P2 acquired by the second acquisition unit 7f. Specifically, the composite image generation unit 7h generates a composite image Fa obtained by combining the processed image P2 in the specific image region A specified by the image region specification unit 7c among the image regions of one frame image F. Generate.
For example, the composite image generation unit 7h specifies the coordinate position of each corner of the quadrilateral area A1 inside the specific image area A in the image area of one frame image F for live view display, and Image data of a combined image Fa is generated by combining the quadrilateral image P2a of the processed image P2 having a shape substantially equal to the quadrangular region A1 generated by the quadrangular image generation unit 7g so as to overlap the quadrangular region A1. .
Then, the composite image generation unit 7 h sequentially outputs the generated image data of each composite image Fa to the memory 2 and stores it in the memory 2.

The display control unit 8 performs control for reading display image data temporarily stored in the memory 2 and displaying the read image data on the display unit 9.
Specifically, the display control unit 8 includes a VRAM (Video Random Access Memory), a VRAM controller, a digital video encoder, and the like. The digital video encoder reads the luminance signal Y and the color difference signals Cb and Cr read from the memory 2 and stored in the VRAM (not shown) under the control of the central control unit 1 through the VRAM controller. Are read out at a predetermined playback frame rate (for example, 30 fps), and a video signal is generated based on these data and output to the display unit 9.
Further, the display control unit 8 causes the display unit 9 to sequentially display the composite image Fa generated by the composite image generation unit 7h. In other words, the display control unit 8 sequentially acquires the image data of the composite image Fa corresponding to one frame image F from the memory 2 and sequentially displays the composite image Fa on the display unit 9 (see FIG. 10). .

  The display unit 9 is composed of, for example, a liquid crystal display panel, and displays an image captured by the imaging unit 3 based on a video signal from the display control unit 8 on a display screen. Specifically, the display unit 9 predetermines a plurality of frame images F, which are captured by the image capturing unit 3 and the image capturing control unit 4 and generated by the image data generating unit 5 in the still image capturing mode and the moving image capturing mode. The live view image is displayed while sequentially updating at the display frame rate. The display unit 9 displays a rec view image of an image recorded as a still image or an image being recorded as a moving image.

The transmitter / receiver unit 10 performs a call with an external user of an external device connected via the communication network N.
Specifically, the transmission / reception unit 10 includes a microphone 10a, a speaker 10b, a data conversion unit 10c, and the like. The transmission / reception unit 10 performs A / D conversion processing on the user's transmission voice input from the microphone 10a by the data conversion unit 10c and outputs the transmission voice data to the central control unit 1. Under the control, voice data such as received voice data outputted and inputted from the communication control unit 11 is D / A converted by the data conversion unit 10c and outputted from the speaker 10b.

The communication control unit 11 transmits and receives data via the communication network N and the communication antenna 11a.
That is, the communication antenna 11a is a predetermined communication method (for example, W-CDMA (Wideband Code Division Multiple Access) method, CDMA2000 method, GSM) adopted by the mobile terminal 100 for communication with a radio base station (not shown). (Global System for Mobile Communications (registered trademark) system, etc.) is an antenna that can transmit and receive data. The communication control unit 11 transmits / receives data to / from the radio base station via the communication antenna 11a through a communication channel set in the communication method according to a communication protocol corresponding to a predetermined communication method.
That is, the communication control unit 11 transmits / receives voice during a call with an external user of the external device to the external device of the communication partner based on the instruction signal output from the central control unit 1 and input, Send and receive e-mail data.
Note that the configuration of the communication control unit 11 is an example and is not limited thereto, and can be arbitrarily changed as appropriate. For example, although not shown, a wireless LAN module is mounted and an access point (Access Point) is provided. It is good also as a structure which can access the communication network N via this.

  The communication network N is a communication network that connects the mobile terminal 100 via a wireless base station, a gateway server (not shown), or the like. The communication network N is a communication network constructed by using a dedicated line or an existing general public line, and applies various line forms such as a LAN (Local Area Network) and a WAN (Wide Area Network). Is possible. The communication network N includes, for example, various communication network networks such as a telephone line network, ISDN line network, dedicated line, mobile communication network, communication satellite line, CATV line network, IP network, VoIP (Voice over Internet Protocol). ) Gateways, Internet service providers, etc. are included.

The operation input unit 12 is for inputting various instructions to the mobile terminal 100 main body.
Specifically, the operation input unit 12 executes a shutter button related to a subject photographing instruction, up / down / left / right cursor buttons and a determination button related to a selection instruction of a mode, a function, etc., making / receiving a call, sending / receiving an e-mail, etc. Various buttons (not shown) such as communication-related buttons related to instructions and numeric buttons and symbol buttons related to text input instructions are provided.
When various buttons are operated by the user, the operation input unit 12 outputs an operation instruction corresponding to the operated button to the central control unit 1. The central control unit 1 causes each unit to execute a predetermined operation (for example, imaging of a subject, incoming / outgoing calls, transmission / reception of an e-mail, etc.) according to an operation instruction output from the operation input unit 12 and input.

  The operation input unit 12 may include a touch panel provided integrally with the display unit 9, and an operation instruction corresponding to the predetermined operation is given to the central control unit based on a predetermined operation of the touch panel by the user. 1 may be output.

<Image registration process>
Next, image registration processing by the mobile terminal 100 will be described with reference to FIGS. 4 and 5.
FIG. 4 is a flowchart illustrating an example of an operation related to the image registration process. FIG. 5 is a diagram schematically illustrating an example of an image related to the image registration process. FIG. 5A schematically illustrates a predetermined image P1 that is a target of the registration process, and FIG. The first square image P1a is schematically shown, and FIG. 5C schematically shows the processed image P2.

  As shown in FIG. 4, first, among a predetermined number of images recorded in the recording unit 6, a predetermined image P1 (see FIG. 5A) is based on a predetermined operation of the operation input unit 12 by the user. When specified (step S1), the square image generation unit 7d duplicates and acquires the image data of the predetermined image P1 from the recording unit 6 (step S2). Thereafter, the square image generation unit 7d performs a square conversion process on the image data, and relatively enlarges the short side to match the long side, or matches the long side to the short side. The first square image P1a (see FIG. 5B) is generated by reducing the image relative to (step S3).

  Next, the feature amount calculation unit 7e performs a predetermined feature extraction process on the image data of the first square image P1a generated by the square image generation unit 7d, and extracts a plurality of feature points. A SIFT feature amount is calculated (step S4).

  Thereafter, the image conversion processing unit 7a duplicates and acquires the image data of the predetermined image P1 to be processed from the recording unit 6, and then the predetermined processing content designated based on the predetermined operation of the operation input unit 12 by the user. The image conversion process (for example, art conversion process) is performed to generate a processed image P2 (see FIG. 5C) (step S5).

The recording unit 6 then processes the “image ID” of the image to be processed, the “feature information” regarding the feature point extracted by the feature amount calculation unit 7e, and the processed image P2 generated by the image conversion processing unit 7a. Are registered in the first processed image table T1 (see FIG. 2) in association with each other (step S6).
Each of the above processes is performed for all images designated as registration targets by the user, and when registration of information regarding all images is completed, the image registration process ends.

<Image display processing>
Next, image display processing by the portable terminal 100 will be described with reference to FIGS.
FIG. 6 is a flowchart illustrating an example of an operation related to the image display process. FIG. 7 is a diagram for explaining image display processing. FIG. 8 is a diagram schematically illustrating an example of an image related to the image display processing. FIG. 8A schematically illustrates a specific image region A in the frame image F, and FIG. ) Schematically represents the quadrilateral area A1 inside the specific image area A, and FIG. 8C schematically represents the second square image P3. FIG. 9 is a diagram schematically illustrating an example of a quadrilateral image P2a related to image display processing. FIG. 10 is a diagram for explaining image display processing.

  As shown in FIG. 6, first, after an imaging mode is selected based on a predetermined operation of the operation input unit 12 by the user, a specific subject S is imaged by the imaging unit 3 under the control of the imaging control unit 4. When performed (step S11), the display control unit 8 displays the live view image on the display unit 9 while sequentially updating the plurality of frame images F,... Generated by the image data generation unit 5 at a predetermined display frame rate. (Step S12; see FIG. 7).

  Next, the first acquisition unit 7b of the image processing unit 7 sequentially acquires low-resolution image data for live view display of a plurality of frame images F,... Sequentially generated by the image data generation unit 5 from the memory 2. (Step S13). Subsequently, the image area specifying unit 7c specifies a quadrilateral frame-shaped specific image included in the low-resolution image data for live view display of one frame image F among the plurality of acquired frame images F,. Region A (see FIG. 8A) is specified (step S14). Thereby, for example, when the framed photograph is captured as the specific subject S, the image area specifying unit 7c specifies the specific image area A having a quadrangular frame shape corresponding to the rectangular frame G.

Next, the square image generation unit 7d identifies the quadrilateral region A1 (see FIG. 8B) within the region surrounded by the corners inside the four vertices of the specific image region A having a quadrilateral frame shape. (Step S15), the image data of the image of the quadrilateral area A1 is subjected to a square conversion process so that the short side is relatively enlarged to match the long side, or the long side is set to the short side. A second square image P3 (see FIG. 8C) is generated by relatively reducing the image so as to match the side (step S16).
Next, the feature amount calculation unit 7e performs a predetermined feature extraction process on the image data of the second square image P3 generated by the square image generation unit 7d, and extracts a plurality of feature points. A SIFT feature amount is calculated (step S17). Then, the feature amount calculation unit 7e associates various pieces of information related to the extracted feature points and temporarily records them in the memory 2 as feature point information D (see FIG. 3).

Next, the dissimilarity calculation unit f1 of the second acquisition unit 7f obtains the feature information of each of the plurality of feature points extracted in advance from the first square image P1a of each processed image P2 from the first processed image table T1. After reading out and acquiring (step S18), the feature point information D temporarily recorded in the memory 2 is acquired, the mutual feature information is compared for each corresponding dimension, and the images are compared according to a predetermined arithmetic expression. The degree of difference is calculated (step S19).
Then, the second acquisition unit 7f determines whether or not the difference calculated by the difference calculation unit f1 is equal to or less than a predetermined determination value (step S20). Here, if it is determined that the difference is equal to or less than the predetermined determination value (step S20; YES), the second acquisition unit 7f has a plurality of features of the first square image P1a used for calculating the difference. Image data of the processed image P2 associated with the point is acquired from the recording unit 6 (step S21).

On the other hand, when it is determined in step S20 that the degree of difference is larger than the predetermined determination value (step S20; NO), the CPU of the central control unit 1 returns the process to step S1 and controls the imaging control unit 4 Below, a specific subject S is imaged by the imaging unit 3. Thereafter, the central control unit 1 executes each processing after step S2 in the same manner as described above.
In addition, when there are a plurality of dissimilarities that are equal to or smaller than the predetermined determination value in the determination in step S20, the second acquisition unit 7f specifies the one having the smallest dissimilarity and calculates the dissimilarity The image data of the processed image P2 associated with the plurality of feature points used in the above is acquired.

After that, the quadrilateral image generation unit 7g specifies the inclinations of the four line segments constituting the quadrilateral in the specific image area A of the quadrilateral frame shape, and then the processed image P2 has the inclinations of the four line segments. Projective transformation processing is performed on the image data of the processed image P2 so as to correspond to the inclinations of the four sides, thereby generating image data of the quadrilateral image P2a (step S22).
Subsequently, the composite image generating unit 7h is substantially the same as the quadrangular region A1 so as to overlap the quadrangular region A1 inside the specific image region A in the image region of one frame image F for live view display. Image data of a composite image Fa obtained by combining the quadrilateral image P2a of the processed image P2 having the same shape is generated (step S23). Then, the composite image generation unit 7 h sequentially outputs the image data of each composite image Fa to the memory 2 and stores it in the memory 2.

  Next, the display control unit 8 acquires the image data of the composite image Fa corresponding to the one frame image F from the memory 2 and displays the composite image Fa on the display unit 9 (step S24; FIG. 10). reference). That is, the display unit 9 displays the composite image Fa in which the portion of the quadrilateral area A1 inside the specific image area A in the one frame image F is replaced with the quadrilateral image P2a of the processed image P2. The

Thereafter, the CPU of the central control unit 1 determines whether or not an instruction to end the image display process has been input (step S25). Specifically, the CPU of the central control unit 1 determines whether or not a subject imaging instruction is input based on, for example, a predetermined operation (for example, full pressing operation) of the shutter button of the operation input unit 12 by the user, It is determined whether or not an instruction to end the image display process is input based on whether or not a mode different from the imaging mode is set based on a predetermined operation of the selection determination button.
If it is determined in step S25 that an instruction to end the image display process has not been input (step S25; NO), the CPU of the central control unit 1 returns the process to step S1 and executes the subsequent processes. Control.
On the other hand, if it is determined that an instruction to end the image display process has been input (step S25; YES), the CPU of the central control unit 1 ends the image display process.

As described above, according to the mobile terminal 100 of the first embodiment, a plurality of feature points are extracted from one frame image F among a plurality of sequentially acquired frame images F,... And similar to the plurality of feature points. The processed image P2 associated with each feature point to be acquired is acquired from the recording unit 6 and corresponds to a plurality of feature points extracted from the one frame image F in the image region of the one frame image F. Since the composite image Fa is generated by compositing the processed image P2 with respect to the image region, the affinity for the content of the specific subject S that has been subjected to the predetermined image conversion processing without using a marker that has nothing to do with the subject. The composite image Fa including the highly processed image P2 can be sequentially displayed. That is, when the plurality of frame images F,... Are sequentially displayed in live view, the feature points similar to the plurality of feature points of one frame image F of the plurality of frame images F,. The live view image (synthesized image) in which a part of the displayed image is replaced with the processed image P2 by synthesizing the processed image P2 associated with each feature point in the frame image F. Fa) can be sequentially displayed, and a more interesting image can be provided to the viewer of the live view image without reducing the affinity for the contents of the actual subject.
In this way, an image having high affinity for the contents of the object can be displayed without degrading the quality of the imaged object itself.

In addition, since a plurality of feature points are extracted from a specific image region A in one frame image F, the feature points similar to the plurality of feature points can be specified appropriately by limiting the extraction range of the feature points. In addition, the processing can be performed at high speed, and the processed image P2 from the recording unit 6 can be appropriately acquired.
Further, since the composite image Fa is generated by combining the processed image P2 in the specific image region A among the image regions of the single frame image F, the single frame image F can be obtained by using the specific image region A. The processed image P2 can be synthesized at an appropriate position in the image area.
Further, a projection conversion process is performed on the processed image P2 so as to correspond to the inclination of the specific image area A having a quadrilateral frame shape to generate a quadrilateral image P2a, and the image area of one frame image F Among them, the composite image Fa is generated by combining the quadrilateral image P2a of the processed image P2 in the specific image region A, so that the specific image region A has a right-sided quadrilateral shape by capturing the specific subject S from an oblique direction. Even if not, the quadrilateral image P2a of the processed image P2 having a shape that matches the inclination of the specific image area A of the quadrilateral frame shape may be combined with the image area of one frame image F. It is possible to make it difficult for the viewer of the composite image Fa to feel uncomfortable.

  Further, the degree of difference between the plurality of feature points extracted from one frame image F and the plurality of feature points recorded in the recording unit 6 is calculated, and the calculated difference degree is equal to or less than a predetermined determination value. Since the processed image P2 associated with a plurality of feature points is acquired from the recording unit 6, the processed image P2 having high affinity for the content of the predetermined image P1 can be properly specified. In particular, a plurality of feature points extracted from the second square image P3 generated by performing the square conversion process on the image of the quadrilateral area A1 inside the specific image area A of the quadrilateral frame shape and the recording unit 6 By calculating the degree of difference from a plurality of feature points extracted in advance from the first square image P1a obtained by converting the predetermined image P1 recorded in the square into a square, each of the horizontal and vertical directions The degree of difference can be calculated by removing the influence of the direction, and the calculation process can be speeded up.

[Embodiment 2]
FIG. 11 is a block diagram illustrating a schematic configuration of the mobile terminal 200 according to the second embodiment to which the present invention is applied.
Note that the portable terminal 200 of the second embodiment has substantially the same configuration as the portable terminal 100 of the first embodiment except for the details described below, and a detailed description thereof is omitted.

As shown in FIG. 11, the second processed image table T2 is recorded in the recording unit 6 of the mobile terminal 200 of the second embodiment.
Like the first processed image table T1, the second processed image table T2 is a table in which “image ID”, “feature information”, and “processed image” are associated with each other. The “feature information” includes information related to a plurality of feature points extracted from the image data itself of the predetermined image P1 to be registered.

The image processing unit 207 includes an image conversion processing unit 7a, a first acquisition unit 7b, a feature amount calculation unit 7e, a second acquisition unit 7f, and a composite image generation unit 7h.
The functions of the image conversion processing unit 7a and the first acquisition unit 7b are the same as those in the first embodiment, and a detailed description thereof is omitted.

The feature amount calculation unit 7e selects effective feature points having a large amount of grayscale information in a region around a pixel such as a corner constituted by a boundary of contrast from one frame image F among a plurality of frame images F,. The feature extraction process to extract is performed.
Specifically, the feature amount calculation unit 7e performs a predetermined feature extraction process on the image data of one frame image F for live view display, for example, to extract a plurality of feature points. SIFT feature values (coordinate positions of each pixel in the XY plane space, m-dimensional feature vectors, etc.) are calculated. Further, the feature amount calculation unit 7e associates various pieces of information related to the extracted feature points and temporarily records them in the memory 2 as feature point information D (see FIG. 3).
In addition, the feature amount calculation unit 7e performs, for example, a predetermined feature extraction process on the image data of the predetermined image P1 to be registered, and extracts a plurality of feature points in the same manner as described above. SIFT feature values (coordinate positions of each pixel in the XY plane space, m-dimensional feature vectors, etc.) at the points are calculated.

  Note that, before extracting the feature points, the feature amount calculation unit 7e specifies a specific subject area A2 (see FIG. 14) including the main subject in advance using, for example, an AF evaluation value, and the like. The feature points may be extracted from within the specific subject area A2.

The difference calculation unit f1 of the second acquisition unit 7f calculates the difference between the plurality of feature points extracted by the feature amount calculation unit 7e and the plurality of feature points recorded in the recording unit 6. Specifically, the degree-of-difference calculation unit f1 reads out and obtains feature information of each of a plurality of feature points extracted from one frame image F from the memory 2 and obtains predetermined information corresponding to each processed image P2. The feature information of each of the plurality of feature points extracted in advance from the image P1 is read from the second processed image table T2 and acquired. Then, the dissimilarity calculation unit f1 associates the feature information of each of the plurality of feature points extracted from one frame image F with the feature information of each of the plurality of feature points extracted in advance from the predetermined image P1. Comparison is made for each dimension, and the degree of difference between the images is calculated according to a predetermined arithmetic expression.
The second acquisition unit 7f determines whether or not the difference calculated by the difference calculation unit f1 is equal to or less than a predetermined determination value. As a result of this determination, when it is determined that the degree of difference is equal to or less than a predetermined determination value, the second acquisition unit 7f is associated with a plurality of feature points of the predetermined image P1 used for calculation of the difference. The image data of the processed image P2 is acquired from the recording unit 6.

The composite image generation unit 7h generates a composite image Fb obtained by combining the processed image P2 with one frame image F among the plurality of frame images F,.
That is, the composite image generation unit 7h acquires the image region corresponding to a plurality of feature points extracted from the one frame image F from the image region of the one frame image F by the second acquisition unit 7f. A synthesized image Fb is generated by synthesizing the processed image P2. Specifically, the composite image generation unit 7h, for example, in the image area of one frame image F for live view display, includes a specific subject area A2 including the main subject (for example, an area corresponding to a photograph or the like). ) Is specified, and image data of a composite image Fb obtained by combining the processed image P2 so as to overlap with the specific subject area A2 is generated.
At this time, if the shape of the specific subject area A2 is not rectangular, the composite image generation unit 7h performs projective transformation processing on the image data of the processed image P2, and the four sides of the processed image P2 The image data of the shape image P2a may be generated, and the quadrilateral image P2a may be combined with the image area of one frame image F.

<Image registration process>
Next, image registration processing by the mobile terminal 200 will be described with reference to FIG.
FIG. 12 is a flowchart illustrating an example of an operation related to the image registration process.
Note that the image registration process performed by the mobile terminal 200 according to the second embodiment is substantially the same as the image registration process performed by the mobile terminal 100 according to the first embodiment except for the details described below, and a detailed description thereof will be omitted.

As shown in FIG. 12, as in the image registration process of the first embodiment, first, among a predetermined number of images recorded in the recording unit 6, first, based on a predetermined operation of the operation input unit 12 by the user. When a predetermined image P1 (see FIG. 5A) is specified (step S1), the feature amount calculation unit 7e duplicates and acquires the image data of the predetermined image P1 from the recording unit 6 (step S52). ).
Next, the feature amount calculation unit 7e performs a predetermined feature extraction process on the acquired image data of the predetermined image P1 to extract a plurality of feature points, and calculates a SIFT feature amount at an effective feature point ( Step S53).

  Thereafter, the image conversion processing unit 7a duplicates and acquires the image data of the predetermined image P1 to be processed from the recording unit 6, and then the predetermined processing content designated based on the predetermined operation of the operation input unit 12 by the user. The image conversion process (for example, art conversion process) is performed to generate a processed image P2 (see FIG. 5C) (step S54).

The recording unit 6 then processes the “image ID” of the image to be processed, the “feature information” regarding the feature point extracted by the feature amount calculation unit 7e, and the processed image P2 generated by the image conversion processing unit 7a. Are registered in the second processed image table T2 in association with each other (step S55).
Each of the above processes is performed for all images designated as registration targets by the user, and when registration of information regarding all images is completed, the image registration process ends.

<Image display processing>
Next, image display processing by the portable terminal 200 will be described with reference to FIGS.
FIG. 13 is a flowchart illustrating an example of an operation related to image display processing. 14 and 15 are diagrams for explaining the image display process.
Note that the image display process performed by the mobile terminal 200 according to the second embodiment is substantially the same as the image display process performed by the mobile terminal 100 according to the first embodiment except for the details described below, and a detailed description thereof will be omitted.

  As shown in FIG. 13, in substantially the same manner as the image registration process of the first embodiment, first, after an imaging mode is selected based on a predetermined operation of the operation input unit 12 by the user, the control is performed by the imaging control unit 4. When the imaging unit 3 captures a specific subject S (step S11), the display control unit 8 displays the plurality of frame images F,... Generated by the image data generation unit 5 at a predetermined display frame rate. The live view image is displayed on the display unit 9 while sequentially updating (step S12; see FIG. 14).

  Next, the first acquisition unit 7b of the image processing unit 207 is for live view display of a plurality of frame images F,... Sequentially generated by the image data generation unit 5 in substantially the same manner as the image registration processing of the first embodiment. Are sequentially acquired from the memory 2 (step S13). Subsequently, the feature amount calculation unit 7e performs predetermined feature extraction processing on the low-resolution image data for live view display of one frame image F among the plurality of acquired frame images F,. A plurality of feature points are extracted, and SIFT feature values at effective feature points are calculated (step S64). Then, the feature amount calculation unit 7e associates various pieces of information related to the extracted feature points and temporarily records them in the memory 2 as the feature point information D.

Next, the dissimilarity calculation unit f1 of the second acquisition unit 7f includes feature information of each of a plurality of feature points extracted in advance from a predetermined image P1 corresponding to each processed image P2 from the second processed image table T2. Is acquired (step S65), the feature point information D temporarily recorded in the memory 2 is acquired, the mutual feature information is compared for each corresponding dimension, and the images are compared according to a predetermined arithmetic expression. Is calculated (step S66).
Then, the second acquisition unit 7f determines whether or not the difference calculated by the difference calculation unit f1 is equal to or less than a predetermined determination value in substantially the same manner as the image registration process of the first embodiment (step S20). ). Here, if it is determined that the degree of difference is equal to or less than the predetermined determination value (step S20; YES), the second acquisition unit 7f calculates the degree of difference in substantially the same manner as the image registration process of the first embodiment. The image data of the processed image P2 associated with a plurality of feature points of the predetermined image P1 used in the above is acquired from the recording unit 6 (step S21).

  On the other hand, if it is determined in step S20 that the degree of difference is greater than a predetermined determination value (step S20; NO), the CPU of the central control unit 1 performs substantially the same as the image registration process of the first embodiment. The processing is returned to step S1, and the imaging of the specific subject S is performed by the imaging unit 3 under the control of the imaging control unit 4. Thereafter, the central control unit 1 executes each processing after step S2 in the same manner as described above.

  Thereafter, the composite image generation unit 7h specifies a specific subject area A2 corresponding to a rectangular photograph that is the main subject in the image area of one frame image F for live view display, and the specific subject Image data of a synthesized image Fb obtained by synthesizing the processed image P2 so as to overlap with the area A2 is generated (step S67). Then, the composite image generation unit 7 h sequentially outputs the image data of each composite image Fb to the memory 2 and stores it in the memory 2.

  Next, the display control unit 8 acquires the image data of the composite image Fb corresponding to the one frame image F from the memory 2, and displays the composite image Fb on the display unit 9 in live view (step S68; FIG. 15). reference). That is, the display unit 9 displays a composite image Fb in which a part of a specific subject area A2 corresponding to a rectangular photograph that is a main subject in one frame image F is replaced with a processed image P2. .

Thereafter, the CPU of the central control unit 1 determines whether or not an instruction to end the image display process has been input in substantially the same manner as the image registration process of the first embodiment (step S25).
Here, if it is determined that an instruction to end the image display process has not been input (step S25; NO), the CPU of the central control unit 1 returns the process to step S1, and controls the execution of the subsequent processes. . On the other hand, if it is determined that an instruction to end the image display process has been input (step S25; YES), the CPU of the central control unit 1 ends the image display process.

As described above, according to the mobile terminal 200 of the second embodiment, when the plurality of frame images F,... Are sequentially displayed in live view, as in the mobile terminal 100 of the first embodiment, the plurality of frame images F, When the feature points similar to the plurality of feature points of one frame image F are recorded in the recording unit 6, the processed image P2 associated with the feature points is included in the frame image F. , The live view image (synthesized image Fb) obtained by replacing a part of the displayed image with the processed image P2 can be sequentially displayed, and the viewer of the live view image can see the real subject. An image with higher interest can be provided without lowering the affinity for the content.
Thus, the synthesized image Fb having high affinity for the contents of the object can be displayed without degrading the quality of the object to be imaged.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, in the first and second embodiments, the first and second processed image tables T1 and T2 are recorded in the recording unit 6 of the mobile terminals 100 and 200. May be recorded in the server 300 connected via the communication network N (see FIG. 16).

That is, as shown in FIG. 16, the portable terminal 100 (200) and the server 300 constitute an image display system that is connected via a predetermined communication network N so that various types of information can be transmitted and received.
The server 300 has, for example, a function of opening a predetermined page on the Internet as a Web (World Wide Web) server, and transmits predetermined information to the mobile terminal 100 in response to an access from the mobile terminal 100. To do.
Specifically, the server 300 includes recording means (not shown) for recording the third processed image table T3, and information registered in the image registration processing in the first and second embodiments is predetermined from the portable terminal 100. And is recorded in the third processed image table T3.
In addition, the server 300 is instructed to read the feature information of each of the plurality of feature points recorded in the third processed image table T3 from the portable terminal 100 via the predetermined communication network N in the image display process. Then, the instructed feature information is read out and transmitted to the portable terminal 100 via the predetermined communication network N. Further, the server 300 is instructed when the readout of the processed image P2 recorded in the third processed image table T3 is instructed from the portable terminal 100 via the predetermined communication network N in the image display processing. The image data of the processed image P2 is read out and transmitted to the portable terminal 100 via the predetermined communication network N.

  Even in such a configuration, similarly to the first and second embodiments, the composite image Fa (Fb) having high affinity for the content of the object is displayed without degrading the quality of the object to be imaged. be able to.

  In the modification, the image registration process is performed on the mobile terminal 100 side. However, the image registration process may be performed on the server 300 side. That is, the server 300 is mounted with an image processing unit including the square image generation unit 7d, the feature amount calculation unit 7e, the image conversion processing unit 7a, and the like according to the first and second embodiments, and the image processing unit. By executing the image registration process, “image ID”, “feature information”, and “processed image” may be associated with each other and registered in the third processed image table T3.

  In the first and second embodiments and the modified examples, each of the first to third processed image tables T1 to T3 includes a plurality of processed images P2 (for example, images processed differently or moving images Image), and any one processed image P2 designated on the basis of a predetermined operation of the operation input unit 12 or the like by the user at the time of image display processing is stored in the first to third processed image tables. You may make it acquire from T1-T3.

  In the first and second embodiments and the modified examples, the first to third processed image tables T1 to T3 are provided, and the processed image P2 generated in advance is recorded in the table. However, during the image display process, the image conversion processing unit 7a performs a predetermined image conversion process on the predetermined image P1 to generate a processed image P2, and the second acquisition unit 7f receives the processed image P2 You may make it acquire.

  Furthermore, the configurations of the portable terminals 100 and 200 and the image display system are merely examples as illustrated in the first and second embodiments and the modified examples, and are not limited thereto, and can be arbitrarily changed as appropriate.

In addition, in the above-described Embodiments 1 and 2 and the modified examples, each function is controlled under the control of the central control unit 1 of the mobile terminal 100 (200), the first acquisition unit 7b, the feature amount calculation unit 7e, The second acquisition unit 7f, the composite image generation unit 7h, and the display control unit 8 are configured to be driven. However, the configuration is not limited to this, and a predetermined program or the like is executed by the CPU of the central control unit 1 It is good also as a structure implement | achieved by doing.
That is, a program including a first acquisition processing routine, a feature point extraction processing routine, a second acquisition processing routine, a generation processing routine, and a display control processing routine is stored in a program memory that stores the program. Then, the CPU of the central control unit 1 may function to sequentially acquire a plurality of images in which the specific subject S is continuously captured by the first acquisition processing routine. Further, the CPU of the central control unit 1 may function so as to extract an image region corresponding to a predetermined image from one image among a plurality of acquired images by the feature point extraction processing routine. Further, the CPU of the central control unit 1 may function to acquire the processed image P2 corresponding to the extracted predetermined image from the recording unit by the second acquisition processing routine. Further, the CPU of the central control unit 1 generates a composite image Fa obtained by combining the acquired processed image P2 at a position where an image area corresponding to a predetermined image in one image is extracted by the generation processing routine. You may make it function. Further, the CPU of the central control unit 1 may be caused to function so as to sequentially display the generated composite image Fa on the display means by the display control processing routine.

  Similarly, other functions may be realized by executing a predetermined program or the like by the CPU of the central control unit 1.

  Furthermore, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory or a portable recording medium such as a CD-ROM is applied in addition to a ROM or a hard disk. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

[Appendix]
Although several embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
<Claim 1>
A recording unit that records a predetermined image including a specific subject in advance and a processed image obtained by performing a predetermined image conversion process on the predetermined image in association with each other;
First acquisition means for sequentially acquiring a plurality of images in which the specific subject is continuously captured;
An extracting unit that extracts an image region corresponding to the predetermined image from one image among the plurality of images acquired by the first acquiring unit;
Second acquisition means for acquiring a processed image corresponding to the predetermined image extracted by the extraction means from the recording means;
Generating means for generating a composite image obtained by combining the processed image acquired by the second acquisition means at a position where an image region corresponding to the predetermined image in the one image is extracted;
Display control means for sequentially displaying live images on the display means, the composite image generated by the generation means;
An image display device comprising:
<Claim 2>
The extraction unit further includes a feature point extraction unit that extracts a plurality of feature points from one image among the plurality of images acquired by the first acquisition unit,
The second acquisition unit acquires the processed image associated with each feature point similar to the plurality of feature points extracted by the feature point extraction unit from the recording unit,
The generating unit is configured to apply the processed image acquired by the second acquiring unit to an image region corresponding to a plurality of feature points extracted from the one image among the image regions of the one image. The image display apparatus according to claim 1, wherein a combined image is generated.
<Claim 3>
A specifying unit for specifying a specific image area in the one image;
The image display apparatus according to claim 2, wherein the feature point extracting unit extracts a plurality of the feature points from the specific image area specified by the specifying unit.
<Claim 4>
The said generation means produces | generates the synthesized image which synthesize | combined the said processed image in the said specific image area specified by the said specification means among the image areas of the said one image. The image display device described.
<Claim 5>
A quadrilateral image is obtained by performing projective transformation processing on the processed image acquired by the second acquisition unit so as to correspond to the inclination of the specific image region of the quadrilateral frame shape specified by the specifying unit. Further comprising projective transformation processing means for generating
The generating unit generates a composite image obtained by combining the quadrilateral image of the processed image generated by the projective transformation processing unit in the specific image region out of the image region of the one image. The image display device according to claim 3 or 4.
<Claim 6>
A calculation unit for calculating a degree of difference between the plurality of feature points extracted by the feature point extraction unit and the plurality of feature points recorded in the recording unit;
The second acquisition unit acquires from the recording unit the processed image associated with a plurality of feature points whose degree of difference calculated by the calculation unit is equal to or less than a predetermined determination value. The image display apparatus as described in any one of Claims 2-5.
<Claim 7>
The recording means records a plurality of feature points extracted in advance from the first square image obtained by converting the predetermined image into a square in association with the processed image,
A square image generating means for generating a second square image by performing a square conversion process on the image of the quadrilateral area inside the specific image area of the quadrilateral frame shape specified by the specifying means;
The calculation unit is configured to discriminate between a plurality of feature points extracted from the second square image by the feature point extraction unit and a plurality of feature points previously extracted from the first square image recorded in the recording unit. The image display device according to claim 6, wherein the degree is calculated.
<Claim 8>
An image display method using an image display device including a recording unit that records a predetermined image including a specific subject in advance and a processed image obtained by performing a predetermined image conversion process on the predetermined image in association with each other Because
A process of sequentially acquiring a plurality of images in which the specific subject is continuously captured;
A process of extracting an image region corresponding to the predetermined image from one image among the plurality of acquired images;
Processing for obtaining a processed image corresponding to the extracted predetermined image from the recording means;
Processing for generating a combined image obtained by combining the acquired processed image at a position where an image region corresponding to the predetermined image in the one image is extracted;
A process of sequentially displaying the generated composite image on a display means;
An image display method comprising:
<Claim 9>
A computer of an image display device including a recording unit that records a predetermined image including a specific subject in advance and a processed image obtained by performing a predetermined image conversion process on the predetermined image.
First acquisition means for sequentially acquiring a plurality of images in which the specific subject is continuously captured;
Extraction means for extracting an image region corresponding to the predetermined image from one image among the plurality of images acquired by the first acquisition means;
Second acquisition means for acquiring a processed image corresponding to the predetermined image extracted by the extraction means from the recording means;
Generating means for generating a composite image obtained by combining the processed image acquired by the second acquisition means at a position where an image region corresponding to the predetermined image in the one image is extracted;
Display control means for sequentially displaying the composite image generated by the generating means on a display means;
A program characterized by functioning as

100, 200 Mobile terminal 1 Central control unit 6 Recording unit T1 First processed image table T2 Second processed image table T3 Third processed image table 7, 207 Image processing unit 7a Image conversion processing unit 7b First acquisition unit 7c Image area specifying unit 7d Square image generation unit 7e Feature amount calculation unit 7f Second acquisition unit f1 Dissimilarity calculation unit 7g Quadrilateral image generation unit 7h Composite image generation unit 8 Display control unit 9 Display unit N Communication network

Claims (9)

An extraction means for extracting a specific image region in a quadrilateral frame shape of a predetermined image including a specific subject;
Image processing means for creating a processed image obtained by performing a predetermined image conversion process on the predetermined image;
Projective transformation processing means for generating a quadrilateral image by performing a projective transformation process on the processed image so as to correspond to the inclination of the specific image region of the quadrilateral frame shape ;
Generating means for generating a synthesized image obtained by synthesizing the projective transformation image of the processed image in the specific image region;
Recording means for associating and recording the predetermined image and the processed image;
First acquisition means for sequentially acquiring a plurality of images in which the specific subject is continuously captured;
Feature point extraction means for extracting a plurality of feature points from one image among the plurality of images acquired by the first acquisition means;
Obtaining the processed image corresponding to the predetermined image from the recording means and the processed image associated with each feature point similar to the plurality of feature points extracted by the feature point extracting means Second acquisition means for acquiring from,
The generating means combines the processed image acquired by the second acquiring means with an image area corresponding to a plurality of feature points extracted by the feature point extracting means in the one image. An image display device characterized by generating the image. The extraction means, from among the acquired plurality of images by said first acquisition means, and extracts an image region corresponding to the predetermined image from one image,
Before Symbol generating means, the position of the image region corresponding to the predetermined image in the one image is extracted, generating a composite image of the processed image acquired by the second acquisition means The image display device according to claim 1, wherein: The image display apparatus according to claim 2 , wherein the generation unit sequentially displays the generated composite image on a display unit . A specifying unit for specifying a specific image area in the one image;
The image display apparatus according to claim 2 , wherein the feature point extracting unit extracts a plurality of the feature points from the specific image area specified by the specifying unit. It said generating means of the image area of the one image, in claim 4, characterized in that to generate a composite synthetic image the processed image in the specific image region specified by the specifying means The image display device described. A calculation unit for calculating a degree of difference between the plurality of feature points extracted by the feature point extraction unit and the plurality of feature points recorded in the recording unit;
The second acquisition unit acquires from the recording unit the processed image associated with a plurality of feature points whose degree of difference calculated by the calculation unit is equal to or less than a predetermined determination value. The image display apparatus as described in any one of Claims 1-5 . The recording means records a plurality of feature points extracted in advance from the first square image obtained by converting the predetermined image into a square in association with the processed image,
Further comprising a square image generating means for generating a second square image by performing a square conversion processing on the image of the inside of the quadrilateral region of the quadrilateral frame shape of said particular image area,
The calculation unit is configured to discriminate between a plurality of feature points extracted from the second square image by the feature point extraction unit and a plurality of feature points previously extracted from the first square image recorded in the recording unit. The image display device according to claim 6 , wherein the degree is calculated. An extraction process for extracting a specific image area in a quadrilateral frame shape of a predetermined image including a specific subject;
Image processing for creating a processed image obtained by performing predetermined image conversion processing on the predetermined image;
A projective transformation process for generating a quadrilateral image by performing a projective transformation process on the processed image so as to correspond to the inclination of the specific image region of the quadrilateral frame shape ;
A generation process for generating a composite image obtained by combining the projection conversion image of the processed image in the specific image region;
A recording process in which the predetermined image and the processed image are associated and recorded in a recording unit;
A first acquisition process for sequentially acquiring a plurality of images in which the specific subject is continuously captured;
A feature point extraction process for extracting a plurality of feature points from one image among the plurality of images acquired by the first acquisition process;
Obtaining the processed image corresponding to the predetermined image from the recording means and the processed image associated with each feature point similar to the plurality of feature points extracted by the feature point extraction processing from the recording means A second acquisition process acquired from
The generation process is a composite image obtained by combining the processed image acquired by the second acquisition process with an image region corresponding to a plurality of feature points extracted by the feature point extraction process in the one image. An image display method characterized by generating an image. The computer of the image display device
An extraction means for extracting a specific image region in a quadrilateral frame shape of a predetermined image including a specific subject;
Image processing means for creating a processed image obtained by performing a predetermined image conversion process on the predetermined image;
Projective transformation processing means for generating a quadrilateral image by performing a projective transformation process on the processed image so as to correspond to the inclination of the specific image region of the quadrilateral frame shape ;
Generating means for generating a synthesized image obtained by synthesizing the projective transformation image of the processed image in the specific image region;
Recording means for associating and recording the predetermined image and the processed image;
First acquisition means for sequentially acquiring a plurality of images in which the specific subject is continuously captured;
Feature point extraction means for extracting a plurality of feature points from one image among the plurality of images acquired by the first acquisition means;
Obtaining the processed image corresponding to the predetermined image from the recording means and the processed image associated with each feature point similar to the plurality of feature points extracted by the feature point extracting means Function as a second acquisition means for acquiring from
The generating means combines the processed image acquired by the second acquiring means with an image area corresponding to a plurality of feature points extracted by the feature point extracting means in the one image. A program characterized by generating <br/>.