JP2013214158A - Display image retrieval device, display control system, display control method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a moving image to be displayed as information associated with a photographing position without downloading a large file.SOLUTION: A display image retrieval device comprises: an image feature amount calculation unit 23 that calculates a photographed image feature amount indicating a feature of a photographed image on the basis of image data of the photographed image; an image feature amount comparison unit 26 that acquires specific area information indicating a specific area with a feature matching or similar to that of a specific image in the photographed image on the basis of specific image feature amount information indicating the feature of the specific image and the photographed image feature amount; and a specific image motion setting unit 31 that outputs motion information for controlling a movement of a motion image included in the specific area and the specific area information to a display unit 15.

Description

  The present invention relates to a display image search device, a display control system, a display control method, and a program that search for content corresponding to a captured image and display the content obtained by the search on a display unit.

There is a system that uses a virtual reality (VR) technology to display a real space in which a user exists in a virtual space displayed from a user viewpoint and displays the real space on a mobile terminal.
Also, using augmented reality (AR) technology, based on position information indicating the current position of the user and an image captured by the mobile device at this position, information associated with the captured position is obtained. There is a system in which a captured image is displayed on a mobile device and provided to a user. As a system using this augmented reality technology, there are Sekai Camera (Tonchidot Corporation, http://secamera.com/) and Layar (http://layer.com/), which are attracting attention.

For example, as a technique using augmented reality technology, a marker such as a two-dimensional barcode is photographed, and a content uniquely associated with the two-dimensional barcode is synthesized and displayed on the photographed image (for example, , See Patent Document 1).
Further, as an image recognition device that recognizes the content of captured image data, there is a device that extracts a keyword representing the content of captured image data according to the captured position (see, for example, Patent Document 2).

JP 2008-249407 A JP 2007-41762 A

  As described above, by using the AR technology, it is possible to display information associated with the taken position together with an image obtained by photographing information associated with the taken position. If moving image information can be displayed as information associated with a captured position, a user can enjoy a number of virtual experiences while viewing the moving image superimposed on the image captured by the user, and the user's enjoyment is also improved.

  However, in order to display a moving image as information associated with the shooting position, it is necessary to download a moving image file. Video files have a large data capacity, and downloading video files on mobile devices places a heavy burden on network traffic. Also, it takes a lot of time to download a video, and it takes time for the video to start moving on the mobile device.

  In addition, when displaying the information associated with the taken position together with the image taken with the information associated with the taken position, if a video file is downloaded and superimposed on the taken image, the lighting environment, etc. In many cases, the color of a captured image differs from that of a downloaded image due to the influence of the above.

  In view of the above-described problems, the present invention provides a display image search device, a display control system, and display control that can display a moving image as information associated with a captured position without downloading a large-capacity file. It is an object to provide a method and a program.

  In order to solve the above-described problem, a display image search device according to the present invention includes an input unit that inputs image data of a captured image captured by the capturing unit, and a capturing that indicates the characteristics of the captured image based on the image data. Based on the image feature amount calculation unit for calculating the image feature amount, the specific image feature amount information indicating the feature of the specific image that is the search target, and the captured image feature amount, An image feature amount comparison unit that acquires specific region information indicating a similar specific region, a specific image operation setting that outputs motion information for controlling the motion of a motion image included in the specific region and the specific region information to a display unit A section.

  In the above-described display image search device, the specific image operation setting unit further includes a storage unit that stores the operation image region information indicating the operation image region corresponding to the operation image in the specific region and the operation information. When the motion image is moved and displayed, synthesis information for synthesizing a background image on at least a part of the motion image area is output to the display unit.

  In the above-described display image search device, the specific image operation setting unit includes a color corresponding to the captured image based on background image color information indicating the color of the background image and captured image color information indicating the color of the captured image. Then, correction information for correcting the color of the background image is output to the display unit.

  In the above-described display image search device, the specific image operation setting unit outputs control information for controlling the movement of the motion image to the display unit based on the operation instruction information for instructing the motion of the motion image.

  A display control system according to the present invention includes one of the above display image search devices, and a display device that is connected to the display image search device via a network and includes the photographing unit and the display unit. The display device includes a communication unit that receives information output from the display image search device, and the operation image included in the specific region from the captured image based on the specific region information received from the display image search device. And a control unit that controls movement of the motion image based on the motion information.

  A display control method according to the present invention is a display control method for displaying an image to be operated on a captured image captured by a capturing unit, wherein image data of the captured image is input, and the captured image is captured based on the image data. Based on the specific image feature amount information indicating the feature of the specific image to be searched and the captured image feature amount, the captured image is matched or similar to the specific image. The specific area information indicating the specific area is acquired, and the motion information for controlling the motion of the motion image included in the specific area and the specific area information are output to the display unit.

  The program according to the present invention includes an input means for inputting image data of a photographed image photographed by the photographing unit to the computer, and an image feature quantity calculation for computing a photographed image feature quantity indicating the feature of the photographed image based on the image data. Means for acquiring specific region information indicating a specific region having a feature matching or similar to the specific image in the captured image based on the specific image feature amount information indicating the feature of the specific image to be searched and the captured image feature amount; This is a program for functioning as image feature amount comparison means for performing, and as specific image action setting means for outputting the action information for controlling the movement of the action image included in the specified area and the specified area information to the display unit.

  According to the present invention, it is possible to provide a visual effect as if a part of a captured image starts moving without downloading a moving image after recognizing a registered image from the captured images. In addition, after recognizing the registered image, the color of the registered image area that appears in the registered image and the captured image is affected by the lighting environment at the time of shooting by cutting out the motion part and operating it instead of superimposing the moving image. Even if they are different, it is possible to give a visual effect as if a part of the photographed image has started to move.

  Further, according to the present invention, since a part of the captured image is cut out and operated, the operation range covers the entire range that can be displayed by the display unit, and the operation can be controlled in a wider range. In addition, by storing an image to be synthesized in the area where the motion image was drawn, it is possible to prevent the area where the motion image was drawn from being lost, and to be synthesized in the area where the motion image was drawn. By performing the same color conversion on the image as in other areas, it is possible to give a visual effect as if a part of the captured image started to move naturally.

It is a figure which shows an example of a picked-up image. It is a figure which shows an example of a display. It is a figure which shows the other example of a display. It is a functional block diagram of an image search display system. It is a figure which shows an example of a specific image position table. It is a figure which shows an example of specific image data. It is a figure which shows an example of the picked-up image feature-value. It is a figure which shows an example of the specific image feature-value table. It is a reference figure for demonstrating position conversion. It is a figure which shows an example of a specific image additional information table. It is a figure which shows an example of specific image operation information. It is a figure which shows an example of an operation | movement image table. It is a figure which shows an example of a structure of a registration apparatus. It is a flowchart which shows an example of operation | movement of a registration apparatus. It is a reference figure for explaining an example in the case of creating an operation picture based on a specific picture. It is a flowchart which shows the other example of operation | movement of a registration apparatus. FIG. 10 is a reference diagram for explaining an example when a specific image is created from an operation image and a background image. It is a sequence diagram which shows an example of the display control method. It is explanatory drawing of the display apparatus provided with the sensor. It is a figure which shows the other example of description of operation | movement. It is a sequence diagram which shows the other example of the display control method. It is a reference diagram for explaining color correction.

Hereinafter, an image search and display system according to an embodiment of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a diagram illustrating an example of a captured image. FIG. 1 is an image captured by the display device 1. This image includes, for example, an image of a certain downtown area, and includes an image of a large building B1 on the right side of the screen and an image of a building B2 smaller than the building B1 at the center of the screen. A large display is installed in this building B1. Building B2 is provided with a large rental advertisement space for posting rental advertisements.
A cat image O1 is displayed on the large display of the building B1. Note that the large display of the building B1 has a long right side and a short left side that is distorted in the captured image. In addition, regarding the rental advertisement space of the building 2, the upper side and the lower side are curved on the photographed image.

FIG. 2 is a diagram illustrating an example of display. The display in FIG. 2 is displayed by the display device 1 based on the photographed image in FIG.
In this figure, the cat image O2 is the one in which the cat image O1 (motion image) in FIG. Note that the movement includes a movement such that the motion image jumps forward or a movement that retracts backward (for example, movement in the normal direction of the display surface of the large display of the building B1). Thus, in the image search and display system, a specific image (the cat image in FIG. 6A) is stored in advance, and the display apparatus 1 operates an action image (cat image O1) whose features match or are similar to the specific image. ) Control the movement.
In this figure, a region R1 represents a region (operation image region) where the cat image O1 of FIG. 1 was displayed. In this region R1, an image (a part of the image bk1 in FIG. 2) is synthesized. Thus, in the image search / display system, the background image of the specific image (see FIG. 15C) is stored in advance, and the display device 1 synthesizes the background image in the motion image area.

FIG. 3 is a diagram illustrating another example of display. The display in FIG. 3 is displayed by the display device 1 based on the photographed image in FIG.
In this figure, a cat image O3 is obtained by further enlarging / reducing and rotating the cat image O2 of FIG. In addition, the rotation includes those having the normal line of the screen as the rotation axis and those having the axis in the screen as the rotation axis. As described above, the motion of the motion image includes, for example, movement, enlargement / reduction, and rotation.
In this figure, a region R2 represents a region where the cat image O2 of FIG. 2 was displayed. For example, the cat image O3 moves from the region R2 to the position of the cat image O3, and moves from there to rotate or enlarge.

FIG. 4 is a functional block diagram of the image search / display system.
As shown in FIG. 4, the image search / display system includes a display device 1 and a display image search device 2. In the present embodiment, the display device 1 and the display image search device 2 are communicably connected via a network NW.
In the present embodiment, the display device 1 and the display image search device 2 are different devices, and will be described as an example in which the display device 1 and the display image search device 2 are connected via the network NW. However, the present invention is not limited to this. I can't. For example, the display device 1 may include the display image search device 2 inside. The network NW may be either wired or wireless. For example, any network that performs communication using a telephone line, an Internet line, a wireless LAN communication method, or the like may be used.

<About display device 1>
As the display device 1, for example, a mobile device such as a mobile phone with a camera or a camera with a communication function can be used. The display device 1 is equipped with a function for setting an AR mode in which content obtained by using augmented reality technology is combined with a captured image and displayed. This AR mode is set below. The functions and operations used in the case will be mainly described.

The display device 1 includes a terminal control unit 11, a photographing unit 12, a position information detection unit 13, a timing unit 14, a display unit 15, a communication unit 16, a storage unit 17, and an operation unit 18.
The imaging unit 12 receives light from the subject, photoelectrically converts the received light, generates image data, and outputs the image data. That is, the photographing unit 12 photographs a subject and outputs image data.
The photographing unit 12 outputs photographed image data D (n) and real-time reproduction image data as image data. The real-time reproduction image data is image data output from the photographing unit 12 in order to display an image photographed by the photographing unit 12 on the display unit 15 in real time. As a result, the user can take an image of the subject displayed on the display unit 15 while viewing the image displayed on the display unit 15.
On the other hand, when the image based on the real-time reproduction image data is displayed on the display unit 15, the image capturing unit 12 detects the captured image data D (n) based on the image capturing instruction from the user input by the operation unit 18. Is image data obtained by photographing.

The position information detection unit 13 uses, for example, GPS (Global Positioning System) to detect and output shooting position information P (n) indicating a position where the display device 1 is present at the time of shooting. The photographing position information P (n) indicates an absolute position on the earth determined by, for example, latitude, longitude, and altitude.
In the present embodiment, the time of shooting may be, for example, the timing when the user presses the shutter button of the operation unit 18 or the timing when the captured image data D (n) is generated.

  The timekeeping unit 14 measures the date and time and outputs shooting date / time information T (n) indicating the date and time measured at the time of shooting.

  The terminal control unit 11 comprehensively controls the display device 1. The terminal control unit 11 captures the captured image data D (n) {n = 1, 2...} Output from the capturing unit 12 and indicates the position where the captured image data D (n) is captured. Information P (n) and shooting date / time information T (n) indicating the date / time when the shot image data D (n) was shot are associated with each other and stored in the storage unit 17. Note that n is a captured image ID (Identification) that is identification information for identifying captured image data captured by the capturing unit 12 from other captured image data.

In addition, the terminal control unit 11 inputs real-time reproduction image data continuously captured by the imaging unit 12 from the terminal control unit 11 and displays an image captured by the imaging unit 12 on the display unit 15 in real time. Here, the terminal control unit 11 cuts out a part of the image (cat image O1 in FIG. 1) and controls the movement of the cut-out image (motion image) (see FIGS. 2 and 3). Further, the terminal control unit 11 synthesizes an image with the background of the cut image (see FIGS. 2 and 3).
Specifically, the terminal control unit 11 acquires motion image area information, motion images, motion information, and synthesis information from the display image search device 2. The terminal control unit 11 combines the background image indicated by the combination information with the action image area indicated by the acquired action image area information. The terminal control unit 11 controls the motion of the motion image based on the motion information. Note that the terminal control unit 11 may control the movement of the motion image based on a user operation. In addition, the terminal control unit 11 may generate an operation image by cutting out an image from the operation image area of the captured image.

  The display unit 15 is a liquid crystal display, for example, and displays an image based on information input from the terminal control unit 11.

  The communication unit 16 receives information transmitted from the display image search device 2 via the network NW, outputs the information to the terminal control unit 11, and retrieves the information input from the terminal control unit 11 via the network NW. Transmit to device 2.

  The storage unit 17 stores the captured image data D (n) generated by the imaging unit 12, the imaging position information P (n), and the imaging date / time information T (n) in association with each other. The storage unit 17 also stores a program for operating the display device 1.

  The operation unit 18 includes, for example, a power button for inputting power ON / OFF, a shutter button for inputting a photographing instruction from the user, and the like.

<About the display image search device 2>
The display image search device 2 includes a communication unit 21, a first storage unit 22, an image feature amount calculation unit 23, a position information comparison unit 24, a second storage unit 25, an image feature amount comparison unit 26, and an image. An identification unit 27, a third storage unit 28, a fourth storage unit 29, a specific image additional information acquisition unit 30, a specific image operation setting unit 31, and a fifth storage unit 32 are provided.

The communication unit 21 is an interface for inputting / outputting information to / from the display device 1, and the captured image data D (n) transmitted from the display device 1 via the network NW and the captured position information P (n ) And shooting date / time information T (n). The communication unit 21 uses the photographed image data D (n) as the image feature amount calculation unit 23 and the specific image operation setting unit 31, and the photographing position information P (n) as the position information comparison unit 24 and the specific image additional information acquisition unit 30. In addition, the shooting date / time information T (n) is output to the specific image additional information acquisition unit 30, respectively. The communication unit 21 transmits the operation data input from the specific image operation setting unit 31 to the display device 1 via the network NW.
Note that, when the display device 1 and the display image search device 2 are connected via the network NW as in the present embodiment, the communication unit 21 functions to transmit and receive information to and from the display device 1 as described above. Have On the other hand, although not shown, when the display image search device 2 is built in the display device 1, it has a function of inputting / outputting information to / from the display device 1. That is, the communication unit 21 functions as an input unit that inputs information from the display device 1 and an output unit that outputs information to the display device 1 (display unit 15).

The first storage unit 22 stores a specific image position table 220 registered in advance by an administrator of the display image search device 2, for example.
As shown in FIG. 5, the specific image position table 220 is a table that associates a specific image ID with specific image position information indicating a position where additional information related to the specific image can be provided. The specific image position information includes information indicating latitude, longitude, altitude, and position error allowable distance.
The specific image ID is identification information for identifying the specific image. Note that the specific image data to which the specific image ID is assigned is stored in the first storage unit 22 in association with the specific image ID.

For example, the first storage unit 22 stores specific image data 100, 200, and 300 having specific image IDs 1, 2, and 3 as shown in FIG.
As shown in FIG. 6A, the specific image data 100 is assigned a specific image ID1, and is a cat image.
Further, as shown in FIG. 6B, the specific image data 200 is a dog image to which a specific image ID2 is assigned.
Furthermore, as shown in FIG. 6C, the specific image data 300 is assigned a specific image ID3 and is a rocket image.
The specific image data in FIGS. 6A to 6C are different in feature points and feature amounts of the feature points.

Returning to FIG. 5, latitude, longitude, and altitude are information indicating the specific image position corresponding to the specific image data of the specific image ID. The specific image position is, for example, a position where a poster or a large display in which specific image data is represented is installed.
The position error allowable distance is information indicating a certain range that is regarded as a specific image position by latitude, longitude, and altitude. This position error allowable distance is, for example, a distance indicating the radius of a circle centered on the specific image position.

  Returning to FIG. 4, the image feature amount calculation unit 23 calculates an image feature amount (hereinafter referred to as a captured image feature amount) indicating the feature of the image based on the captured image data D (n). This photographed image feature amount is obtained by quantifying image features such as color components, brightness, and edge images. Note that a point (for example, a pixel) in which the image feature amount has a numerical value equal to or greater than a certain threshold is referred to as a feature point. Further, the position of the feature point in the captured image is indicated by (X, Y) coordinate values (see FIG. 9B). The image feature amount calculation unit 23 generates feature point information in which the captured image feature amount and the (X, Y) coordinate value are associated with each feature point. The image feature amount calculation unit 23 generates captured image feature amount information including feature point information for all feature points in the captured image.

An example of the captured image feature amount is shown in FIG. As illustrated in FIG. 7, the image feature amount calculation unit 23 calculates a captured image feature amount based on the captured image data D (n), for example, and obtains captured image feature amount groups C1, C2, and C3.
This photographed image feature quantity group C1 is a set of feature points calculated from an image area corresponding to the character image of “movie title” displayed on the large display of the building B1 in the photographed image data D (n). It is.
The photographed image feature quantity group C2 is a set of feature points calculated from an image area corresponding to a cat image displayed on the large display of the building B1 in the photographed image data D (n).
The photographed image feature quantity group C3 is a set of feature points calculated from an image area corresponding to a cat image displayed in the rental advertisement space of the building B2 in the photographed image data D (n) image.
Note that the photographed image feature quantity groups C2 and C3 are a set of feature points calculated from an image area corresponding to a cat image, and the distribution of the feature points is similar. Further, the feature points of the photographed image feature quantity group C2 are particularly similar to the feature points of the specific image data in FIG.

Returning to FIG. 4, the position information comparison unit 24 receives the shooting position information P (n) from the communication unit 21. The position information comparison unit 24 refers to the specific image position table 220 and searches for the corresponding specific image ID based on the input shooting position information P (n). That is, the position information comparison unit 24 searches for specific image data that includes the shooting position indicated by the shooting position information P (n) within the range indicated by the specific image position information.
In other words, the position information comparison unit 24 obtains the specific image data corresponding to the shooting position information from all the specific image data stored in the first storage unit 22 by searching, and thereby the image feature amount comparison unit. 26 is performed to limit the comparison target to that corresponding to the shooting position information.

More specifically, the position information comparison unit 24 uses the position indicated by the shooting position information P (n) as the center with the latitude, longitude, and altitude indicated in the specific image position table 220 as the radius. It is determined whether there is a specific image ID that falls within the specific image position range. When the position information comparison unit 24 obtains a specific image ID in which the position indicated by the shooting position information P (n) is within the specific image position range by the search, the specific image ID obtained by the search is compared with the image feature amount. To the unit 26.
In the specific image feature amount table 250, the specific image data of the specific image ID that is not associated with the specific latitude, longitude, altitude, and position error allowable distance is excluded from the search result candidates by the position information comparison unit 24. First, the position information comparison unit 24 outputs the image feature amount comparison unit 26.

The second storage unit 25 stores, for example, a specific image feature amount table 250 registered in advance by an administrator of the display image search device 2.
As shown in FIG. 8, the specific image feature amount table 250 is a table that associates a specific image ID with specific image feature amount information indicating the characteristics of the specific image. The specific image feature amount information includes information indicating the feature point ordinate, the feature point abscissa, and the feature point feature amount of the feature point in the specific image.
The feature point ordinate and the feature point abscissa are information indicating the position of the feature point indicating the specific feature amount in the image of the corresponding specific image ID. The feature point feature amount is information indicating the feature amount of the feature point indicated by the feature point ordinate and the feature point abscissa. Note that the number of feature point feature quantities associated with each specific image ID is determined according to each specific image ID. This feature amount may be extracted from various viewpoints, for example, and when a plurality of types of feature amounts are extracted, the feature amount is a vector. In the drawing, for simplification, one kind of feature amount for feature points is shown.

  Returning to FIG. 4, the image feature amount comparison unit 26 reads the feature amount corresponding to the specific image ID obtained by the search of the position information comparison unit 24 with reference to the specific image feature amount table 250, and this specific image. The feature amount corresponding to the ID and the captured image feature amount of the captured image feature amount information generated by the image feature amount calculation unit 23 are compared, and the feature amount included in the image of the captured image data D (n) A specific image (also referred to as a similar specific image) having a similar feature amount is detected.

More specifically, for example, the image feature amount comparison unit 26 reads a feature amount for each specific image ID, and determines whether or not the read feature amount matches the captured image feature amount of the captured image feature amount information. judge. Here, “match” means a case where the value of the feature value is within the threshold value T1. When there are a plurality of types of feature amounts (for example, two types of color components and brightness) for one feature point, the image feature amount comparison unit 26 compares the same types of feature amounts. . The image feature amount comparison unit 26 may count the number of types of feature amounts whose compared feature amount values are within the threshold value T1, and may determine that they are similar when the counted number is equal to or greater than the threshold value.
The image feature quantity comparison unit 26 counts the number of feature points determined to match. For example, the image feature amount comparison unit 26 divides the counted number of feature points by the total number of feature points of the specific image ID, and sets the divided value as the feature amount similarity. Note that the image feature amount comparison unit 26 may calculate the feature amount similarity using another method. The image feature amount comparison unit 26 stores the feature amount similarity and the specific image ID in association with each other.
The image feature amount comparison unit 26 performs the above processing for all the specific image IDs. The image feature amount comparison unit 26 extracts the specific image ID corresponding to the maximum feature amount similarity, and detects the specific image of the extracted specific image ID as the similar specific image.
The image feature amount comparison unit 26 outputs the specific image ID of the detected similar specific image, the specific image feature amount information corresponding to the specific image ID, and the captured image feature amount information to the image identification unit 27.

Based on the specific image feature amount information and the captured image feature amount information input from the image feature amount comparison unit 26, the image identification unit 27 selects a specific image and an image in a captured image similar to the specific image (both similar images). The conversion matrix A indicating the correspondence between the positions is calculated.
For example, as shown in FIG. 9A, the position of the original specific image in the image is indicated by (x, y) coordinate values. Further, as shown in FIG. 9B, the position in the captured image is indicated by (X, Y) coordinate values.

  The image identification unit 27 extracts feature points from the specific image feature amount information, and extracts feature points that match the feature points from the captured image feature amount information. Here, “match” refers to a case where the value of the feature value is within a threshold value, but this threshold value may be different from or equal to the threshold value T1. The image identification unit 27 substitutes the feature point ordinate and the feature point abscissa for (x, y) in the following equation (1) for the feature points extracted from the specific image feature amount information. For the feature points extracted from the captured image feature amount information, the image identification unit 27 substitutes the (X, Y) coordinate values for the extracted feature points into (X, Y) of the following equation (1). When there are a plurality of feature points extracted from the captured image feature amount information, the average value of the (X, Y) coordinate values may be substituted into (X, Y) in the following equation (1). .

As a result of the substitution, the image identification unit 27 generates a mathematical expression having each component of the matrix A _xy as an unknown. The image identification unit 27 repeats the above process a predetermined number of times, and generates a mathematical expression for the number of times. The image identification unit 27 calculates the matrix A _xy by solving the generated mathematical formulas simultaneously.

The image identification unit 27 refers to the motion information (see FIG. 11) to determine whether or not the motion image has a movement amount in the z direction. Here, the z direction is a coordinate orthogonal to the feature point ordinate and the feature point abscissa. When it is determined that there is no movement amount in the z direction in the motion image, the image identification unit 27 sets the matrix A _xy as the transformation matrix A. On the other hand, when it is determined that the motion image has a movement amount in the z direction, the image identification unit 27 calculates the matrix A _xyz and sets the calculated matrix A _xyz as the conversion matrix A.

Here, the matrix A _xyz will be described.
As shown in FIG. 11, the motion image is set in three dimensions (x, y, z) corresponding to the coordinate values of the specific image. This is because, for example, the operation image can describe a three-dimensional operation such that the operation image pops out from the display surface (including the print surface). However, the above formula lacks information for expressing the jumping out or retracting in the z direction with a captured image. Therefore, the image identification unit 27 converts the (x, y, z) coordinate value obtained by adding the z coordinate value to the (x, y) coordinate value into a coordinate value in the captured image, a matrix A _xyz (formula (6)). ) Is calculated.
This matrix A _xyz is obtained from the condition of the expression (3), as in the expression (2). Note that L is a positive value.

具体的に、方程式に書き下すと、式（４）となる。ここで、Ａ_１１、Ａ_１２、Ａ_２１、Ａ_２２は、式（１）の行列Ａ_ｘｙの成分である。

Specifically, when written in an equation, equation (4) is obtained. Here, A ₁₁ , A ₁₂ , A ₂₁ , A ₂₂ are components of the matrix A _{xy in} Expression (1).

Image identification unit 27 solves Equation (4) is a quadratic equation of ^{L 2,} and calculates the solution of ^{L 2} a (simply referred to as ^{L 2).} The image identification unit 27 calculates i _z and j _z by using the calculated L ² using Expression (5).

The image identification unit 27 generates Expression (6) based on the calculated i _z and j _z and each component of the matrix A _xy .

  The conversion matrix A calculated by the image identification unit 27 is a conversion rule for converting the position of the original specific image to a position in the photographing. However, the conversion rule may not be a matrix, and the conversion rule may be provided for each part of the specific image. For example, the image identification unit 27 may calculate a conversion matrix A for each x coordinate value or each y coordinate value. Note that when the position of the specific image is converted in the captured image according to the conversion matrix A, the area where the specific image is located in the captured image is hereinafter referred to as an operation image area. That is, the image identification unit 27 calculates information indicating the position of the motion image area that appears in the captured image. In other words, the image identification unit 27 determines a motion image region having a matching or similar feature to the specific image in the captured image based on the feature amount of the specific image and the feature amount of the captured image. Note that the image in the motion image area in the captured image is the motion image. However, the present invention is not limited to this, and the image in the motion image area may be another image. For example, an image obtained by converting a specific image by the conversion matrix A may be used.

The image identification unit 27 outputs the conversion matrix A and the specific image ID of the similar specific image to the specific image additional information acquisition unit 30.
The third storage unit 28 includes a specific image data storage area 280 that is an area for storing image data of an original specific image and a feature amount extracted image.

The fourth storage unit 29 stores a specific image additional information table 290 registered in advance by the administrator of the display image search device 2, for example.
As shown in FIG. 10, the specific image additional information table 290 includes a specific image ID, specific image additional information indicating additional information related to the specific image, and specific image additional information corresponding to the specific image ID at the time of shooting. It is a table which matches imaging condition information which is information for further specifying according to conditions. This photographing condition information includes latitude, longitude, altitude, allowable position error distance, year / month / day, and time.

The latitude, longitude, altitude, and position error allowable distance are information for allowing the specific image additional information corresponding to the specific image ID to be further specialized according to the position at the time of shooting.
The year / month / day and time are information for allowing the specific image additional information corresponding to the specific image ID to be further specialized according to the date and time at the time of shooting.
The content URL is a URL (Uniform Resource Locator) that provides content related to a specific image via the Internet.
The moving image file name is information indicating a moving image file associated with the specific image. Note that the moving image file indicated by the moving image file name is stored in the fourth storage unit 29.
The number of slide show images and the slide show image display frame rate are information indicating the number of slide show images associated with the specific image and the frame rate for displaying the slide show.

The specific image additional information acquisition unit 30 reads the specific image additional information corresponding to the specific image ID input from the image identification unit 27 from the specific image additional information table 310. The specific image additional information acquisition unit 30 reads specific image additional information corresponding to the shooting position information and the shooting date / time information input from the communication unit 21 from the specific image additional information table 310. The specific image additional information acquisition unit 30 outputs the specific image ID, the specific image additional information, and the conversion matrix A to the specific image operation setting unit 31.
The specific image operation setting unit 31 performs the following processes (1) to (4) based on information input from the specific image additional information acquisition unit 30 and the communication unit 21.

(1) Calculation of Motion Image Region The specific image motion setting unit 31 reads the motion image region information corresponding to the specific image ID input from the specific image additional information acquisition unit 30 from the motion image table 320 (FIG. 12). The specific image operation setting unit 31 extracts a mask image (FIG. 15B) of the specific image, which is a mask image included in the operation image region information. The mask image is, for example, an image in which the bit value at a position with a pixel value is “1” and the bit value at a position without a pixel value is “0” (transparent).
The specific image operation setting unit 31 calculates a conversion matrix A from the left with respect to the extracted mask image, thereby generating a mask image of the captured image. The mask image of the photographed image indicates an operation image area. That is, the specific image operation setting unit 31 calculates an operation image area.

(2) Generation of Motion Image The specific image motion setting unit 31 extracts an image of the calculated motion image area from the captured image data D (n) input from the communication unit 21 (for example, the cat image in FIG. 1). O1). That is, the specific image operation setting unit 31 generates an operation image. Note that, when the display device generates a motion image, the specific image motion setting unit 31 does not have to generate a motion image.

(3) Generation of Background Image The specific image operation setting unit 31 reads, from the operation image table 320, operation image area information (see FIG. 11) corresponding to the specific image ID input from the specific image additional information acquisition unit 30. The specific image operation setting unit 31 extracts a background image (FIG. 15C) of the specific image. The specific image operation setting unit 31 calculates a conversion matrix A from the left with respect to the extracted background image, thereby generating a background image to be combined with the operation image region (for example, an image of the region R1 in FIG. 2).

(4) Generation of Motion Information The specific image motion setting unit 31 reads the specific image motion information corresponding to the specific image ID input from the specific image additional information acquisition unit 30 from the motion image table 320. The specific image operation setting unit 31 calculates the transformation matrix A from the left with respect to the (x, y, z) coordinate values of the operation image included in the read specific image operation information. Accordingly, the specific image operation setting unit 31 generates operation information indicating the movement of the operation image in the captured image.

  The specific image additional information acquisition unit 30 outputs the information indicating the calculated motion image region, the generated motion image, the composite information indicating the background image, and the motion information to the communication unit 21 as motion data.

FIG. 11 is a diagram illustrating an example of specific image operation information. The specific image operation information is information representing the movement of the specific image. Note that the motion information generated based on the specific image motion information is information representing the motion image motion. In this figure, Time indicates the elapsed time, and x, y, and z indicate, for example, the x, y, and z coordinate values of the center of the specific image.
In the specific image operation information, for example, as shown in FIG. 11, the position of the specific image corresponding to the elapsed time from the reference time is defined. Note that the position where the specific image at the initial position is after the elapse of time (for example, milliseconds) from the reference time is defined. For example, the specific image operation information shown in FIG. 11 includes coordinates when the specific image at the position of the coordinate (100, 0, 0) when the elapsed time is “1000” milliseconds and the elapsed time “2000” milliseconds. 100, 0, 100). That is, the specific image operation information in FIG. 11 represents that the specific image jumps out (or retracts) in the z direction. Note that the description method of the operation of the image is not limited to this. In addition, when there are a plurality of specific images, specific image operation information is described for each specific image.

Next, an example of the operation image table 320 will be described with reference to FIG.
As shown in FIG. 12, the operation image area information and the specific image operation information are stored in correspondence with the specific image ID. Here, for the specific image of the specific image ID1, the action image area information (1_a.png) and the specific image action information (activation description 1_a) are associated with each other. In FIG. 12, for simplification, one piece of information indicating an image is stored as operation image area information, but two pieces of information indicating a mask image of a specific image and information indicating a background image are stored. Yes.

The display device 1 inputs operation data from the display image search device 2 via the communication unit 16 and outputs the information to the terminal control unit 11 of the display device 1. Based on the received motion data, the terminal control unit 11 synthesizes a background image in the motion image area of the captured image data D (n) and moves the motion image based on the motion information. Thereby, it seems to the user that the motion image is cut out from the captured image, and the cut-out motion image starts to move.
In addition, about the elapsed time which is not prescribed | regulated in operation | movement information, the terminal control part 11 carries out the linear interpolation of a position, for example. Thereby, it seems to the user that the motion image moves continuously.

Next, an example of the registration apparatus 5 that registers the operation image area information and the specific image operation information in the operation image table 320 will be described with reference to FIG. Here, the registration device 5 is described as being separate from the display image search device 2, but the function of the registration device 5 may be realized in a server as the display image search device 2.
As illustrated in FIG. 13, the registration device 5 includes a registration processing unit 51, an operation input unit 52, a display unit 53, and an image storage unit 54.

  The registration processing unit 51 executes registration processing in the operation image table 320. The operation input unit 52 includes a keyboard, a mouse, and the like, and accepts operator input. The display unit 53 includes a liquid crystal display or the like, and displays a source image and a processed image. The image storage unit 54 stores an image to be registered as a specific image.

  The registration of the motion image area information and the specific image motion information in the motion image table 320 includes a process of cutting out the motion image from the specific image and registering the motion image area information (hereinafter referred to as a cutout registration process), and a motion image. There is a process of registering a specific image by combining the motion image indicated by the area information and the background image indicated by the composite information (hereinafter referred to as a composite registration process).

First, an example of the cutout registration process will be described with reference to FIG. FIG. 14 is a flowchart for explaining an example of the cutout registration process.
The operator designates an image (step ST101). The operator designates a specific image in this image (step ST102). When a specific image to be registered is designated, the registration processing unit 51 generates motion image area information based on the designated information and registers it in the motion image table 320 (step ST103).

  For example, in step ST101, the operator designates an image as shown in FIG. The operator designates a cat image portion as the specific image (FIG. 6A). In this case, the registration processing unit 51 generates a mask image shown in FIG. The mask image is, for example, an image in PNG (Portable Network Graphics) format. For example, the mask image has a value in a cat image portion, and the other portions are transparent (no value). The registration processing unit 51 registers the specific image and the mask image as operation image area information in association with the specific image ID.

If the motion image area information is registered in step ST103, the operator then describes and inputs the motion image motion (step ST104). As a result, the registration processing unit 51 generates specific image operation information, and registers the generated specific image operation information in association with the specific image ID in step S103 (step ST105).
Next, the registration processing unit 51 extracts the feature amount of the specific image (step ST106), and registers it as the feature amount of the specific image in the specific image feature amount table 250 (step ST107). Then, the registration processing unit 51 determines whether or not to end the registration process (step ST108). If the registration process is not ended, the registration processing unit 51 proceeds to a registration process for the next image (step ST109) and returns to step ST101. .

Next, an example of processing for registering a background image by combining the background image and the mask image (composite registration processing) will be described with reference to FIG. FIG. 16 is a flowchart illustrating an example of the composition registration process.
The operator designates an image (for example, FIG. 17) including a background image (step ST201). The operator designates a mask image (for example, FIG. 15B) (step ST202). The registration processing unit 51 combines the image specified in step ST201 and the mask image specified in step ST202 (step ST203). Thereby, the registration process part 51 produces | generates a background image (for example, FIG.15 (c)). The registration processing unit 51 registers the generated background image in the motion image table 320 by including it in the motion image area information including the mask image specified in step ST202 (step ST204).

Next, an example of an image search display method according to the present embodiment will be described with reference to FIG. FIG. 18 is a flowchart for explaining an example of the image search display method according to the present embodiment.
The display image search device 2 calculates the transformation matrix A based on the captured image feature amount and the feature amount of the specific image. The display image search device 2 calculates an operation image area based on the conversion matrix and the mask image of the specific image. That is, the position of the specific image on the captured image is calculated (step ST11). The display image search device 2 transmits motion data including the motion image region calculated in step ST11 to the display device 1 (step ST12).

The display device 1 receives the operation data transmitted in step S12 (step S13). The display device 1 specifies the motion image area of the captured image based on the motion image area included in the motion data received in step S13. That is, the display device 1 sets the position of the specific image in the captured image (step ST14).
The display image search device 2 generates an operation image and transmits operation data including the generated operation image to the display device 1 (step ST15). The display device 1 receives the operation image transmitted in step ST15 (step S16).
The display image search device 2 generates a background image and transmits operation data including composite information indicating the generated background image to the display device 1 (step ST17). The display device 1 receives the synthesis information transmitted in step ST17 (step S18).

The display device 1 supplements the motion image region by combining (superimposing) the background image received in step S18 on the motion image region specified in step ST14. The display device 1 superimposes the operation image received in step S16 on the combined image. Here, the display device 1 divides, for example, the layer of the combined image and the layer of the motion image. Thereby, the display apparatus 1 cuts out an operation | movement image (step ST19).
The display image search device 2 generates operation information and transmits operation data including the generated operation information to the display device 1 (step ST20). The display device 1 receives the operation information transmitted in step ST20 (step S21).
The display device 1 reads the received operation information (step ST22). The display device 1 moves the motion image cut out in step ST19 based on the motion information read in step ST22 (step S23).

Thus, in the present embodiment, the display device 1 can display a moving image related to the captured image without downloading a moving image file from the display image search device 2. Further, the image displayed on the display unit 15 of the display device 1 is a natural image that matches the captured image because the motion image cut out from the captured image captured by the imaging unit 12 of the display device 1 is moved. Even if the brightness and color of the image registered in the display image search device 2 and the image captured by the display device 1 are different due to the influence of the surrounding environment, the image is not affected. Further, in the present embodiment, since a part of the captured image starts to move, a unique visual effect can be given to the user, and various services using such a visual effect can be expected.
In addition, for example, a plurality of operation images and operation descriptions corresponding to captured images captured by the display device 1 are prepared, and operation images provided from the display image search device 2 to the display device 1 are selected or randomly selected for each user. And the description of the operation may be switched. In this way, the user can see images of various movements with respect to the same captured image. In addition, various services using this can be expected.

  By the way, the description of the operation information as shown in FIG. 11 described above is that the operator of the display image search device 2 provides a predetermined operation on the display image search device 2 side, and the motion of the operation image is It is uniquely determined according to the description of the operation information. However, the present invention is not limited to this, and the image search and display system may change the movement of the motion image in accordance with various types of user input. This increases the enjoyment of the user.

[Second Embodiment]
Here, the input by the user is not limited to the input by the keyboard or buttons of the operation unit 18. For example, when the contact sensor 19 is present on the display unit 15 of the display device 1, an operation such as touching, sliding, or flicking the display unit 15 is input, and a gravity sensor 20 is provided on the display device 1. In such a case, an operation in which the user changes the direction of the display device 1 or the user shakes the display device 1 with a predetermined movement is input. Furthermore, if various sensors are provided, input using these sensors can be performed. That is, the detection results of the contact sensor 19, the gravity sensor 20, and other various sensors are operation instruction information that instructs the movement of the operation image.
Here, an example based on a physical model will be described as an example of processing for changing the motion of an operation image by input using such a sensor.
As shown in FIG. 19, when the display device 1 includes the gravity sensor 20, the following physical models are registered in the specific image operation setting unit 31 and the terminal control unit 11. Specifically, it is assumed that the coordinate (X, Y) of the motion image changes (X0, Y0), (X1, Y1), (X2, Y2),. When measuring the direction (gx, gy) of

Can be updated as. For example, this physical model can be described and transmitted as shown in FIG.

  When receiving the description of the operation as shown in FIG. 20, the terminal control unit 11 updates and displays the operation image according to the physical model. That is, the terminal control unit 11 calculates the position of the motion image after Δt seconds according to the above formula and displays it on the photographing unit 12. In addition, when there is an input in the operation image from the contact sensor 19,

The operation may be stopped with a finger and moved with a finger.

[Third Embodiment]
Next, another example of the image search / display method according to the present embodiment will be described. FIG. 21 is a flowchart for explaining another example of the image search and display method according to the present embodiment. When FIG. 21 is compared with FIG. 18, the processes of steps ST42 to ST45 and ST47 of FIG. 21 are different. The processes in steps S31 to ST41 in FIG. 21 are the same as the processes in steps S11 to S21 in FIG.

Image identification unit 27 extracts sample points for color correction (step ST42), the information of the extracted sample points to calculate the transformation matrix A _c of the color correction (step ST43). The specific image operation setting unit 31 sends the transformation matrix _{A c} calculated in step ST43 to the communication unit 21, communication unit 21 transmits the transformation matrix _{A c} to the display device 1 (step ST44). The communication unit 16 of the display device 1 receives the transformation matrix _{A c} (step ST45).
The display device 1 moves the motion image cut out in step ST39 based on the motion information read in step ST46. The display device 1, using the transformation matrix A _c received in step ST45, the background image received at step ST38 and color correction, synthesizes the background image color correction operation image area (step S47) .

In this example, the background image is synthesized with the area after the motion image is cut out and moved, and the color of the background image is corrected so that more natural synthesis can be performed. That is, when the motion image area information as shown in FIG. 15B is transmitted from the display image search device 2 to the display device 1 and the motion image is moved and displayed on the display unit 15 of the display device 1, the motion image is displayed. The image displayed in the area disappears, and for example, white or black. For this reason, when the motion image is cut out and moved, the motion image region appears white or the like, resulting in an unnatural image.
For example, even when a background image is received from the display image search device 2, the received background image may not match the surrounding color.

  Therefore, in this example, in step ST42 to step ST45, the color correction conversion matrix generated by the display image search device 2 is sent to the display device 1 to perform color correction of the background image. As a result, the motion image area after the motion image has moved appears as a natural image.

When performing color conversion, the image identification unit 27 calculates a conversion matrix A indicating the color correspondence between the specific image and the similar image. Here, the image identification unit 27 calculates the transformation matrix (A _c · A _s ) as the transformation matrix A. Here, the matrix A _s is a conversion matrix (first embodiment) indicating the correspondence between positions, and the matrix A _c is a conversion matrix indicating the correspondence between colors. Note that (A _c · A _s ) represents that A _c is calculated for the color information of each coordinate converted by the matrix A _s .
More specifically, the image identification unit 27 extracts sample points for color correction from the original specific image and its similar image. In this embodiment, as shown in FIG. 22 (a), the image identification unit 27 is based on the sample points p1, p2, p3,... Defined on the specific image. From FIG. 22B, sample points P1, P2, P3... Corresponding to the sample points p1, p2, p3. Incidentally, (X, Y) coordinates of the sample points P1 is obtained by calculating the matrix _{A s} from the left (x, y) coordinate values.

The image identification unit 27, color information of the sample points p1, p2, p3 · · · in the specific image _{(r 1, g 1, b} 1), (r 2, g 2, b 2), (r 3, g _{3, b 3)} to perform a color conversion of .... More specifically, the image identification unit 27 outputs the color information (r ₁ , g ₁ , b ₁ ), (r ₂ , g ₂ , b ₂ ), (r ₃ , g ₃ , b ₃ ). , Color information (R ₁ , G ₁ , B ₁ ), (R ₂ , G ₂ , B ₂ ), (R ₃ , G ₃ , B ₃ ). It calculates a matrix a _c as converted to ....
Hereinafter, an example of a method for calculating the matrix _Ac by the image identification unit 27 will be described.
For example, the color information (r ₁ , g ₁ , b ₁ ) and (r ₂ , g ₂ , b ₂ ) of the sample points p1 and p2 in the specific image are shown as follows.

Further, for example, the color information (R ₁ , G ₁ , B ₁ ) and (R ₂ , G ₂ , B ₂ ) of the sample points P1 and P2 in the specific image included in the captured image data D (n) are It shall be shown as follows.

The image identification unit 27 calculates the color correction conversion coefficient F that makes the color information of the corresponding sample point p1 and the color information of P1, and the color information of p2 and the color information of P2 close to each other.
Note that the following equation (11) associates the color information of the corresponding sample point p1 with the color information of P1, and the color information of p2 and the color information of P2.
The image identification unit 27 calculates a matrix E that minimizes the expression shown in Expression (11), whereby the following Expression (13) is obtained for the matrix C and the matrix D in Expression (12) shown below. Is used to calculate the optimal solution of the matrix A _{c in} equation (14). The image identification unit 27 calculates the matrix (A _c · A _s ) as the transformation matrix A.

  Note that techniques described in the following papers such as the RANSAC method may be used.

M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” Commun. ACM, no.24, vol.6, pp.381-395, June 1981.

The display image search device 2 sends the above-described color correction matrix A _c to the display device 1, and the terminal control unit 11 of the display device 1 performs synthesis processing based on the matrix A _c when performing the synthesis processing. Perform color conversion. Specifically, the display device 1 calculates the matrix _Ac from the left with respect to the background image. The display device 1 superimposes the calculated background image on the motion image area. Thereby, the color conversion of the background portion can be performed in accordance with the shooting environment of the shot image sent from the display device 1.
In the present embodiment, the display image retrieval device 2, using the matrix A _c, the image may be the background image after the color conversion of the background image.

  As shown in FIG. 15B, when background information is transparent and an action image is cut out, composition may be performed using the technique described in the following paper.

W. AU AND R. TAKEI, Image inpainting with the Navier-Stokes equations.
Available at Final Report APMA 930 SFU, 2002.

In addition, this invention is not restricted to the above-mentioned embodiment, For example, the following structures may be sufficient.
For example, the display device 1 may cut out the motion image from the motion image region of the captured image. In this case, the display image search device 2 may not generate an operation image.
Further, the display image search device 2 calculates an inverse matrix of the transformation matrix A from the left with respect to the motion image, thereby generating an image corresponding to the specific image (also referred to as a captured specific image) for the motion image. Good. In this case, the display image search device 2 may transmit operation data including the generated captured specific image, the conversion matrix A, and the specific image operation information to the display device 1. The display device 1 generates an operation image by converting the received captured specific image with the conversion matrix A. The display device 1 calculates the transformation matrix A from the left for the (x, y, z) coordinate values of the motion image included in the specific image motion information. Thereby, the display device 1 generates motion information indicating the motion of the motion image in the captured image.

  Further, the display image search device 2 may transmit to the display device 1 an operation image obtained by calculating the conversion matrix A from the left with respect to the specific image. In other words, the display image search device 2 may change the position of the specific image and convert it into the same shape as the shape of the motion image area as the motion image.

  The synthesis includes complementation and interpolation. Here, the term “complement” refers to, for example, redefining a value in an undefined value range from a value in a surrounding value range. Further, combining the background image with the motion image area includes setting a predetermined pixel value in the motion image area. Further, the synthesis information may be information indicating, for example, linear interpolation. In this case, the display device 1 performs, for example, linear interpolation on the motion image area based on the surrounding pixel values. Further, the background image indicated by the synthesis information may be a light color.

In addition, a program for realizing the above steps is recorded on a computer-readable recording medium, and a program for realizing the functions of the display device 1 and the display image search device 2 is computer-readable. The estimated value of the shape information of the object to be detected may be calculated by reading the program recorded on the recording medium and causing the computer system to read and execute the program.
The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  DESCRIPTION OF SYMBOLS 11 ... Terminal control part, 12 ... Image pick-up part, 13 ... Position information detection part, 14 ... Time measuring part, 15 ... Display part, 16 ... Communication part, 17 ... Memory | storage 18, operation unit 21, communication unit 22, first storage unit 23, image feature amount calculation unit 24, position information comparison unit 25, second Storage unit 26... Image feature amount comparison unit 27... Image identification unit 28... Third storage unit 29 29 Fourth storage unit 30. 31 ... specific image operation setting unit, 32 ... fifth storage unit, 320 ... operation image table

Claims (7)

An input unit for inputting image data of a photographed image taken by the photographing unit;
An image feature amount calculation unit that calculates a captured image feature amount indicating the feature of the captured image based on the image data;
An image for acquiring specific area information indicating a specific area that matches or is similar to the specific image in the captured image based on the specific image feature amount information indicating the characteristic of the specific image to be searched and the captured image feature amount A feature comparison unit;
A specific image operation setting unit that outputs operation information for controlling movement of an operation image included in the specific region and the specific region information to a display unit;
A display image retrieval apparatus comprising: A motion image area information indicating a motion image area corresponding to the motion image in the specific area, and a storage unit for storing the motion information;
The specific image operation setting unit includes:
The composition information for synthesizing a background image to at least a part of the motion image area is output to the display unit when the motion image is moved and displayed. Display image search device. The specific image operation setting unit includes:
Based on background image color information indicating the color of the background image and captured image color information indicating the color of the captured image, correction information for correcting the color of the background image to a color corresponding to the captured image, The display image search apparatus according to claim 2, wherein the display image search apparatus outputs to a display unit. The specific image operation setting unit includes:
The control information for controlling the motion of the motion image is output to the display unit based on the motion instruction information for instructing the motion of the motion image. Display image search device. The display image search device according to any one of claims 1 to 4,
A display device that is connected to the display image search device via a network and includes the photographing unit and the display unit;
The display device
A communication unit that receives information output from the display image search device;
The image processing apparatus further includes a control unit that acquires the motion image included in the specific area from the captured image based on the specific area information received from the display image search device and controls movement of the motion image based on the motion information. A display control system. A display control method for displaying an operating image on a captured image captured by a capturing unit,
Input the image data of the captured image,
Based on the image data, a captured image feature amount indicating the characteristics of the captured image is calculated,
Based on the specific image feature amount information indicating the characteristics of the specific image to be searched and the captured image feature amount, specific area information indicating a specific area that matches or is similar to the specific image in the captured image is acquired;
A display control method comprising: outputting operation information for controlling movement of an operation image included in the specific region and the specific region information to a display unit. Computer
Input means for inputting image data of a photographed image photographed by the photographing unit;
An image feature amount calculating means for calculating a photographed image feature amount indicating a feature of the photographed image based on the image data;
An image for acquiring specific area information indicating a specific area that matches or is similar to the specific image in the captured image based on the specific image feature amount information indicating the characteristic of the specific image to be searched and the captured image feature amount Feature comparison means,
Specific image operation setting means for outputting operation information for controlling movement of an operation image included in the specific region and the specific region information to a display unit;
Program to function as.

Images