JP2017016166A - Image processing apparatus and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technology of presenting an AR image formed by superimposing AR content on a desired background image, without any limitation of location or time.SOLUTION: An image processing apparatus includes: an operation receiving unit which receives an operation for selecting an image file; a background acquisition unit which acquires an image of the image file selected in the selection operation, as a background image, from a background holding unit holding the image file; a marker detection unit which detects a marker from the acquired background image; a reference calculation unit which sets a three-dimensional coordinate space, on the basis of real information on the detected marker and image information, and calculates a background imaging position and a background imaging direction; a position determination unit which determines a position on the three-dimensional coordinate space where an AR content to be displayed is arranged virtually; and a display processing unit which causes a display unit to display a composite image formed by superimposing an image viewed from the background imaging position in the background imaging direction of the AR content to be arranged in the position determined on the three-dimensional coordinate space, on the background image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to augmented reality (AR) image processing technology.

  Currently, various applications using a technique called augmented reality are provided. The AR technology displays three-dimensional content and two-dimensional content (such as characters and designs) in a superimposed manner in the real world obtained via a camera or a head-mounted display (HMD) of a mobile device such as a smartphone. For example, the following Non-Patent Document 1 proposes a method of generating a three-dimensional content to be superimposed on the real world with an image obtained by actually capturing a real object existing in the real world. In addition, there is a form in which the virtual three-dimensional model generated by using computer graphic (CG) technology is used for the three-dimensional content, instead of the photographed image.

  Hereinafter, a composite image in which 3D content and 2D data are superimposed on the real world may be referred to as an AR image. In addition, an image of the real world on which 3D content or 2D content is superimposed may be described as a background image. In addition, 3D content and 2D content superimposed on the background image may be referred to as AR content.

  Non-Patent Document 2 below proposes a remote control system for a mobile robot using AR technology. In this proposed method, an image in which a three-dimensional model of a robot is superimposed on an extended free viewpoint image using AR technology is presented to the operator via the HMD. This extended free viewpoint image is generated based on the omnidirectional image group and distance image group acquired at a plurality of points by the mobile robot, and the position and orientation of the robot at the time of acquisition. Also, in Patent Document 1 below, as a method for generating three-dimensional content to be superimposed, a converted image for three-dimensional display of a target object is generated with a single camera without using dedicated equipment such as an acceleration sensor. A method has been proposed. In this proposed technique, the user designates a plurality of image coordinates constituting the rotation axis of the camera for a plurality of images including the target object, and the conversion target image is geometrically transformed so that the designated image coordinates coincide.

International Publication No. 2011/148595

Yusuke Uno, “Presentation of Virtual Real Objects Based on View-Dependent Texture Geometry for Augmented Reality”, Nara Institute of Science and Technology, Graduate School of Information Science, Master's Thesis, March 15, 2013 Yuko Ueda et al. “Remote Mobile Robot Operation Interface Using Extended Free Viewpoint Image Generation”, IEICE Technical Report, vol.112, no.386, pp.217-222, January 2013

  In the above-described proposed method and existing applications, a real-time video obtained from an imaging device in real time is used as a background image on which three-dimensional content or two-dimensional content is superimposed. As a result, the user can visually recognize an AR image in which AR content exists in the real world captured by the imaging apparatus in real time.

  However, with the above-described method, the real world where the AR content is to be superimposed is limited to the location where the imaging device is present. For example, a user can view an AR image in which AR content is superimposed on his / her room with his / her camera-equipped mobile terminal only when the user is in his / her room with the mobile terminal. The AR image cannot be seen in places other than the user's room such as in a train.

  The present invention has been made in view of such circumstances, and it is an object of the present invention to provide a technique for presenting an AR image in which an AR content is superimposed on a desired background image without restrictions on place or time.

  Each aspect of the present invention employs the following configurations in order to solve the above-described problems.

  The first aspect relates to an image processing apparatus. The image processing apparatus according to the first aspect is a selection operation accepted by an operation accepting unit from an operation accepting unit that accepts an operation of selecting an image file and a background holding unit that holds an image file that stores a background image in which a marker is captured. Based on a background acquisition unit that acquires an image of the selected image file as a background image, a marker detection unit that detects a marker from the background image acquired by the background acquisition unit, and real information and image information about the detected marker A reference calculation unit that sets a three-dimensional coordinate space, calculates a background shooting position and a background shooting direction, and a position determination unit that determines a position in a three-dimensional coordinate space where an AR content to be displayed is virtually arranged And a visual image from the background photographing position and background photographing direction of the AR content arranged at the determined position in the three-dimensional coordinate space are superimposed on the background image. It has a display unit for displaying the synthesized image on the display unit.

  The second aspect relates to an image processing method. The image processing method according to the second aspect receives an image file selection operation, and from a background holding unit that stores an image file that stores a background image in which a marker is captured, the image file selected by the received selection operation. An image is acquired as a background image, a marker is detected from the acquired background image, a three-dimensional coordinate space is set based on the detected reality information and image information, and the detected reality information and image related to the detected marker Based on the information, the background shooting position and the background shooting direction are calculated, the position in the three-dimensional coordinate space where the AR content to be displayed is virtually arranged is determined, and the position is arranged at the determined position in the three-dimensional coordinate space. And displaying a composite image in which a visual image from the background shooting position and background shooting direction of the AR content is superimposed on the background image on the display unit.

  Another aspect of the present invention may be a program that causes at least one computer to execute the method of the second aspect, or a computer-readable recording medium that records such a program. May be. This recording medium includes a non-transitory tangible medium.

  According to each aspect described above, it is possible to provide a technique for presenting an AR image in which AR content is superimposed on a desired background image without any restrictions on place or time.

It is a figure which shows notionally the process structural example of the image processing apparatus in this embodiment. 5 is a flowchart illustrating an operation example of the image processing apparatus according to the present embodiment. 1 is a diagram conceptually illustrating a hardware configuration example of an image processing apparatus according to a first embodiment. It is a figure which shows notionally the process structural example of the image processing apparatus in 1st Embodiment. It is a figure which shows notionally the example of several content image matched with each content imaging | photography direction hold | maintained at a content holding part. 3 is a flowchart illustrating an operation example of the image processing apparatus according to the first embodiment. It is a figure which shows notionally the hardware structural example of the image processing apparatus in 2nd Embodiment. 12 is a flowchart illustrating a part of an operation example of the image processing apparatus according to the second embodiment. It is a figure which shows notionally the process structural example of the image processing apparatus in a 2nd modification. It is a figure which shows the example of the background image preserve | saved at an image file. It is a figure which shows the example of the background image preserve | saved at an image file. It is a figure which shows the example of the background image preserve | saved at an image file. It is a figure which shows the example of the synthesized image (AR image) with which the content image was superimposed on the background image shown by FIG. 10A. It is a figure which shows the example of the synthesized image (AR image) with which the content image was superimposed on the background image shown by FIG. 10B. It is a figure which shows the example of the synthesized image (AR image) with which the content image was superimposed on the background image shown by FIG. 10C. It is a figure which shows the movement of AR content on an AR image. It is a figure which shows the movement of AR content on an AR image. It is a figure which shows the movement of AR content on an AR image.

  Embodiments of the present invention will be described below. In addition, embodiment mentioned below is an illustration and this invention is not limited to the structure of the following embodiment.

  FIG. 1 is a diagram conceptually illustrating a processing configuration example of an image processing apparatus according to the present embodiment. As illustrated in FIG. 1, the image processing apparatus 100 includes an operation reception unit 101, a background acquisition unit 102, a marker detection unit 103, a reference calculation unit 104, a position determination unit 105, a display processing unit 106, and the like. The image processing apparatus 100 has a hardware configuration similar to that of the image processing apparatus 10 in the detailed embodiment described later, and each processing unit described above is realized by processing a program in the same manner as the image processing apparatus 10. The In addition, the image processing apparatus 100 may include a background holding unit 107 and a display unit 108. However, the background holding unit 107 and the display unit 108 may be included in another computer (not shown) connected to be communicable with the image processing apparatus 100. The display unit 108 is a device that displays a screen corresponding to drawing data, such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display.

  The operation reception unit 101 receives an image file selection operation.

  The background acquisition unit 102 acquires, as a background image, the image of the image file selected by the selection operation received by the operation reception unit 101 from the background storage unit 107 that stores the image file that stores the background image on which the marker is reflected. Here, moving image data or still image data is stored in the image file, and the background acquisition unit 102 acquires a background image as still image information from the moving image data or still image data.

  The marker detection unit 103 detects a marker from the background image acquired by the background acquisition unit 102. Here, the marker is an image or an object arranged in the real world and is called an AR marker or the like. However, in the present embodiment, if a certain reference point and three directions orthogonal to each other from the reference point can be obtained from this marker regardless of the reference direction, the specific of this marker can be obtained. The form is not limited. For example, the marker detection unit 103 holds in advance the marker actual information (shape information and the like), and detects the marker from the background image based on the reality information.

  The reference calculation unit 104 sets a three-dimensional coordinate space, and calculates a background shooting position and a background shooting direction based on the actual information and image information regarding the detected marker. The real information on the marker means known information in the real world of the marker, and indicates the actual shape of the marker, for example. Moreover, the image information regarding a marker means the information in the background image of a marker, and shows the shape in the background image of a marker. The reference calculation unit 104 sets a three-dimensional coordinate space at a position corresponding to the reference point recognized from the detected marker. The set position in the three-dimensional coordinate space is not particularly limited. In addition, the reference calculation unit 104 calculates the background shooting position and the background shooting direction by comparing the actual information about the marker and the image information. The background shooting position and the background shooting direction mean the position and orientation (posture) of the imaging apparatus that shot the background image of the image file.

  The position determination unit 105 determines the position in the three-dimensional coordinate space where the AR content to be displayed is virtually arranged. The position determination unit 105 can determine this position arbitrarily or fixedly. The AR content to be displayed may be the three-dimensional model proposed in Non-Patent Documents 1 and 2 above, or a plurality of content images obtained by photographing a real object from a plurality of directions are displayed by switching. It may be a three-dimensional content that is simulated in the above, or a planar two-dimensional content.

  The display processing unit 106 displays on the display unit 108 a composite image in which a visual image from the background photographing position and background photographing direction of the AR content arranged at the determined position in the three-dimensional coordinate space is superimposed on the background image. Let When the AR content is the above-described 3D model or 2D content, the display processing unit 106 obtains the visual image using a well-known method based on the information of the 3D model or 2D content. Can do. On the other hand, when the AR content is a three-dimensional content that is simulated by switching and displaying a plurality of content images obtained by photographing a real object from a plurality of directions, a method for acquiring the visual image is implemented in detail. It will be described later as a form.

  FIG. 2 is a flowchart showing an operation example of the image processing apparatus 100 in the present embodiment. As shown in FIG. 2, the image processing method in the present embodiment is executed by at least one computer such as the image processing apparatus 100, and includes the processing steps (S21) to (S27).

  In (S21), the computer accepts an image file selection operation.

  In (S22), the computer acquires, as a background image, an image of the image file selected by the selection operation accepted in (S21) from a background holding unit that holds an image file in which each marker stores a background image. . The marker and the image file are as described above. The content of (S22) is the same as the processing content of the background acquisition unit 102.

  In (S23), the computer detects a marker from the background image acquired in (S22). The content of (S23) is the same as the processing content of the marker detection unit 103.

In (S24), the computer sets a three-dimensional coordinate space based on the actual information and image information regarding the marker detected in (S23).
In (S25), the computer calculates a background shooting position and a background shooting direction based on the actual information and the image information regarding the marker detected in (S23). The contents of (S24) and (S25) are the same as the processing contents of the reference calculation unit 104.

  In (S26), the computer determines a position in the three-dimensional coordinate space where the AR content to be displayed is virtually arranged. The content of (S26) is the same as the processing content of the position determination part 105. FIG.

  In (S27), the computer captures the background photographing position and background photographing calculated in (S25) of the AR content arranged at the position determined in (S26) in the three-dimensional coordinate space set in (S24). The visual image from the direction is displayed on the display unit as a composite image superimposed on the background image acquired in (S22). The content of (S27) is the same as the processing content of the display processing unit 106.

  The display unit and the background holding unit may be included in at least one computer that is an execution subject of the image processing method, or another computer (not shown) connected to be communicable with the at least one computer. ) May have. The embodiment of the present invention may be a program that causes at least one computer to execute the above-described image processing method, or may be a computer-readable recording medium that records such a program. .

  Further, the execution order of the processing steps of the image processing method in the present embodiment is not limited to the order shown in FIG. For example, if (S26) is after (S24), it may be executed before (S25), or may be executed in parallel with (S25).

  As described above, in this embodiment, the image of the image file selected by the selection operation by the user from the image files held in the background holding unit is acquired as a background image (superimposed image), and the background image Then, a composite image on which a visual image of the AR content is superimposed is displayed. Therefore, according to the present embodiment, the user performs an operation of selecting an image file in which a desired background image is stored from among the held image files, so that the AR content is superimposed on the desired background image. An image (AR image) can be seen. As a result, the user can view the AR image of the desired background image anytime and anywhere.

  As described above, the present inventors have solved the new idea of eliminating the limitation of the background image on which the AR content is superimposed and presenting the AR image in which the AR content is superimposed on the desired background image without limitation of place or time. I found. Then, as described above, a plurality of background images each containing a marker are stored as a plurality of image files so as to be a superimposed image, and when a user wants to view an AR image, the user can select a desired image from the background image. A new user interface has been found to select an image file for storing a background image. Such a user interface has a completely different idea from the user interface of an existing AR application that presupposes that AR content is superimposed on real-time video.

  According to such a user interface in the present embodiment, the following problems in the existing AR application can be solved. When a portable terminal with a camera is used as a device that captures and displays an AR image, the user must always shoot the real world on which AR content is superimposed on the portable terminal. In this case, in order to view the AR image, the user needs to always raise his arm, and cannot view the AR image with an easy posture. Furthermore, since the portable terminal is held by the user's hand, it is difficult to completely fix the position of the portable terminal, and the background image is blurred.

  According to the present embodiment, since the image of the image file selected by the selection operation by the user is used as the background image (superimposed image), the user does not mind camera shake and has an easy posture. You can see the image.

  The visual image of the AR content is a background image in which the AR content placed at a certain position in the three-dimensional coordinate space set based on the marker detected from the background image is calculated based on the marker. This is a form visually recognized from the background photographing position and the background photographing direction. Therefore, the user can recognize the situation as if the AR content existed in the real world shown in the desired background image by viewing the displayed AR image.

  Hereinafter, the details of the above-described embodiment will be described. Below, 1st Embodiment and 2nd Embodiment are illustrated as detailed embodiment. In the following, the same contents as those of the above-described embodiment will be omitted as appropriate, and the contents different from those of the above-described embodiment will be mainly described.

[First Embodiment]
The image processing apparatus according to the first embodiment uses one content image among a plurality of content images obtained by photographing a real object from a plurality of directions as a visual image of the AR content, and uses an AR image (composite image). Is generated.

〔Device configuration〕
FIG. 3 is a diagram conceptually illustrating a hardware configuration example of the image processing apparatus 10 in the first embodiment. The image processing apparatus 10 is a so-called computer. The form of the image processing apparatus 10 such as a portable type or a stationary type is not limited. The image processing apparatus 10 includes, for example, a CPU (Central Processing Unit) 11, a memory 12, an input / output interface (I / F) 13, a communication device 14, and the like that are connected to each other via a bus. The memory 12 is a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk, or the like. The communication unit 14 exchanges signals with other computers and devices. A portable recording medium or the like can be connected to the communication unit 14.

  The input / output I / F 13 can be connected to user interface devices such as the display device 15 and the input device 16. The display device 15 displays a screen corresponding to drawing data processed by a CPU 11 or a GPU (Graphics Processing Unit) (not shown) such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display. Device. The display device 15 corresponds to the display unit 108 described above. The input device 16 is a device that receives an input of a user operation such as a keyboard and a mouse. The display device 15 and the input device 16 may be integrated and realized as a touch panel. The display device 15 and the input device 16 may be connected to another computer that is communicably connected to the image processing device 10 via the communication unit 14.

  Further, the image processing apparatus 10 may have other hardware elements not shown in FIG. For example, the image processing apparatus 10 may have a camera corresponding to the imaging unit. In this case, the image processing apparatus 10 can cause the background holding unit 31 to hold an image file of a background image captured by the camera, and can capture a content image of a real object described later with the camera. The hardware configuration of the image processing apparatus 10 is not limited.

[Processing configuration]
FIG. 4 is a diagram conceptually illustrating a processing configuration example of the image processing apparatus 10 in the first embodiment. The image processing apparatus 10 according to the first embodiment includes an operation reception unit 21, a background acquisition unit 22, a marker detection unit 23, a reference calculation unit 24, a position determination unit 25, an image extraction unit 26, a size adjustment unit 27, and a synthesis processing unit 28. A display processing unit 29, a background holding unit 31, a content holding unit 32, and the like. Each of these processing units is realized, for example, by executing a program stored in the memory 12 by the CPU 11. Further, the program is installed from a portable recording medium such as a CD (Compact Disc) or a memory card or another computer on the network via the input / output I / F 13 or the communication unit 14 and is stored in the memory 12. It may be stored.

  The operation reception unit 21 corresponds to the operation reception unit 101 described above. The operation reception unit 21 receives a user operation using the input device 16 for a screen displayed on the display device 15. The operation accepting unit 21 further accepts a content image moving operation with respect to the composite image in addition to the image file selecting operation. The selection operation includes, for example, an operation for selecting one image file from a list of a plurality of image files displayed on the display device 15 and a button operation for confirming the operation. The movement operation is, for example, an operation of dragging and dropping or swiping the content image on the composite image displayed on the display device 15. However, the present embodiment does not limit the specific operation content itself. The content image is an image that is superimposed on the background image as a visual image of the AR content, and details thereof will be described later.

  Furthermore, the operation reception unit 21 can also receive an operation for selecting one real object to be displayed from among a plurality of real objects. A real object is an object or person existing in the real world that is the source of AR content to be superimposed on a background image. The operation receiving unit 21 displays a list of a plurality of real objects corresponding to content images held in a content holding unit 32 described later on the display device 15 and receives a selection operation using the input device 16 for the list. it can.

  The background acquisition unit 22, background holding unit 31, marker detection unit 23, and reference calculation unit 24 are the same as the background acquisition unit 102, background holding unit 107, marker detection unit 103, and reference calculation unit 104 in the above-described embodiment. .

  The position determination unit 25 determines the position in the three-dimensional coordinate space where the AR content to be displayed is virtually arranged, like the position determination unit 105 described above. Further, when the operation accepting unit 21 accepts a moving operation, the position determining unit 25 determines the position after movement of the AR content in the three-dimensional coordinate space indicated by the moving operation. Specifically, the position determination unit 25 converts the moved position indicated by the moving operation on the composite image into a position on the three-dimensional coordinate space based on the marker actual information and the image information, and the converted position is obtained. The position is determined as the position after the AR content is moved.

  The content holding unit 32 holds a plurality of content images obtained by shooting a real object corresponding to the AR content from a plurality of directions in association with each content shooting direction. In the first embodiment, since the plurality of content images are used as a visual image of the AR content, it is desirable that the content holding unit 32 hold a large number of content images taken from all directions with a fine angle width. This is because the content image to be superimposed on the background image can be approximated to the visual image of the real object from the background shooting direction and the background shooting position of the background image.

  FIG. 5 is a diagram conceptually illustrating an example of a plurality of content images associated with each content shooting direction held in the content holding unit 32. In FIG. 5, the content shooting direction is shown in a two-axis coordinate space in the horizontal direction and the vertical direction. As shown in FIG. 5, the content holding unit 32 can also hold each content image in association with position coordinates indicating the content shooting direction.

  In the example of FIG. 5, each content image includes a marker and other backgrounds as well as a real object. In the first embodiment, the content holding unit 32 may hold a content image in which something other than a real object is captured. However, it is desirable that the content holding unit 32 holds a content image in which an image area other than the real object is made transparent. This is because it is easy to generate a composite image in which a content image representing only a visual image of a real object is superimposed on a background image.

  When a marker appears in the content image to be held, the content holding unit 32 holds information on the size and shape of the marker in the real world. The content holding unit 32 holds information about the size of the real world of the real object when the marker is not captured in the content image to be held.

  The content holding unit 32 can also hold a plurality of content images associated with each content shooting direction for each of a plurality of real objects. In this case, the operation reception unit 21 can receive an operation of selecting one real object to be displayed from among a plurality of real objects.

  The image extraction unit 26 holds content associated with a content shooting direction close to the direction from the position determined by the position determination unit 25 to the background shooting position calculated by the reference calculation unit 24 from the content holding unit 32. Extract images. It is desirable that the content holding unit 32 hold a content image associated with a content shooting direction that matches the direction from the position in the three-dimensional coordinate space of the AR content to the background shooting position. This is because the content image matches the visual image of the real object from the shooting position of the background image. When such a matching content image is not held in the content holding unit 32, the image extraction unit 26 extracts a content image held in association with the closest content shooting direction.

  When content images of a plurality of real objects are held in the content holding unit 32, the image extraction unit 26 corresponds to the real object selected by the operation received by the operation receiving unit 21, and the content holding unit 32. One content image is extracted from a plurality of content images held in the.

  In addition, when the operation accepting unit 21 accepts a moving operation, the image extracting unit 26 uses a direction from the moved position determined by the position determining unit 25 to the background photographing position calculated by the reference calculating unit 24. A content image held in the content holding unit 32 in association with the content shooting direction close to is newly extracted. The newly extracted content image is a visual image of a real object that exists at a position after the background image is moved from the shooting position.

  The size adjustment unit 27 is extracted by the image extraction unit 26 based on the actual size of the real object, the real information and image information regarding the marker detected by the marker detection unit 23, and the position determined by the position determination unit 25. The enlarged content image is enlarged or reduced. The size adjusting unit 27 determines the enlargement ratio or reduction ratio of the content image so that the real object shown in the content image is superimposed on the real world shown in the background image in the size of the real world. For example, the size adjustment unit 27 can obtain the reduction ratio of the real world in the background image by using the real information and the image information regarding the marker, and the real size of the real object, the image size in the content image, and the background image The enlargement ratio or reduction ratio of the content image can be determined using the reduction ratio in the real world and the distance between the background shooting position and the AR content position.

  The composition processing unit 28 is a composite image (AR image) in which the visual image from the background photographing position and the background photographing direction of the AR content arranged at the position determined by the position determining unit 25 in the three-dimensional coordinate space is superimposed on the background image. ) Is generated. Specifically, the composition processing unit 28 superimposes the content image extracted by the image extraction unit 26 on the background image acquired by the background acquisition unit 22 as a visual image of the real object (AR content).

  The composition processing unit 28 switches the content image to be superimposed on the background image to the content image newly extracted by the image extracting unit 26 in response to the movement operation received by the operation receiving unit 21. Further, when the content image is enlarged or reduced by the size adjusting unit 27, the composition processing unit 28 superimposes the enlarged or reduced content image on the background image.

  The display processing unit 29 causes the display device 15 to display the combined image generated by the combining processing unit 28.

[Operation example]
The image processing method in the first embodiment will be described below with reference to FIG. FIG. 6 is a flowchart illustrating an operation example of the image processing apparatus 10 according to the first embodiment. In the following description, the image processing apparatus 10 is the execution subject of each method, but the above-described processing units included in the image processing apparatus 10 may be the execution subject. In FIG. 6, the same processing steps as those shown in FIG.

The image processing apparatus 10 accepts a user operation using the input device 16 (S61). The content of (S61) is the same as the processing content of the operation reception part 21. FIG.
When the selection operation is accepted (S62; selection), the image processing apparatus 10 executes (S22) to (S26) as in FIG.

  In this case, the image processing apparatus 10 is held from the content holding unit 32 in association with the content shooting direction close to the direction from the position determined in (S26) to the background shooting position calculated in (S25). A content image is extracted (S63). The content of (S63) is the same as the processing content of the image extraction unit 26.

  Subsequently, the image processing apparatus 10 enlarges or reduces the content image extracted in (S63) so that the real object is superimposed on the real world in the background image with the size in the real world (S64). The content of (S64) is the same as the processing content of the size adjustment unit 27.

  The image processing apparatus 10 superimposes the content image enlarged or reduced in (S64) on the background image acquired in (S22) to generate composite image properties (S65). The content of (S65) is the same as the processing content of the composition processing unit 28.

  The image processing device 10 causes the display device 15 to display the composite image generated in (S65) (S66).

  On the other hand, when the movement operation is accepted (S62; movement), the image processing apparatus 10 determines the position after the movement of the AR content in the three-dimensional coordinate space indicated by the movement operation accepted in (S61). (S67). The content of (S67) is the same as the processing content of the position determination part 25. FIG.

  The image processing apparatus 10 stores the content held in the content holding unit 32 in association with the content shooting direction close to the direction from the moved position determined in (S67) to the background shooting position calculated in (S25). An image is newly extracted (S63). Thereafter, the image processing apparatus 10 executes (S64) and subsequent steps on the newly extracted content image.

  Although not shown in FIG. 6, when content images of a plurality of real objects are held in the content holding unit 32, the image processing apparatus 10 (S 61) An operation for selecting one real object to be displayed can be accepted. In this case, the image processing apparatus 10 extracts one content image from a plurality of content images held in the content holding unit 32 corresponding to the real object selected by the operation (S63).

[Operation and Effect of First Embodiment]
As described above, in the first embodiment, an image of an image file selected by a user's selection operation from among a plurality of image files is used as a background image (superimposed image). In addition, an AR image in which AR content is superimposed on a desired background image can be shown.

  Furthermore, in the first embodiment, a plurality of content images obtained by shooting a real object from a plurality of directions are stored in advance in association with each content shooting direction. An AR image (composite image) is generated using one content image as a visual image of the AR content. As described above, according to the first embodiment, since a plurality of content images associated with each content shooting direction need only be prepared, compared to a mode of generating a three-dimensional model that requires specialized CG technology. AR content can be easily created. Therefore, according to the first embodiment, the threshold for development and use of the platform using the AR technology can be lowered, and it can be easily used by general users.

  Furthermore, in the first embodiment, a content image held in association with a content shooting direction that is close to the direction from the position of the AR content in the three-dimensional coordinate space to the shooting position of the background image is used as a visual image of the real object. It is superimposed on the background image. Therefore, according to the first embodiment, the content image superimposed on the background image can be approximated to the visual image of the real object from the shooting position and shooting direction of the background image, and the real object appears in the real world reflected in the background image. It is possible to enhance the realism of the space where is superimposed.

  Furthermore, in the first embodiment, the position after movement in the three-dimensional coordinate space indicated by the movement operation is determined, and is associated with the content shooting direction close to the direction from the position after the movement to the shooting position of the background image. The content image to be held is newly extracted, and the content image to be superimposed on the background image is switched to the newly extracted content image. Therefore, when a moving operation is performed on the content image on the AR image, the appearance of the real object on the AR image changes according to the position after the movement. Thus, according to the first embodiment, the real object as the AR content on the AR image can be moved to the user while maintaining the realism.

  Furthermore, in the first embodiment, the superimposed content image is enlarged or reduced so that the real object is superimposed on the real world in the size of the real world and reflected in the background image. Therefore, according to the first embodiment, the size relationship in the real world between the real object and the background reflected in the background image can be appropriately recognized by the user.

  Further, in the first embodiment, if a plurality of content images associated with each content shooting direction are held for each of a plurality of real objects, the AR content to be superimposed is not limited to the user. A desired real object can be selected.

[Second Embodiment]
The image processing apparatus according to the second embodiment selectively executes a method for acquiring an image of an image file as a background image and a method for acquiring an image frame of a real-time video as a background image. Hereinafter, the image processing apparatus 10 according to the second embodiment will be described focusing on the content different from the first embodiment. In the following description, the same contents as those in the first embodiment are omitted as appropriate.

〔Device configuration〕
FIG. 7 is a diagram conceptually illustrating a hardware configuration example of the image processing apparatus 10 in the second embodiment. The image processing apparatus 10 according to the second embodiment further includes a camera 17 corresponding to an imaging unit in addition to the configuration of the first embodiment. The camera 17 is connected to the input / output I / F 13 or the communication unit 14.

[Processing configuration]
The image processing apparatus 10 in the second embodiment has a processing configuration similar to that of the first embodiment shown in FIG.

  The operation receiving unit 21 can also receive an operation requesting acquisition of real-time video from the camera 17. In response to this operation reception, the operation reception unit 21 can also activate the camera 17.

  When the requested operation is accepted by the operation accepting unit 21, the background obtaining unit 22 sequentially obtains an image frame of a real-time video obtained from the camera 17 as a background image in accordance with the accepted requested operation. Thereby, the background image as the superimposed image is switched from the image of the image file to the image frame of the real-time video.

  The display processing unit 29 causes the display device 15 to display a real-time video on which the content image as described above is superimposed as a visual image.

[Operation example]
The image processing method in the second embodiment will be described below with reference to FIG. FIG. 8 is a flowchart illustrating a part of an operation example of the image processing apparatus 10 according to the second embodiment. In the following description, the image processing apparatus 10 is the execution subject of each method, but the above-described processing units included in the image processing apparatus 10 may be the execution subject. In FIG. 8, the same processing steps as those shown in FIG. 6 are denoted by the same reference numerals as those in FIG.

When the image processing apparatus 10 according to the second embodiment receives an operation requesting acquisition of a real-time video in (S61) illustrated in FIG. 6, the image processing apparatus 10 performs the operation illustrated in FIG.
The image processing apparatus 10 acquires an image frame of a real-time video obtained from the camera 17 as a background image (S81). Thereafter, (S23) and subsequent steps are executed on the background image acquired in (S81) as in the first embodiment.

  The image processing apparatus 10 displays the real-time video on which the content image is superimposed as a visual image on the display device 15 by repeatedly executing the operation shown in FIG. 8 for each image frame of the real-time video obtained from the camera 17. Can do. The image processing apparatus 10 may thin out an image frame used as a background image from all image frames constituting a real-time video. Further, the image processing apparatus 10 can omit (S23) to (S64) when the marker arrangement does not change between adjacent image frames.

[Operations and effects in the second embodiment]
In the second embodiment, the background image on which the AR content is superimposed is switched from an image file image to a real-time video image frame in accordance with a user operation. Then, a content image held in association with a content shooting direction close to the direction from the AR content position to the shooting position of the image frame is superimposed on the image frame of the real-time image as a visual image of the real object. Thereby, according to 2nd Embodiment, the real-time image | video with which the content image was superimposed as a visual recognition image can be displayed on the display apparatus 15. FIG.

[First Modification]
In the first embodiment described above, since the user can select a background image, there is a possibility that the user may select an image file that holds a background image on which a marker is not accidentally shown. Such an error may occur because there are many image files stored in the computer, and among them, there are many image files other than the image file that stores the appropriate background image. If the marker is not reflected in the background image, the AR content cannot be appropriately superimposed on the background image. Therefore, each processing unit of the image processing apparatus 10 in the first embodiment may perform the following processing.

The display processing unit 29 causes the display device 15 to display that no marker is detected when no marker is detected from the background image by the marker detection unit 23.
The operation accepting unit 21 accepts an operation of reselecting an image file other than the image file in which no marker is detected.
The background acquisition unit 22 newly acquires an image of the image file newly selected by the reselection operation as a background image.

  As a result, the user can recognize whether the selected image file stores a background image on which AR content cannot be superimposed, and can reselect the image file. The image file can be selected.

[Second Modification]
The time and place for generating the image file for storing the background image are arbitrary. In other words, the user can cause the background holding unit 31 to hold a plurality of image files showing a plurality of different places for use in the background image. Under such circumstances, it may be more convenient to provide multiple types of markers. On the other hand, the image processing apparatus 10 needs to know the actual information (shape information, size information) of the marker that appears in the background image.

  FIG. 9 is a diagram conceptually illustrating a processing configuration example of the image processing apparatus 10 in the second modification. The image processing apparatus 10 according to the second modification further includes a marker information holding unit 35 in addition to the configurations of the first embodiment and the second embodiment. The marker information holding unit 35 is also realized in the same manner as other processing units. The marker information holding unit 35 may be included not in the image processing apparatus 10 but in another computer connected to be communicable with the image processing apparatus 10.

  The marker information holding unit 35 holds information on a plurality of types of markers each including shape information and size information. In this case, the marker detection unit 23 acquires the shape information and size information of one kind of marker from the marker information holding unit 35, and is acquired by the background acquisition unit 22 using the acquired shape information and size information. Marker is detected from the background image. The technique for specifying one type of marker is not limited. For example, the marker detection unit 23 can try whether or not a marker can be sequentially detected for each marker type held in the marker information holding unit 35. The marker detection unit 23 can also display a screen for allowing the user to select a marker type on the display device 15 and specify the marker type according to the user's selection operation.

  In this way, it is possible to allow the user to use a plurality of types of markers when shooting the background image, and the convenience of the AR platform can be improved.

[Third Modification]
Each processing unit of the image processing apparatus 10 according to the second embodiment may perform the following processing.

The operation receiving unit 21 receives a snapshot operation in a state where a real-time video on which a visual image is superimposed is displayed on the display device 15.
The background acquisition unit 22 fixedly holds the image frame as a background image in response to the reception of the snapshot operation by the operation reception unit 21. At this time, the background acquisition unit 22 can also stop the camera 17 after holding the image frame fixedly.
The display processing unit 29 causes the display device 15 to display a composite image in which a content image is superimposed as a visual image on a fixedly held background image, instead of displaying a real-time video on which the visual image is superimposed.

  In this way, the AR image on which the content image is superimposed can be displayed on the display device 15 by using the image frame obtained when the snapshot operation by the user is accepted as a fixed background image. . According to this, the user can see the AR image in which the content image is superimposed on the image obtained by stopping the real-time video at a desired timing. Furthermore, according to this, it is possible to prevent the position of the superimposed content image from being shaken due to camera shake or the like, making it difficult to see the displayed AR image, and to allow the user to view the AR image with an easy posture. be able to.

[Supplement]
The image processing apparatus 10 according to the second embodiment selectively executes a method for acquiring an image of an image file as a background image and a method for acquiring an image frame of a real-time video as a background image. However, the image processing apparatus 10 does not have a method of acquiring an image of an image file as a background image, and may execute a method of acquiring an image frame of a real-time video as a background image. In this case, the image processing apparatus 10 may not have the operation receiving unit 21.

  Examples will be given below to describe the above-described embodiments in more detail. The present invention is not limited in any way by the following examples.

  10A, 10B, and 10C are diagrams respectively showing examples of background images stored in the image file. As shown in these figures, each background image includes a marker. According to these examples, the same marker is included in each background image, but the marker included in each background image may not be the same as long as the image processing device 10 knows the marker. As shown in these examples, the background holding unit 31 holds a plurality of image files for storing different background images.

  Hereinafter, (S63) and (S65) shown in FIGS. 6 and 8 and further specific examples of the processing contents of the image extraction unit 26 and the composition processing unit 28 will be described. In this case, the content holding unit 32 holds, as information indicating the content shooting direction, a unit vector indicating the camera direction from the real object when shooting the real object in association with each content image.

  The image processing apparatus 10 (image extraction unit 26) calculates a unit vector indicating the direction from the position of the AR content in the three-dimensional coordinate space to the shooting position of the background image. The image processing apparatus 10 (image extraction unit 26) calculates the inner product of the unit vector calculated from the background image and each unit vector of each content image held in the content holding unit 32, and the inner product value is maximum. A unit vector of the content image is specified. The image processing apparatus 10 (image extraction unit 26) extracts the content image held in association with the specified unit vector from the content holding unit 32.

  When synthesizing the content image and the background image, the image processing apparatus 10 (compositing processing unit 28) first generates a plate-like three-dimensional object in the three-dimensional coordinate space. The image processing apparatus 10 (the composition processing unit 28) pastes the content image extracted as described above as a texture on a plate-shaped three-dimensional object.

  11A, 11B, and 11C are diagrams illustrating examples of a composite image (AR image) in which a content image is superimposed on the background image illustrated in FIGS. 10A, 10B, and 10C. As shown in these drawings, the superimposed content image is switched according to the shooting position and shooting direction of the background image, and the real object is visually recognized from the shooting position and shooting direction of the background image.

  12A, 12B, and 12C are diagrams illustrating movement of AR content on an AR image. In FIG. 12A, the AR content is arranged on the marker. In FIG. 12B, the AR content is moved backward from the state of FIG. 12A. In FIG. 12C, the AR content is moved from the state of FIG. The content has been moved to the left with respect to the marker. The user can view the AR image as shown in FIGS. 12B and 12C by performing an operation of moving the content image in a state where the AR image shown in FIG. 12A is displayed on the display device 15. . In FIG. 12B, the content image is reduced in size compared to the content image shown in FIG. 12A according to the position after movement. In FIG. 12C, the content image is switched from the content image shown in FIG. 12A according to the position after movement.

  In addition, in the some flowchart used by the above-mentioned description, although several process (process) is described in order, the execution order of the process performed by each embodiment is not restrict | limited to the order of the description. In each embodiment, the order of the illustrated steps can be changed within a range that does not hinder the contents. Moreover, each above-mentioned embodiment and each modification can be combined in the range with which the content does not conflict.

  A part or all of each of the above embodiments and modifications may be specified as in the following supplementary notes. However, each embodiment and each modification are not limited to the following description.

(Appendix 1)
An operation reception unit for receiving an image file selection operation;
A background acquisition unit for acquiring, as a background image, an image of an image file selected by a selection operation received by the operation reception unit, from a background holding unit for storing an image file storing a background image in which a marker is reflected;
A marker detection unit for detecting a marker from the background image acquired by the background acquisition unit;
A reference calculation unit that sets a three-dimensional coordinate space based on real information and image information regarding the detected marker, and calculates a background shooting position and a background shooting direction;
A position determining unit that determines a position in the three-dimensional coordinate space in which AR (Augmented Reality) content to be displayed is virtually arranged;
Display processing for causing a display unit to display a composite image in which a visual image from the background photographing position and the background photographing direction of the AR content arranged at the determined position in the three-dimensional coordinate space is superimposed on the background image And
An image processing apparatus comprising:
(Appendix 2)
From a content holding unit that holds a plurality of content images obtained by shooting a real object corresponding to the AR content from a plurality of directions in association with each content shooting direction, from the position determined by the position determination unit An image extraction unit for extracting a content image held in association with the content shooting direction close to the direction to the background shooting position;
A composition processing unit that superimposes the content image extracted by the image extraction unit on the background image as the visual image;
The image processing apparatus according to appendix 1, further comprising:
(Appendix 3)
The operation accepting unit further accepts an operation for moving the content image with respect to the composite image,
The position determination unit determines a position after movement of the AR content in the three-dimensional coordinate space indicated by the accepted movement operation;
The image extraction unit newly creates a content image held in the content holding unit in association with the content shooting direction close to the direction from the moved position determined by the position determination unit to the background shooting position. Extract and
The composition processing unit switches the content image to be superimposed on the background image to the newly extracted content image.
The image processing apparatus according to appendix 2.
(Appendix 4)
Enlarging or extracting the extracted content image based on the real size of the real object, the real information and the image information regarding the marker detected by the marker detection unit, and the position determined by the position determination unit Size adjustment part to reduce,
Further comprising
The composition processing unit superimposes the content image adjusted by the size adjusting unit on the background image as the visual image.
The image processing apparatus according to appendix 2 or 3.
(Appendix 5)
The content holding unit holds the plurality of content images associated with the content shooting directions for each of a plurality of real objects,
The operation accepting unit further accepts an operation of selecting one real object to be displayed from the plurality of real objects,
The image extraction unit extracts one content image from the plurality of content images held in the content holding unit corresponding to a real object selected by the accepted operation;
The image processing apparatus according to any one of appendices 2 to 4.
(Appendix 6)
The display processing unit displays on the display unit that no marker is detected when the marker is not detected from the background image by the marker detection unit,
The operation accepting unit accepts an operation of reselecting an image file other than the image file in which the marker is not detected,
The background acquisition unit newly acquires an image of an image file newly selected by the operation as the background image.
The image processing apparatus according to any one of appendices 1 to 5.
(Appendix 7)
The operation accepting unit accepts an operation requesting acquisition of a real-time video from the imaging unit,
The background acquisition unit sequentially acquires an image frame of the real-time video obtained from the imaging unit as the background image in response to the operation received by the operation reception unit,
The display processing unit causes the display unit to display a real-time video in which the content image is superimposed as the visual image.
The image processing apparatus according to any one of appendices 2 to 5.
(Appendix 8)
The operation accepting unit accepts a snapshot operation in a state where a real-time video on which the visual image is superimposed is displayed on the display unit,
The background acquisition unit fixedly holds the image frame as the background image in response to reception of the snapshot operation by the operation reception unit,
The display processing unit displays the composite image in which the content image is superimposed on the fixedly held background image as the visual image instead of displaying the real-time video on which the visual image is superimposed. Let
The image processing apparatus according to appendix 7.
(Appendix 9)
The marker detection unit acquires shape information and size information of one type of marker from a marker information holding unit that holds information of a plurality of types of markers each including shape information and size information, and the acquired shape information and Detecting the marker using size information;
The image processing apparatus according to any one of appendices 1 to 8.

(Appendix 10)
In an image processing method executed by at least one computer,
Accept image file selection operations,
An image of the image file selected by the accepted selection operation is acquired as a background image from a background holding unit that holds an image file that stores a background image showing a marker,
Detecting a marker from the acquired background image;
Based on real information and image information regarding the detected marker, a three-dimensional coordinate space is set,
Based on the actual information and image information about the detected marker, calculate a background shooting position and a background shooting direction,
Determining a position in the three-dimensional coordinate space where the AR (Augmented Reality) content to be displayed is virtually arranged;
Displaying a composite image in which a visual image from the background photographing position and the background photographing direction of the AR content arranged at the determined position in the three-dimensional coordinate space is superimposed on the background image on a display unit;
An image processing method.
(Appendix 11)
From the content holding unit that holds a plurality of content images obtained by shooting a real object corresponding to the AR content from a plurality of directions in association with each content shooting direction, the determined in the three-dimensional coordinate space Extracting a content image held in association with the content shooting direction close to the direction from the position to the background shooting position;
Superimposing the extracted content image on the background image as the visual image;
The image processing method according to appendix 10, further including:
(Appendix 12)
Accepting an operation of moving the content image with respect to the composite image;
Determining a position after movement of the AR content in the three-dimensional coordinate space indicated by the accepted movement operation;
Newly extracting a content image held in the content holding unit in association with the content shooting direction close to the direction from the determined moved position to the background shooting position,
Switching the content image to be superimposed on the background image to the newly extracted content image;
The image processing method according to claim 11, further comprising:
(Appendix 13)
The extracted content image is enlarged or reduced based on the real size of the real object, the real information and the image information regarding the detected marker, and the determined position in the three-dimensional coordinate space. To
The image processing method according to appendix 11 or 12, further including:
(Appendix 14)
Receiving an operation of selecting one real object to be displayed from among a plurality of real objects;
Further including
The content holding unit holds the plurality of content images associated with the content shooting directions for each of the plurality of real objects,
In the extraction of the content image, one content image is extracted from the plurality of content images held in the content holding unit corresponding to the real object selected by the accepted operation.
The image processing method according to any one of appendices 11 to 13.
(Appendix 15)
If the marker is not detected from the background image, display that the marker is not detected on the display unit,
Accepting an operation to reselect an image file other than the image file in which the marker is not detected,
An image of the image file newly selected by the operation is newly acquired as the background image.
The image processing method according to any one of appendices 10 to 14, further including:
(Appendix 16)
Accepting an operation requesting acquisition of real-time video from the imaging unit,
In response to the accepted operation, the image frame of the real-time video obtained from the imaging unit is sequentially acquired as the background image,
Displaying the real-time video on which the content image is superimposed as the visual image on the display unit;
The image processing method according to any one of appendices 11 to 14, further including:
(Appendix 17)
Accepting a snapshot operation in a state where the real-time video on which the visual image is superimposed is displayed on the display unit,
In response to accepting the snapshot operation, the image frame is fixedly held as the background image,
Instead of displaying the real-time video on which the visual image is superimposed, the composite image in which the content image is superimposed as the visual image on the fixedly held background image is displayed on the display unit.
The image processing method according to appendix 16, further including:
(Appendix 18)
Obtaining shape information and size information of one type of marker from a marker information holding unit that holds information of a plurality of types of markers each including shape information and size information;
Further including
The detection of the marker detects the marker using the acquired shape information and size information.
18. The image processing method according to any one of appendices 10 to 17.

(Appendix 19)
A background acquisition unit that sequentially acquires an image frame of a real-time video obtained from the imaging unit as a background image;
A marker detection unit for detecting a marker from the background image acquired by the background acquisition unit;
A reference calculation unit that sets a three-dimensional coordinate space based on real information and image information regarding the detected marker, and calculates a background shooting position and a background shooting direction;
A position determining unit that determines a position in the three-dimensional coordinate space in which AR (Augmented Reality) content to be displayed is virtually arranged;
From a content holding unit that holds a plurality of content images obtained by shooting a real object corresponding to the AR content from a plurality of directions in association with each content shooting direction, from the position determined by the position determination unit An image extraction unit for extracting a content image held in association with the content shooting direction close to the direction to the background shooting position;
A composition processing unit that superimposes the content image extracted by the image extraction unit on the background image as the visual image;
A display processing unit that displays on the display unit a real-time video in which the content image is superimposed as the visual image;
An image processing apparatus comprising:
(Appendix 20)
An operation accepting unit that accepts a snapshot operation in a state where the real-time video on which the visual image is superimposed is displayed on the display unit;
Further comprising
The background acquisition unit fixedly holds the image frame as the background image in response to reception of the snapshot operation by the operation reception unit,
The display processing unit displays the composite image in which the content image is superimposed on the fixedly held background image as the visual image instead of displaying the real-time video on which the visual image is superimposed. Let
The image processing apparatus according to appendix 19.

(Appendix 21)
Obtain sequentially the image frame of the real-time video obtained from the imaging unit as a background image,
Detecting a marker from the acquired background image;
Based on real information and image information regarding the detected marker, a three-dimensional coordinate space is set,
Based on the actual information and image information regarding the detected marker, calculate a background shooting position and a background shooting direction of the background image,
Determining a position in the three-dimensional coordinate space where the AR (Augmented Reality) content to be displayed is virtually arranged;
From the content holding unit that holds a plurality of content images obtained by shooting a real object corresponding to the AR content from a plurality of directions in association with each content shooting direction, the determined in the three-dimensional coordinate space Extracting a content image held in association with the content shooting direction close to the direction from the position to the background shooting position;
Superimposing the extracted content image as the visual image on the background image;
Displaying the real-time video on which the content image is superimposed as the visual image on the display unit;
An image processing method.
(Appendix 22)
Accepting a snapshot operation in a state where the real-time video on which the visual image is superimposed is displayed on the display unit,
In response to accepting the snapshot operation, the image frame is fixedly held as the background image,
Instead of displaying the real-time video on which the visual image is superimposed, the composite image in which the content image is superimposed as the visual image on the fixedly held background image is displayed on the display unit.
The image processing method according to claim 21, further comprising:

(Appendix 23)
A program that causes at least one computer to execute the image processing method according to any one of Supplementary Notes 10 to 18, 21, and 22.

10, 100 Image processing apparatus 11 CPU
DESCRIPTION OF SYMBOLS 12 Memory 15 Display apparatus 16 Input device 17 Camera 21, 101 Operation reception part 22, 102 Background acquisition part 23, 103 Marker detection part 24, 104 Reference | standard calculation part 25, 105 Position determination part 26 Image extraction part 27 Size adjustment part 28 Composition Processing unit 29, 106 Display processing unit 31, 107 Background holding unit 32 Content holding unit 35 Marker information holding unit 108 Display unit

Claims (11)

An operation reception unit for receiving an image file selection operation;
A background acquisition unit for acquiring, as a background image, an image of an image file selected by a selection operation received by the operation reception unit, from a background holding unit for storing an image file storing a background image in which a marker is reflected;
A marker detection unit for detecting a marker from the background image acquired by the background acquisition unit;
A reference calculation unit that sets a three-dimensional coordinate space based on real information and image information regarding the detected marker, and calculates a background shooting position and a background shooting direction;
A position determining unit that determines a position in the three-dimensional coordinate space in which AR (Augmented Reality) content to be displayed is virtually arranged;
Display processing for causing a display unit to display a composite image in which a visual image from the background photographing position and the background photographing direction of the AR content arranged at the determined position in the three-dimensional coordinate space is superimposed on the background image And
An image processing apparatus comprising: From a content holding unit that holds a plurality of content images obtained by shooting a real object corresponding to the AR content from a plurality of directions in association with each content shooting direction, from the position determined by the position determination unit An image extraction unit for extracting a content image held in association with the content shooting direction close to the direction to the background shooting position;
A composition processing unit that superimposes the content image extracted by the image extraction unit on the background image as the visual image;
The image processing apparatus according to claim 1, further comprising: The operation accepting unit further accepts an operation for moving the content image with respect to the composite image,
The position determination unit determines a position after movement of the AR content in the three-dimensional coordinate space indicated by the accepted movement operation;
The image extraction unit newly creates a content image held in the content holding unit in association with the content shooting direction close to the direction from the moved position determined by the position determination unit to the background shooting position. Extract and
The composition processing unit switches the content image to be superimposed on the background image to the newly extracted content image.
The image processing apparatus according to claim 2. Enlarging or extracting the extracted content image based on the real size of the real object, the real information and the image information regarding the marker detected by the marker detection unit, and the position determined by the position determination unit Size adjustment part to reduce,
Further comprising
The composition processing unit superimposes the content image adjusted by the size adjusting unit on the background image as the visual image.
The image processing apparatus according to claim 2. The content holding unit holds the plurality of content images associated with the content shooting directions for each of a plurality of real objects,
The operation accepting unit further accepts an operation of selecting one real object to be displayed from the plurality of real objects,
The image extraction unit extracts one content image from the plurality of content images held in the content holding unit corresponding to a real object selected by the accepted operation;
The image processing apparatus according to claim 2. The display processing unit displays on the display unit that no marker is detected when the marker is not detected from the background image by the marker detection unit,
The operation accepting unit accepts an operation of reselecting an image file other than the image file in which the marker is not detected,
The background acquisition unit newly acquires an image of an image file newly selected by the operation as the background image.
The image processing apparatus according to claim 1. The operation accepting unit accepts an operation requesting acquisition of a real-time video from the imaging unit,
The background acquisition unit sequentially acquires an image frame of the real-time video obtained from the imaging unit as the background image in response to the operation received by the operation reception unit,
The display processing unit causes the display unit to display a real-time video in which the content image is superimposed as the visual image.
The image processing apparatus according to claim 2. The operation accepting unit accepts a snapshot operation in a state where a real-time video on which the visual image is superimposed is displayed on the display unit,
The background acquisition unit fixedly holds the image frame as the background image in response to reception of the snapshot operation by the operation reception unit,
The display processing unit displays the composite image in which the content image is superimposed on the fixedly held background image as the visual image instead of displaying the real-time video on which the visual image is superimposed. Let
The image processing apparatus according to claim 7. The marker detection unit acquires shape information and size information of one type of marker from a marker information holding unit that holds information of a plurality of types of markers each including shape information and size information, and the acquired shape information and Detecting the marker using size information;
The image processing apparatus according to claim 1. In an image processing method executed by at least one computer,
Accept image file selection operations,
An image of the image file selected by the accepted selection operation is acquired as a background image from a background holding unit that holds an image file that stores a background image showing a marker,
Detecting a marker from the acquired background image;
Based on real information and image information regarding the detected marker, a three-dimensional coordinate space is set,
Based on the actual information and image information about the detected marker, calculate a background shooting position and a background shooting direction,
Determining a position in the three-dimensional coordinate space where the AR (Augmented Reality) content to be displayed is virtually arranged;
Displaying a composite image in which a visual image from the background photographing position and the background photographing direction of the AR content arranged at the determined position in the three-dimensional coordinate space is superimposed on the background image on a display unit;
An image processing method.   A program for causing at least one computer to execute the image processing method according to claim 10.

Images