WO2015072091A1 - Image processing device, image processing method, and program storage medium - Google Patents

Abstract

An objective of the present invention is to provide a technology whereby an AR image formed by superpositioning AR content on a desired background image is presented without restriction on place or time. Provided is an image processing device comprising: a manipulation acceptance means for accepting a selection manipulation on an image file; a background acquisition means for acquiring, from a background retaining means for retaining image files, an image of the image file selected with the selection manipulation as a background image; a marker detection means for detecting a marker from the acquired background image; a reference computation means for setting a three-dimensional coordinate space and computing a background photography location and a background photography direction on the basis of reality information and image information relating to the detected marker; a location determination means for determining a location in the three-dimensional coordinate space where AR content to be displayed is virtually positioned; and a display control means for causing a display means to display a composite image formed by superpositioning on the background image a viewing image, to be seen from the background photography location and background photography direction, of the AR content positioned at the determined location in the three-dimensional coordinate space.

Description

Image processing apparatus, image processing method, and program storage medium

The present invention relates to augmented reality (AR) image processing technology.

Currently, various applications using technology called augmented reality are being provided. The AR technology displays three-dimensional content and two-dimensional content (such as characters and designs) in a superimposed manner in the real world obtained via a camera or a head-mounted display (HMD) of a mobile device such as a smartphone. For example, the following Non-Patent Document 1 proposes a method of generating a three-dimensional content to be superimposed on the real world with an image obtained by actually capturing a real object existing in the real world. In addition, there is a form in which the virtual three-dimensional model generated by using computer graphic (CG) technology is used for the three-dimensional content, instead of the photographed image.

Hereinafter, a composite image in which 3D content and 2D data are superimposed on the real world may be referred to as an AR image. In addition, an image of the real world on which 3D content or 2D content is superimposed may be described as a background image. In addition, 3D content and 2D content superimposed on the background image may be referred to as AR content.

The following Non-Patent Document 2 proposes a remote control system for a mobile robot using AR technology. In this proposed method, an image in which a three-dimensional model of a robot is superimposed on an extended free viewpoint image using AR technology is presented to the operator via the HMD. This extended free viewpoint image is generated based on the omnidirectional image group and distance image group acquired at a plurality of points by the mobile robot, and the position and orientation of the robot at the time of acquisition. Further, in Patent Document 1 below, as a method for generating three-dimensional content to be superimposed, a converted image for three-dimensional display of a target object can be obtained with a single camera without using a dedicated device such as an acceleration sensor. A generation method has been proposed. In this proposed method, the conversion target image is geometrically transformed so that the specified image coordinates match by allowing the user to specify a plurality of image coordinates constituting the rotation axis of the camera for a plurality of images including the target object. The

International Publication No. 2011/148595

In the above-described proposed method and existing applications, a real-time video obtained from an imaging device in real time is used as a background image on which 3D content or 2D content is superimposed. As a result, the user can visually recognize an AR image in which AR content exists in the real world captured by the imaging apparatus in real time.

However, with the method as described above, the real world where the AR content is to be superimposed is limited to the location of the imaging device at that time. For example, a user can view an AR image in which AR content is superimposed on his / her room with his / her camera-equipped mobile terminal only when the user is in his / her room with the mobile terminal. . The user cannot see the AR image in a place other than the user's room such as in a train.

The present invention has been made in view of such circumstances. One of the objects of the present invention is to provide a technique for presenting an AR image in which AR content is superimposed on a desired background image without restrictions on place and time.

In each aspect of the present invention, the following configurations are adopted in order to solve the above-described problems.

The first aspect of the present invention relates to an image processing apparatus. An image processing apparatus according to a first aspect of the present invention is received by an operation receiving unit from an operation receiving unit that receives an image file selection operation and a background holding unit that stores an image file that stores a background image in which a marker is captured. Background acquisition means for acquiring an image of the image file selected by the selection operation as a background image, marker detection means for detecting a marker from the background image acquired by the background acquisition means, and actual information and image information regarding the detected marker And a reference calculation means for calculating a background shooting position and a background shooting direction, and a position in the three-dimensional coordinate space where the AR content to be displayed is virtually arranged is determined. Position determining means, background shooting position and background shooting method of AR content arranged at determined position in three-dimensional coordinate space Viewed image from having a display control means for displaying on the display means a composite image superimposed on the background image.

The second aspect of the present invention relates to an image processing method. The image processing method according to the second aspect of the present invention receives an image file selection operation, and is selected by the received selection operation from a background holding unit that holds an image file that stores a background image in which a marker is captured. An image of an image file is acquired as a background image, a marker is detected from the acquired background image, a three-dimensional coordinate space is set based on the actual information and image information regarding the detected marker, and the reality regarding the detected marker Based on the information and the image information, the background photographing position and the background photographing direction are calculated, the position on the three-dimensional coordinate space where the AR content to be displayed is virtually arranged is determined, and the determined position in the three-dimensional coordinate space Display on the display means a composite image in which the visual image from the background shooting position and the background shooting direction of the AR content arranged on the background image is superimposed on the background image. , Including that.

The third aspect of the present invention relates to a program. According to a third aspect of the present invention, there is provided a program comprising: an operation receiving unit that receives a selection operation of an image file; and a background holding unit that holds an image file that stores a background image in which a marker is captured. A background acquisition unit that acquires an image of an image file selected by an accepted selection operation as a background image, a marker detection unit that detects a marker from the background image acquired by the background acquisition unit, and the detected marker The reference calculation means for setting the three-dimensional coordinate space based on the real information and the image information and calculating the background photographing position and the background photographing direction and the AR (Augmented Reality) content to be displayed are virtually arranged. A position determining means for determining a position in the three-dimensional coordinate space; and a position determining unit arranged at the determined position in the three-dimensional coordinate space. Display control means for viewing images displayed on the display means a composite image superimposed on the background image from the background photographing position and the background photographing direction of the AR content is to operate with.
The third aspect of the present invention is also realized by a computer-readable recording medium that records the program. This recording medium includes a non-transitory tangible medium.

According to each aspect described above, it is possible to provide a technique for presenting an AR image in which AR content is superimposed on a desired background image without any restrictions on place or time.

FIG. 1 is a diagram conceptually illustrating a processing configuration example of an image processing apparatus according to the first embodiment of the present invention. FIG. 2 is a flowchart showing an operation example of the image processing apparatus according to the first embodiment of the present invention. FIG. 3 is a diagram conceptually illustrating a hardware configuration example of an image processing apparatus according to the second embodiment of the present invention. FIG. 4 is a diagram conceptually illustrating a processing configuration example of the image processing apparatus according to the second embodiment of the present invention. FIG. 5 is a diagram conceptually illustrating an example of a plurality of content images held in the content holding unit and associated with various content shooting directions. FIG. 6 is a flowchart showing an operation example of the image processing apparatus according to the second embodiment of the present invention. FIG. 7 is a diagram conceptually illustrating a hardware configuration example of an image processing apparatus according to the third embodiment of the present invention. FIG. 8 is a flowchart showing a part of an operation example of the image processing apparatus according to the third embodiment of the present invention. FIG. 9 is a diagram conceptually illustrating a processing configuration example of the image processing apparatus according to the third modification of the present invention. FIG. 10A is a diagram illustrating an example of a background image stored in an image file. FIG. 10B is a diagram illustrating an example of a background image stored in an image file. FIG. 10C is a diagram illustrating an example of a background image stored in an image file. FIG. 11A is a diagram showing an example of a composite image (AR image) in which a content image is superimposed on the background image shown in FIG. 10A. FIG. 11B is a diagram showing an example of a composite image (AR image) in which a content image is superimposed on the background image shown in FIG. 10B. FIG. 11C is a diagram illustrating an example of a composite image (AR image) in which a content image is superimposed on the background image illustrated in FIG. 10C. FIG. 12A is a diagram illustrating movement of AR content on an AR image. FIG. 12B is a diagram illustrating movement of AR content on an AR image. FIG. 12C is a diagram illustrating movement of AR content on an AR image.

Hereinafter, embodiments of the present invention will be described. In addition, embodiment mentioned below is an illustration and this invention is not limited to the structure of the following embodiment.

[First Embodiment]
FIG. 1 is a diagram conceptually illustrating a processing configuration example of an image processing apparatus according to the first embodiment of the present invention. As illustrated in FIG. 1, the image processing apparatus 100 includes an operation reception unit 101, a background acquisition unit 102, a marker detection unit 103, a reference calculation unit 104, a position determination unit 105, a display control unit 106, and the like. The image processing apparatus 100 has the same hardware configuration as the image processing apparatus 10 in a detailed embodiment described later. Similar to the image processing apparatus 10, the image processing apparatus 100 is controlled by a program, thereby realizing the above-described processing units. In addition, the image processing apparatus 100 may include a background holding unit 107 and a display unit 108. However, another computer (not shown) connected to be communicable with the image processing apparatus 100 may include the background holding unit 107 and the display unit 108. The display unit 108 is a device that displays a screen generated based on drawing data, such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display.

The operation reception unit 101 receives an image file selection operation.

The background acquisition unit 102 acquires, as a background image, the image of the image file selected by the selection operation received by the operation reception unit 101 from the background holding unit 107 that stores the image file that stores the background image in which the marker is reflected. Here, moving image data or still image data is stored in the image file. The background acquisition unit 102 acquires a background image as still image information from moving image data or still image data.

The marker detection unit 103 detects a marker from the background image acquired by the background acquisition unit 102. Here, the marker is an image drawn on an object arranged in the real world or an object arranged in the real world. The marker is called an AR marker or the like. However, in the present embodiment, if a certain reference point and three directions perpendicular to each other from the reference point can be obtained from this marker regardless of the reference direction, the specific of this marker can be obtained. The form is not limited. For example, the marker detection unit 103 holds in advance marker actual information (such as shape information). And the marker detection part 103 detects a marker from a background image based on this real information.

The reference calculation unit 104 sets a three-dimensional coordinate space based on the detected real information and image information about the marker, and calculates the background shooting position and the background shooting direction. The actual information related to the marker means known information in the real world of the marker. The actual information regarding the marker indicates, for example, the actual shape of the marker. Moreover, the image information regarding a marker means the information in the background image of a marker. The image information regarding the marker indicates the shape of the marker in the background image. The reference calculation unit 104 sets a three-dimensional coordinate space at a position corresponding to the reference point recognized from the detected marker. The position where the three-dimensional coordinate space is set is not particularly limited. In addition, the reference calculation unit 104 calculates the background shooting position and the background shooting direction by comparing the actual information about the marker and the image information. The background shooting position and the background shooting direction mean the position and orientation (posture) of the imaging apparatus that shot the background image of the image file.

The position determination unit 105 determines the position in the three-dimensional coordinate space where the AR content to be displayed is virtually arranged. The position determination unit 105 can determine this position arbitrarily or fixedly. The AR content to be displayed may be a three-dimensional model proposed in Non-Patent Documents 1 and 2 above. The AR content to be displayed may be three-dimensional content that is simulated by switching and displaying a plurality of content images obtained by photographing a real object from a plurality of directions. The AR content to be displayed may be planar two-dimensional content.

The display control unit 106 displays a composite image in which a visual image from the background shooting position and the background shooting direction of the AR content arranged at the determined position in the three-dimensional coordinate space is superimposed on the background image. 108 is displayed. When the AR content is the above-described three-dimensional model or two-dimensional content, the display control unit 106 obtains the visual image using a known method based on the information of the three-dimensional model or the two-dimensional content. Can do. A method for acquiring the visual image when the AR content is a three-dimensional content that is simulated by switching and displaying a plurality of content images obtained by photographing a real object from a plurality of directions is described in detail. Will be described later.

FIG. 2 is a flowchart showing an operation example of the image processing apparatus 100 in the present embodiment. The image processing method in the present embodiment is executed by at least one computer such as an image processing apparatus 100 as shown in FIG. The computer may have a hardware configuration similar to that of the image processing apparatus 10 described later. The image processing method includes processing steps from step S21 to step S27.

In step S21, the computer accepts an image file selection operation.

In step S22, the computer acquires the image of the image file selected by the selection operation accepted in step S21 as the background image from the background holding unit 107 that holds the image files in which the markers each store the background image. The marker and the image file are as described above. The content of step S22 is the same as the processing content of the background acquisition unit 102.

In step S23, the computer detects a marker from the background image acquired in step S22. The content of step S23 is the same as the processing content of the marker detection unit 103.

In step S24, the computer sets a three-dimensional coordinate space based on the real information and image information regarding the marker detected in step S23.
In step S25, the computer calculates a background shooting position and a background shooting direction based on the actual information and the image information regarding the marker detected in step S23. The contents of steps S24 and S25 are the same as the contents of processing of the reference calculation unit 104.

In step S26, the computer determines a position on the three-dimensional coordinate space where the AR content to be displayed is virtually arranged. The content of step S26 is the same as the processing content of the position determination unit 105.

In step S27, the computer causes the display unit 108 to display a composite image in which the visual image of the AR content to be arranged is superimposed on the background image. The position where the AR content is arranged is the position determined in step S26 in the three-dimensional coordinate space set in step S24. The visual image of the AR content is a visual image of the AR content arranged at the above-described position from the background shooting position and the background shooting direction calculated in step S25. The background image is the background image acquired in step S22. The content of step S27 is the same as the processing content of the display control unit 106.

At least one computer that is the execution subject of the image processing method may include the display unit 108 and the background holding unit 107. Another computer (not shown) connected to be communicable with the at least one computer may include the display unit 108 and the background holding unit 107. In addition, each embodiment of the present invention may be realized by a program that causes at least one computer to execute the above-described image processing method. Each embodiment of the present invention may be realized by a computer-readable recording medium that records such a program.

Further, the execution order of the processing steps of the image processing method in the present embodiment is not limited to the order shown in FIG. For example, step S26 may be executed before step S25 if it is after step S24. Step S26 may be executed in parallel with step S25 after step S24.

Thus, in this embodiment, the image of the image file selected by the user's selection operation from the image files held in the background holding unit 107 is acquired as the background image (superposed image). A composite image in which a visual image of the AR content is superimposed on the background image is displayed. Therefore, according to the present embodiment, the user performs an operation of selecting an image file in which a desired background image is stored from among the held image files, so that the AR content is superimposed on the desired background image. An image (AR image) can be seen. As a result, the user can view the AR image of the desired background image anytime and anywhere.

As described above, the present inventors have solved the new idea of eliminating the limitation of the background image on which the AR content is superimposed and presenting the AR image in which the AR content is superimposed on the desired background image without limitation of place or time. I found. Then, as described above, the present inventors are a new user who allows a user to select an image file in which a desired background image is stored from a plurality of image files when he / she wants to view an AR image. Found the interface. The plurality of image files are image files in which a plurality of background images, each of which includes a marker, are stored so as to be a superimposed image. Such a user interface has a completely different concept from the user interface of an existing AR application that is based on the assumption that AR content is superimposed on real-time video.

According to such a user interface in the present embodiment, the following problems in the existing AR application can be solved. When a portable terminal with a camera is used as a device that captures and displays an AR image, the user must always shoot the real world on which AR content is superimposed on the portable terminal. In this case, in order to view the AR image, the user needs to always raise his arm. Therefore, the user cannot see the AR image with an easy posture. Furthermore, since the portable terminal is held by the user's hand, it is difficult to completely fix the position of the portable terminal. Therefore, the background image is blurred.

According to the present embodiment, since the image of the image file selected by the selection operation by the user is used as the background image (superimposed image), the user does not mind camera shake and has an easy posture. You can see the image.

The visual image of the AR content is set based on the marker detected from the background image, and the AR image placed at the position in the three-dimensional coordinate space is calculated based on the marker. Represents the field of view visually recognized from the background photographing position and the background photographing direction. Therefore, the user can recognize the situation as if the AR content existed in the real world shown in the desired background image by viewing the displayed AR image.

Hereinafter, further details of the above-described first embodiment will be described. Below, 2nd Embodiment and 3rd Embodiment are illustrated as detailed embodiment. Hereinafter, the same contents as those in the first embodiment are appropriately omitted. The following description will focus on the content different from the first embodiment described above.

[Second Embodiment]
The image processing apparatus according to the second embodiment uses one content image among a plurality of content images obtained by photographing a real object from a plurality of directions as a visual image of the AR content, and uses an AR image (composite). Image).

〔Device configuration〕
FIG. 3 is a diagram conceptually illustrating a hardware configuration example of the image processing apparatus 10 according to the second embodiment. The image processing apparatus 10 is a so-called computer. The form of the image processing apparatus 10 such as a portable type or a stationary type is not limited. The image processing apparatus 10 includes, for example, a CPU (Central Processing Unit) 11, a memory 12, an input / output interface (I / F) 13, a communication unit 14, and the like that are connected to each other via a bus. The memory 12 is at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), and a hard disk. The communication unit 14 exchanges signals with other computers and devices. A portable recording medium 18 or the like can be connected to the communication unit 14.

The input / output I / F 13 can be connected to the display device 15 and a user interface device such as the input device 16. The display device 15 is generated based on drawing data processed by a CPU 11 or a GPU (Graphics Processing Unit) (not shown) such as an LCD (Liquid Crystal Display) or a CRT (Cathode Ray Tube) display. It is a device that displays a screen. The display device 15 corresponds to the display unit 108 described above. The input device 16 is a device that receives an input of a user operation, such as a keyboard and a mouse. The display device 15 and the input device 16 may be integrated. The display device 15 and the input device 16 may be realized as a touch panel, for example. The display device 15 and the input device 16 may be connected to another computer that is communicably connected to the image processing device 10 via the communication unit 14.

Further, the image processing apparatus 10 may have other hardware elements not shown in FIG. For example, the image processing apparatus 10 may have a camera corresponding to the imaging unit. In this case, the image processing apparatus 10 can cause the background holding unit 31 to hold the image file of the background image captured by the camera. Then, the image processing apparatus 10 can capture a content image of a real object described later with the camera. The hardware configuration of the image processing apparatus 10 is not limited.

[Processing configuration]
FIG. 4 is a diagram conceptually illustrating a processing configuration example of the image processing apparatus 10 according to the second embodiment. The image processing apparatus 10 according to the second embodiment includes an operation reception unit 21, a background acquisition unit 22, a marker detection unit 23, a reference calculation unit 24, a position determination unit 25, an image extraction unit 26, a size adjustment unit 27, and a synthesis processing unit. 28, a display control unit 29, and the like. The image processing apparatus 10 further includes a background holding unit 31, a content holding unit 32, and the like. These processing units are realized, for example, by executing a program stored in the memory 12 by the CPU 11. In addition, the program is installed from a portable recording medium 18 such as a CD (Compact Disc) or a memory card or another computer on the network via the input / output I / F 13 or the communication unit 14, and the memory 12. May be stored.

The operation reception unit 21 corresponds to the operation reception unit 101 described above. The operation reception unit 21 receives a user operation using the input device 16 for a screen displayed on the display device 15. The operation accepting unit 21 further accepts a content image moving operation with respect to the composite image in addition to the image file selecting operation. The selection operation includes, for example, an operation for selecting one image file from a list of a plurality of image files displayed on the display device 15 and a button operation for confirming the operation. The movement operation is, for example, an operation of dragging and dropping or swiping the content image on the composite image displayed on the display device 15. However, the present embodiment does not limit the specific operation content itself. The content image is an image that is superimposed as a visual image of the AR content on the background image. The content image will be described in detail later.

Furthermore, the operation accepting unit 21 can accept an operation for selecting one real object to be displayed from among a plurality of real objects. A real object is an object or person existing in the real world that is the source of AR content superimposed on a background image. The operation accepting unit 21 can cause the display device 15 to display a list of a plurality of real objects related to content images held in the content holding unit 32 described later. And the operation reception part 21 can receive selection operation using the input device 16 with respect to the list | wrist.

The background acquisition unit 22, the background holding unit 31, the marker detection unit 23, and the reference calculation unit 24 are the same as the background acquisition unit 102, the background holding unit 107, the marker detection unit 103, and the reference calculation unit 104 in the first embodiment described above. It is the same.

The position determination unit 25 determines the position in the three-dimensional coordinate space where the AR content to be displayed is virtually arranged, like the position determination unit 105 described above. Furthermore, when the movement operation is received by the operation reception unit 21, the position determination unit 25 determines a position after movement of the AR content on the three-dimensional coordinate space indicated by the movement operation. Specifically, the position determination unit 25 converts the moved position indicated by the moving operation on the composite image into a position in the three-dimensional coordinate space based on the marker actual information and image information. The position determination unit 25 determines the converted position as the position after movement of the AR content.

The content holding unit 32 holds a plurality of content images respectively associated with the content shooting directions obtained by shooting a real object corresponding to the AR content from a plurality of directions. The content shooting direction is the direction in which the content image is shot. In the second embodiment, since the plurality of content images are used as visually recognized images of the AR content, it is desirable that the content holding unit 32 hold a large number of content images taken from all directions with a fine angular width. The reason is that the content image superimposed on the background image can be approximated to the visual image of the real object from the background shooting direction and the background shooting position of the background image.

FIG. 5 is a diagram conceptually showing an example of a plurality of content images held in the content holding unit 32 and associated with various content shooting directions. In FIG. 5, the content shooting direction is shown in a two-axis coordinate space in the horizontal direction and the vertical direction. As shown in FIG. 5, the content holding unit 32 can also hold a plurality of content images each associated with a position coordinate indicating the content shooting direction.

In the example of FIG. 5, each content image includes a marker and other backgrounds along with the real object. In the second embodiment, the content holding unit 32 may hold a content image in which something other than a real object is captured. However, it is desirable that the content holding unit 32 holds a content image in which an image area other than the real object is made transparent. This is because, as a content image, it is easy to generate a composite image in which only a visual image of a real object is superimposed on a background image.

The content holding unit 32 holds information on the size and shape of the marker in the real world when a marker appears in the content image to be held. The content holding unit 32 holds information about the size of the real world of the real object when the marker is not captured in the content image to be held.

The content holding unit 32 can also hold a plurality of content images associated with various content shooting directions for each of the plurality of real objects. In this case, the operation reception unit 21 can receive an operation of selecting one real object to be displayed from among a plurality of real objects.

The image extraction unit 26 holds the content image associated with the content shooting direction close to the direction from the position determined by the position determination unit 25 to the background shooting position calculated by the reference calculation unit 24 from the content holding unit 32. To extract. It is desirable that the content image associated with the content shooting direction that coincides with the direction from the position of the AR content in the three-dimensional coordinate space to the background shooting position is held in the content holding unit 32. The reason is that the content image matches the visual image of the real object from the shooting position of the background image. When such a matching content image is not held in the content holding unit 32, the image extraction unit 26 extracts a content image held in association with the closest content shooting direction.

When content images of a plurality of real objects are held in the content holding unit 32, the operation receiving unit 21 receives an operation of selecting one real object from the plurality of real objects. The image extraction unit 26 extracts one content image from a plurality of content images associated with the selected real object that is retained in the content retention unit 32.

Further, when a movement operation is received by the operation reception unit 21, the position determination unit 25 determines a position after the movement. The image extraction unit 26 is held in the content holding unit 32 in association with a content shooting direction that is close to the direction from the moved position determined by the position determination unit 25 to the background shooting position calculated by the reference calculation unit 24. New content images are extracted. The newly extracted content image is a visual image of a real object that exists at a position after the background image is moved from the shooting position.

The size adjustment unit 27 is extracted by the image extraction unit 26 based on the actual size of the real object, the real information and image information regarding the marker detected by the marker detection unit 23, and the position determined by the position determination unit 25. The enlarged content image is enlarged or reduced. The size adjusting unit 27 determines the enlargement ratio or reduction ratio of the content image so that the real object shown in the content image is superimposed on the real world shown in the background image in the size of the real world. For example, the size adjusting unit 27 can obtain the reduction ratio of the real world in the background image by using the real information and the image information regarding the marker. The size adjustment unit 27 uses the real size of the real object, the image size in the content image, the reduction ratio of the real world in the background image, and the distance between the background shooting position and the AR content position. The enlargement or reduction ratio of the image can be determined.

The composition processing unit 28 is a composite image in which the visual image from the background photographing position and the background photographing direction of the AR content arranged at the position determined by the position determining unit 25 in the three-dimensional coordinate space is superimposed on the background image. (AR image) is generated. Specifically, the composition processing unit 28 superimposes the content image extracted by the image extraction unit 26 on the background image acquired by the background acquisition unit 22 as a visual image of the real object (AR content).

The composition processing unit 28 switches the content image to be superimposed on the background image to the content image newly extracted by the image extraction unit 26 in response to the movement operation received by the operation reception unit 21. Further, when the content image is enlarged or reduced by the size adjusting unit 27, the composition processing unit 28 superimposes the enlarged or reduced content image on the background image.

The display control unit 29 causes the display device 15 to display the composite image generated by the composition processing unit 28.

[Operation example]
The image processing method according to the second embodiment will be described below with reference to FIG. FIG. 6 is a flowchart illustrating an operation example of the image processing apparatus 10 according to the second embodiment. In the following description, the image processing apparatus 10 executes each processing step (that is, step). However, among any of the above-described processing units included in the image processing apparatus 10, a processing unit corresponding to the processing step to be executed may execute the processing step. 6, the same processing steps as those shown in FIG. 2 are denoted by the same reference numerals as those in FIG.

The image processing apparatus 10 accepts a user operation using the input device 16 (step S61). The content of step S61 is the same as the processing content of the operation reception part 21. FIG.
When the selection operation is accepted (“selection” in step S62), the image processing apparatus 10 executes the processing steps from step S22 to step S26 as in FIG.

In this case, the image processing apparatus 10 extracts a content image from the content image held in the content holding unit 32 (step S63). In step S63, the image processing apparatus 10 may extract a content image associated with a content shooting direction that is close to the direction from the position determined in step S26 to the background shooting position calculated in step S25. The content of step S63 is the same as the processing content of the image extraction unit 26.

Subsequently, the image processing apparatus 10 enlarges or reduces the content image extracted in step S63 so that the real object is superimposed on the real world in the size of the real world and reflected in the background image (step S64). The content of step S64 is the same as the processing content of the size adjustment unit 27.

The image processing apparatus 10 generates a composite image by superimposing the content image enlarged or reduced in step S64 on the background image acquired in step S22 (step S65). The content of step S65 is the same as the processing content of the composition processing unit 28.

The image processing device 10 causes the display device 15 to display the composite image generated in step S65 (step S66).

When the moving operation is accepted (“move” in step S62), the image processing apparatus 10 determines the position after the movement of the AR content in the three-dimensional coordinate space indicated by the moving operation accepted in step S61. (Step S67). The content of step S67 is the same as the processing content of the position determination unit 25.

The image processing apparatus 10 newly extracts a content image from the content image held in the content holding unit 32 (step S63). In this case, the image processing apparatus 10 may extract a content image associated with a content shooting direction that is close to the direction from the moved position determined in step S67 to the background shooting position calculated in step S25. Thereafter, the image processing apparatus 10 executes step S64 and subsequent steps on the newly extracted content image.

When content images of a plurality of real objects are held in the content holding unit 32, the image processing apparatus 10 performs an operation of selecting one real object to be displayed from the plurality of real objects in step S61. Can also be accepted. In this case, in step S63, the image processing apparatus 10 extracts one content image associated with the real object selected by the operation from the plurality of content images held in the content holding unit 32. . The above operation in the case where content images of a plurality of real objects are held in the content holding unit 32 is not shown in FIG.

[Operation and Effect of Second Embodiment]
As described above, in the second embodiment, an image of an image file selected by a user's selection operation from among a plurality of image files is used as a background image (superimposed image). Without limitation, an AR image in which AR content is superimposed on a desired background image can be shown.

Furthermore, in the second embodiment, a plurality of content images obtained by photographing a real object from a plurality of directions and respectively associated with the content photographing directions are stored in advance. Then, an AR image (composite image) is generated using one content image among the plurality of held content images as a visual image of the AR content. As described above, according to the second embodiment, since it is only necessary to prepare a plurality of content images associated with the content shooting direction, compared to a mode of generating a three-dimensional model that requires specialized CG technology, AR content can be easily created. Therefore, according to the second embodiment, the threshold for development and use of a platform using the AR technology can be reduced. Furthermore, the platform using the AR technology can be easily used by general users.

Furthermore, in the second embodiment, the held content image associated with the content shooting direction close to the direction from the AR content position to the background image shooting position in the three-dimensional coordinate space is the visual recognition of the real object. The image is superimposed on the background image. Therefore, according to the second embodiment, the content image to be superimposed on the background image can be approximated to the visual image of the real object from the shooting position and shooting direction of the background image. Therefore, it is possible to enhance the realism of the space in which the real object is superimposed on the real world shown in the background image.

Furthermore, in the second embodiment, the position after movement in the three-dimensional coordinate space indicated by the movement operation is determined. Furthermore, a content image to be held that is associated with a content shooting direction that is close to the direction from the determined position after movement to the shooting position of the background image is newly extracted. The content image to be superimposed on the background image is switched to the newly extracted content image. Therefore, when a movement operation is performed on the content image on the AR image, the appearance of the real object on the AR image changes according to the position after the movement. As described above, according to the second embodiment, the user can move the real object as the AR content on the AR image while maintaining the reality.

Furthermore, in the second embodiment, the content image to be superimposed is enlarged or reduced so that the real object is superimposed on the real world in the background image with the size in the real world. Therefore, according to the second embodiment, the size relationship in the real world between the real object and the background reflected in the background image can be appropriately recognized by the user.

In the second embodiment, a plurality of content images associated with various content shooting directions can be held for each of the plurality of real objects. In this case, the user can select not only a desired background image but also a desired real object as an AR content to be superimposed.

[Third Embodiment]
The image processing apparatus according to the third embodiment selectively operates a function of acquiring an image of an image file as a background image and a function of acquiring an image frame of a real-time video as a background image. Hereinafter, the image processing apparatus 10 according to the third embodiment will be described focusing on contents different from those of the second embodiment. In the following description, the same contents as those in the second embodiment are omitted as appropriate.

〔Device configuration〕
FIG. 7 is a diagram conceptually illustrating a hardware configuration example of the image processing apparatus 10 according to the third embodiment. The hardware configuration of the image processing apparatus 10 according to the third embodiment may be the same as the hardware configuration of the image processing apparatus 10 according to the second embodiment. The image processing apparatus 10 according to the third embodiment further includes a camera 17 that corresponds to an imaging unit in addition to the configuration of the second embodiment. That is, the camera 17 is further connected to the image processing apparatus 10 in the present embodiment. The camera 17 is connected to the input / output I / F 13 or the communication unit 14.

[Processing configuration]
The image processing apparatus 10 according to the third embodiment has the same functional configuration as that of the second embodiment shown in FIG.

The operation accepting unit 21 can also accept an operation for requesting acquisition of a real-time video from the camera 17 (hereinafter, a request operation). In response to accepting this operation, the operation accepting unit 21 can also activate the camera 17.

When the operation request is received by the operation reception unit 21, the background acquisition unit 22 sequentially acquires the real-time video image frame obtained from the camera 17 as a background image according to the received request operation. Thereby, the background image as the superimposed image is switched from the image of the image file to the image frame of the real-time video.

The display control unit 29 causes the display device 15 to display a real-time video in which the content image as described above is superimposed as a visual image.

[Operation example]
Hereinafter, an image processing method according to the third embodiment will be described with reference to FIG. FIG. 8 is a flowchart illustrating a part of an operation example of the image processing apparatus 10 according to the third embodiment. In the following description, the image processing apparatus 10 executes each processing process. However, among the above-described processing units included in the image processing apparatus 10, a processing unit corresponding to the processing step to be executed may execute the processing step. In FIG. 8, the same processing steps as those shown in FIG. 6 are denoted by the same reference numerals as in FIG.

The image processing apparatus 10 according to the third embodiment first executes step S61. When the image processing apparatus 10 receives an operation requesting acquisition of real-time video in step S61, the image processing apparatus 10 performs the operation shown in FIG.
The image processing apparatus 10 acquires an image frame of a real-time video obtained from the camera 17 as a background image (step S81). Thereafter, the processing steps after step S23 are executed on the background image acquired in step S81, as in the second embodiment described above.

The image processing apparatus 10 displays the real-time video on which the content image is superimposed as a visual image on the display device 15 by repeatedly executing the operation shown in FIG. 8 for each image frame of the real-time video obtained from the camera 17. be able to. The image processing apparatus 10 does not have to use all the image frames constituting the real-time video as the background image. For example, the image processing apparatus 10 may perform thinning by removing some image frames from the image frames constituting the real-time video according to a predetermined method. The image processing apparatus 10 may use the image frame after thinning as a background image. Further, the image processing apparatus 10 can omit the processing steps from step S23 to step S64 when the marker arrangement does not change between adjacent image frames.

[Operation and Effect in Third Embodiment]
In the third embodiment, the background image on which the AR content is superimposed is switched from the image of the image file to the image frame of the real-time video according to the operation by the user. Then, a held content image associated with a content shooting direction close to the direction from the AR content position to the shooting position of the image frame is superimposed on the image frame of the real-time image as a visual image of the real object. Is done. Thereby, according to 3rd Embodiment, the real-time image | video with which the content image was superimposed as a visual recognition image can be displayed on the display apparatus 15. FIG.

[First Modification]
The first modification is a modification of the second embodiment of the present invention. The third embodiment of the present invention can be modified as in this modification. In the second embodiment described above, since the user can select a background image, the user may erroneously select an image file in which a background image without a marker is recorded. There are many image files stored in the computer. Among them, there are many image files other than an image file for storing an appropriate background image. Therefore, such an error can occur. If the marker is not reflected in the background image, the AR content cannot be appropriately superimposed on the background image. Therefore, the following processing unit of the image processing apparatus 10 according to the second embodiment may perform the following processing.

The display control unit 29 causes the display device 15 to display that no marker is detected when no marker is detected from the background image by the marker detection unit 23.
The operation accepting unit 21 accepts an operation of reselecting an image file other than the image file in which no marker is detected.
The background acquisition unit 22 newly acquires an image of the image file newly selected by the reselection operation as a background image.

Thereby, the user can recognize whether or not a background image that cannot superimpose the AR content is stored in the selected image file. Furthermore, the user can reselect the image file. Therefore, the user can perform an image file selection operation with peace of mind.

[Second Modification]
The second modification is a modification of the second embodiment of the present invention. The third embodiment of the present invention can be modified as in this modification. The time and place for generating the image file for storing the background image are arbitrary. In other words, the user can hold a plurality of image files in which photographs showing a plurality of different places are recorded in the background holding unit 31 in order to use the plurality of image files as a background image. Under such circumstances, it may be more convenient to provide multiple types of markers. On the other hand, the image processing apparatus 10 needs to know the actual information (shape information and size information) of the marker that appears in the background image.

FIG. 9 is a diagram conceptually illustrating a processing configuration example of the image processing apparatus 10A according to the second modification. The image processing apparatus 10A according to the second modification further includes a marker information holding unit 35 in addition to the configurations of the second and third embodiments. The marker information holding unit 35 is also realized in the same manner as other processing units. Instead of the image processing apparatus 10 </ b> A, another computer that is communicably connected to the image processing apparatus 10 </ b> A may have the marker information holding unit 35.

The marker information holding unit 35 holds marker information including shape information and size information for each of a plurality of marker types. In this case, the marker detection unit 23 acquires shape information and size information of the specified type of marker from the marker information holding unit 35. The marker detection unit 23 detects a marker from the background image acquired by the background acquisition unit 22 using the acquired shape information and size information. The method for specifying the marker type is not limited. For example, the marker detection unit 23 can test whether or not a marker can be sequentially detected for each type of marker held in the marker information holding unit 35. In addition, the marker detection unit 23 can cause the display device 15 to display a screen for allowing the user to select a marker type. The marker detection unit 23 can also specify the marker type in response to an operation of selecting the marker type by the user.

In this way, it is possible to allow the user to use a plurality of types of markers when taking a background image. The convenience of the AR platform can be improved.

[Third Modification]
The third modification is a modification of the third embodiment of the present invention. The following processing unit of the image processing apparatus 10 according to the third embodiment may perform the following processing.

The operation reception unit 21 receives a snapshot operation in a state where a real-time video on which a visual image is superimposed is displayed on the display device 15.
The background acquisition unit 22 fixedly holds the image frame of the real-time video obtained by the camera 17 as the background image in response to reception of the snapshot operation by the operation reception unit 21. At this time, the background acquisition unit 22 can also stop the camera 17 after holding the image frame fixedly.
The display control unit 29 causes the display device 15 to display a composite image in which a content image is superimposed as a visual image on a fixedly held background image, instead of displaying a real-time video on which the visual image is superimposed.

In this way, the AR image on which the content image is superimposed can be displayed on the display device 15 by using the image frame obtained when the snapshot operation by the user is accepted as a fixed background image. . According to this, the user can see the AR image in which the content image is superimposed on the image in which the real-time video is stopped at a desired timing. Further, according to this, it is possible to prevent the displayed AR image from becoming difficult to see due to the position of the superimposed content image being shaken by hand shake or the like, and the AR image can be displayed in an easy posture for the user. Can be seen.

[Supplement]
The image processing apparatus 10 according to the third embodiment selectively operates a function of acquiring an image of an image file as a background image and a function of acquiring an image frame of a real-time video as a background image. However, the image processing apparatus 10 may operate the function of acquiring the image frame of the real-time video as the background image without having the function of acquiring the image of the image file as the background image. In this case, the image processing apparatus 10 may not have the operation receiving unit 21.

The above-described embodiments will be described in more detail with the following implementation examples. The present invention is not limited by the following implementation examples.
[Example of implementation]

FIG. 10A, FIG. 10B, and FIG. 10C are diagrams respectively showing examples of background images stored in the image file. As shown in these figures, each background image includes a marker. In these examples, the same marker is included in each background image, but the marker included in each background image may not be the same as long as it is a marker known by the image processing apparatus 10. As shown in these examples, the background holding unit 31 holds a plurality of image files for storing different background images.

Hereinafter, further specific examples of the processing contents of steps S63 and S65 shown in FIGS. 6 and 8 and the image extraction unit 26 and the composition processing unit 28 will be described. In this case, the content holding unit 32 holds a content image. Each content image held by the content holding unit 32 is associated with a unit vector representing the direction from the real object to the camera 17 when photographing the real object as information representing the content photographing direction.

The image processing apparatus 10 (image extraction unit 26) calculates a unit vector indicating a direction from the position of the AR content in the three-dimensional coordinate space to the shooting position of the background image. The image processing apparatus 10 (image extraction unit 26) calculates the inner product of the unit vector calculated from the background image and each unit vector of each content image held in the content holding unit 32. The image processing apparatus 10 (image extraction unit 26) specifies the unit vector of the content image that has the maximum calculated inner product value. The image processing apparatus 10 (image extraction unit 26) extracts the held content image associated with the specified unit vector from the content holding unit 32.

When synthesizing the content image and the background image, the image processing apparatus 10 (the synthesis processing unit 28) first generates a transparent plate-shaped three-dimensional object defined in the three-dimensional coordinate space. Also good. This three-dimensional object is a virtual object. The image of the three-dimensional object is superimposed on the background image. The image processing apparatus 10 (the composition processing unit 28) may paste the content image extracted as described above as a texture on a plate-shaped three-dimensional object. As a result, the image processing apparatus 10 (the composition processing unit 28) superimposes the content image on the background image. The image processing apparatus 10 (the composition processing unit 28) determines the position of the three-dimensional object and the content image so that the image of the three-dimensional object to which the content image is pasted is substantially a visual image of the real object at the background shooting position. What is necessary is just to set the method to paste. The method of superimposing the content image on the background image is not limited to the above example.

11A, 11B, and 11C are diagrams illustrating examples of a composite image (AR image) in which a content image is superimposed on the background image illustrated in FIGS. 10A, 10B, and 10C.
As shown in these drawings, the superimposed content image is switched according to the shooting position and shooting direction of the background image. The image of the real object in the superimposed content image has the same form as the real object installed in the space where the background image is captured, which is viewed from the shooting position and shooting direction of the background image.

12A, 12B, and 12C are diagrams illustrating movement of AR content on an AR image. In FIG. 12A, the AR content is arranged on the marker. In FIG. 12B, compared with the state shown in FIG. 12A, the AR content is moved backward with respect to the marker. In FIG. 12C, compared with the state shown in FIG. 12A, the AR content is moved to the left with respect to the marker. The user can view the AR image as shown in FIGS. 12B and 12C by performing an operation of moving the content image in a state where the AR image shown in FIG. 12A is displayed on the display device 15. . In FIG. 12B, the content image is reduced in size compared to the content image shown in FIG. 12A according to the position of the AR content after movement. In FIG. 12C, the content image is switched from the content image shown in FIG. 12A according to the position after movement.

In the plurality of flowcharts used in the above description, a plurality of steps (processes) are described in order, but the execution order of the steps executed in each embodiment is not limited to the description order. In each embodiment, the order of the illustrated steps can be changed within a range that does not hinder the contents. Moreover, each above-mentioned embodiment and each modification can be combined in the range with which the content does not conflict.

Part or all of the above embodiments can be specified as in the following supplementary notes. However, each embodiment is not limited to the following description.

(Appendix 1)
An operation reception unit for receiving an image file selection operation;
A background acquisition unit for acquiring, as a background image, an image of an image file selected by a selection operation received by the operation reception unit, from a background holding unit for storing an image file storing a background image in which a marker is reflected;
A marker detection unit for detecting a marker from the background image acquired by the background acquisition unit;
A reference calculation unit that sets a three-dimensional coordinate space based on real information and image information regarding the detected marker, and calculates a background shooting position and a background shooting direction;
A position determining unit that determines a position in the three-dimensional coordinate space in which AR (Augmented Reality) content to be displayed is virtually arranged;
Display control for causing a display unit to display a composite image in which a visual image from the background photographing position and the background photographing direction of the AR content arranged at the determined position in the three-dimensional coordinate space is superimposed on the background image. And
An image processing apparatus comprising:
(Appendix 2)
From a content holding unit that holds a plurality of content images obtained by shooting a real object corresponding to the AR content from a plurality of directions in association with each content shooting direction, from the position determined by the position determination unit, the background An image extraction unit for extracting a content image held in association with the content shooting direction close to the direction to the shooting position;
A composition processing unit that superimposes the content image extracted by the image extraction unit on the background image as the visual image;
The image processing apparatus according to appendix 1, further comprising:
(Appendix 3)
The operation accepting unit further accepts an operation for moving the content image with respect to the composite image,
The position determination unit determines a position after movement of the AR content in the three-dimensional coordinate space indicated by the accepted movement operation;
The image extraction unit newly extracts a content image held in the content holding unit in association with the content shooting direction close to the direction from the moved position determined by the position determination unit to the background shooting position. And
The composition processing unit switches the content image to be superimposed on the background image to the newly extracted content image.
The image processing apparatus according to appendix 2.
(Appendix 4)
Enlarging or extracting the extracted content image based on the real size of the real object, the real information and the image information regarding the marker detected by the marker detection unit, and the position determined by the position determination unit Size adjustment part to reduce,
Further comprising
The composition processing unit superimposes the content image adjusted by the size adjusting unit on the background image as the visual image.
The image processing apparatus according to appendix 2 or 3.
(Appendix 5)
The content holding unit holds the plurality of content images associated with each content shooting direction for each of a plurality of real objects,
The operation accepting unit further accepts an operation of selecting one real object to be displayed from the plurality of real objects,
The image extraction unit extracts one content image from the plurality of content images held in the content holding unit in relation to a real object selected by the accepted operation;
The image processing apparatus according to any one of appendices 2 to 4.
(Appendix 6)
The display control unit causes the display unit to display that no marker is detected when the marker detection unit does not detect the marker from the background image,
The operation accepting unit accepts an operation of reselecting an image file other than the image file in which the marker is not detected,
The background acquisition unit newly acquires an image of an image file newly selected by the operation as the background image.
The image processing apparatus according to any one of appendices 1 to 5.
(Appendix 7)
The operation accepting unit accepts an operation requesting acquisition of a real-time video from the imaging unit, and the background acquisition unit is the real-time video obtained from the imaging unit in response to the operation accepted by the operation accepting unit. Sequentially obtained as the background image,
The display control unit causes the display unit to display a real-time video in which the content image is superimposed as the visual image.
The image processing apparatus according to any one of appendices 2 to 5.
(Appendix 8)
The operation accepting unit accepts a snapshot operation in a state where a real-time video on which the visual image is superimposed is displayed on the display unit,
The background acquisition unit fixedly holds the image frame as the background image in response to reception of the snapshot operation by the operation reception unit,
The display control unit displays the composite image in which the content image is superimposed as the visual image on the fixedly held background image on the display unit, instead of displaying the real-time video on which the visual image is superimposed. Let
The image processing apparatus according to appendix 7.
(Appendix 9)
The marker detection unit acquires shape information and size information of one type of marker from a marker information holding unit that holds information of a plurality of types of markers each including shape information and size information, and the acquired shape information and Detecting the marker using size information;
The image processing apparatus according to any one of appendices 1 to 8.

(Appendix 10)
In an image processing method executed by at least one computer,
Accept image file selection operations,
An image of the image file selected by the accepted selection operation is acquired as a background image from a background holding unit that holds an image file that stores a background image showing a marker,
Detecting a marker from the acquired background image;
Based on real information and image information regarding the detected marker, a three-dimensional coordinate space is set,
Based on the actual information and image information about the detected marker, calculate a background shooting position and a background shooting direction,
Determining a position in the three-dimensional coordinate space where the AR (Augmented Reality) content to be displayed is virtually arranged;
Displaying a composite image in which a visual image from the background photographing position and the background photographing direction of the AR content arranged at the determined position in the three-dimensional coordinate space is superimposed on the background image on a display unit;
An image processing method.
(Appendix 11)
The determined position in the three-dimensional coordinate space from a content holding unit that holds a plurality of content images obtained by shooting a real object corresponding to the AR content from a plurality of directions in association with each content shooting direction. A content image held in association with the content shooting direction that is close to the direction from the background shooting position to
Superimposing the extracted content image on the background image as the visual image;
The image processing method according to appendix 10, further including:
(Appendix 12)
Accepting an operation of moving the content image with respect to the composite image;
Determining a position after movement of the AR content in the three-dimensional coordinate space indicated by the accepted movement operation;
Newly extracting a content image held in the content holding unit in association with the content shooting direction close to the direction from the determined moved position to the background shooting position,
Switching the content image to be superimposed on the background image to the newly extracted content image;
The image processing method according to claim 11, further comprising:
(Appendix 13)
The extracted content image is enlarged or reduced based on the real size of the real object, the real information and the image information regarding the detected marker, and the determined position in the three-dimensional coordinate space. To
The image processing method according to appendix 11 or 12, further including:
(Appendix 14)
Receiving an operation of selecting one real object to be displayed from among a plurality of real objects;
Further including
The content holding unit holds the plurality of content images associated with the content shooting directions for each of the plurality of real objects,
In the extraction of the content image, one content image is extracted from the plurality of content images held in the content holding unit in relation to the real object selected by the accepted operation.
The image processing method according to any one of appendices 11 to 13.
(Appendix 15)
If the marker is not detected from the background image, display that the marker is not detected on the display unit,
Accepting an operation to reselect an image file other than the image file in which the marker is not detected,
An image of the image file newly selected by the operation is newly acquired as the background image.
The image processing method according to any one of appendices 10 to 14, further including:
(Appendix 16)
Accepting an operation requesting acquisition of real-time video from the imaging unit,
In response to the accepted operation, the image frame of the real-time video obtained from the imaging unit is sequentially acquired as the background image,
Displaying the real-time video on which the content image is superimposed as the visual image on the display unit;
The image processing method according to any one of appendices 11 to 14, further including:
(Appendix 17)
Accepting a snapshot operation in a state where the real-time video on which the visual image is superimposed is displayed on the display unit,
In response to accepting the snapshot operation, the image frame is fixedly held as the background image,
Instead of displaying the real-time video on which the visual image is superimposed, the composite image in which the content image is superimposed as the visual image on the fixedly held background image is displayed on the display unit.
The image processing method according to appendix 16, further including:
(Appendix 18)
Obtaining shape information and size information of one type of marker from a marker information holding unit that holds information of a plurality of types of markers each including shape information and size information;
Further including
The detection of the marker detects the marker using the acquired shape information and size information.
18. The image processing method according to any one of appendices 10 to 17.

(Appendix 19)
A background acquisition unit that sequentially acquires an image frame of a real-time video obtained from the imaging unit as a background image;
A marker detection unit for detecting a marker from the background image acquired by the background acquisition unit;
A reference calculation unit that sets a three-dimensional coordinate space based on real information and image information regarding the detected marker, and calculates a background shooting position and a background shooting direction;
A position determining unit that determines a position in the three-dimensional coordinate space in which AR (Augmented Reality) content to be displayed is virtually arranged;
From a content holding unit that holds a plurality of content images obtained by shooting a real object corresponding to the AR content from a plurality of directions in association with each content shooting direction, from the position determined by the position determination unit, the background An image extraction unit for extracting a content image held in association with the content shooting direction close to the direction to the shooting position;
A composition processing unit that superimposes the content image extracted by the image extraction unit on the background image as the visual image;
A display control unit that causes a display unit to display a real-time video in which the content image is superimposed as the visual image;
An image processing apparatus comprising:
(Appendix 20)
An operation accepting unit that accepts a snapshot operation in a state where the real-time video on which the visual image is superimposed is displayed on the display unit;
Further comprising
The background acquisition unit fixedly holds the image frame as the background image in response to reception of the snapshot operation by the operation reception unit,
The display control unit displays the composite image in which the content image is superimposed as the visual image on the fixedly held background image on the display unit, instead of displaying the real-time video on which the visual image is superimposed. Let
The image processing apparatus according to appendix 19.

(Appendix 21)
Obtain sequentially the image frame of the real-time video obtained from the imaging unit as a background image,
Detecting a marker from the acquired background image;
Based on real information and image information regarding the detected marker, a three-dimensional coordinate space is set,
Based on the actual information and image information regarding the detected marker, calculate a background shooting position and a background shooting direction of the background image,
Determining a position in the three-dimensional coordinate space where the AR (Augmented Reality) content to be displayed is virtually arranged;
The determined position in the three-dimensional coordinate space from a content holding unit that holds a plurality of content images obtained by shooting a real object corresponding to the AR content from a plurality of directions in association with each content shooting direction. A content image held in association with the content shooting direction that is close to the direction from the background shooting position to
Superimposing the extracted content image as the visual image on the background image;
Displaying the real-time video on which the content image is superimposed as the visual image on the display unit;
An image processing method.
(Appendix 22)
Accepting a snapshot operation in a state where the real-time video on which the visual image is superimposed is displayed on the display unit,
In response to accepting the snapshot operation, the image frame is fixedly held as the background image,
Instead of displaying the real-time video on which the visual image is superimposed, the composite image in which the content image is superimposed as the visual image on the fixedly held background image is displayed on the display unit.
The image processing method according to claim 21, further comprising:

(Appendix 23)
A program that causes at least one computer to execute the image processing method according to any one of Supplementary Notes 10 to 18, 21, and 22.
While the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.
This application claims the priority on the basis of Japanese application Japanese Patent Application No. 2013-233805 for which it applied on November 12, 2013, and takes in those the indications of all here.

10, 10A, 100 Image processing apparatus 11 CPU
DESCRIPTION OF SYMBOLS 12 Memory 15 Display apparatus 16 Input apparatus 17 Camera 18 Portable recording medium 21, 101 Operation reception part 22, 102 Background acquisition part 23, 103 Marker detection part 24, 104 Reference | standard calculation part 25, 105 Position determination part 26 Image extraction part 27 Size adjustment unit 28 Composition processing unit 29, 106 Display control unit 31, 107 Background holding unit 32 Content holding unit 35 Marker information holding unit 108 Display unit

Claims (11)

1. Operation accepting means for accepting an image file selection operation;
   A background acquisition unit that acquires an image of an image file selected by a selection operation received by the operation reception unit as a background image from a background holding unit that stores an image file that stores a background image showing a marker;
   Marker detection means for detecting a marker from the background image acquired by the background acquisition means;
   Reference calculation means for setting a three-dimensional coordinate space and calculating a background shooting position and a background shooting direction based on the actual information and image information regarding the detected marker;
   Position determining means for determining a position in the three-dimensional coordinate space in which AR (Augmented Reality) content to be displayed is virtually arranged;
   Display control for causing a display unit to display a composite image in which a visual image from the background photographing position and the background photographing direction of the AR content arranged at the determined position in the three-dimensional coordinate space is superimposed on the background image. Means,
   An image processing apparatus comprising:
2. From a content holding unit that holds a plurality of content images obtained by shooting a real object corresponding to the AR content from a plurality of directions in association with each content shooting direction, from the position determined by the position determination unit, the background Image extracting means for extracting a content image held in association with the content shooting direction close to the direction to the shooting position;
   Composition processing means for superimposing the content image extracted by the image extraction means on the background image as the visual image;
   The image processing apparatus according to claim 1, further comprising:
3. The operation accepting unit further accepts a moving operation of the content image with respect to the composite image,
   The position determining means determines a position after movement of the AR content on the three-dimensional coordinate space indicated by the accepted movement operation;
   The image extraction unit newly extracts a content image held in the content holding unit in association with the content shooting direction close to the direction from the moved position determined by the position determination unit to the background shooting position. And
   The composition processing means switches the content image to be superimposed on the background image to the newly extracted content image.
   The image processing apparatus according to claim 2.
4. Enlarging or extracting the extracted content image based on the real size of the real object, the real information and the image information relating to the marker detected by the marker detection means, and the position determined by the position determination means Size adjustment means to reduce,
   Further comprising
   The composition processing unit superimposes the content image adjusted by the size adjusting unit on the background image as the visual image;
   The image processing apparatus according to claim 2.
5. The content holding unit holds the plurality of content images associated with each content shooting direction for each of a plurality of real objects,
   The operation accepting unit further accepts an operation of selecting one real object to be displayed from the plurality of real objects,
   The image extracting means extracts one content image from the plurality of content images held in the content holding means in relation to a real object selected by the accepted operation;
   The image processing apparatus according to claim 2.
6. The display control means causes the display means to display that no marker is detected when the marker is not detected from the background image by the marker detection means,
   The operation accepting unit accepts an operation of reselecting an image file other than the image file in which the marker is not detected,
   The background acquisition unit newly acquires an image of an image file newly selected by the operation as the background image.
   The image processing apparatus according to claim 1.
7. The operation accepting means accepts an operation requesting acquisition of real-time video from the imaging means,
   The background acquisition unit sequentially acquires an image frame of the real-time video obtained from the imaging unit as the background image in response to the operation received by the operation reception unit,
   The display control means causes the display means to display a real-time video in which the content image is superimposed as the visual image.
   The image processing apparatus according to claim 2.
8. The operation accepting unit accepts a snapshot operation in a state where a real-time video on which the visual image is superimposed is displayed on the display unit,
   The background acquisition unit holds the image frame fixedly as the background image in response to reception of the snapshot operation by the operation reception unit,
   The display control unit displays the composite image in which the content image is superimposed as the visual image on the fixedly held background image on the display unit, instead of displaying the real-time video on which the visual image is superimposed. Let
   The image processing apparatus according to claim 7.
9. The marker detection means acquires shape information and size information of one type of marker from marker information holding means for holding information of a plurality of types of markers each including shape information and size information, and the acquired shape information and Detecting the marker using size information;
   The image processing apparatus according to claim 1.
10. Accept image file selection operations,
    From a background holding unit that holds an image file that stores a background image showing a marker, an image of the image file selected by the accepted selection operation is acquired as a background image,
    Detecting a marker from the acquired background image;
    Based on real information and image information regarding the detected marker, a three-dimensional coordinate space is set,
    Based on the actual information and image information about the detected marker, calculate a background shooting position and a background shooting direction,
    Determining a position in the three-dimensional coordinate space where the AR (Augmented Reality) content to be displayed is virtually arranged;
    Displaying a composite image in which a visual image from the background photographing position and the background photographing direction of the AR content arranged at the determined position in the three-dimensional coordinate space is superimposed on the background image on a display unit;
    Image processing method.
11. Computer
    Operation accepting means for accepting an image file selection operation;
    A background acquisition unit that acquires, as a background image, an image of an image file selected by a selection operation received by the operation reception unit from a background holding unit that stores an image file that stores a background image in which a marker is reflected;
    Marker detection means for detecting a marker from the background image acquired by the background acquisition means;
    Reference calculation means for setting a three-dimensional coordinate space and calculating a background shooting position and a background shooting direction based on the actual information and image information regarding the detected marker;
    Position determining means for determining a position in the three-dimensional coordinate space in which AR (Augmented Reality) content to be displayed is virtually arranged;
    Display control for causing a display unit to display a composite image in which a visual image from the background photographing position and the background photographing direction of the AR content arranged at the determined position in the three-dimensional coordinate space is superimposed on the background image. Means,
    Recording medium for storing a program to be operated.

Images