JP2016131342A - Device, method, and program for model railroad appreciation, dedicated display monitor, and scene image data for synthesis - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device for model railroad appreciation which allows a user of a model railroad where a model vehicle can travel and move to view the model vehicle as if the model vehicle traveled and moved continuously from the installation space of the model railroad to a video space displayed on a display.SOLUTION: The device for model railroad appreciation comprises a camera, a display, device position etc. detection means, model detection means, scene image display means, vehicle model extraction means, and image synthesis means for synthesizing a vehicle model area to a scene image. The device, method and program for model railroad appreciation, a dedicated display monitor, and scene image data for synthesis are characterized in that a model vehicle viewed by a user looks as if the model vehicle traveled continuously from on the model railroad onto a railroad actually existing in the scene image.SELECTED DRAWING: Figure 3

Description

The present invention is a railway model in which the vehicle model can travel and can be visually recognized by the user as if the vehicle model continuously travels and moves from the space where the railway model is installed to the image space displayed on the display. The present invention relates to a railway model ornamental device.

The railway model is a small and modeled version of an actual railway, and the user can view a part or the whole of the model from a free distance and angle. This is one of the real pleasures of the railway model.

On the other hand, the railroad model is pursuing the reality of not only the vehicle model but also the layout including the tracks and surrounding scenes (sometimes referred to as scenery section or diorama). The ultimate form of reality is to replace the layout with the actual scene image and synthesize the image of the vehicle model. However, simply synthesizing the image, the user only sees a screen independent of the railway model, and in view of the above-mentioned excitement, it is no longer a railway model.

Therefore, in the present invention, while the user can move freely, the railway model can be viewed from a free distance and angle, and at any point, by combining the vehicle model with the scene image actually taken outdoors, Reality can now be pursued. At this time, the same vehicle model is moved from the model world to the real world so that the user can continuously see the vehicle model that the user is looking directly at and the vehicle model that is synthesized with the actual scene image.恰 also made it possible to display as if entering.

In contrast to the conventional image composition technology, the user focuses on viewing the inside of the display. In the present invention, however, the user focuses on viewing the continuity inside and outside the display. This is one of the major features. Except for the case of the present invention in which the vehicle model traveling on the track and the layout are simultaneously viewed, it can be said that the motive for enabling the continuity between the inside and outside of the display to be appreciated was scarce. For example, Patent Document 1 describes an image composition device that composes two input images having different sizes based on a plurality of composition modes to produce an output image of a predetermined size.

In recent years, research on free viewpoint image generation has been actively conducted with the improvement of computer processing capability. For example, Patent Document 2 describes a free viewpoint video generation system that generates a video viewed from an arbitrary viewpoint using a two-dimensional video and a background mask that extracts a background region of the two-dimensional image. Further, Patent Document 3 uses image-based rendering to facilitate the interpolation of ray information even when the subject is distributed over a wide range in real space and the parallax due to the difference in depth is large. Thus, a method of generating a free viewpoint image that can generate an image of an arbitrary viewpoint in real space close to reality is described. Further, Non-Patent Document 1 describes a method of browsing a free viewpoint video on a tablet PC.

JP 2001-86406 A JP 2007-52644 A JP 2008-15756 A Junya Kashiwakuma, Itaru Kitahara, Yoshinari Kameda, Yuichi Ohta, A Virtual Camera Controlling Method Using Multi-Touch Gestures for Capturing Free-viewpoint Video; 11th European Interactive TV Conference, pp.67-74 (2013.6)

However, the prior art has the following problems.

The conventional image composition technology was mainly intended to display the result of image composition on the display. For example, an object that the user was viewing outside the display is displayed on a display such as a portable terminal or a head-mounted display. The motivation to display the scenes as if they were entering the scene image was poor. At this time, the motive for expressing the continuity between the object existing outside the display and the object displayed in the display was poor. For this reason, there is no conventional image composition technique that emphasizes continuity between a scene that the user visually recognizes on the display and a peripheral scene that the user visually recognizes without using the display. For example, the image composition apparatus described in the cited document 1 also synthesizes two input images having different sizes, but does not consider continuity with the outside of the display.

On the other hand, when enjoying a railway model, not only the reality of the vehicle model but also the reality of the layout, which is a scene model, is often important. In the world of layouts composed of miniature models, users can watch the vehicle model sprinting while freely changing the viewpoint. However, it takes time and money to make a layout, and it is difficult to disassemble and reconfigure what has been once configured, and it has not been easy to enjoy.

Therefore, the present invention solves the problems of the conventional image composition technology and the railway model, and uses a scene image (a still image or a moving image) that is actually taken, thereby providing a visual effect similar to or higher than that of the layout. A railroad model ornamental device that can be easily obtained and that allows a moving vehicle model to enter the world of a scene image actually taken from the world of the layout composed of the model. It is an object of the present invention to provide a method, a program, a dedicated display monitor, and scene image data for composition.

In order to solve the above-mentioned problems, the railroad model viewing device according to claim 1 is a device for viewing a railroad model on which the vehicle model travels, and detects a camera, a display, a vehicle model, or a track model. A model detecting means for detecting the position of the apparatus, a position detecting device for detecting the position or orientation of the apparatus, a scene image including a real line photographed in advance, and the real line and the line model continuously. A scene image display means for aligning the image so that it can be seen and displayed on the display, a vehicle model extraction means for extracting a vehicle model area from an image obtained by photographing the railway model with the camera, and the vehicle model in the scene image Image synthesizing means for synthesizing the area, and the vehicle model visually recognized by the user seems to run continuously on the actual track from the track model. Characterized in that it was.

Next, the railway model ornamental apparatus according to claim 2 is the railway model ornamental apparatus according to claim 1, wherein the scene image display means has a size of the vehicle model or the track model visually recognized by the user. The scene image can be zoomed and displayed.

Next, the railway model ornamental device according to claim 3 is the railway model ornamental device according to claim 1, wherein the viewpoint for specifying the positional relationship in the model space between the vehicle model or the railroad model and the viewpoint of the user. A positional relationship specifying unit; and a free viewpoint image generating unit that generates a free viewpoint image from image data obtained by photographing a scene including the existing track, and the actual position exists in the positional relationship specified by the viewpoint positional relationship specifying unit. The free viewpoint image generation unit automatically generates a scene image when the track is viewed, and the scene image display unit can display the scene image on the display.

Next, the railway model ornamental apparatus according to claim 4 is the railway model ornamental apparatus according to claim 1, wherein the scene image display means captures the railway model with the camera, or the user's facial expression. Alternatively, an optimum scene image is automatically selected and displayed from a plurality of scene images including a real track taken in advance based on biological information.

Next, the railway model viewing method according to claim 5 is a railway model viewing method for viewing a railway model on which the vehicle model travels using a camera and a display. A position detection step for detecting the position, posture, or orientation of the apparatus, and a scene image including a real line photographed in advance are aligned so that the real line and the line model can be seen continuously. A scene image display step for displaying on the display; a vehicle model extraction step for extracting a vehicle model area from an image obtained by photographing the vehicle model with the camera; and an image synthesis for synthesizing the vehicle model area with the scene image. The vehicle model visually recognized by the user is continuously run from the track model to the actual track. Characterized in that the so that.

Next, the railway model viewing program according to claim 6 is a device for viewing a railway model on which a vehicle model including a camera and a display travels. It is a program for executing image composition processing so that it appears to run continuously on an existing track, model detecting means for detecting a vehicle model or a track model, and the position or orientation or orientation of the device. Detecting device position detection means, scene image including pre-existing real line is aligned so that the real line and the line model can be seen continuously, and displayed on the display Display means, vehicle model extraction means for extracting a vehicle model area from an image obtained by photographing the vehicle model with the camera, and imaging the vehicle model area in the scene image Is a program for causing the model railroad ornamental device as image synthesizing means, for forming.

Next, a dedicated display monitor according to a seventh aspect is a dedicated display monitor provided with a display used in the railway model viewing apparatus according to the first aspect.

Next, the scene image data for composition according to claim 8 is scene image data for composition obtained by photographing the scene including the existing track used in the railway model ornamental apparatus according to claim 1.

By replacing the layout with an actual scene image instead of a model, the reality of the railway model increases, and for example, it becomes possible to enjoy the feeling of running a vehicle model on an existing track. In addition, it is possible to appreciate the appearance of the vehicle model entering the actual scene world from the world composed of models, and to enjoy the fun of the model and the beauty of the scene at the same time. Furthermore, when the traveling vehicle model is a steam locomotive, by using augmented reality technology, for example, the state that the steam locomotive model running on the track in the scene image is emitting smoke or steam can be viewed. You can enjoy it.

FIG. 1 is a block diagram schematically showing an example of the most basic hardware configuration of a railway model ornamental device. FIG. 2 is an explanatory diagram showing a situation where the vehicle model 11 (steam locomotive) is traveling on the track model 12. FIG. 3 is an explanatory view showing an example of the apparatus for viewing a railway model of the present invention using a smartphone or a tablet-type terminal. FIG. 4 is an explanatory view showing an example of a railway model ornamental device of the present invention using a head mounted display. FIG. 5 is an explanatory view showing an example of the railway model viewing apparatus of the present invention using a smartphone or a tablet-type terminal when the vehicle model 11 is viewed obliquely from above and front. FIG. 6 is an explanatory diagram showing the relationship between the model space and the scene space. FIG. 7 is a flowchart showing an example of a basic processing flow of a program for causing a model railway device to function.

Hereinafter, an example of a railway model viewing apparatus according to the best mode of the present invention will be described with reference to the drawings. In addition, as much as possible, the same code | symbol is attached | subjected to the same part and the overlapping description is abbreviate | omitted. Further, the description overlapping with the description in the means for solving the problem is omitted as much as possible.

FIG. 1 is a block diagram schematically showing an example of the most basic hardware configuration of a railway model ornamental device.
The CPU 6 loads and executes a control program stored in a ROM (Read-Only Memory) 3 using a RAM (Random-Access Memory) 4 as a work area.
A user interface (User Interface) 1 is composed of a touch panel or the like, and is used for initial setting, activation / termination, etc.
The camera (Camera) 2 is a digital system, and a vehicle model area is extracted from an image obtained by photographing a railway model with the camera (Camera 2), and an image obtained by synthesizing the extracted vehicle model area and a scene image is displayed (Display) 5. To display. The model detection means may detect the vehicle model or the track model by recognizing the image from the image of the railroad model taken by the camera (Camera) 2 or may include a marker on the vehicle model or the track model. Alternatively, it may be detected by recognizing the marker.
For this purpose, it is necessary for the device position detection means to detect the position / posture / azimuth of the railway model ornamental device. Therefore, the posture is detected by an acceleration sensor 8 and a gyro sensor 9, and the orientation is detected by a magnetic field sensor 10. Regarding the position, since the positional relationship in the model space between the vehicle model or track model and the user's viewpoint is important, the position is estimated from the size and angle at which the vehicle model or track model can be seen.
A data storage device (Data storage) 7 stores scene image data for synthesis, data related to a standard pattern used for detection of a vehicle model or a track model.

FIG. 2 is an explanatory diagram showing a situation where the vehicle model 11 (steam locomotive) is traveling on the track model 12. As a layout, simple tree models 13 are arranged.

There are three main methods for the scene image display means to display a scene image on the display as if the vehicle model has entered.
The first method is a method of displaying a scene image including an actual track by detecting a marker provided on the track of the railway model. This is the same method as that used in the marker-type augmented reality technology.
The second method is a method of displaying a scene image including an actual track by detecting the size, position, inclination, or the like of the railroad model railroad.
The third method detects the position of the user's viewpoint with respect to the track in the space constituted by the railroad model, and displays a scene image including the actual track when the actual track is viewed from the same positional relationship. It is a method to do.
First, the position of the user's viewpoint with respect to the track in the space constituted by the railroad model is detected from the position, size, or inclination of the railroad model track that is visible from the user's viewpoint. Next, a scene image when an actual track is viewed from the detected position of the user's viewpoint is displayed on the display. The easiest way to do this is to take a scene image of a real railway track from multiple viewpoints, and use the scene image when viewed from the viewpoint closest to the user's viewpoint position. It is a method to do. Further, the viewpoint can be adjusted with higher accuracy by the free viewpoint image generation means. At this time, in order to display the track in the scene image so as to be continuously visible with the track of the model train, it is desirable to record information on the position, direction, curvature, etc. of the track in the scene image data for synthesis. .

The scene image display means automatically selects an optimal scene image from an image obtained by photographing a railroad model with a camera, or a plurality of scene images including a real track that has been photographed in advance based on a user's facial expression or biological information. Describes how to select and display.
First, as a method based on an image obtained by photographing a railway model, there is a simple method in which a marker is placed on a track model and an optimal scene image corresponding to a point on the track is displayed. In addition, when it is desired to avoid the marker being conspicuous, a symbolic layout model may be installed to perform image recognition. Thus, for example, a forest scene can be displayed by simply placing a simple tree model.
Next, as a method based on the user's facial expression or biological information, an optimal scene image corresponding to the user's facial expression or biological information at that time is displayed. For example, a user's facial expression is read by image recognition, a bright scene is displayed when a bright expression such as a smile is displayed, a nostalgic scene is displayed when a sad expression is displayed, and a heart rate or When the blood pressure is high, a calm rural landscape may be displayed while playing BGM.
Bars, restaurants, museums, etc. with railroad models and diorama are still popular among the general public as well as railroad enthusiasts. Therefore, scene image data can also be selected according to each TPO (time, place, case). For example, in a bar or restaurant, a scene image data collection may be selected according to the atmosphere of the store.

Next, a method for synthesizing the vehicle model region so that the vehicle model enters the displayed scene image will be described. First, the vehicle model extracting means captures a space formed by the railway model viewed by the user with a camera from the user's viewpoint, and extracts a moving vehicle model region from the captured image. The moving object extraction method in the image is not an essential part of the present invention, and is well known and need not be limited to a specific method. Various methods such as a method using an optical flow and a method using feature point extraction can be used. When there is a moving object other than the vehicle model area, it may occur that the moving object is extracted by mistake. A method for solving this problem by template matching or threshold processing is known, but in the present invention, it is solved by limiting the range in which the vehicle model travels from the positional relationship between the vehicle model and the track model. You can also. Finally, the image synthesis means synthesizes the extracted vehicle model area with the scene image.

FIG. 3 is an explanatory view showing an example of the apparatus for viewing a railway model of the present invention using a smartphone or a tablet-type terminal. A scene image including an actual overpass is displayed on the display 15 of the smartphone or tablet terminal 14, and the area of the vehicle model 11 is combined with the scene image and displayed. The scene image is zoomed and displayed based on the size of the vehicle model 11, and the scene image including the existing track photographed in advance is aligned so that the existing track and the track model can be seen continuously. And displayed on the display 15. Moreover, the smoke 5 emitted from the chimney of the vehicle model 11 is displayed by using augmented reality technology.

FIG. 4 is an explanatory view showing an example of a railway model ornamental device of the present invention using a head mounted display. A scene image including an actual overpass is displayed on the head-mounted display 17, and an area of the vehicle model is combined with the scene image and displayed. At this time, the smoke emitted from the chimney of the vehicle model is displayed by using augmented reality technology. In addition, as a video projection method of the head mounted display, various methods and forms such as a method using a liquid crystal display, a method using a projector, a method using a prism, a method using a contact lens, and the like can be considered. Since it is not a part, only the display image is shown.

FIG. 5 is an explanatory view showing an example of the railway model viewing apparatus of the present invention using a smartphone or a tablet-type terminal when the vehicle model 11 is viewed obliquely from above and front.
In the present invention, while the user can move freely, the railway model can be viewed from a free distance and angle, and reality can be pursued. At this time, the same vehicle model is moved from the model world to the real world so that the user can continuously see the vehicle model that the user is looking directly at and the vehicle model that is synthesized with the actual scene image. It is possible to display the kite as it enters. In FIG. 5, the free viewpoint image generation means automatically generates a scene image when the actual track is viewed from the viewpoint, and the scene image is zoomed based on the size of the vehicle model 11 and the width of the track model 12. It can be seen that the continuity between the inside and outside of the display is well maintained.

FIG. 6 is an explanatory diagram showing the relationship between the model space and the scene space.
The viewpoint positional relationship specifying means specifies the positional relationship in the model space between the vehicle model or track model and the user's viewpoint 18. For example, in the model space shown in the upper part of FIG. 6, when the vehicle model 11 is seen at the position of the vector (Δx, Δy, Δz) from the user's viewpoint 18, the vehicle model 11 is in the same user's viewpoint in the scene space shown in the lower part. 18 is visible at a position of D ÷ d × (Δx, Δy, Δz).
In addition, when there is a difference in the size when displaying an image photographed by the camera on the display depending on a model such as a smartphone, a tablet-type terminal, or a head-mounted display, the display size needs to be corrected. It is also necessary to keep the distance between the user's viewpoint and the display as constant as possible.
When using a dedicated display monitor, the display size can be changed by specifying the distance to the user standing or sitting in front of the monitor and automatically detecting the user's position. To do.

FIG. 7 is a flowchart showing an example of a basic processing flow of a program for causing a model railway device to function.
First, the model detection means detects a vehicle model or a track model (S1). Next, the device position detection means detects the position, posture or orientation of the railway model viewing device (S2). Next, the scene image display means aligns the scene image including the existing track photographed in advance so that the existing track and the track model can be continuously viewed, and displays them on the display (S3). ). Next, the vehicle model extraction means extracts a vehicle model region from an image obtained by photographing the vehicle model with a camera (S4). Next, the image synthesizing unit synthesizes the vehicle model area with the scene image (S5). Finally, an end determination is made (S6). If the determination result is NO, the process is repeated from S2, and if YES, the process ends.

Bars, restaurants, museums, etc. with railroad models and diorama are still popular among the general public as well as railroad lovers. By holding a smartphone or tablet-type terminal over the model railroad, you will be able to enjoy the scene as a more realistic layout and will attract more customers. In addition, for example, by making it possible to download application software for causing the railway model ornamental device of the present invention to function free of charge, this not only attracts customers but also promotes the sales of the railway model itself.

1 User Interface
2 Camera
3 ROM (Read-Only Memory)
4 RAM (Random-Access Memory)
5 Display
6 CPU
7 Data storage
8 Accelerometer
9 Gyro Sensor
10 Magnetic field sensor
11 Vehicle Model 12 Track Model 13 Tree Model 14 Smartphone or Tablet Terminal 15 Display 16 Smoke 17 Head Mount Display 18 User's Viewpoint

For the display, a dedicated display monitor provided with a display used in the railway model viewing apparatus according to claim 1 may be used .

Claims (8)

An apparatus for appreciating a railway model on which a vehicle model travels,
A camera,
Display,
A model detecting means for detecting a vehicle model or a track model;
A device position detection means for detecting the position, posture or orientation of the device;
A scene image display means for displaying a scene image including an existing track photographed in advance so that the existing track and the track model can be continuously viewed and displayed on the display,
Vehicle model extraction means for extracting a vehicle model area from an image obtained by photographing the railway model with the camera;
Image synthesizing means for synthesizing the vehicle model region with the scene image;
With
The railway model viewing apparatus, wherein the vehicle model visually recognized by a user is allowed to continuously run from the track model onto the existing track. The railroad model appreciation according to claim 1, wherein the scene image display means can zoom and display the scene image based on the size of the vehicle model or the track model visually recognized by the user. Equipment. A viewpoint positional relationship specifying means for specifying a positional relationship in the model space between the vehicle model or the track model and the viewpoint of the user;
Free viewpoint image generation means for generating a free viewpoint image from image data obtained by photographing a scene including the actual track,
With
The free viewpoint image generation unit automatically generates a scene image when the actual track is viewed in the positional relationship specified by the viewpoint positional relationship specifying unit, and the scene image display unit can display the scene image on the display. The railway model ornamental device according to claim 1, wherein The scene image display means is an image obtained by photographing the railway model with the camera, or an optimal scene image from a plurality of scene images including a real track previously photographed based on a user's facial expression or biological information. The railway model ornamental device according to claim 1, wherein is automatically selected and displayed. In the railway model appreciation method of appreciating the railway model on which the vehicle model runs using a camera and display,
A model detection step for detecting a vehicle model or a track model;
A device position detection means for detecting the position, posture or orientation of the device;
A scene image display step for displaying a scene image including an existing track photographed in advance, aligning the actual track and the track model so that they can be seen continuously, and displaying them on the display,
An image synthesis step of synthesizing the vehicle model region with the scene image;
A vehicle model extraction step for extracting a vehicle model region from an image obtained by photographing the vehicle model with the camera;
Run
The railroad model viewing method characterized in that the vehicle model visually recognized by a user is allowed to continuously run on the actual track from the track model. In a device for viewing a railway model on which a vehicle model equipped with a camera and a display travels,
The vehicle model visually recognized by the user is a program for executing an image composition process so that the vehicle model appears to continuously run on the actual track from the track model,
Model detection means for detecting a vehicle model or a track model,
A device position detection means for detecting the position, posture or orientation of the device;
A scene image display means for displaying a scene image including an existing track photographed in advance so that the existing track and the track model can be continuously viewed and displayed on the display,
Vehicle model extraction means for extracting a vehicle model region from an image obtained by photographing the vehicle model with the camera;
Image synthesizing means for synthesizing the vehicle model region with the scene image;
As a program for causing the railway model ornamental device to function. A dedicated display monitor provided with a display used in the railway model ornamental device according to claim 1. The scene image data for composition which image | photographed the scene containing the said actual track | line used for the railway model ornamental apparatus of Claim 1.