WO2021075611A1 - Method and device for projecting content - Google Patents

Abstract

A method and device for projecting content according to an embodiment: tracks a first position of a train; downloads content for slots in at least one tunnel corresponding to the operation schedule of the train; and projects content allocated to one of the slots on the basis of correction information corresponding to the slot according to the result of determining whether the train has entered the slot on the basis of a second position tracked according to the operation of the train. Here, the operation schedule is determined by matching the first position and the time tracking the first position with one of the operation schedules of a plurality of trains.

Description

Method and apparatus for projecting content

The following embodiments relate to a method and apparatus for projecting content.

Currently, media systems of urban railways and/or subway trains are mostly composed of LCD (Liquid Crystal Display) screens or LED (Light-Emitting Diode) panels. Since this type of media system cannot distract the attention of passengers concentrated on mobile devices such as smart phones, the effect of notifying major contents and/or delivering advertisements for safety and the like is very low.

According to an embodiment, content may be reproduced by correcting content in real time according to a changing tunnel type.

According to an embodiment, by measuring a tunnel to determine a projectable area, content can be effectively reproduced in places and sections within a tunnel with many restrictions.

According to an embodiment, it is possible to organize and manage content more conveniently through a web cloud.

According to an embodiment, a method of projecting content includes tracking a first location of a train; For at least one slot in the tunnel corresponding to the operation schedule of the train-the operation schedule is determined by matching the first position and the time of tracking the first position with any one of operation schedules of a plurality of trains. Downloading content; Tracking a second position according to the operation of the train; Determining whether the train has entered any one of the slots based on the second position; And projecting the content allocated to the one slot based on the correction information corresponding to the one slot according to the determination result.

The step of tracking the second location may include determining whether the second location corresponds to the at least one tunnel section.

The train includes a projector installed outside the train, a sleeper sensor that counts a first number of sleepers installed on a track on which the train runs, a lighting sensor that counts a second number of lights installed on the track, and It may include at least one of a gyro sensor that detects motion, an acceleration sensor that detects acceleration of the train, an RGB camera that photographs an image projected by the projector, and a depth camera that measures the depth of the projection surface of the image. have.

The determining whether the second position corresponds to the at least one tunnel section is based on at least one of a first number of sleepers installed on a track on which the train runs and a second number of lights installed on the track. Thus, it may include determining whether the second location corresponds to the at least one tunnel section.

The step of tracking the second position according to the operation of the train may include determining whether the train is operated; And tracking the second position according to the operation of the train according to the determination of whether the train is running.

The step of determining whether one of the slots has entered may include a depth value at the second position, a first number of sleepers counted to the second position, and a first number of lights counted to the second position. It may include the step of determining whether one of the slots has been entered based on at least one of the two numbers.

The method of projecting the content includes a depth value of the at least one tunnel collected by a depth sensor, a position of the at least one tunnel estimated by a first number of sleepers counted by a sleeper sensor, and counted by a lighting sensor. The method may further include acquiring the positions of the slots based on at least one of a position of the at least one tunnel estimated by the second number of lights and an acceleration of the train detected by an acceleration sensor.

The correction information may be fixed in any one of the slots.

The method of projecting the content may further include calculating the correction information by using images captured of patterns projected at different depths, different positions, or different postures within the at least one tunnel.

In the calculating of the correction information, patterns projected at different depths, different positions, or different postures in the at least one tunnel by a projector installed outside the train may be used by the camera inside the train to open the window of the train. The camera inside the train based on the position of the window based on the first images captured through the first images, the original images of the patterns, and the second images captured by the RGB camera installed outside the train, and the Generating a three-dimensional model representing the positional relationship between the projectors; And calculating the correction information based on the 3D model.

Projecting the content allocated to one of the slots includes a distance from a projection surface on which the content is to be projected in the at least one tunnel and a curvature of the projection surface based on correction information corresponding to the one slot. It may include the step of projecting the content by changing at least one of the degrees.

At least one of an advertisement exposure position and an advertisement exposure time for each of the slots may be determined differently from each other.

According to an embodiment, a method of projecting content includes: downloading content for at least one tunnel corresponding to an operation schedule of the train by using a first module of a first vehicle included in a train; Tracking a position of a second vehicle in a tunnel according to the operation of the train by using a second module of a second vehicle preceding the first vehicle; Collecting information on a projection surface inside the tunnel at a location within the tunnel of the second vehicle using the second module; Obtaining correction information generated based on the projection surface information by using a first module of the first vehicle; Tracking the position of the first vehicle in the tunnel according to the operation of the train by using the first module; And projecting the content allocated to the tunnel based on the correction information by using the first module.

The operation schedule of the train may be determined by matching the location of the first vehicle and the time of tracking the location of the first vehicle with any one of operation schedules of a plurality of trains.

The second module of the second vehicle includes a projector installed outside the second vehicle, a sleeper sensor that counts the first number of sleepers installed on a track on which the second vehicle travels, and a second lighting device installed on the track. An illumination sensor that counts the number, a gyro sensor that detects the movement of the second vehicle, an acceleration sensor that detects the acceleration of the second vehicle, an RGB camera that photographs an image projected by the projector, and a projection surface of the image. It may include at least one of a depth camera that measures the depth of.

The step of tracking the position of the second vehicle in the tunnel may include at least one of a first number of sleepers installed on a track on which the second vehicle travels and a second number of lights installed on the track, the second It may include determining whether the location of the vehicle corresponds to the at least one tunnel section.

In the collecting of the projection surface information inside the tunnel, the projection surface information inside the tunnel is collected using images photographed with patterns projected at different depths, different positions, or different postures within the at least one tunnel. It may include the step of.

The obtaining of the correction information generated based on the projection surface information may include indicating a positional relationship between an RGB camera included in the first module of the first vehicle and a camera inside the first vehicle based on the projection surface information. Generating a three-dimensional model; And obtaining the correction information based on the 3D model.

According to an embodiment, the device for projecting content tracks a first position of a train, and the operation schedule of the train-the operation schedule is the operation of a plurality of trains based on the time of tracking the first position and the first position. Determined by matching any one of the schedules-Download content for at least one slot in the tunnel corresponding to -, track a second location according to the operation of the train, and based on the second location, the train A processor that determines whether one of the slots has been entered; And a projector for projecting the content allocated to the one slot based on the correction information corresponding to the one slot according to the determination result.

According to one side, the content can be reproduced by correcting the content in real time according to the changing tunnel shape.

According to one side, by measuring a tunnel to determine a projection area, content can be effectively reproduced in places and sections within the tunnel with many restrictions.

According to one side, it is possible to organize and manage content more conveniently through the web cloud.

1 is a diagram illustrating a configuration of a system for projecting content according to an embodiment.

2 is a view showing the configuration of a train according to an embodiment.

3 is a flowchart illustrating a method of projecting content according to an exemplary embodiment.

4 is a diagram for describing a method of calculating correction information according to an exemplary embodiment.

5 is a flowchart illustrating a method of projecting content according to another embodiment.

Fig. 6 is a block diagram of a device for projecting content according to an exemplary embodiment.

Specific structural or functional descriptions disclosed in this specification are exemplified only for the purpose of describing embodiments according to a technical concept, and the embodiments may be implemented in various different forms and are limited to the embodiments described herein. It doesn't work.

Terms such as first or second may be used to describe various components, but these terms should be understood only for the purpose of distinguishing one component from other components. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

When a component is referred to as being "connected" or "connected" to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in the middle. It should be. On the other hand, when a component is referred to as being "directly connected" or "directly connected" to another component, it should be understood that there is no other component in the middle. Expressions describing the relationship between components, for example, "between" and "just between" or "neighboring to" and "directly neighboring to" should be interpreted as well.

Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate that the specified features, numbers, steps, actions, components, parts, or combinations thereof exist, but one or more other features or numbers, It is to be understood that the presence or addition of steps, actions, components, parts or combinations thereof does not preclude the possibility of preliminary exclusion.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the relevant technical field. Terms as defined in a commonly used dictionary should be construed as having a meaning consistent with the meaning of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in the present specification. Does not.

1 is a diagram illustrating a configuration of a system for projecting content according to an exemplary embodiment. Referring to FIG. 1, the system 100 according to an embodiment may include a server 110, a train 130, and a content management server 150.

The server 110 may exchange train information with the train 130 in real time by synchronization with the train 130. The server 110 may deliver the scheduled content received from the content management server 150 to the train 130. The server 110 may control train operation and content provision. The server 110 may control, for example, which content should be projected and played back from which vehicle of which train, which line, which line. The server 110 may remotely manage a projectile (eg, a webcam or a camera) installed on a train.

In addition, when the train starts running, the server 110 receives a location from the train, selects one of the train location and the time of tracking the location of the plurality of train schedules. , It is possible to specify the corresponding train based on the selected operation schedule.

The server 110 may control the reproduction of content by section/time in the train 130 in real time based on the cloud. The server 110 may be, for example, a single server computer or a similar system, or may be a plurality of servers arranged in one or more server banks or other arrangements. The server 110 may be placed in a single facility, or may be a server “cloud” distributed among many different geographic locations. The server 110 may be, for example, a web cloud server.

The train 130 may receive content from the server 110 and project the received content onto the projection surface 145 in the tunnel 140 to reproduce the content. The projection surface 145 on which the content is reproduced may correspond to, for example, slot(s). The'slot(s)' may correspond to a predetermined area(s) or section(s) or a projection suitable section(s) of the content in order to project the content in the tunnel. One content or multiple content may be played in one slot, or one content may be played in multiple slots. An advertisement exposure position for each slot and an advertisement exposure time for each slot may be different from each other. In this case, the advertisement cost for each slot may be set differently for each train and/or for each vehicle of each train in consideration of the characteristics of the main passengers of the corresponding track.

The server 110 may slot, for example, a section in which the depth information of the window area of the train 130 in the tunnel 140 is constant (eg, 10 seconds or more).

The train 130 is, for example, a projector that projects content onto the projection surface 145 of the tunnel 140, an RGB camera that photographs an image projected by the projector, and the depth to the projection surface 145 on which the image is projected. In addition, the train 130 may further include a sensor for estimating the position of the train 130. For example, the train 130 may include a sleeper sensor that counts the first number of sleepers installed on the track on which the train 130 runs and/or a lighting sensor that counts the second number of lights installed on the track. I can. In addition, the train 130 may further include a sensor for estimating the movement of the train 130. For example, the train 130 may include a gyro sensor that detects the movement of the train 130 and/or an acceleration sensor that detects the acceleration of the train 130. In some cases, the train 130 may further include a camera (or webcam) for photographing content projected on the projection surface 145 through the window of the train 130.

The gyro sensor, acceleration sensor, RGB camera, and depth camera may be configured as a single product, for example, a real sense camera. The sleeper sensor may correspond to a distance sensor that recognizes sleepers installed on a track at regular intervals. As an example, the sleeper sensor may include a distance sensor (eg, a laser sensor, etc.) having a predetermined specification to recognize the sleeper. According to an embodiment, the specification related to the sampling rate of the sleeper sensor may be determined in consideration of the speed of the train. In addition, the lighting sensor may correspond to an image sensor that recognizes lighting installed on a track. As an example, the lighting sensor may include an image sensor (eg, an RGB camera, etc.) having a predetermined specification to recognize lighting.

The server 110 collects depth information at a corresponding position of the train 130 by a depth camera or a depth sensor installed outside the train 130, and the number of sleepers collected by the sleeper sensor You can estimate the location. Accordingly, the server 110 may obtain depth information according to the location of the train in advance.

The train 130 may manage a module including a projector installed outside the train 130 and download schedules and contents from the server 110. In addition, the train 130 may download slot information on slots in at least one tunnel 140 and information on a projection section in the tunnel together.

Hereinafter, “content” may be multimedia content including video content and/or video, and sound. The content may be, for example, advertisement content, public service content, or emergency notification content. The train 130 may provide various information such as weather, current location, etc. along with the content through other layers in addition to the content layer corresponding to the content. The train 130 may provide various information along with the content by synthesizing and rendering another layer with the content layer in real time at the time when the content is projected.

Depending on the embodiment, a special button for projecting emergency notification content may be provided in the engine room of the train 130. For example, in the case of an accident or fire of the train 130, emergency notification content indicating an escape direction based on the current location by pressing a special button by an engineer will be projected on the projection surface 145 or other inner walls of the tunnel. I can.

The train 130 may be, for example, a city railroad, a subway, a general railroad, a light railroad and/or a wide area railroad, and may include at least one vehicle. Each system of the train 130 can be modularized. The train 130 may include, for example, an Uninterrupted Power Supply (UPS), control power through the UPS, and/or drive the module upon emergency power off.

The content management server 150 may upload content to be played on the train 130 to the server 110. In addition, the content management server 50 may update the content schedule corresponding to the content to be played on the train 130 to the server 110.

The server 110 and/or the content management server 150 may manage content and schedule for each slot. In addition, the server 110 and/or the content management server 150 may manage log data.

The server 110 and the content management server 150 may be distinguished from each other as separate entities as shown in FIG. 1, or may be integrated into one server according to embodiments.

The projection device described below may correspond to the train 130 itself, or may be a modulator separately provided for each vehicle of the train 130. The moderator may be separately provided for each number of windows of a vehicle, for example. The configuration of the train 130 will be described in detail with reference to FIG. 2 below.

2 is a diagram showing the configuration of a train according to an embodiment. Referring to FIG. 2, a train including a first vehicle 210 and a second vehicle 230 is shown.

Real sense cameras and projectors may be installed outside each vehicle 210 and 230 of the train. The real sensor camera and projector can be composed of one module. The module may be installed under the left and right windows of each vehicle. For example, the 1-1 module 221 and the 1-2 module 223 may be installed under the left window of the first vehicle 210. Such a module may be installed under the right window in addition to the lower left window of the first vehicle 210. Likewise, the 2-1 module 231 and the 2-2 module 233 may be installed under the left window of the second vehicle 230.

Each vehicle (210, 230) of the train recognizes the sleepers installed on the track on which the vehicle is running, and a sleeper sensor that counts the first number of the recognized sleepers, and the lights installed on the track, and the recognized lighting A lighting sensor that counts the second number of may be included. The sleeper sensor and the light sensor may be installed outside each vehicle 210 and 230. In addition, an RGB camera for capturing an image projected by the projector, and a depth camera for measuring the depth of the image with the projection surface may be installed outside each vehicle 210 and 230.

According to an embodiment, each vehicle (210, 230) included in the train is whether the vehicles included in any train are included in the same train by the first number of counted sleepers and/or the second number of lights, or It is also possible to determine which vehicle the vehicle is connected to on the same train. Each of the vehicles 210 and 230 may recognize that each vehicle is included in the same train if, for example, the increase in the first number of counted sleepers or the second number of lights is constant compared to time. In addition, each vehicle 210 and 230 may determine the positional relationship between the preceding and following vehicles in the corresponding train by the total amount of the counted first number and/or the second number of lights.

The sleeper sensor-based algorithm (vProbe) according to an embodiment may be implemented as shown in Table 1.

3 is a flowchart illustrating a method of projecting content according to an exemplary embodiment. Referring to FIG. 3, the projection device according to an embodiment tracks a first position of a train (310). The first position may correspond to the initial position of the train. The first position may be a position in which the train is stopped or may be a position in which the train is running. The projection device, for example, calculates the distance from the base window by the total number of sleepers recognized and/or counted through the sleeper sensor while moving from the departure point or the base window of the train according to the operation schedule of the train, and The first location according to the train operation schedule can be identified. For example, the projection apparatus may count the total number of sleepers by recognizing or determining the sleepers based on the average and standard deviation of a moving window rather than one frame of sleeper data.

The projection device downloads content for slots in at least one tunnel corresponding to the train operation schedule (320). In this case, the operation schedule is determined by matching the first position and the time of tracking the first position with any one of operation schedules of the plurality of trains. The projection device may, for example, download the content and/or the operation schedule for the slots in the tunnel from the server.

The projection device tracks the second position according to the operation of the train (330). The second position may correspond to the position of the train changed from the first position according to the operation of the train. In step 330, the projection device may determine whether the second location corresponds to at least one tunnel section. The projection device, for example, based on at least one of the first number of sleepers installed on the track on which the train runs and the second number of lights installed on the track, whether the second position corresponds to at least one tunnel section Can be judged. In this case, the number of accumulated counted sleepers and/or the number of accumulated counted lights corresponding to the distance between each station and each tunnel according to the train operation schedule may be known in advance. For example, the projection device may determine whether the second position corresponds to at least one tunnel section by comparing the number of previously known sleepers up to the first tunnel of Line 2 with the number of currently accumulated counted sleepers. have.

In addition, in step 330, the projection device determines whether the train is running, and if it is determined that the train is running, it may track a second position according to the train's operation. The projection device may determine whether the train is running and/or whether the train is stopped using, for example, at least one of an acceleration sensor, a lighting sensor, and a sleeper sensor. The projection device may determine whether the train is running based on, for example, whether there is a change in the acceleration detected by the acceleration sensor, or whether there is a change in the count value from the lighting sensor or the sleeper sensor. The projection device may determine that the train is running when there is a change in acceleration or a change in the count value of each sensor.

Alternatively, the projection device may track the second location using a Wi-Fi receiver. For example, the projection device may generate a database by receiving a signal from an access point (AP) installed in each station using a Wi-Fi receiver, and collecting a unique ID of the AP from the received signal. In the database, the unique ID of the AP and information indicating the station in which the AP is installed may be matched and stored. Thereafter, when the train enters a specific station, a unique ID may be obtained by receiving a signal from an AP installed at the station, and information of the station stored by matching the unique ID from the database may be obtained. Through the above-described method, the projection device can determine which station the current location corresponds to.

Based on the second position, the projection device determines whether the train has entered any one of the slots (340). The projection device is, for example, based on at least one of the depth value at the second position, the first number of sleepers counted up to the second position, and the second number of lights counted up to the second position. It can be determined whether or not the slot has been entered.

According to the determination result, the projection apparatus projects the content allocated to one slot based on correction information corresponding to one slot (350). The projection device may change at least one of a distance from a projection surface to project content and a degree of curvature of the projection surface in at least one tunnel based on correction information corresponding to any one slot to project the content. I can. The correction information may be calculated in advance by a correction relationship based on the depth and the position of the projector. The projection apparatus may set different correction information for each section within a slot, or may dynamically select correction information within a single slot.

For example, even if the projection surface in the tunnel on which the content is projected is a single tunnel, the flatness of the projection surface itself and the degree of curvature of the projection surface may be different depending on the location of the tunnel. Accordingly, in one embodiment, correction information for correcting screen distortion occurring on the projection surface due to various causes is calculated in advance, and content is projected based on the correction information, so that passengers of the train can view the projected content more clearly. can do. In this case, the correction information may be, for example, a correction value for correcting a screen distortion that occurs according to a train speed, a vibration according to a track road surface, a projection distance inside a tunnel, and a degree of curvature of the tunnel surface.

The correction information may be determined identically for slots included in one tunnel, or may be determined differently for each of the slots included in one tunnel. The correction information may be fixed for each slot.

According to an embodiment, the projection device may photograph content projected on the projection surface while the train is running and feed back the quality of the content to the server. The quality of the content fed back to the server may be utilized by the server and/or the content management server to branch a single slot. Depending on the embodiment, a camera or webcam capable of photographing content projected on the projection surface or slot(s) on the projection surface in the tunnel through the window may be installed inside the window of each vehicle, that is, inside the vehicle.

4 is a diagram for describing a method of calculating correction information according to an exemplary embodiment. According to an embodiment, the correction information is determined corresponding to the slot in the tunnel.

At this time, before determining the correction information, 3D modeling (for example, pre-calibration) the positional relationship of the projector and sensor installed in the train based on the window of the train, and the train Operations for generating slots may precede by determining a section suitable for projection in the tunnel in which the is operated.

The image projected into the tunnel through a projector installed at the bottom of the train is intended to be viewed by passengers through a window. Therefore, it is important how the image projected into the tunnel by the projector is viewed from a viewpoint inside the train. In one embodiment, through pre-calibration, a three-dimensional position of a projector and/or a sensor (eg, a three-dimensional real sensor) may be modeled based on a window of a train.

The pre-calibration process can be performed as follows while the train is stopped. For example, a plurality of temporary walls having different distances (depths), different positions, or different postures at the vehicle base of the train may be used. In this case, it can be assumed that the distance between each temporary wall and the projector of the train is known in advance, and the size of the original pattern projected on the temporary wall through the projector is also known in advance. Patterns projected on different temporary walls through a projector installed at the bottom of the train can be photographed by an RGB camera installed at the bottom of the train and a webcam inside the train. A webcam inside the train can capture patterns through the train's window.

The server or projection device captures images of patterns projected on different temporary walls, more specifically, the first images captured by a webcam inside the train through the window of the train, and the patterns captured by an RGB camera installed outside the train. The positional relationship of a webcam, an RGB camera, and/or a projector may be 3D modeled based on a window by using the relationship between the second images and the original image of the patterns. In this case, the webcam may take an image including a window inside the train, and the location of the webcam may be used to determine a position based on the window.

According to an embodiment, when a structure that accurately knows a position relative to a window is used, pre-calibration may be performed without an image captured through a webcam inside a train. However, even in this case, a plurality of temporary walls having different depths, different positions, or different postures may still be used. The server or projection device uses, for example, an image shot at a right angle to a window of patterns displayed on temporary walls through an RGB camera, and an image shot at a different rotation or distance, and uses an RGB camera and/or projector based on the window. The positional relationship of can be modeled in 3D.

For convenience of explanation, embodiments in which a plurality of temporary walls are used have been described, but pre-calibration may be performed by photographing a plurality of times while placing a single temporary wall at different depths, positions, or postures. Through photographing a plurality of temporary walls or multiple photographing of a single temporary wall, a scale of a pattern may be calculated, and the calculated scale may be used for 3D modeling based on a window.

According to an embodiment, in order to obtain a three-dimensional position of the projector in the pre-calibration step, first, a camera looking at a window in the subway is installed, and then the coordinates of the camera must be obtained based on the window position. When the 3D positions of the four points of the window are defined as X _w ⁿ and the positions of the four points of the window captured from the camera image _{are defined as x cam} ⁿ , the rotation matrix (R _cam ) and the movement matrix (T) satisfying Equation 1 _cam ). Here, K _cam is an internal parameter of the camera.

For example, since the position of the window is used as a reference, X _w ⁿ may be determined corresponding to the origin of the 3D coordinate system based on the window. If you know the size of the window, you can specify the location of four points in the 3D coordinate system based on the window. After shooting with a camera so that all four points of the window are visible, the positions of the four points in the camera image can be specified _{as x cam} ^n. In this case, since x _cam ⁿ , K _cam and X _w ⁿ _{are known, the rotation matrix R cam} and the movement matrix T _cam of the camera can be calculated in the window-based 3D coordinate system using Equation 1.

Thereafter, a temporary wall is erected outside the window at a certain distance (d), a pattern is projected using a projector, the pattern is captured with a camera, and a 3D position value is obtained with a depth information camera. When the 3D location of the points included in the pattern projected on the temporary wall is X _d ⁿ and the location of the points included in the pattern captured from the camera image is x _pattern ⁿ _{, X d} ⁿ that satisfies Equation 2 can be calculated. I can.

For example, since x _pattern ⁿ , K _cam and [R _cam T _{cam ] are known, the 3D position X d} ⁿ of points included in the pattern projected on the temporary wall can be calculated.

_{According to the value of the 3D pattern position information X d} ⁿ obtained through the above-described process, the server or the projection device may obtain rotation information and position information of the projector.

Here, x _proj ⁿ is the position of the pattern in the image projected by _{the projector, K proj} is the internal parameter of _{the projector, R proj} is the rotation matrix of the projector, and T _proj is the movement matrix of the projector.

For example, since x _proj ⁿ , K _proj and X _d ⁿ _{are known, the rotation matrix R proj} and the movement matrix T _proj of the projector can be calculated in a window-based 3D coordinate system using Equation 3.

In addition, the server or the projection device may obtain rotation information and position information of the depth information camera by using Equation 4.

Here, x _depth ⁿ is the position of the pattern in the image captured by the depth information camera, K _depth is an internal parameter of the depth information cafe _{, R depth is the rotation matrix of the depth information cafe, and T depth} is the depth information cafe. It is the movement matrix of Ra.

For example, since x _depth ⁿ , K _depth and X _d ⁿ _{are known, a rotation matrix (R depth} ) and a movement matrix (T _depth ) of a depth information camera can be calculated in a window-based 3D coordinate system using Equation 4. have.

The pre-calibration information may include a rotation matrix and a movement matrix of a projector and/or a depth information camera in a window-based 3D coordinate system.

In addition, the slots in the tunnel may be predetermined, for example, by the following method while the train is running.

The projection device is, for example, the depth value of at least one tunnel collected by the depth sensor during the operation of the train, the position of at least one tunnel estimated by the first number of sleepers counted by the sleeper sensor, and the lighting sensor. At least one of the position of at least one tunnel estimated by the second number of lights counted by and the acceleration of the train detected by the acceleration sensor may be collected. The server or the projection device may create a slot by determining a section suitable for projection in the tunnel based on the information collected from the projection device.

For example, a server or a projection device provides an interface that allows the administrator to select a projection suitable section by visualizing the depth to the tunnel corresponding to the window area of the train, and can create a slot according to the administrator's selection. have.

Alternatively, the server or the projection device may automatically determine a section suitable for projection in the tunnel by using a deep learning technique in which the collected information is input. In this case, a deep learning model may be pre-trained based on information on a previously generated slot and information collected in correspondence with the corresponding slot.

Alternatively, the server or the projection device may automatically determine a section suitable for projection in the tunnel by using a clustering technique (eg, K-means clustering) in which the collected information is input.

The generated slot includes location information (eg, sleeper sensor information, global time information, etc.) and depth information (eg, depth information of a visible area from a window) corresponding to a section suitable for projection in the tunnel. can do.

The server or the projection device may generate correction information for each slot based on the pre-calibration information and the slot information. For example, the server or the projection device corrects based on the depth information included in the slot information and the projector and/or depth information included in the pre-calibration information 3D position of the camera (e.g., a movement matrix and a rotation matrix). Correction information for each slot can be calculated through a relational expression.

According to an embodiment, the server or the projection device may acquire depth information of a window reference tunnel by changing depth information received from a depth information camera for each slot into a window-based coordinate system using pre-calibration information. For example, since the pre-calibration information includes a movement matrix and a rotation matrix of a depth information camera, depth information measured in a 3D coordinate system based on a depth information camera may be converted into a 3D coordinate system based on a window. In this case, the depth information of the tunnel based on the window may include _{3D information X t} ^{n of the inner surface of the tunnel in the 3D coordinate system based on the window.}

The server or the projection device may determine a projection target area to actually project the content based on the depth information of the tunnel based on the window. For example, the server or the projection device may select a rectangular area having a flatness of a certain or more within the tunnel. In this case, the projection target area information may include _{3D information X s} ⁿ of four corners of a rectangular area in a 3D coordinate system based on a window.

The server or the projection device may calculate an area corresponding to the projection target area in the projected image of the projector by using the pre-calibration information. For example, referring to Equation 5, the server or the projection device is based on the movement matrix and rotation matrix of the projector included in the pre-calibration information, _{from the 3D information X s} ⁿ of the projection target area to the projection target area in the projection image. We can calculate _{the location x screen} ⁿ of the corresponding four points.

For example, since K _proj , R _proj , T _proj and X _s ⁿ _{are known, the positions x screen} ⁿ of four points corresponding to the projection target area in the projection image can be calculated using Equation 5.

The server or the projection device may generate homography information that converts the positions of the four corners of the actual projected content into x _screen ⁿ as correction information for each slot.

As described below, when a train enters a specific slot, the server or the projection device converts the content (for example, warping) using the correction information of the corresponding slot to project the content without distortion when viewed from the window. I can.

4, as the train 410 moves, contents 403 and 405 projected at different locations or distances within the tunnel 401 by the projector 412 installed outside the train 410 are shown. do. The projection device converts, warps, or corrects the content based on the correction information included in the slot information and projects it to provide an image (eg, a flat image, etc.) in a desired form from the viewpoint of passengers in the window.

As described above with reference to FIG. 3, correction information for each slot may be generated in advance and stored in a server or the like. In this case, as the projection device enters a specific slot, it may obtain correction information of the corresponding slot from the server. Of course, according to an embodiment, the projection device may generate correction information in real time, or may obtain correction information generated by a projection device of a preceding vehicle in the same train.

According to the embodiment, in order to establish a system that can be operated at all times in special environments such as vibration and dust according to the train speed and the track road surface, the projection surface reduces the screen distortion caused by the projection distance inside the tunnel and the degree of curvature of the tunnel surface. It can be automatically corrected based on the tunnel data.

An RGB camera 414 for photographing contents 403 and 405 may be installed at a location adjacent to the external projector 412 of the train 410. In addition, a webcam 416 capable of photographing the contents 403 and 405 projected on the projection surface through the window of the train may be further installed in the train 410. According to an embodiment, the image quality may be inspected through the RGB camera 414 and/or the webcam 416. Various quality management protocols may be performed, such as recalculating correction information of a specific slot or calculating correction information for each divided slot after a specific slot is divided into a plurality of slots according to the inspection result of the image quality.

5 is a flowchart illustrating a method of projecting content according to another exemplary embodiment. Referring to FIG. 5, the projection apparatus according to an embodiment downloads content for at least one tunnel corresponding to a train operation schedule by using a first module of a first vehicle included in a train (operation 510 ). The train operation schedule may be determined, for example, by matching the location of the first vehicle and the time of tracking the location of the first vehicle with any one of the operation schedules of the plurality of trains.

The projection device may download content from, for example, a server or a content management server. The first module of the first vehicle includes, for example, a projector installed outside of the first vehicle, a sleeper sensor that counts the first number of sleepers installed on the track on which the first vehicle travels, and the second lighting installed on the track. A lighting sensor that counts the number, a gyro sensor that detects the movement of the first vehicle, an acceleration sensor that detects the acceleration of the first vehicle, an RGB camera that shoots an image projected by the projector, and measures the depth of the projection surface of the image. It may include a depth camera and the like.

The projection device tracks the position of the second vehicle in the tunnel according to the operation of the train using the second module of the second vehicle preceding the first vehicle (520). The second module of the second vehicle includes, for example, a projector installed outside the second vehicle, a sleeper sensor that counts the first number of sleepers installed on the track on which the second vehicle travels, and a second light source installed on the track. A light sensor that counts the number, a gyro sensor that detects the movement of a second vehicle, an acceleration sensor that detects the acceleration of the second vehicle, an RGB camera that shoots an image projected by the projector, and measures the depth of the projection surface of the image. It may include a depth camera and the like. The projection device is, for example, based on at least one of the first number of sleepers installed on the track on which the second vehicle travels and the second number of lights installed on the track, the position of the second vehicle is at least one tunnel section. It can be determined whether or not it corresponds to.

In an embodiment, the first vehicle and the second vehicle may be vehicles included in the same train or may be vehicles included in different trains. When the first vehicle and the second vehicle are vehicles included in different trains, the second vehicle is a vehicle included in a preceding train running on the same track, and the first vehicle is a vehicle included in a subsequent train following the preceding train. Can be In step 520, the projection device tracks the position of the second vehicle in the tunnel based on, for example, the number of sleepers and/or the number of lights detected by the sleeper sensor and the lighting sensor included in the second module. can do. The position of the second vehicle in the tunnel may be transmitted to the first vehicle through a communication interface of the projection device.

The projection device uses the second module to collect information on the projection surface inside the tunnel at the location of the second vehicle in the tunnel (530 ). Correction information may be generated based on the projection surface information.

The projection device acquires correction information generated based on the projection surface information by using the first module of the first vehicle (540 ). The projection device uses the first module to track the position of the first vehicle in the tunnel according to the operation of the train (550). The projection device may track the position of the first vehicle in the tunnel in the same manner as in step 520. The projection device projects the content allocated to the tunnel based on the correction information using the first module (560).

In an embodiment, each of the first module and the second module may be installed in various forms, such as being installed in units of a vehicle of a train or corresponding to windows of each vehicle.

6 is a block diagram of an apparatus for projecting content according to an exemplary embodiment. Referring to FIG. 6, a device (hereinafter, referred to as a'projection device') 600 for projecting content according to an embodiment includes a processor 610 and a projector 620. The projection device 600 may further include a communication interface 630 and a memory 640. The processor 610, the projector 620, the communication interface 630, and the memory 640 may communicate with each other through a communication bus 605.

The processor 610 tracks the first position of the train. The processor 610 downloads contents for slots in at least one tunnel corresponding to the train operation schedule. The operation schedule is determined by matching the first position of the train and the time of tracking the first position with any one of the operation schedules of the plurality of trains. The train operation schedule may be determined by, for example, a server. The processor 610 may receive an operation schedule of a corresponding train from a server through the communication interface 630.

The processor 610 tracks the second position according to the operation of the train. The processor 610 determines whether the train has entered any one of the slots based on the second position.

The projector 620 projects the content allocated to any one slot based on the correction information corresponding to any one slot according to the result of the processor 610's determination that the train has entered any one of the slots. do. The correction information may be calculated in advance by a server or the like, or may be calculated in real time by the processor 610.

The projector 620 may project different contents allocated to slots in the tunnel. The content may be video content or multimedia content including video, audio, and the like.

The communication interface 630 may receive content downloaded from the server. In addition, the communication interface 630 may exchange information with other vehicles of the corresponding train. In addition, the communication interface 630 may exchange various information corresponding to slots in a corresponding tunnel in addition to the server and the operation schedule.

The memory 640 may store content allocated to any one slot. The memory 640 may store content projected by the projector 620. The memory 640 is a high-speed random access memory, such as, for example, a dynamic random access memory (DRAM), a static random access memory (SRAM), a double data rate synchronous dynamic random access memory (DDR RAM), or another high-speed access solid state memory device. It may include high-speed random access memory. The memory 640 may include volatile memory and nonvolatile memory. The memory 640 may selectively include at least one storage device located remotely from the processor 610.

In addition, the processor 610 may perform at least one method described above with reference to FIGS. 1 to 5 or an algorithm corresponding to at least one method. The processor 610 may execute a program and control the projection device 600. The program code executed by the processor 610 may be stored in the memory 640. The projection device 600 may be, for example, a module included in each vehicle of a train or a train.

The embodiments described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods, and components described in the embodiments include, for example, a processor, a controller, a central processing unit (CPU), a graphics processing unit (GPU), and an ALU ( arithmetic logic unit), digital signal processor, microcomputer, field programmable gate array (FPGA), programmable logic unit (PLU), microprocessor, Application Specific Integrated Circuits (ASICS), or instructions ( Instructions) may be implemented using one or more general purpose computers or special purpose computers, such as any other device capable of executing and responding to instructions.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

As described above, although the embodiments have been described by the limited drawings, various modifications and variations are possible from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved. Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (20)

1. Tracking the first position of the train;

   For at least one slot in the tunnel corresponding to the operation schedule of the train-the operation schedule is determined by matching the time of tracking the first position and the first position with any one of operation schedules of a plurality of trains. Downloading content;

   Tracking a second position according to the operation of the train;

   Determining whether the train has entered any one of the slots based on the second position; And

   Projecting the content allocated to any one slot based on the correction information corresponding to the one slot according to the determination result

   Containing, a method of projecting content.
2. The method of claim 1,

   The step of tracking the second location

   Determining whether the second location corresponds to the at least one tunnel section

   Containing, a method of projecting content.
3. The method of claim 2,

   The above train

   A projector installed outside the train, a sleeper sensor that counts the first number of sleepers installed on the track on which the train runs, a lighting sensor that counts the second number of lights installed on the track, and detects the movement of the train Projecting content, including at least one of a gyro sensor, an acceleration sensor that detects acceleration of the train, an RGB camera that photographs an image projected by the projector, and a depth camera that measures a depth with a projection surface of the image How to.
4. The method of claim 2,

   The step of determining whether the second location corresponds to the at least one tunnel section

   Based on at least one of a first number of sleepers installed on a track on which the train runs and a second number of lights installed on the track, determining whether the second position corresponds to the at least one tunnel section step

   Containing, a method of projecting content.
5. The method of claim 1,

   The step of tracking the second location according to the operation of the train

   Determining whether the train is running; And

   Tracking a second position according to the operation of the train according to the determination of whether to operate

   Containing, a method of projecting content.
6. The method of claim 1,

   The step of determining whether one of the slots has entered

   Whether the slot is entered based on at least one of a depth value at the second position, a first number of sleepers counted up to the second position, and a second number of lights counted up to the second position The steps to judge

   Containing, a method of projecting content.
7. The method of claim 1,

   The depth value of the at least one tunnel collected by the depth sensor, the position of the at least one tunnel estimated by the first number of sleepers counted by the sleeper sensor, and the second number of lights counted by the lighting sensor. Obtaining the positions of the slots based on at least one of the position of the at least one tunnel estimated by and the acceleration of the train detected by an acceleration sensor

   A method for projecting content further comprising.
8. The method of claim 1,

   The correction information is

   A method of projecting content, which is fixed in any one of the slots.
9. The method of claim 1,

   Calculating the correction information by using images photographed of patterns projected at different depths, different positions, or different postures within the at least one tunnel

   A method for projecting content further comprising.
10. The method of claim 9,

    The step of calculating the correction information

    First images captured by a camera inside the train through a window of the train by projecting patterns projected at different depths, different positions, or different postures in the at least one tunnel by a projector installed outside the train, 3 representing the positional relationship between the camera inside the train and the projector based on the position of the window based on the original image of the patterns and the second images captured by the RGB camera installed outside the train. Generating a dimensional model; And

    Calculating the correction information based on the 3D model

    Containing, a method of projecting content.
11. The method of claim 1,

    Projecting the content allocated to any one of the slots

    Projecting the content by changing at least one of a distance from a projection surface to project the content and a degree of curvature of the projection surface within the at least one tunnel based on correction information corresponding to the one slot

    Containing, a method of projecting content.
12. The method of claim 1,

    At least one of an advertisement exposure position and an advertisement exposure time for each of the slots is determined differently from each other.
13. Downloading content for at least one tunnel corresponding to an operation schedule of the train by using a first module of a first vehicle included in the train;

    Tracking a position of a second vehicle in a tunnel according to the operation of the train by using a second module of a second vehicle preceding the first vehicle;

    Collecting information on a projection surface inside the tunnel at a location within the tunnel of the second vehicle using the second module;

    Obtaining correction information generated based on the projection surface information by using a first module of the first vehicle;

    Tracking the position of the first vehicle in the tunnel according to the operation of the train by using the first module; And

    Projecting the content allocated to the tunnel based on the correction information by using the first module

    Containing, a method of projecting content.
14. The method of claim 13,

    The schedule for the above train is

    The method of projecting content, which is determined by matching the location of the first vehicle and the time of tracking the location of the first vehicle with any one of operation schedules of a plurality of trains.
15. The method of claim 13,

    The second module of the second vehicle

    A projector installed outside the second vehicle, a sleeper sensor that counts a first number of sleepers installed on a track on which the second vehicle travels, a lighting sensor that counts a second number of lights installed on the track, the second 2 At least one of a gyro sensor that detects the movement of the vehicle, an acceleration sensor that detects the acceleration of the second vehicle, an RGB camera that photographs an image projected by the projector, and a depth camera that measures the depth with the projection surface of the image. A method of projecting content, including one.
16. The method of claim 13,

    The step of tracking the position of the second vehicle in the tunnel

    Based on at least one of the first number of sleepers installed on the track on which the second vehicle travels and the second number of lights installed on the track, whether the position of the second vehicle corresponds to the at least one tunnel section Steps to determine whether

    Containing, a method of projecting content.
17. The method of claim 13,

    Collecting information on the projection surface inside the tunnel

    Collecting information on the projection surface inside the tunnel by using images photographed of patterns projected at different depths, different positions, or different postures within the at least one tunnel

    Containing, a method of projecting content.
18. The method of claim 13,

    The step of obtaining correction information generated based on the projection surface information

    Generating a 3D model representing a positional relationship between an RGB camera included in a first module of the first vehicle and a camera inside the first vehicle based on the projection surface information; And

    Obtaining the correction information based on the 3D model

    Containing, a method of projecting content.
19. A computer program combined with hardware and stored in a computer-readable recording medium to execute the method of claim 9.
20. Tracking the first position of the train, and corresponding to the operation schedule of the train-the operation schedule is determined by matching the time of tracking the first position and the first position with any one of the operation schedules of a plurality of trains- Download content for at least one slot in the tunnel, track a second location according to the operation of the train, and whether the train has entered any one of the slots based on the second location A processor that determines the; And

    A projector that projects the content allocated to any one slot based on the correction information corresponding to the one slot according to the determination result

    Containing, a device for projecting content.

Images