KR20210013900A - Simulation system of digestion experience using VR - Google Patents

Abstract

The preset invention relates to a fire extinguishing experience simulation system using VR, a fire extinguisher for experience, and a fire extinguishing experience (fire extinguisher experience) simulator which uses a training fire extinguisher that mimics a shape of a real fire extinguisher and a head-up display (HMD) through programs, etc., thereby being able to experience firefighting in the same order as in actual firefighting activities indoors.

Description

Simulation system of digestion experience using VR

The present invention is a firefighting experience simulation system using VR, using a training fire extinguisher and a head-up display (HMD) that mimic the shape of an actual fire extinguisher through a fire extinguishing experience simulation system using VR, a fire extinguisher, and a program. It relates to a fire extinguishing experience (fire extinguisher experience) simulator.

As a prior art of a fire fighting training simulator, for example, an indoor set is prepared in a cargo compartment such as a truck, and a fire is actually generated so that an extinguishing agent can be injected into the actual flame (Prior Art 1).

In addition, another conventional technology is a virtual reality experience simulator using a head mounted display (HMD), and a simulator is constructed by connecting the HMD and a graphic workstation with a cable (Prior Art 2).

In addition, as another conventional technology, there is a wearable device that is scheduled to provide an immersive viewing experience to the experiencer by configuring an HMD using a smart phone, etc., and displaying an image corresponding to the experiencer's comprehensive field of view on the display. Technology 3).

Conventional technologies 1 and 2 both require large-scale structures, for example, there is room for improvement in terms of mobility and simplicity of the simulator, and both are intended to be used by experts, regardless of the age of the experienced person. It doesn't mean it's easy to use.

In addition, wearable computing technology (prior art 3 above), which enables various immersive viewing experiences in VR (virtual reality) space, is in progress, but this technology presupposes immersion in virtual reality (VR) space. For example, using the tools used in actual activities is not very helpful in sharing the sense of reality and actually experiencing useful experiences.

The present invention is to provide a fire extinguishing experience simulation system using VR that allows a user to easily experience the actual fire extinguishing activity step with a sense of reality.

Other objects of the present invention will become apparent to those skilled in the art with reference to the following embodiments and the best embodiments and the accompanying drawings.

Next, in order to easily understand the outline of the present invention, an embodiment according to the present invention is illustrated.

As a first embodiment, a fire extinguishing experience simulation system using VR includes a display for displaying an image, a first communication unit for computer and communication processing, and an image camera for capturing an actual image, and a head mounted display mounted on the experienced person (HMD), a second communication unit that communicates between the first communication unit of the HMD, a fire extinguisher body, and when the HMD is in a virtual reality (VR) mode, an operation lever serving as a switch for injecting a virtual reality fire extinguishing agent and the The HMD includes a hose that functions as an ejection device for ejecting a virtual reality fire extinguishing agent from the end, a first sensor that detects an operation of the operation lever by the experienced person, and a second sensor that detects the movement of the end of the hose. Including an experiential fire extinguisher that also functions as a control,

The computer of the HMD receives a function as a control unit for controlling the operation of the HMD, wherein the control unit,

A real image/virtual reality image conversion unit that switches the real image display mode/virtual reality display mode, and in the case of the real image display mode, displays the real image captured by the image camera on the display;

When the HMD is switched from the real image display mode to the virtual reality image display mode, as the second embodiment, each detection of the first sensor and the second sensor of the experience fire extinguisher input through a first communication unit Based on the information, the hose held in the hand of the experienced person,

At the end, having a virtual reality image generation unit that displays a simulation image including an image in which the virtual reality extinguishing agent is sprayed and an image where the flame of virtual reality calms down, and an experience fire extinguisher imitating a fire extinguisher actually used in the first embodiment (It also functions as an HMD controller), as well as actual fire extinguishing activities, it is possible to operate the lever or change the direction of the hose end by hand. Thus, the experienced person can get used to the actual fire extinguisher. In addition, instead of entering the virtual reality space immediately, the actual image (for example, the actual image including the training fire extinguisher taken with the smartphone camera constituting the HMD) is displayed on the display, and the experiencer shows the location of the training fire extinguisher and his or her hand. It is possible to start a simulation of firefighting activity in a virtual reality (VR) space in the form of repeating the confirmed reality after checking the back.

Therefore, it is possible to perform fire extinguishing work (fire extinguisher experience) in the same way as the actual fire extinguishing stage while sharing a sense of reality. Therefore, the experiences can experience the actual stage of fire extinguishing with a sense of reality while sharing the reality.

In addition, the main components of the fire extinguishing experience simulation system of this type are a fire extinguisher for experience and an HMD, and the configuration is simplified.

Therefore, it is possible to easily implement a simulator with excellent mobility (portability setting) and simplicity (for example, a point that can be performed easily indoors without individual support from staff).

In addition, the training fire extinguisher, for example, the same as the actual fire extinguisher, for example, can be provided with a device equipped with a sensor and a communication unit, but is not limited thereto, and the weight of the training fire extinguisher is lightened, and a small fire extinguisher for children , It can be modified appropriately, such as slightly changing the shape of the fire extinguisher body or the design attached to the surface.

In a second embodiment dependent on the first embodiment, a third communication unit for communicating between the first communication units of the HMD is provided, and the actual image display mode input through the first and third communication units In the virtual reality image display mode, according to the image data of the real image and the image data of the virtual reality image in the virtual reality image display mode, the experiencer may further have a monitor that displays an image as seen through the display of the HMD as a monitor image.

According to the second embodiment, since the image that the experiencer is watching through the HMD display is displayed on the monitor as it is, one person or multiple people (spectators) other than the experiencer can share the same experience through the same video as the experiencer. .

Therefore, for example, the leader of the fire extinguishing activity can give appropriate advice to the experienced person, and it is also possible to create a more realism, such as raising the audience's shouts. Further, the monitor may be, for example, a projection type display device having one large screen that is used in common by several people, or may be a tablet display of each of the audience.

In a third embodiment, the virtual reality image generator holds the end of the hose displayed in the virtual reality space and fixes the position of the wrist rest of the hand of the experiencer, and according to the detection signal of the second sensor It is possible to change the direction of the tip of the hose in the virtual reality space around the point of the wrist.

According to the third embodiment, by holding the end of the hose displayed in the virtual reality space, fixing the position of the wrist rest of the experiencer, and measuring the amount of displacement caused by rotation of the wrist around the point, You can specify the direction.

In this case, since there is no need to consider the movement in the depth direction (forward and backward) and the movement in the vertical direction (vertical direction) of the hand, the detection burden of the sensor can be reduced, and the burden of image processing for image generation in the virtual reality image generator It is reduced.

Also, when using a fire extinguisher, the basics of using a fire extinguisher are to adjust the direction of the jet by the movement of the wrist with the tip of the hose while spraying the fire extinguishing agent vigorously toward the center of the flame, for example, at an appropriate distance (i.e., unnecessary movement of the arm and body, etc. It means that you can accurately hit the fire extinguishing agent without movement), and you can definitely experience this basic movement. Thus, the experienced person can actually learn the basics of useful digestive activities.

In a fourth embodiment, when the virtual reality image generation unit is switched from the real image display mode to the virtual reality image display mode, the image before the fire extinguishing activity is displayed in the virtual reality space after viewing the image before the fire extinguishing activity. After controlling to approach and stop the experiencer, the simulation image is displayed, and when the flame calms down and decreases to a predetermined degree, the video after the fire extinguishing activity including the flame that is calmed down and reduced to a predetermined degree approaches and stops. Can be controlled.

Since the third embodiment above does not take into account the movement of the hand with the hose in the depth direction (front-to-back direction), it may be difficult for the experienced person to grasp the sense of distance to the flame that is the object to be extinguished in the virtual reality space. (For example, a video of a flame superimposed on a background image) so that the experiencer can approach the experiencer at an appropriate timing, and through this, the experiencer himself advanced toward the flame (i.e., moved to an appropriate distance). It's like you can get a sense.

For example, when switching from the real image display mode to the virtual reality frostbite mode, the fire extinguishing activity is not immediately started, but the video is approached to the experiencer, and the experiencer first approaches the fire target flame to the distance the extinguishing agent reaches. It is important to learn the basics of digestive activities. In addition, you can learn the basics that it is important to extinguish the fire reliably by proceeding with extinguishing activities and by approaching closer without being vigilant at the timing when the flame has calmed down to a certain level.

In this embodiment, a simplified configuration is adopted that does not detect hand movement in the depth direction (forward and backward direction), while the experienced person can reliably learn the matters to be noted in the fire extinguishing operation in the depth direction (forward and backward direction). have.

In the fifth adoption example subordinate to any one of the first to fourth,

The virtual reality image generator may display a background image included in an image prior to the fire extinguishing activity of the fire phenomenon and a panoramic image for viewing the experiencer as a background image included in the simulation image.

In the fifth aspect, a wide field of view panoramic image (for example, a 360-degree panoramic image) that the experiencer can look around is not a flat image as an image displayed in a virtual reality space, so that the experiencer can experience immersive digestion activities. In addition, the panoramic image is not only a pole (or partial circumference) type, but also a full celestial shape, such as covering the entire field of view from the left and right heads to feet, as well as a full circumference of the sky, such as covering the upper half of the field of view on a dome (hemispherical) screen such as a planetarium. You can appropriately adopt the video. In addition, if a separate image with a proper parallax can be provided for the left and right eyes of the experiencer, a panoramic image of the whole celestial sphere can be provided.

In addition, in a sixth embodiment subordinate to any one of the first to fifth embodiments, the control unit evaluates the firefighting activity of the experienced person according to a predetermined criterion after the firefighting activity in the virtual reality space is finished. , It may further include an evaluation unit for displaying the information indicating the evaluation on the display of the HMD.

In the sixth embodiment, the evaluation by the evaluation unit (eg, “evaluation result” and “demonstration evaluation”) after the fire extinguishing experience is completed is displayed on the display of the HMD (or the display surface of the monitor when equipped with a monitor as in the second embodiment). Can be displayed.

For example, the evaluation is made according to the time spent on the fire and the direction of injection of the fire extinguishing agent, so that the experienced person (and the audience) does not have a serious effect on the sense of the game, and can actually help the fire extinguishing experience. It can also motivate those who can give evaluation to actively participate in digestion activities using VR.

In the seventh embodiment, the experience fire extinguisher is a fire extinguisher for experience in the configuration of a fire extinguishing experience simulation system (that is, a component of the system) using a dependent VR according to any one of the first to sixth embodiments.

The experience fire extinguisher has a shape and structure that mimics an actual fire extinguisher, and is equipped with a sensor that detects the movement of the hands of the experienced person and a communication unit that transmits the detection information of the sensor to the HMD, and many general people use the experience fire extinguisher. Therefore, you can easily experience fire extinguishing activities using VR indoors.

In the eighth embodiment, the program is a program for operating the computer provided in the HMD of the fire extinguishing experience simulation system using VR as the control unit in any one of the first to sixth embodiments.

In the eighth embodiment, for example, it is not necessary to load a program into the computer of the HMD to prepare the HMd, the components of the fire extinguishing experience simulation system using VR, and a dedicated wearable computer (HMD). Therefore, it makes it easy to use by many general users. This can contribute to the spread of a fire extinguishing experience simulation system using VR.

Those skilled in the art will readily understand that the illustrated embodiments according to the present invention can be changed without departing from the spirit of the present invention.

1(a) is a diagram showing a basic configuration example of a fire extinguishing experience simulation system using VR (virtual reality) according to the present invention.
1(b) is a diagram showing an example of a fire extinguishing activity in a fire extinguishing experience simulation using VR.
2(a) to 2(d) show that each experiencer fixes the wrist rest of the hand holding the end of the hose and rotates the wrist around the point, and changes the direction of the fire extinguishing agent sprayed from the end of the hose. It is a figure which shows an example to make.
3 is a view for explaining an example of a fire extinguishing experience step using VR and a location where a flame in virtual reality is visible, respectively, in FIGS. 3(a) to 3(d).
4A to 4E are diagrams showing examples of images visually recognized by the experiencer and the audience (including the leader) during a fire extinguishing experience using VR, respectively.
5 is a diagram showing an example of the overall configuration of a fire extinguishing experience simulation system using VR (virtual reality).
6(a) and 6(b) are views for explaining the display of a panoramic image,
6(c) is a diagram showing a basic configuration example of a flame simulator.
7 is a flow chart showing an example of a control method of a controller (computer) of an HMD in a fire extinguishing experience simulation system using VR (virtual reality).

The embodiments of the present invention presented below are used to easily understand the present invention. Accordingly, those skilled in the art should note that the present invention is not unduly limited by the embodiments described below.

In the following description, an example in which a head mounted display (hereinafter referred to as HMD) is used as a device for experiencing virtual reality (VR) is described, but the HMD is not limited to being mounted on a person's head. It is good to be attached to the body, and in a broad sense, it can be interpreted in the same meaning as a computer (wearable computer) that can be learned.

In addition, a portable mobile terminal (mobile terminal) such as an integrated smart phone and tablet PC, which is installed integrally with the body of goggles or glasses, is a head-mounting strip where a stationary computer in which the computer part is delayed apart from the display with the head mounted HMD. It is largely divided into a mobile type that constitutes an HMD by installing detachably on the support member of

In the present invention, any one may be used, but in particular, the goal is to disseminate the system by allowing everyone regardless of age group to easily experience the fire extinguishing activity at low cost.From this point of view, the following description uses a mobile HMD. Explain an example.

1(a) is a diagram showing a basic configuration example of a fire extinguishing experience simulation system using VR (virtual reality) according to the present invention, and FIG. 1(b) is a view showing a fire fighting example in a fire extinguishing experience simulation using VR. The Y-axis is set in the front direction (vertical direction) of the experienced person (U) in the real space so that it is displayed on the upper left of Fig. 1(a), and the X-axis is set in the left-right direction orthogonal to the Y-axis so that it is perpendicular to the Y-axis. A three-dimensional coordinate space is defined by each axis in which the Z axis is set in the direction. This three-dimensional coordinate space can be applied similarly to virtual reality (VR) space in appearance.

The experienced person U is wearing the HMD 10 on his head so that it is displayed on the upper left of Fig. 1(a). The HMD (10) goggles are constructed by detachably installing the support member (12) of a portable mobile terminal (mobile handset) 16 head-mounting strip such as a smart phone or tablet PC, and also a headphone so that you can hear the sound of burning fire. (14) is also installed.

In this configuration, a mobile terminal 16 such as a smart phone is equipped, and the display surface of the display (for example, a liquid crystal display) is the eye 1 (left eye 1L, right eye) of the experienced person U equipped with the HMD 10. The virtual reality image is projected on the company's display located in front of (1R)), allowing the experienced person (U) to experience digestive activities using VR (virtual reality). The display provided in the mobile terminal 16 may display a different image (parallax image) for each left and right eye, but here, in order to reduce the burden on the computer equipped with the mobile terminal 16, as shown in the example of FIG. 1(a) HMD (10) is a configuration with goggles.

In addition, the mobile terminal 16 and the computer (CPU, MPU, etc.) operate according to, for example, a preloaded program (application program), and the HMD 10 constituting a digestive experience simulation system using VR (virtual reality). It functions as a control unit that controls the operation of ). In addition, in Fig. 1(b), the control unit is indicated as “HMD control unit 200.” Also, in the following description, it may be simply referred to as “control unit 200”.

In addition, a communication unit that performs the communication processing of the HMD 10 (receiving processing of the detection signal from the sensor coming from the experience fire extinguisher 20, transmitting processing to the monitor 30 of the image data displayed on the HMD 10 display, etc.) Communication unit: not shown in Fig. 1(a), Fig. 1(b) includes a real-view camera (to be described later) capable of capturing the real view of the real space surrounding the HMD 10 and what is indicated by the HMd communication unit 300 do.

In addition, as shown in the lower left corner of Fig. 1(a), an experience fire extinguisher 20 that also serves as a controller for the HMD 10 is prepared in advance. This experience fire extinguisher 20 is a communication unit (second communication unit: for example, each of the first and second sensors (S1, S2) to communicate between the communication unit (first communication unit) of the HMD 10) has a communication function. When present, the part having the communication function becomes the second communication unit) and the fire extinguisher main body 22 and the HMD 10, in the case of a virtual reality (VR) display mode, an operation lever 24 that serves as a switch to inject a virtual reality fire extinguishing agent. ), the virtual reality fire extinguishing agent (5), the hose (26) that functions as an ejection device ejected from the end.According to the experienced person (U), the first sensor (S1) and the hose (26) end 28 for detecting the operation of the operation lever 24 ) Is provided with a second sensor (S2) for detecting the movement.

In addition, the training fire extinguisher 20, for example, the same as the actual fire extinguisher, for example, may be provided by mounting a device including a sensor and a communication unit, but is not limited thereto, and the weight of the training fire extinguisher 20 is lightened. , As a small fire extinguisher for children, the shape of the fire extinguisher body 22 or the design attached to the surface of the fire extinguisher can be changed slightly, and other modifications can be appropriately achieved.

This training fire extinguisher 20 has a function as a controller for the HMD 10, but serves as an experience tool (experience tool) through which actual fire extinguisher work can be experienced. Here, in a conventional virtual reality space such as a battle game using an immersive HMD, for example, in order to control the movement of the jukdo, the user moves the jukdo in the virtual reality space with a toy jukdo-type controller. The toy jukdo is simply acting as a controller and does not work as an experience tool (experience tool) for the purpose of getting used to the actual use of jukdo. Accordingly, the experience fire extinguisher of the present invention combines the function of a controller and a function as an experience tool (experience tool), and is a novel and useful structure for a fire extinguishing experience simulation.

That is, the experience fire extinguisher 20 has a shape and structure that mimics an actual fire extinguisher, and also includes sensors (S1, S2) that detect the movement of the hands of the experienced person, and a communication unit that transmits detection information of the sensor to the HMD. , It becomes possible for many ordinary people to experience fire extinguishing activities using VR conveniently indoors by using the experience fire extinguisher 20.

Further, as shown on the right side of Fig. 1(a), a monitor 30 is installed near the experienced person U. The monitor 30 is equipped with a communication unit (third communication unit) 32 for communicating between the communication unit (first communication unit) of the HMD 10, as specified in Fig. 1(b). The monitor 30 is not shown but is equipped with an image processing device.

The image processing device (not shown) provided in the monitor is input through the HMD 10 communication unit (first communication unit) and the monitor 30 communication unit (third communication unit) 32 (that is, transmitted/received through wireless communication). Based on the image data of the real image when the HMD 10 is in the real image display mode and the image data of the virtual reality image in the virtual reality image display mode), the experienced person (U) is An image as seen through the display of the HMD 10 is displayed as a monitor image. In the example of FIG. 1(a), the experience fire extinguisher 20 and the video near the left hand wrist of the experience user U, that is, the actual image taken by the external camera 103 of FIG. 5 This is being illuminated. In addition, the sound that the experienced person U is listening to is output from a speaker (not shown) equipped with a monitor 30.

In addition, in the example of FIG. 1(a), a leader (tc) in charge of professional advice on firefighting activities and an experience applicant (ad) waiting for the order of experience report a monitor image projected on the monitor 30 as a spectator. have. With the monitor 30 installed, one person or multiple people (spectators) other than the experienced person can share the same experience through the same video as the experience person (U). Therefore, for example, the leader of the fire extinguishing activity (tc) can give appropriate advice to the experienced person (U), and it is also possible to produce a more realistic feeling, such as the spectator (ad) raising a scream.

In addition, the monitor 30 may be, for example, a projection type display device with one large screen used in common by several people, or may be a tablet display of each of the audience.

As shown in Fig. 1(a), after the experiencer confirms the fire extinguisher main body 22 and the image (real image) of his or her hand (right hand 3R), the process proceeds to Fig. 1(b). That is, the HMD 10 mode is switched from the real image display mode to the virtual reality mode by the H MD control unit 200 (specifically, the real image/virtual reality image conversion unit 212 of FIG. 5: to be described later).

1(b) shows a fire extinguishing experience simulation using VR. When the HMD control unit 200 (specifically, the virtual reality image generation unit 210 of FIG. 5) is switched from the real image display mode to the virtual reality image display mode, the HMD control unit 200 performs image synthesis (image rendering). Video VR (1) including the end (28) of the hose (26) held in the hands of the experienced person (U), and the video VR (2) and virtual reality sprayed with a virtual reality fire extinguishing agent (5) from the end (28) of the hose (26) As shown in the background object 9, Fig. 1(b), the image VR (3) in which the flame 7 is calmed down, in this case, the background including the multi-tap for octopus-type wiring as a firearm is shown in Fig. VR (4). The frame image VR 4 for the virtual reality exhibition is created by superimposing it, and the image including the generated frame image VR 4 is displayed on the display of the HMD 10 as a simulation image, and in parallel with this, the frame image VR 4 ) Data is transmitted to the monitor 30. The monitor 30 displays the same image as the image viewed by the experienced person U on the monitor.

The generation of the frame image VR 4 is performed by the virtual reality image generation unit 210 shown in FIG. 5. In addition, the virtual reality image generation unit 210 is transmitted through wireless communication from the experience fire extinguisher 20 (that is, the experience fire extinguisher 20) through the second communication unit and the HMD 10 first communication unit, the HMD control unit 200 ), based on the detection information and elapsed time information of the experience fire extinguisher 201 sensor (S1) and the second sensor (S2), the virtual reality fire extinguishing agent (5) from the end (28) of the hose (26) The sprayed image VR (2) creates an image VR (3) in which the spark 7 of virtual reality calms down (details will be described later).

As shown in Figs. 1(a) and (b), the fire extinguishing experience simulation system using VR (virtual reality) of the present embodiment is an experience fire extinguisher 20 (controller of the HMD 10) imitating a fire extinguisher that is actually used. Use the user's hand 3 (left hand 3L right hand (3R)) to perform the operation of lever (24) or changing the direction of the end (28) of the hose (26) as in actual fire extinguishing activity can do. Therefore, the experienced person (U) can get used to the actual fire extinguisher.

In addition, not immediately entering the virtual reality space, as shown in Fig.1 (a), training first taken with a camera of a real image (for example, a mobile terminal 16 such as a smartphone constituting the HMD 10) The actual image including the fire extinguisher) is displayed on the display, and the experienced person (U) checks the location of the training fire extinguisher 20 and his/her hand (right hand (3R)), and then in the confirmed real state, virtual reality (VR) It is possible to start a simulation of fire fighting activity in space.

Therefore, it is possible to extinguish the flame in the same way as the actual extinguishing activity step while sharing the sense of reality. Therefore, the experienced person (U) can experience the actual stage of digestion activity with a sense of reality while sharing a sense of reality.

In addition, the main components of the fire extinguishing experience simulation system shown in Figs. 1 (a) and (b) are the fire extinguisher 20 and HMD 10 for experience (and a monitor 30 is provided as needed). It is simplified. Therefore, it is possible to easily implement a simulator with excellent mobility (portability setting) and simplicity (for example, a point that can be performed easily indoors without individual support from staff).

The fire extinguishing experience simulation system using VR (virtual reality) of the present embodiment operates the actual tools of the experience fire extinguisher 20, unlike the existing immersive game equipment, It is possible to experience a realistic fire extinguishing activity through the perspective of a VR image in a virtual reality space that is changed in conjunction with each other, and it can be said to be a virtual fire fighting experience simulator accompanied by a real reality.

Next, referring to FIG. 2, FIGS. 2(a) to 2(d) show that each experiencer fixes the wrist rest of the hand holding the end of the hose and rotates the wrist around the point, at the end of the hose. It is a diagram showing an example of changing the direction of an injecting extinguishing agent.

As described in Fig. 1(b), the spray direction of the virtual reality extinguishing agent 5 sprayed from the end 28 of the hose 26 is the hand of the experienced person U holding the end 28 of the hose 26. It changes with movement. The movement of the hand is detected by the second sensor S2 of the end 28 of the hose 26 of the fire extinguisher 20 for experience. For example, if you use a 6-axis sensor (a composite inertial sensor, an object that senses movement in the X, Y, and Z axis directions and rotation around each axis θX, θy, θZ) as the second sensor S2, Complex hand movements can be detected in detail.

Of course, such a detection method may be used in the present invention, but in this case, since there is a lot of detection information, the image processing for generating a simulated image is complicated and the processing capability of the mobile terminal 16 constituting the H MD10 is exceeded. It is also assumed, and it is undeniable that it is expensive when providing a powerful mobile terminal. In order to promote the dissemination of the fire extinguishing experience simulation system, it is desirable that it can be easily used by many ordinary people at as low cost as possible.

In this respect, in the example of FIG. 2, in consideration of the reduction of the burden of image processing, the HMD control unit 200, that is, the virtual reality image generation unit 210 of FIG. 5, ends the hose 26 displayed in the virtual reality space. Hold (28) and fix the position of the hands of the experienced person (U), for example, the right hand (3R)), and the wrist rest (K-1), and according to the detection signal from the second sensor (S2), the wrist rest (K-1) ) To the center (specifically, by detecting the variation caused by the rotation of the wrist centered on the point (K-1)), change the hose 26 and the end 28 in the virtual reality space.

Figure 2 (a) corresponds to the virtual reality fire extinguishing agent 5 and the virtual reality flame 7 ejected from the right hand (3R) and the end 28 of the hose 26 in the virtual reality space up and down (vertical direction) It shows an image when viewed along the Z-axis (VR image viewed from the plane viewed in the Z-axis direction). 3(b) shows a state in which the VR image is projected on the monitor 30.

As can be seen from Figs. 2(a) and (b), the position of the wrist rest K-1 of the right hand 3R in the virtual reality space is fixed, so even if the right hand 3R moves in the real space, the virtual Movement in real space is not reflected and is always at the same position as the point (K-1). In the example of Fig. 3(a), the rotation (rotation) around the Z-axis, that is, the rotation (rotation) clockwise/counterclockwise from the left and right as viewed from the experienced person (U) is the experience fire extinguisher 20 sensor (S2). ).

FIG. 2(c) is a VR image viewed from a plane as seen in the X-axis direction corresponding to the left-right direction of the image as in FIG. 2(a), and FIG. 2(d) shows the VR image being projected on the monitor 30 It is a drawing showing the state that there is. Likewise, the point (K-1) is fixed, and the rotation (rotation) around the X-axis, that is, the rotation (rotation) clockwise/counter-clockwise in the vertical direction as viewed from the experienced person (U) is a fire extinguisher for experience (20). ) It is detected by the sensor S2.

The motion detected in this way is transformed by rotation (rotation) in the Z-axis and X-axis directions (of course, it is possible to detect the motion in the other axis. In this case, the sensing information other than the motion in the two-axis directions above is the image) It is not used for generation) to reduce the processing burden of the HMD control unit 200.

In other words, by holding the end 28 of the hose 26 displayed in the virtual reality space, the position of the wrist rest K-1 of the right hand 3R of the user's hand is fixed, and the point K-1 is the center. For example, by measuring the amount of displacement due to the rotation of the wrist with an angular velocity sensor (gyro sensor, etc.), the direction of the end of the hose 26 can be specified, while the hand depth direction (the Y-axis direction, which is the front and rear direction) Therefore, since it is not necessary to consider the movement (translation) according to the movement (translation) and the vertical direction (vertical direction), the detection burden of the second sensor S2 can be reduced, and the virtual reality image generation of the HMD controller 200 The burden of image processing for image generation by the unit 210 (see FIG. 5) is also reduced.

In addition, the basics when using a fire extinguisher are the movement of the hand (right hand (3R)) wrist with the end of the hose (26) while spraying the flame (7), for example, vigorously towards the center (5) at an appropriate distance. By adjusting the direction of the ejection, that is, without unnecessary movements such as large movements of the arm and body, the fire extinguishing agent can be accurately matched to the fire, and this basic movement can be experienced reliably. Therefore, the experienced person (U) is actually useful and can learn the basics of digestive activities.

Next, refer to FIG. 3. 3(a) to 3(d) are diagrams for explaining an example of a fire extinguishing experience step using VR and a location where a flame in virtual reality is visible, respectively. In FIG. 3, objects such as those in the above drawings are indicated by the same reference numerals. In addition, this point is the same also in other drawings.

In Fig. 3(a), the experienced person (U) equipped with the HMD 10 first pulls out the safety pin 13 of the experience fire extinguisher 20, and then holds the hand (here, the right hand 3R) on the lever 24 (not holding it. Therefore, the detection signal of the first sensor S1 is off), and the end 28 of the hose 26 is held in the other hand (here, the left hand 3L).

As described above, in the state of FIG. 3 (a), the mobile terminal 16 acting as a goggle displays an actual image captured by the camera (for example, see the monitor image in FIG. 1 (a)), and the experienced person (U) Can be confirmed by admitting the experience fire extinguisher 20 or the location of one's hand.

Next, the HMD 10 is switched from the real image display mode to the virtual reality image display mode. 3(b) shows an image before the fire extinguishing activity in a virtual reality space (for example, an image around an octopus-type wiring in an office that is burning). The distance from the experiencer (U) to the flame (7) is L0. Then, the video approaches and stops next to the experiencer (U).

In this case, the distance from the experienced person (U) to the flame (7) is L1 (L1 <L0).

Approaching the firearm image to the experienced person (U) is, as described above, when the movement of the hand holding the end of the hose 26 (28) in the depth direction (forward and backward) is not considered, in the experienced person (U), fire fighting It may be difficult to grasp the sense of distance to the target flame (7).

Taking this into consideration, the firearm image (the image of the firework superimposed on the background image) in the virtual reality space is made accessible to the experienced person (U) at an appropriate timing. 7) by moving forward (ie, moving to an appropriate distance) so that you can get a sense of it.

For example, when switching from the actual image display mode to the virtual reality image display mode, the fire extinguishing activity does not start immediately, but the firearm image is approached to the experienced person (U), and the experienced person (U) first reaches the flame (7) to be extinguished. You can learn the basics of fire extinguishing that it is important to approach the distance that the Korean fire extinguishing agent (5) reaches.

Next, a simulation image of the fire extinguishing activity is displayed as shown in FIG. 3(c), and a picture showing the appearance of the flame 7 over time by spraying (spray) of the fire extinguishing agent 5 in the virtual reality space is shown. .

Next, as shown in Fig. 3(d), the flame 7 calms down and becomes small by a predetermined degree (for example, the size of the flame becomes about 30% of the initial state) and the flame 7 calms down and becomes smaller by a predetermined degree. The video after the fire extinguishing activity, including the, comes closer to the experienced person (U) and stops. It is important to extinguish the flame (7) more closely without being vigilant that the flame (7) has become small, and to approach the image including the embers at the time when the flame (7) has subsided to a certain level by conducting extinguishing activities. You can learn the basics. In time, the flame is completely extinguished and the experiential simulation ends accordingly.

In this way, even when a simplified configuration that detects hand movement in the depth direction (forward and backward) is adopted, the experienced person (U) can reliably learn the precautions for fire extinguishing operation in the depth direction (forward and backward). have.

Next, refer to FIG. 4. 4(a) to 4(E) are diagrams each showing an example of a video visually recognized by the experiencer and the audience (including the leader) during a fire extinguishing experience using VR. FIG. 4 shows an image displayed on the HMD 10 (display of) on the left and an image displayed on the monitor 30 on the right.

In Fig. 4(a), a search image is displayed when starting the simulation. In Fig. 4(b), first, the real image (real image) shown in Fig. 1(a) is displayed.

In FIG. 4(c), the actual image (real image) is converted to a VR image (virtual image). The HMD 10 displays an image representing an ember (Figure VR 2L showing the left eye flame 7L, and Figure VR 2R showing the right eye flame 7R). The monitor 30 shows the figure VR 2 showing the flame 7.

4(d) shows that the HMD 10 displays a simulation image (ie, left eye frame image VR(FL), right eye frame image VR(FR)) during fire extinguishing activity. Monitor 30 frame image VR(F) is Is displayed.

4(E) is a simulation image (ie, left-eye frame image VR (FL), right-eye frame image VR (FR)) in a state in which the flame 7 calms down and becomes about 30% of the initial size in the HMD 10. Is displayed The monitor 30 frame image VR(F) is displayed.

Next, refer to FIG. 5. 5 is a diagram showing an example of the overall configuration of a fire extinguishing experience simulation system using VR (virtual reality). The configuration of the main body (HMD main body) 101 of the HMD 10 is shown on the right side of FIG. 5, and the configuration of the HMD 10 controller (HMD controller) 200 is shown on the left side of FIG. 5.

The HMD main body 101 includes an image processing unit 107 and an external camera (image camera) that photographs the outside 103 a display driving unit 109 a display area 16L for the left eye 1L of the display and a display area for the right eye 1R of the display. (16R) and headphones (14) (including left ear speaker (1L) and right ear speaker (1L)) Experienced person (U) My camera (eye detection camera) that detects the movement of the gaze (view point) (13) Sensor unit ( There is 15). The sensor unit 15 includes a gaze detection unit 17 and a motion detection unit 2 and 1 that detects a hand movement according to a sensor signal from an inertial sensor 19 including a gyro sensor and an acceleration sensor.

In addition, the sound of burning fire during the simulation of the fire extinguishing experience through the headphones 14 is output to the ear of the experienced person (U), and through this, the experienced person (U) can experience a more realistic fire extinguishing activity.

In addition, the eyes captured by the internal camera 13 may be analyzed by the gaze detector 17 to detect the gaze position (view point position) of the experienced person U. For example, it is possible to present an alternative such as "Stop simulation, run away without firefighting" on the display of the HMD 10 and execute the gaze of the experienced person U and a desired motion.

A control unit (HMD control unit) 200, a virtual reality image generation unit 210, a real image/virtual reality image conversion unit 212, a storage unit 220, an evaluation unit 230, a timer 240, an image signal generation unit 250 ), an audio processing unit 260 and an audio signal generation unit 270.

A control unit (HMD control unit) 200 A guide image generation unit 21 and a panorama image generation unit 213 for generating a search (guide) image as shown in Fig. 4(a), and an image generation unit of the end of the hose and the wrist It is the fire image of 214, the fire extinguishing agent spraying (squirting) image generation part 216. It has a combustion image generator (ie, flame simulator 216) that generates a combustion image. In addition, the panoramic image will be described later.

Here, the real image/virtual reality image conversion unit 212 switches the real image display mode/virtual reality display mode in the HMD 10, and in the case of the real image display mode, the real-view imaging camera (external camera, 103) The actual image is displayed on the display of the HMD 10.

In addition, when the HMD 10 is switched from the actual image display mode to the virtual reality image display mode, the virtual reality image generation unit 210 includes a fire extinguisher for experience 20 (a second communication unit) and a communication unit (first communication unit) of the HMD. Each detection information of the experience fire extinguisher 201 sensor (S1) and the second sensor (S2) input through 300 (the lever ON/OFF signal shown in Fig. 5) is applied to the rotation (rotation) around the X-axis and Z-axis. Depending on the information θX, θZ) representing the variation caused by the virtual reality, the video (Fig. 1(b), the image VR(2)) and the virtual reality are injected from the end 28 of the hose 26 held by the user U. It operates to display a simulation image (frame image VR(F) in FIG. 1(b)) including an image where the flames of reality calm down (image VR(3) in FIG. 1(b)).

In addition, the storage unit 220 is made of RAM, ROM, EEP ROM, and the like, and includes image generation data 221 and a system program 222 and an application program 223.

The application program 223 is a program that functions as a control unit (HMD control unit) 200 with a computer (CPU, MPU) equipped with an HMD 10 of a fire extinguishing experience simulation system using VR. Therefore, it is necessary to load the application program 223, the HMD 10 into a computer (CPU, MPU) and prepare the components of the fire extinguishing experience simulation system using the HMD 10 and VR, and a dedicated wearable computer (HMD) for water use. none. Therefore, it makes it easy to use by many general users. This can contribute to the spread of fire extinguishing experiences using VR.

In addition, as described above, it has a control unit (HMD control unit) 200 and an evaluation unit 230. After the firefighting activity in the virtual reality space is finished, the evaluation unit 230 evaluates the firefighting activity of the experienced person U according to a predetermined criterion, and displays information indicating the evaluation on the display of the HMD 10.

For example, an evaluation by the evaluation unit 230 (eg, “evaluation result” and “evaluation indicating strengths”) after the fire extinguishing experience is completed is displayed on the display of the HMD 10 and the display surface of the monitor 30 .

Specifically, the evaluation is made according to, for example, the time taken for digestion, the direction of injection of the digestive agent, etc., so that the experienced person (and the spectators) can have a really helpful digestion experience without seriously affecting the game sense. In addition, it can motivate the experienced person (U) who can give evaluation to actively participate in fire extinguishing activities using VR, and contributes to the spread of the system.

Next, referring to FIG. 6, FIGS. 6(a) and 6(b) are diagrams for explaining the display of a panoramic image, and FIG. 6(c) is a diagram illustrating a basic configuration example of a flame simulator. First, generation of a panoramic image will be described.

As described with reference to FIG. 5, the virtual reality image generation unit 210 includes a panorama image generation unit 213. For example, the panoramic image generating unit 213 generates and displays a panoramic image that the experienced person U can see as a background image included in the image before the fire extinguishing activity and the background image included in the simulation image. have.

Here, the image displayed in the virtual reality space is not a flat image, but a panoramic image with a wide field of view (for example, a 360-degree panoramic image) that the experiencer can look around, allowing the experiencer (U) to experience immersive digestion activities. have. In addition, the panoramic image is not only a pole (or partial circumference) type, but also a full celestial shape, such as covering the entire field of view from the left and right heads to feet, as well as a full circumference of the sky, such as covering the upper half of the field of view on a dome (hemispherical) screen such as a planetarium. You can appropriately adopt the video.

In addition, if a separate image with a proper parallax can be provided for the left and right eyes of the experienced person (U), a panoramic image of the entire celestial sphere can be provided. However, for example, when realizing a three-dimensional image using a parallax image, the burden of image processing increases, so it is necessary to perform the panoramic image generation process within a range that does not burden the control unit (HMD control unit) 200. .

6(a) shows that the hemispherical coordinate space (MP) is set, and the eyes (1) of the experienced person (U) (the right eye (1R) supports a wide viewing angle at a position separated by a reference distance (L) from the left (1L)). A curved virtual screen (image projection area) SE is set, for example, in the left-right direction according to the movement of the head of the experience user U (or movement of the eyes described in Fig. 5). By rotating (rotating) it, it is possible to realize a display by a panoramic image of a wide field of view that the experienced person U can see in the horizontal direction.

In addition, the movement of the head Ha of the experienced person U can be measured by, for example, an inertial sensor mounted on the main body 101 of the HMD 10 shown in FIG. . For example, using a 6-axis sensor (a composite inertial sensor that detects movement in the X, Y, and Z axis directions and rotation around each axis θX, θy, θZ) details the movement of the head (Ha). It is possible to detect.

In addition, the combustion image generating unit (flame simulator) 216 described in FIG. 5 has, for example, the same configuration as in FIG. 6(c). That is, the flame fragment database 501 created based on the combustion model has a combustion image data generation unit 503 and a timer 505, so that the combustion image data generation unit 503 receives sensor information from the experience fire extinguisher 20 (Detection signals from sensors (S1, S2)) and the timer 505 in consideration of the elapsed time information measured by accessing the flame fragment database 501 and synthesizing the required size of spark fragments, the image of the flame (7) that changes every moment Is created.

As described in FIGS. 2(a) and 2(b), the direction of ejection of the fire extinguishing agent 5 depends on the wrist rotation (rotation) of the hand (right hand 3R) holding the end 28 of the hose 26 However, since the spread of the jet is known in advance, if the direction of the jet is determined, it is possible to predict how much the extinguishing agent per unit time fits within the range of the flame 7. According to this prediction result, combustion image data can be generated using the above configuration.

Next, referring to FIG. 7, FIG. 7 is a flowchart illustrating an example of a control method of a controller (computer) of an HMD in a fire extinguishing experience simulation system using VR (virtual reality). In addition, the steps shown in FIG. 7 must first follow the simulation procedures shown in FIGS. 3 and 4.

The controller of the HMD (computer 200 operating according to the program) first displays a guide (instruction) image such as a job description (step S10), and then, an image captured by the external camera 103 of FIG. 5 (for example, For example, an experiential fire extinguisher 20 as shown in Fig. 4B and a captured image around the wrist) are displayed (step S20).

Next, when the fire extinguishing experience using VR starts (step S30), a VR video image (for example, a diagram showing the fire phenomena before the fire extinguishing activity, as shown in Fig. 4(c) before fire fighting) is displayed (step S40 ) Next, for example, it is determined whether the experienced person U selects the start of the digestive activity (step S50). Here, for example, the HMD 10 displays, for example,'run away' and'start'. Three alternatives are presented and the experiencer (U) selects one of the two.

In the case of YES in step S50, as described in Fig. 4(b), the firearm image is controlled to approach (approach) the experienced person U so that the depth can be felt (step S60), and the experienced person U ) The presence or absence of a lever operation (lever 24 ON/OFF) is detected (step S70).

In the case of YES in step S70, as described in Fig. 4(c), time is reduced by the display of the VR image discharging the hose 26, the end 28, the extinguishing agent 5, and the extinguishing agent 5 sprayed. A fireworks VR image that is evolving over time is displayed (step S80), and in parallel with this, the same VR image is transmitted to the monitor 30 and displayed on the monitor 30 (step S90).

Next, as described in Fig. 4 (d), when the flame 7 is evolved to a size of about 30% or less of the initial state (step S100), in order to make the experienced person U feel more realistic, the image of the furnace Is brought closer to the experienced person (step S10).

When the flame is extinguished (step S120), evaluation (generating an evaluation result) is performed, and the evaluation (evaluation result) is displayed on the display of the HMD 10 and the display surface of the monitor 30 (step S130). , For example, the time limit for fire extinguishing activities can be evaluated as 30 seconds and the time to completion of the fire. For example, if it can be digested within 5 seconds, it is evaluated as 100 points, and if it is more than 15 seconds and less than 20 seconds, it is evaluated as 70 points.

As described above, according to the fire extinguishing experience simulation system, program and experience fire extinguisher using VR of the present invention, it is excellent in mobility and convenience, so that it is possible to receive appropriate guidance from leaders so that the fire fighting experience can be performed in the same order as the actual fire fighting activity. A new fire fighting experience simulator can be realized.

In addition, it is possible to motivate the experience of fire extinguishing by giving a score according to the fire extinguishing time and the direction of injection, etc. In addition, the available time of extinguishing agent injection is the same as that of a real fire extinguisher, but also by approaching the feeling of use of the actual fire extinguisher, it can actually provide helpful fire extinguishing training.

It can also be used by elementary school students or younger, such as using goggles, and through this, it becomes a simulator that can be used regardless of age. You can also share this experience with the audience (audience) and experiencers using the monitor.

In addition, for example, you can experience fire extinguishing actions in a virtual reality (VR) space, such as approaching a fire up to a distance of 3 to 5 meters, starting a fire extinguishing activity, and getting closer to the fire and extinguishing it completely as the fire extinguishing activity progresses and the flame decreases. It can be (Fig. 4(b) to Fig. 4(d)).

In addition, by configuring the mobile terminal constituting the HMD 10 without connecting to the Internet, it is possible to prevent security degradation.

In the above, the present invention has been described according to the embodiments, but the present invention is not limited to the above embodiments, and various modifications and applications are possible. For example, the background image and the flame image appearing in the virtual reality space can be selected from multiple images, for example, by regularly updating the image to be selected, a better situation can be set and practical firefighting activities can be easily performed. There will be.

The present invention is not limited to the above-described exemplary embodiments, and those skilled in the art can easily change the above-described exemplary embodiments to the scope included in the claims.

Claims (4)

A head-mounted display (HMD) mounted on an experienced person with a display for displaying an image, a computer, and a camera for real-view images that perform communication processing with a first communication unit, and
The second communication unit and the fire extinguisher body to communicate between the first communication unit of the HMD, and in the HMD virtual reality (VR) mode, an operation lever serving as a switch to inject a virtual reality fire extinguishing agent and the virtual reality fire extinguishing agent are terminated. An experiential fire extinguisher serving as the HMD controller with a hose functioning as an ejection device ejecting from, a first sensor detecting operation of the operation lever by the experienced person, and a second sensor detecting movement of the end of the hose. And, the computer of the HMD is configured with a control unit that controls the operation of the HMD.
The control unit includes a real image/virtual reality image conversion unit that switches a real image display mode/virtual reality display mode and, in the case of the real image display mode, displays a real image captured by the real-view image camera on the display,
When the HMD is switched from the real image display mode to the virtual reality image display mode, based on each detection information of the first sensor and the second sensor of the experience fire extinguisher input through the second and first communication units And a virtual reality image generator for displaying a simulation image including an image in which the fire extinguishing agent of the virtual reality is sprayed from the end of the hose held in the hand of the experiencer and an image in which the flame of the virtual reality is calmed down. Fire extinguishing experience simulation system using VR. The method of claim 1,
And a third communication unit for communicating between the first communication unit of the HMD, and image data of the real image in the real image display mode input through the first and third communication units, and in the virtual reality image display mode A fire extinguishing experience simulation system using VR, characterized in that it further comprises a monitor that displays an image as a monitor image as the user sees through the display of the HMD according to image data of a virtual reality image. The method according to claim 1 or 2,
The virtual reality image generation unit holds the end of the hose displayed in the virtual reality space and fixes the position of the wrist rest of the hand of the experiencer, and centers on the point of the wrist according to the detection signal of the second sensor. Fire extinguishing experience simulation system using VR, characterized in that changing the direction of the tip of the hose in the virtual reality space. The method of claim 3,
When the virtual reality image generator is switched from the real image display mode to the virtual reality image display mode, after displaying the image before the fire extinguishing activity in the virtual reality space, the image before the fire extinguishing activity approaches and stops. After the control is performed, the simulation image is displayed, and when the flame calms down and decreases to a predetermined degree, the image after the fire extinguishing activity including the flame that is calmed down and reduced to a predetermined degree is controlled to come closer to and stop the user. Fire extinguishing experience simulation system using VR.

Images