KR20220036016A - Solution for making of art gallery employing virtual reality - Google Patents

Abstract

The embodiment relates to an art gallery implementation solution using virtual reality.
Specifically, this solution includes at least one artist terminal that receives virtual work videos from multiple different artists or directly produces and supplies them. In addition, the solution includes a management information processing device that receives each virtual work video provided by the artist terminal in an integrated manner, categorizes and manages each artist and company, and distributes them differently to each company distributed in various locations. .
In addition, the solution includes at least one company terminal that individually receives the virtual work images distributed by the management information processing device, virtualizes them, and displays them in a pre-registered digital picture frame within each company.
In this case, the management information processing device a) groups the virtual work images so that at least one virtual work image of a plurality of different authors is included. And, b) each virtual work video is sold to a company to earn profit. c) At this time, the profit is divided into the first profit paid to the author of the virtual work video, the second profit paid by the brokerage, and the third profit excluding the first profit and the second profit, and the third profit is The profit is settled to be distributed to the artists belonging to the relevant work group.
Therefore, when viewing a work of art, etc. through this, a virtual reality image corresponding to the viewer's reaction to the work is provided, making the viewing of the work of art more effective.
In addition, virtual reality images are provided to coffee shops or restaurants, where these VRs are displayed through digital frames.
Therefore, this further solves the problem that while most of the VR industry is focused only on the gaming industry and competition is fierce, the use of VR in the art field is somewhat low.

Description

Solution for making of art gallery employing virtual reality}

The content disclosed in this specification relates to the field of technology applying virtual reality, and more specifically, to technology in which artists, etc., produce virtual work videos and display these virtual work videos in art galleries, coffee shops, restaurants, etc. .

Unless otherwise indicated herein, the material described in this section is not prior art to the claims of this application, and is not admitted to be prior art by inclusion in this section.

Recently, as the spread of Internet communication networks and computers has rapidly spread, the field using computers is not limited to corporate computerization work but is rapidly expanding to all industrial fields. This trend of the information revolution is no exception in the cultural aspect, and it has become possible to visit the world's leading art museums online and enjoy works of art through the Internet while sitting in one's room.

Meanwhile, the VR industry, which has recently become an issue, is also applied to the art field, but most of it is focused only on the game industry, so competition is fierce, but the use of VR in the art field is somewhat low.

On the other hand, as the exhibition and convention industry develops in general, many types of exhibitions are being held according to various cultural needs. The general form of viewing is that the exhibits are simply displayed according to the will of the organizer, and the audience participates in the form of viewing from a passive position.

In addition, various types of exhibitions and viewings observed by others are shared online in various forms of video, audio, and written information, providing a type of indirect experience.

Due to the development of the exhibition convention industry and diverse cultural needs, exhibitions and expositions are held to show various types of content in various industries. And the number of viewers who use it is increasing day by day, not only for the satisfaction of personal needs, but also for the participation or experiential learning of children and students.

However, events such as various exhibitions and fairs so far have been conducted in such a way that viewers simply watch passively. In other words, it was a viewing method in which the organizer displayed or displayed and screened the exhibition content at will, and the viewer unilaterally accepted it.

Therefore, in a situation where exhibition culture is rapidly developing, there is an urgent need for a viewing method in which viewers actively produce content, moving away from the passive viewing method. This is because not only does it satisfy the needs of viewers, but the economic ripple effect from recycling the resulting UCC (User Created Contents) is enormous.

The prior art document against this background is the patent document below.

(Patent Document 1) KR101317047 Y1

(Patent Document 2) KR101893431 Y1

For reference, Patent Document 1 above is about emotion recognition technology using facial expressions, and Patent Document 2 is about an experiential art museum operation system using smart devices.

The disclosed content is an art gallery using virtual reality that solves the problem that most of the VR industry is focused only on the gaming industry, so competition is fierce, but the use of VR in the art field is somewhat low, and allows viewing of art works, etc. more effectively. We want to provide implementation solutions.

Additionally, the solution allows visitors to experience exhibitions and convention events held in various forms by directly producing content rather than simply viewing them, thereby breaking away from the traditional passive viewing and satisfying the needs of viewers. Of course, we aim to mass produce indirect experiential content through recycling of the produced UCC and achieve economic effects accordingly.

In addition, as mentioned above, the solution solves the underutilization of VR in the art field and provides VR to coffee shops or restaurants, where such VR can be displayed through a digital picture frame.

An art gallery implementation system using virtual reality according to an embodiment is,

An artist terminal that receives virtual work videos from multiple different artists or produces and supplies them directly;

a management information processing device that mediates the virtual work video supplied by the artist terminal for each artist and gallery, matches the virtual work video for each art with a plurality of different emotional states, and distributes the virtual work video to each different gallery; and

a gallery terminal that transforms the virtual work video distributed by the management information processing device into virtual reality and displays it in each gallery; It includes,

The gallery terminal is,

a) When viewing a work corresponding to each of the plurality of different works of art, the viewer's face is continuously recognized,

b) detecting the visitor's emotional state matched to the change state of facial elements in the recognized visitor's face,

c) A different matched virtual work video according to the detected emotional state of the visitor is produced around the corresponding work for each art,

Detecting the emotional state of the visitor (b)),

b-1) Extract metadata of each facial element from the recognized visitor's face for each facial element,

b-2) Estimate the size and location of the face element from the metadata of each extracted face element,

b-3) Detecting the change state of facial elements by applying the size and position of each estimated facial element,

b-4) detecting a visitor emotional state matched to the detected facial element change state; It is characterized by including.

A gallery implementation system using virtual work videos in a service company according to another embodiment,

At least one artist terminal that receives virtual work videos from multiple different artists or produces and supplies them directly;

A management information processing device that receives each virtual work video provided by the artist terminal in an integrated manner, categorizes and manages them through mediation by artist and company, and distributes them differently to each company distributed in various locations; and

At least one company terminal that individually receives the virtual work images distributed by the management information processing device, virtualizes them and displays them in a pre-registered digital picture frame within each company; It includes,

The management information processing device,

a) Group virtual work videos to include at least one virtual work video from multiple different artists,

b) Each virtual work video is sold to a company to earn profit, and

c) The profit is divided into the first profit paid to the author of the virtual work video, the second profit paid by the brokerage, and the third profit excluding the first profit and the second profit, and the third profit It is characterized by settlement for distribution to the artists belonging to the relevant work group.

According to embodiments, when viewing a work of art, etc., a virtual reality image corresponding to the viewer's reaction to the work is provided, making the viewing of the work of art more effective.

In addition, virtual reality images are provided to coffee shops or restaurants, where these VRs are displayed through digital frames.

Therefore, this further solves the problem that while most of the VR industry is focused only on the gaming industry and competition is fierce, the use of VR in the art field is somewhat low.

And, through scientific support for young artists, we create a social atmosphere in which art and science convergence culture can be easily accessed through digital art exhibitions based on digital technology in the digital era.

In addition, we will revitalize digital cultural content and create convergence content jobs by discovering various fields of scientific convergence.

Additionally, through the VR art exhibition experience, convergent thinking about the interactive relationship between digital and humans is promoted among young people who are defined as digital nomads.

In addition, in various viewings, viewers can participate directly and create their own content, thereby maximizing the direct experience by making it a practical experiential viewing rather than a simple viewing. And organizers of exhibitions can maximize the effectiveness of exhibition promotion by increasing the actual experience of viewers. Additionally, the final generated UCC can be used as indirect experience content for third parties, which has an economic effect.

1 is a diagram schematically showing a system to which a method for implementing an art gallery using virtual reality according to an embodiment is applied.
Figure 2 is a block diagram showing the configuration of an information processing device for providing exhibition services applied to the system of Figure 1.
FIG. 3A is a flow chart sequentially showing a method of implementing an art gallery using virtual reality according to an embodiment.
Figure 3b is a diagram showing the 3D space and model applied to the implementation method of Figure 3a
Figure 4 is a conceptual diagram of a gallery implementation system using virtual work videos in a service company according to another embodiment.
Figure 5 is a diagram illustrating the system described in Figure 4 as a whole.

Figure 1 is a diagram schematically illustrating an art gallery implementation system using virtual reality according to an embodiment.

As shown in Figure 1, the system according to one embodiment is a business in which various artists directly produce VR content, provide it in various forms through galleries, etc., and license each VR content to generate profit. It is related to the model.

In this case, the VR content is produced as follows, for example.

In other words, the VR content is required because there is a need to give more value than the paintings drawn by artists themselves.

For example, VR (virtual reality) and AR (augmented reality) technologies, as examples of art content, are combined with trick art content to enable dynamic and dynamic expression by viewing it through HMD devices. Alternatively, artists may directly draw VR content using VR content authoring tools.

So, ordinary people can experience augmented reality through new trick art content by combining 3D images with the current picture and calculated coordinates.

A model that uses such VR content is, for example, when a viewer stands in front of a work of art in a gallery, the viewer's face is recognized, their emotional state is identified, and a virtual reality image matching the viewer is produced.

And, the system of one embodiment produces content such as a movie using these virtual reality images. In this case, the movie is provided as experiential content beyond a simple passive viewing in an art museum.

For this purpose, the system includes, for example, an artist terminal, a management information processing device, and a gallery terminal.

In this case, the artist terminal receives virtual work videos for each different artist or directly produces and supplies them.

In addition, the management information processing device mediates the virtual work video supplied by the artist terminal for each artist and gallery, and distributes the virtual work video to each different gallery by matching a number of different emotional states and art.

Additionally, the gallery terminal transforms the virtual work video distributed by the management information processing device into virtual reality in each gallery and displays it.

As a practical example, the management information processing device includes a user terminal 100, an exhibition service provision information processing device 300, a CCTV 400, a reader 500, an output device 600, an RFID tag 700, and a guide. Includes a robot 800 and a beacon 900.

Figure 1 is a diagram showing such a management information processing device.

At this time, each component of FIG. 1 is generally connected through a network (network, 200). For example, as shown in FIG. 1, the user terminal 100 is connected to the exhibition service providing information processing device 300 through the network 200.

In addition, the exhibition service provision information processing device 300 uses the user terminal 100, CCTV 400, reader 500, output device 600, RFID tag 700, and guide robot ( 800) and beacon 900.

In addition, the CCTV 400 is connected to the exhibition service providing information processing device 300 through the network 200, and the reader 500 is also connected to the exhibition service providing server 300 through the network 200. Additionally, the output device 600, RFID tag 700, guide robot 800, and beacon 900 may also be connected to the exhibition service providing information processing device 300 through the network 200.

Here, the network refers to a connection structure that allows information exchange between each node, such as a plurality of terminals and information processing devices. Examples of such networks include RF, 3rd Generation Partnership Project (3GPP) network, LTE network, It includes 5GPP (5rd Generation Partnership Project) network, WIMAX (World Interoperability for Microwave Access) network, Internet, LAN, Wireless LAN, WAN, PAN, Bluetooth network, NFC network, satellite broadcasting network, analog broadcasting network, DMB network, etc. It is not limited to this.

More specifically, the user terminal 100 is, for example, a terminal of a visitor viewing an exhibition using an exhibition service-related web page, app page, program, or application. In addition, the user terminal 100 transmits location data including GPS of the user terminal 100 to the exhibition service providing information processing device 300, and provides content including guidance comments, text, images, videos, etc. corresponding thereto. It is a terminal that outputs. Additionally, as another example, the user terminal 100 is a terminal that directly outputs or overlays AR content or VR content streamed in real time from the exhibition service providing information processing device 300 on the screen. Additionally, the user terminal 100 may be a terminal that provides visitor information to the exhibition service provision information processing device 300 and receives and outputs customized guidance content corresponding thereto.

Here, the user terminal 100 is implemented as a computer that can connect to a remote information processing device or terminal through a network. At this time, the computer may include, for example, a laptop, desktop, laptop, etc. equipped with a navigation system and a web browser. Additionally, the user terminal 100 is implemented as a terminal that can connect to a remote information processing device or terminal through a network. In addition, the user terminal 100 is, for example, a wireless communication device that guarantees portability and mobility, and includes navigation, PCS, GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), Includes all types of handheld wireless communication devices such as PDA, IMT (International Mobile Telecommunication)-2000, CDMA-2000, W-CDMA, Wibro terminal, smartphone, smartpad, tablet PC, etc. can do.

The exhibition service providing information processing device 300 is a server that provides exhibition service web pages, app pages, programs, or applications. In addition, the exhibition service providing information processing device 300 includes a user terminal 100, CCTV 400, reader 500, output device 600, RFID tag 700, guidance robot 800, and beacon 900. ) monitors and supervises the output from ) and controls each component by transmitting the corresponding control signal. Additionally, the exhibition service provision information processing device 300 analyzes information collected from the user terminal 100 and transmits content or control signals corresponding to the analysis results. In addition, the exhibition service providing information processing device 300 performs lighting control for route guidance of viewers, guidance on at least one booth arranged in the exhibition hall, exhibition hall management, theft prevention, and exhibition lighting control. Alternatively, the exhibition service providing information processing device 300 collects the location of the user terminal 100 through GPS, RFID tag coordinates, beacon signals, etc., and then provides exhibition guide content to the user terminal 100. In addition, the exhibition service provision information processing device 300 monitors the movement control, position control, and guidance control of the guidance robot 800, and provides a recovery path to the guidance robot 800 when an error occurs, so that the manager recognizes the error. and correct the error of the guidance robot (800). In addition, the exhibition service providing information processing device 300 analyzes an image containing at least one frame streamed in real time from the CCTV 400, and controls object tracking when the analyzed image matches the previously stored image. In addition, the exhibition service provision information processing device 300 collects the inquiries entered by the guidance robot 800 and transmits them in real time to at least one booth management terminal (not shown) located in the area where the inquiries were collected. Collect answers.

Here, the exhibition service providing information processing device 300 is implemented as a computer that can connect to a remote information processing device or terminal through a network. The computer may include, for example, a laptop, desktop, or laptop equipped with a navigation system and a web browser.

FIG. 2 is a block diagram showing the configuration of an exhibition service providing information processing device applied to the management information processing device of FIG. 1.

As shown in FIG. 2, the exhibition service provision information processing device 300 according to one embodiment includes a motion sensor 301, a photographing means 302, a profile database unit 303, a content database unit 304, It includes a control unit 305, a display unit 306, and output means 307.

Here, the motion sensor 301 generates depth image information about the visitor, recognizes the visitor, and measures movement. Specifically, the motion sensor 301 includes an infrared source module and a CMOS infrared camera sensor module. Additionally, the CMOS infrared camera sensor module includes an image processor. The infrared source module radiates infrared rays toward museum visitors, and the CMOS infrared camera sensor module collects the reflected infrared rays and calculates the distance to the visitor to recognize the visitor. At this time, the CMOS infrared camera sensor module may include an image processor for calculating the distance to the visitor.

The profile database unit 303 stores a profile consisting of key information for recognizing visitors. Specifically, the profile database unit 303 stores a profile consisting of a plurality of key range information. That is, in order for the motion sensor 301 to recognize the visitor, a certain amount of time is required depending on the size (key range) of the visitor, and the error range is also large, so an appropriate key range is determined and the motion sensor 301 is used accordingly. Performing the recognition process allows for more accurate and faster visitor recognition. At this time, the profile database unit 303 stores key information necessary for the motion sensor 301 to recognize the visitor and provides related information as needed. Specifically, the motion sensor 301 can recognize the visitor based on the selection of the key range input by the visitor, and the selection process for the key range selected by the visitor is performed by the visitor selecting the key range through a predetermined input device. A selection input can be received, and the motion sensor 301 can effectively recognize the visitor based on this input information (selected key range).

The content database unit 304 stores VR panorama content and VR object content related to historic sites or artifacts. That is, the content database unit 304 stores historical site 3D background content, historic site panorama content, historic site information content, 3D three-dimensional shape content of the external appearance of the relic, relic information content, etc. related to the historic site or relics to be provided to visitors, and provides necessary information. Relevant information may be provided accordingly.

The display unit 306 displays the content of the content database unit 304 so that visitors can view it. Specifically, the display unit 306 may be of a screen type or image display type that can effectively provide VR content.

The control unit 305 controls the motion sensor 100 to recognize the visitor based on key information stored in the profile database unit 303. And the control unit 305 controls the display unit 306 to change and display the location and content of the content stored in the content database unit 304 according to the movement of the viewer. Additionally, the control unit 305 may control the display unit to display detailed content corresponding to the view point when the movement of the visitor recognized through the motion sensor 301 is a selection motion for the view point. In other words, if the movement of the visitor recognized by the motion sensor 301 in the pre-provided virtual reality (VR) panoramic content is a selection motion for a view point that is a specific part of the VR content, the control unit 305 moves to the corresponding view point. The display unit can be controlled to display the corresponding detailed content (enlarged image or description content).

The photographing means 302 creates a video image by photographing the visitor or the visitor's face, and the control unit 305 controls the display unit 306 to display the image captured through the photographing means 302 together with the content. In addition, the position of the content and the captured image can be controlled to change according to the movement (gesture) of the visitor measured through the motion sensor 301. That is, the control unit 305 displays the visitor video image obtained through the photographing means 302 on the VR content and VR object content basically provided to the visitor, and the motion sensor 301 recognizes the visitor's movement, and detects the movement. In the case of movement for the corresponding VR content and VR object content or movement for the captured image, the display state corresponding to each movement can be changed.

The output means 307 outputs related content and video images displayed by the display unit 306 in the form of photos, etc. That is, the output means 307 converts the related content and the captured video into an image according to the control operation of the control unit 305, and then, if the viewer selects, outputs the image in the form of a photo or the like and provides it to the viewer. Through this process, visitors who are provided with this system can take a commemorative photo by appropriately placing their image in a state where a virtual historic site or relic is displayed, and can take a commemorative photo as if they had visited the relevant historical site, etc. You can receive the same effect.

Additionally, it may further include a speaker for providing a description of the content to the visitor and a microphone for collecting the visitor's voice information, and the control unit 305 may be configured to control the visitor's movement or movement measured through the motion sensor 301. Keywords included in the voice information collected through the microphone are analyzed, and based on this, detailed explanations are controlled to be provided to the audience through the speaker. That is, the control unit 305 analyzes the visitor's voice information collected from the microphone by keyword and displays questions and answers or detailed explanations about the historic site or artifact included in the content being provided on the display unit 306. In addition, it can be controlled so that voice information is provided through a speaker.

Figures 3a and 3b are diagrams for explaining an art gallery implementation system using virtual reality (an example of a business model of the above-described VR content) according to an embodiment.

Specifically, FIG. 3A is a flow chart sequentially showing the operation of an art gallery implementation system using virtual reality according to an embodiment. And, Figure 3b is a diagram showing the 3D space and model applied to this system.

As shown in FIG. 3A, the art gallery exhibition system using virtual reality according to one embodiment first assumes the following configuration.

In other words, the above configuration includes an artist terminal that receives virtual work videos from a number of different artists or directly produces and supplies them, the virtual work videos supplied by the artist terminal are brokered by each artist and gallery, and a plurality of virtual work videos are provided for the virtual work videos. It includes a management information processing device that matches different emotional states and art such as art and distributes them to different galleries (S301).

In this case, for example, a total of 7 emotions are judged: anger, disgust, fear, happiness, sadness, surprise, and expressionless, and each emotion is segmented into high, medium, and low emotions. A total of 21 emotions are judged through discrimination.

As a specific example, the change values of the facial expression of the first facial image and the facial expression of the second facial image, for example, the change values of the corners of the eyes and mouth, which have different amounts of change depending on each emotion, are compared and analyzed to determine the high, It is judged as average or low.

Here, the amount of expression change in the facial image is changed by detecting the corners of the eyes and mouth of the first and second facial images, respectively, and matching the corners of the eyes and mouth of the first facial image and the corners of the eyes and mouth of the second facial image, respectively. The value is calculated, and if the change value is above a certain level, it is judged as high, if it is within a certain level, it is judged as normal, and if it is below a certain level, it is judged as low.

In addition, the configuration includes a gallery terminal that virtualizes the virtual work video distributed by the management information processing device and displays it in each gallery.

In this state, the system according to one embodiment continuously recognizes the viewer's face when viewing the work corresponding to each of the plurality of different works of art (S302).

Then, the visitor's emotional state matching the facial element change state in the recognized visitor's face is detected (S303).

Therefore, different matched virtual reality images for each detected emotional state of the viewer are produced around the corresponding work of art (see FIG. 3b) (S304). At this time, the virtual reality image is an AR or VR image that responds to the emotional state of the viewer. These videos are produced to suit each art exhibition theme or item.

In this case, the emotional state detection (S303) consists of the following operations.

First, the metadata of each facial element, that is, the attribute value, is extracted for each facial element from the recognized visitor's face.

Next, the size and position of the face element are estimated from the metadata of each face element, for example, by applying the YORO format. In other words, the overall size and position of each facial element are estimated from the immediate metadata of each facial element.

Then, the change state of the face element is detected by applying the size and position of each face element estimated in this way.

Therefore, the visitor's emotional state matching the detected facial element change state is detected.

Accordingly, through this, in one embodiment, when viewing a work of art, etc., a virtual reality image corresponding to the viewer's reaction to the work is provided, making the viewing of the work of art more effective.

As above, one embodiment has the following features.

That is, in one embodiment,

An artist terminal that receives virtual work videos from multiple different artists or produces and supplies them directly;

a management information processing device that mediates the virtual work video supplied by the artist terminal for each artist and gallery, matches the virtual work video for each art with a plurality of different emotional states, and distributes the virtual work video to each different gallery; and

a gallery terminal that transforms the virtual work video distributed by the management information processing device into virtual reality and displays it in each gallery; It includes.

And, in this case, the gallery terminal,

a) When viewing a work corresponding to each of the plurality of different works of art, the viewer's face is continuously recognized,

b) detecting the visitor's emotional state matched to the change state of facial elements in the recognized visitor's face,

c) A different matched virtual work video according to the detected emotional state of the visitor is produced around the corresponding work for each art,

Detecting the emotional state of the visitor (b)),

b-1) Extract metadata of each facial element from the recognized visitor's face for each facial element,

b-2) Estimate the size and location of the face element from the metadata of each extracted face element,

b-3) Detecting the change state of facial elements by applying the size and position of each estimated facial element,

b-4) Detect the visitor emotional state matched to the detected facial element change state.

Accordingly, through this, in one embodiment, when viewing a work of art, etc., a virtual reality image corresponding to the viewer's reaction to the work is provided, making the viewing of the work of art more effective.

Therefore, this further solves the problem that most of the VR industry is focused only on the gaming industry, so competition is fierce, but the use of VR in the art field is somewhat low.

And, through scientific support of young artists, digital technology-based

Through digital art exhibitions, we create a social atmosphere where art and science convergence culture can be easily accessed.

In addition, the revitalization and convergence of digital cultural contents through the discovery of various fields of scientific convergence.

Create content jobs.

Additionally, through the VR art exhibition experience, convergent thinking about the interactive relationship between digital and humans is promoted among young people who are defined as digital nomads.

Additionally, the system according to this embodiment allows the production of a movie using the above-described virtual reality image information, for example, a movie whose content is the experience of viewing a work of art.

For this purpose, the system according to the embodiment has the following configuration.

In other words, the system receives the turning point in the beginning, progression, and conclusion of the story as an option for each interactive content in the exhibition hall according to the exhibition theme of each art through the corresponding mobile app from the user terminal.

In this case, the interactive content means registration of spectator ID and spectator information, scenario selection, video or photo shooting, CG produced by a computer installed in the exhibition hall, one's own comments, poly recording, and sounds such as selected BGM.

Next, the selected results are combined and systematically connected as a movie.

Then, the produced virtual reality image information (including the visitor's face and virtual reality image) is filled in the corresponding turning point or preset sound is recorded to produce the movie.

So, for example, the embodiment goes beyond simply allowing the viewer to passively view the exhibits when viewing an exhibition hall, and allows the viewer to actually directly produce video creations related to the exhibition theme and interact with the video creations stored for each topic. Let's exchange them to create an advanced form of UCC.

For example, the production of the above film occurs as follows.

First, visitors download and install the mobile app on their smartphones.

In this case, the mobile app is downloaded before visiting the exhibition hall, and information is obtained through advertisements, articles, etc. Then, by running this mobile app, you are guided through the exhibition hall structure, exhibition information, programs, etc.

Next, run the aforementioned mobile app on site.

Then, visit the exhibition hall. In an art museum, you can obtain information about each corner while viewing corners such as understanding of a specific work of art, the history of the work of art, and the principles of the work of art through the corresponding management information processing device.

So, it is a case of viewing as a simple method of obtaining information, but another method is possible for viewers to actively participate and produce content on their own when necessary. In other words, it becomes a direct experience type of viewing. Specifically, it involves creating video scenarios and creating content.

Next, create a video scenario. To this end, the viewer constructs a scenario for the video he or she will produce through, for example, guidance graphics installed in the scenario zone, guidance videos, and mobile app guidance pop-up windows installed on smartphones.

In this case, the scenario consists of the beginning of the story, progress (turning points 1 to n), and conclusion. For example, if you are planning to create and experience a video on the subject of historical exploration of a work of art, 1) the story begins in a format where the story progresses differently depending on the choice of historical exploration companion and historical exploration site. And 2) If the historical exploration site is selected as a specific area, the turning point of the story unfolds differently depending on the selection of the time, place, and situation in which the artwork was created. In other words, the audience can freely select and construct a story by combining the number of n cases. And 3) The conclusion of the story is to insert your own opinions and impressions about the movie you made.

The scenarios constructed in this way are systematically connected as a movie. At this time, the installed video has a blank space for each turning point, and the viewer creates and fills the video, sound, and audio for this part to complete the movie.

At this time, sounds and music can be inserted. If there is a part without music or sound in the empty space in the scenario you selected, you can create it. And you can do chromakey shooting. With the concept of appearing in your own movie, you can have yourself appear as the main character in a specific scene that appears in an empty space in the scenario you choose. The chromakey studio is equipped with several props against a blue screen background and has professional cinematographers filming visitors.

Next, after the movie is produced, it is transmitted to the viewer's smartphone. Then, share your work with members by uploading it to UCC, etc.

On the other hand, the system according to the embodiment enables extraction of facial elements more effectively by performing the case of extracting facial elements as described above.

Specifically, the system according to the embodiment first creates a data distribution based on the area of each facial element and the metadata of each facial element, including time data, and specifies the activity area of each facial element within the image to determine whether to monitor it. Set up and analyze big data patterns.

Then, each facial element is classified by applying the shape characteristics of each facial element preset from the analysis results of the big data pattern.

Therefore, through this, facial elements are extracted more effectively than in the above-described embodiment.

On the other hand, the system according to the embodiment enables more effective recognition through the following configuration when recognizing face information.

For this purpose, the system according to the embodiment has the following configuration.

First, before segmenting the face area, the system calculates deep learning parameters for the face area through deep learning learning for 2D face area segmentation from only a plurality of different RGB images in advance to create a deep learning model for face 2D area segmentation. Build.

In this case, the facial 2D region segmentation deep learning model generates a binary mask by extracting representative thresholds for each 2D face region through mask R-CNN type deep learning learning for 2D face region segmentation from multiple different RGB images in advance. By doing so, it becomes a mask R-CNN based facial 2D region segmentation deep learning model.

Then, face 2D region segmentation is performed from the RGB image from the RGB-D camera using the face 2D region segmentation deep learning model.

Next, depth information of the face is obtained by performing a logical AND between the face 2D segmentation result and the depth data from the RGB-D camera.

Then, perspective transformation is applied to the depth information of the face to perform 3D object region segmentation of the face, so that xyz and RGB information of the face can be obtained.

Therefore, metadata of each facial element is extracted for each facial element by applying the xyz and RGB information of the acquired face.

Meanwhile, the display method according to the embodiment is specified by building the deep learning model described above in the following manner.

That is, the above exhibition method first collects only RGB images of visitors through an RGB-D camera.

Next, learning data is created by pre-defining and displaying the face area in the RGB image and specifying the type of face.

Then, a preset face segmentation deep learning model is applied to the learning data to segment only the face area.

Then, the dataset resulting from these segmentation results is applied to a deep learning model for preset 2D face region segmentation and learned.

Next, deep learning parameters for the face area are calculated from the learning results.

So, a deep learning model for facial 2D region segmentation is constructed from deep learning parameters related to these facial regions.

Therefore, through this, face recognition is performed more effectively than in the above-described embodiment.

Meanwhile, the content provision usage cost processing system applied to this system will be explained as an example.

The system first uses a point accumulation card provided by the mobile carrier or company to which the customer subscribes to settle the amount of use of VR content, etc., a card and RF chip that can be used in cultural spaces or related business models, Build a settlement and operation system using IC chips.

In this state, when a request to process the usage amount using a customer card, etc. is made to the operating information processing device, the usage amount is processed, the usage amount is accumulated and stored, and the usage amount is stored in the customer card.

In addition, when VR content operators request sales inquiries using business models or convenience terminals in cultural spaces, the sales amount is searched and provided.

In addition, a sales inquiry function is provided to notify the sales amount and settlement process to terminals related to the business model after processing the sales amount and settlement so that the sales amount can be confirmed and settlement approval can be made.

So, with the settled data, the operator of the convenience facility within the cultural space receives the settled amount in cash from the cultural space operator and completes the settlement process.

Additionally, the operating system for personal investment and promotion applied to this system is explained as an example.

The operating system first provides cultural content promotional information and investment information to the customer terminal through an operating information processing device, and stores the investment and promotional information. Then, the customer checks cultural content information such as VR content produced by artists through the customer terminal and deposits the investment according to the individual's will. Therefore, the operating information processing device processes the investment confirmation after confirming the investment deposit, stores the investor information in the cultural content DB, and generates an authentication number for the admission ticket or coupon provided to the investor.

So, when a coupon authentication number is generated, the coupon authentication number is stored in the authentication number DB and is provided when an admission ticket or coupon is provided through the customer terminal by the application. The customer checks the admission ticket or coupon through the customer terminal. It will be used.

Next, when the convenience facility operator requests confirmation of the coupon authentication number for the admission ticket or coupon to be used by the customer using a convenience facility terminal, etc., the operating information processing device processes the coupon authentication number confirmation request and verifies the authentication number.

So, when the authentication number is confirmed, a confirmation notification is sent, the coupon is approved at the convenience facility terminal, and the admission ticket or coupon is processed for use.

Figure 4 is a conceptual diagram of a gallery implementation system using virtual work videos in a service company according to another embodiment.

As shown in Figure 4, the gallery implementation system using virtual work videos in these service companies includes a number of different artists (e.g. art artists) (400-1, 400-2, ...) and management information processing. It includes a device 410 and businesses such as coffee shops (420-1, 420-2, ...).

Here, the artist (400-1, 400-2, ...) receives virtual work videos differently for each virtual work type or company type, or produces them directly using an authoring tool and supplies them to the management information processing device.

In addition, the management information processing device 410 receives virtual work videos for each artist in an integrated manner, categorizes and manages them through the mediation of each artist and company, and distributes them differently to each company distributed in various locations. In this case, the management information processing device 410 updates the virtual work video distributed to the company periodically or according to the company's request.

In addition, the business terminals (420-1, 420-2, ...) are installed in businesses such as coffee shops or restaurants, and individually receive virtual work videos distributed by the management information processing device 410. It is displayed in virtual reality in a pre-registered digital frame within each company.

In this case, the management information processing device 410 further performs the following operations.

That is, the management information processing device 410,

a) Group virtual work videos so that they contain at least one virtual work video from multiple different artists (400-1, 400-2, ...).

b) Then, each virtual work video is sold to a company to make a profit.

c) In addition, at this time, the profit is the first profit paid to the author (400-1, 400-2, ...) of the virtual work video, the second profit paid by the brokerage, and the first profit It is divided into third revenue excluding second revenue, and the third revenue is settled to be distributed to the artists belonging to the relevant work group.

FIG. 5 is a diagram illustrating the entire system described in FIG. 4.

As shown in Figure 5, the system includes a number of different writer terminals (500-1, 500-2, ...) connected to each other through a private network, a management information processing device 510, and a company terminal ( Includes 520-1, 520-2, ...).

In this case, the artist terminals 500-1, 500-2, ... are at least one terminal such as a mobile terminal or computer that receives virtual work videos from a plurality of different artists or directly produces and supplies them.

In addition, the management information processing device 510 receives each virtual work video provided by the writer terminals (500-1, 500-2, ...) in an integrated manner and categorizes and manages them through mediation by writer and company, It is distributed differently for each company distributed in various locations. Additionally, the management information processing device 510 is connected to a portal site, etc. to post each virtual work video or service it through a blog.

In addition, the company terminals (520-1, 520-2, ...) individually receive the virtual work images distributed by the management information processing device 510 and transform them into virtual reality in a pre-registered digital picture frame within each company. It is a terminal such as at least one computer that is displayed.

Meanwhile, additionally, when the system displays virtual work videos of such artists in coffee shops, etc., it creates interesting and engaging images by displaying different virtual work videos around the customer according to the emotional state of the customer who viewed the virtual work video in a digital frame. Make sure you feel the atmosphere.

For this purpose, the system has the following configuration.

That is, the management information processing device 510 first matches the virtual work images supplied by the artist terminals 500-1, 500-2, ... to a plurality of different emotional states and art.

And, in this state, the company terminals (520-1, 520-2, ...) perform the following operations.

a) First, the face of the customer watching the virtual work video displayed by the digital frame is continuously recognized.

b) Then, the customer's emotional state matching the change state of facial elements in the recognized customer's face is detected.

c) Next, a different matched virtual work video according to the detected emotional state of the customer is displayed around the work that corresponds to each art.

In this case, detecting (b)) the customer emotional state is accomplished as follows.

b-1) First, metadata of each face element is extracted for each face element from the recognized customer face.

b-2) Then, the size and position of the face element are estimated from the metadata of each extracted face element.

b-3) Next, the change state of the facial element is detected by applying the size and position of each estimated facial element.

b-4) Then, the customer emotional state matching the detected facial element change state is detected.

The face extraction (b-1)) is performed, for example, as follows.

b-1-1) That is, the face extraction (b-1)) first creates a data distribution based on the area of each face element and the metadata of each face element including time data, and then creates a data distribution of each face element in the image. Analyze big data patterns by specifying the activity area and setting whether to monitor it.

b-1-2) Then, each facial element is classified by applying the shape characteristics of each facial element preset from the analysis results of the big data pattern.

Alternatively, the face extraction (b-1)) performs the following operation as another example.

b-1-1') First, the face extraction (b-1)) is performed on the face area through deep learning learning for 2D face area segmentation from only a plurality of different RGB images in advance before segmenting the face area. Calculate deep learning parameters to build a deep learning model for facial 2D region segmentation.

In this case, the facial 2D region segmentation deep learning model generates a binary mask by extracting representative thresholds for each 2D face region through mask R-CNN type deep learning learning for 2D face region segmentation from multiple different RGB images in advance. By doing so, it becomes a mask R-CNN based face 2D region segmentation deep learning model.

b-1-2') Then, face 2D region segmentation is performed from the RGB image from the RGB-D camera using the face 2D region segmentation deep learning model.

b-1-3') Next, depth information of the face is obtained by performing a logical AND between the face 2D region segmentation result and the depth data from the RGB-D camera.

b-1-4') Then, perspective transformation is applied to the depth information of the face to perform 3D object region segmentation of the face, so that xyz and RGB information of the face can be obtained.

b-1-5') Therefore, metadata of each face element is extracted for each face element by applying the xyz and RGB information of the obtained face.

Additionally, the facial 2D region segmentation deep learning model construction (b-1-1') is performed as follows.

b-1-1'-1) First, the facial 2D region segmentation deep learning model construction (b-1-1') collects only RGB images of visitors through an RGB-D camera.

b-1-1'-2) Then, the face area in the RGB image is predefined and displayed, and the type of face is specified to create learning data.

b-1-1'-3) Next, a preset face segmentation deep learning model is applied to the learning data to segment only the face area.

b-1-1'-4) Then, the dataset resulting from the above segmentation results is applied to a deep learning model for preset 2D face region segmentation and learned.

b-1-1'-5) Then, deep learning parameters for the face area are calculated from the learning results.

b-1-1'-6) Therefore, a facial 2D region segmentation deep learning model is constructed from the deep learning parameters related to the face region.

300: Exhibition service provision information processing device
301: Motion sensor 302: Filming means
303: Profile database unit 304: Content database unit
305: control unit 306: display unit
307: output means
400: Writer terminal
410, 510: Management information processing device
420: Company
520: Company terminal

Claims (12)

An artist terminal that receives virtual work videos from multiple different artists or produces and supplies them directly;
a management information processing device that mediates the virtual work video supplied by the artist terminal for each artist and gallery, matches the virtual work video for each art with a plurality of different emotional states, and distributes the virtual work video to each different gallery; and
a gallery terminal that transforms the virtual work video distributed by the management information processing device into virtual reality and displays it in each gallery; It includes,
The gallery terminal is,
a) When viewing a work corresponding to each of the plurality of different works of art, the viewer's face is continuously recognized,
b) detecting the visitor's emotional state matched to the change state of facial elements in the recognized visitor's face,
c) A different matched virtual work video according to the detected emotional state of the visitor is produced around the corresponding work for each art,
Detecting the emotional state of the visitor (b)),
b-1) Extract metadata of each facial element from the recognized visitor's face for each facial element,
b-2) Estimate the size and location of the face element from the metadata of each extracted face element,
b-3) Detecting the change state of facial elements by applying the size and position of each estimated facial element,
b-4) detecting a visitor emotional state matched to the detected facial element change state; An art gallery implementation system using virtual reality, comprising: In claim 1,
The gallery terminal is,
d) From the customer terminal, through the mobile app, the turning point in the beginning, progression, and conclusion of the story is selected as an option for each interactive content in the exhibition hall according to the exhibition theme of each art,
e) combining the selected results and systematically linking them into a movie, and
f) producing a movie by filling the corresponding turning point with the produced virtual content (including the audience's face and virtual content) or recording preset sound; An art gallery implementation system using virtual reality, comprising: In claim 1,
In b-1) above,
b-1-1) Create a data distribution based on the area of each facial element and the metadata of each facial element including time data, specify the activity area of each facial element within the video, and set whether to monitor it to create a big data pattern. Analyze and
b-1-2) classifying each facial element by applying the shape characteristics of each facial element preset from the analysis results of the big data pattern; An art gallery implementation system using virtual reality, comprising: In claim 1,
In b-1) above,
b-1-1') Before segmenting the face area, deep learning parameters for the face area are calculated through deep learning learning for 2D face area segmentation from only a number of different RGB images in advance, and deep facial 2D area segmentation is performed. Build a learning model,
b-1-2') Perform face 2D region segmentation from the RGB image from the RGB-D camera using the face 2D region segmentation deep learning model,
b-1-3') Obtaining face depth information by performing logical AND between the face 2D region segmentation result and depth data from the RGB-D camera,
b-1-4') By applying perspective transformation to the depth information of the face to perform 3D object region segmentation of the face, xyz and RGB information of the face can be obtained, and
b-1-5') extracting metadata of each facial element by applying the xyz and RGB information of the acquired face; An art gallery implementation system using virtual reality, comprising: In claim 4,
The facial 2D region segmentation deep learning model is,
Mask R-CNN-based face 2D region segmentation by generating a binary mask by extracting representative thresholds for each 2D face region through mask R-CNN type deep learning learning for 2D face region segmentation from multiple different RGB images in advance. with deep learning models; An art gallery implementation system using virtual reality featuring: In claim 4,
The above b-1-1') is
b-1-1'-1) Collect only RGB images of visitors through RGB-D cameras,
b-1-1'-2) Create learning data by pre-defining and displaying the face area in the RGB image and specifying the type of the face,
b-1-1'-3) Apply a preset face segmentation deep learning model to the learning data to segment only the face area,
b-1-1'-4) Apply and learn the dataset resulting from the above segmentation results to a deep learning model for preset 2D face region segmentation,
b-1-1'-5) Calculate deep learning parameters for the face area from the learning results, and
b-1-1'-6) Constructing a face 2D region segmentation deep learning model from deep learning parameters related to the face region; An art gallery implementation system using virtual reality, comprising: At least one artist terminal that receives virtual work videos from multiple different artists or produces and supplies them directly;
A management information processing device that receives each virtual work video provided by the artist terminal in an integrated manner, categorizes and manages them through mediation by artist and company, and distributes them differently to each company distributed in various locations; and
At least one company terminal that individually receives the virtual work images distributed by the management information processing device, virtualizes them and displays them in a pre-registered digital picture frame within each company; It includes,
The management information processing device,
a) Group virtual work videos to include at least one virtual work video from multiple different artists,
b) Each virtual work video is sold to a company to earn profit, and
c) The profit is divided into the first profit paid to the author of the virtual work video, the second profit paid by the brokerage, and the third profit excluding the first profit and the second profit, and the third profit Settlement for distribution to artists belonging to the relevant work group; A gallery implementation system using virtual work videos in a service company featuring . In claim 7,
The management information processing device,
Matching the virtual work video supplied by the artist terminal to a number of different emotional states and art,
The company's terminal is,
a) Continuously recognize the face of the customer viewing the virtual work video displayed by the digital picture frame,
b) detecting the customer emotional state matched to the change state of facial elements in the recognized customer face,
c) Different matched virtual work videos for each detected emotional state of the customer are displayed around the corresponding work for each art,
Detecting the customer emotional state (b)) is,
b-1) Extract metadata of each face element from the recognized customer face for each face element,
b-2) Estimate the size and location of the face element from the metadata of each extracted face element,
b-3) Detecting the change state of facial elements by applying the size and position of each estimated facial element,
b-4) detecting a customer emotional state matched to the detected facial element change state; A gallery implementation system using virtual work videos in a service company featuring . In claim 8,
In b-1) above,
b-1-1) Create a data distribution based on the area of each facial element and the metadata of each facial element including time data, specify the activity area of each facial element within the video, and set whether to monitor it to create a big data pattern. Analyze and
b-1-2) classifying each facial element by applying the shape characteristics of each facial element preset from the analysis results of the big data pattern; A gallery implementation system using virtual work videos in a service company, comprising: In claim 8,
In b-1) above,
b-1-1') Before segmenting the face area, deep learning parameters for the face area are calculated through deep learning learning for 2D face area segmentation from only a number of different RGB images in advance, and deep facial 2D area segmentation is performed. Build a learning model,
b-1-2') Perform face 2D region segmentation from the RGB image from the RGB-D camera using the face 2D region segmentation deep learning model,
b-1-3') Obtaining face depth information by performing logical AND between the face 2D region segmentation result and depth data from the RGB-D camera,
b-1-4') By applying perspective transformation to the depth information of the face to perform 3D object region segmentation of the face, xyz and RGB information of the face can be obtained, and
b-1-5') extracting metadata of each facial element by applying the xyz and RGB information of the acquired face; A gallery implementation system using virtual work videos in a service company, comprising: In claim 10,
The facial 2D region segmentation deep learning model is,
Mask R-CNN-based face 2D region segmentation by generating a binary mask by extracting representative thresholds for each 2D face region through mask R-CNN type deep learning learning for 2D face region segmentation from multiple different RGB images in advance. with deep learning models; A gallery implementation system using virtual work videos in a service company featuring . In claim 10,
The above b-1-1') is
b-1-1'-1) Collect only RGB images of visitors through RGB-D cameras,
b-1-1'-2) Create learning data by pre-defining and displaying the face area in the RGB image and specifying the type of the face,
b-1-1'-3) Apply a preset face segmentation deep learning model to the learning data to segment only the face area,
b-1-1'-4) Apply and learn the dataset resulting from the above segmentation results to a deep learning model for preset 2D face region segmentation,
b-1-1'-5) Calculate deep learning parameters for the face area from the learning results, and
b-1-1'-6) Constructing a face 2D region segmentation deep learning model from deep learning parameters related to the face region; A gallery implementation system using virtual work videos in a service company featuring .

Images