JP2007207278A - Virtual reality space sharing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To naturally change the expression of an object according to an operator's intention without performing complicated operation in a virtual reality space sharing system in which two or more operators can share a virtual reality space through a network and can move the avator object of each operator in the virtual reality space. <P>SOLUTION: Respective client terminals 3 and 4 detect the expressions of the faces of the operators 3A and 4A of the respective client terminals 3 and 4. Data of the detected expression is sent to a server side by expression data transmission means 3b and 4b. The server side 2 instructs to change the expressions of the objects corresponding to the operators on the screens 3d and 4d of the respective client terminals 3 and 4 based on the data of the sent expressions. The respective client terminals 3 and 4 receiving it change the pictures of the objects on the screens 3d and 4d of each according to the instruction from the server. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a virtual reality space sharing system in which a plurality of operators share a virtual reality space via a network, and more particularly to a virtual reality space sharing system that can move an individual object of each operator in the virtual reality space.

  Conventionally, there is a virtual reality space sharing system that allows a plurality of operators to share a virtual reality space and freely move an individual object of each operator in the space to communicate between the operators ( For example, JP-A-9-081782). In such a system, when an operator wants to change the facial expression or movement of an object of his / her own part, a change instruction is issued on the client terminal. Then, the server side that has received this instruction issues an instruction to change the facial expression and movement of the object to the other client terminal, whereby the other client terminal displays the facial expression and movement of the corresponding object on each screen. To change.

  However, in such a conventional virtual reality space sharing system, the operator uses a mouse or the like on the client terminal to select and instruct the desired facial expression, so the operation is troublesome, It was not possible to change the expression of objects naturally.

  The present invention has been made in view of the above points, and provides a virtual reality space sharing system that can naturally change the facial expression of an object as desired by an operator without performing a complicated operation. With the goal.

  In the present invention, in order to solve the above-described problem, in a virtual reality space sharing system in which a plurality of operators can share a virtual reality space via a network, and a separate object of each operator can be moved in the virtual reality space. A server that provides the virtual reality space; a client that receives the provision of the virtual reality space from the server via the network; a facial expression detection unit that detects facial expressions of the operator of the client; and the client Facial expression data transmitting means for transmitting the detected facial expression data to the server side, and in the server, based on the transmitted facial expression data, the facial expression of the object corresponding to the operator on the screen of each client A facial expression change command means for giving a command to change the Virtual reality space sharing system and having an image changing means for changing the image of the object on each of the screen in response to a command from the server is provided.

  In such a virtual reality space sharing system, first, a virtual reality space is provided to each client by a server via a network. On each client side, facial expression detection means detects the facial expression of the operator of the client. Then, the facial expression data transmitting means sends the detected facial expression data to the server side.

  In the server, the facial expression change commanding unit issues a command to change the facial expression of the object corresponding to the operator on the screen of each client based on the transmitted facial expression data. In each client that receives this, the image changing means changes the image of the object on its own screen in response to a command from the server.

  In the present invention, the facial expression of the operator of each client is detected, and the image of the object on the screen of each client is changed according to the detected facial expression. Can be changed naturally as the operator wishes. Therefore, more realistic communication can be achieved.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a concept of functions of the virtual reality space sharing system of this embodiment. A server terminal 2 and a large number of client terminals 3 and 4 are connected to the network 1. Each client terminal 3, 4 is provided with facial expression detection means 3 a, 4 a comprising a CCD camera and facial expression detection circuit. These facial expression detection means 3a and 4a detect whether the facial expression of the operators 3A and 4A of the client terminals 3 and 4 is, for example, a smile, an angry face, a crying face, or a surprised face. Expression data detected by the expression detection means 3a, 4a is sent to the server by the expression data transmission means 3b, 4b. However, in this case, it is preferable to transmit facial expression data only when the facial expressions of the operators 3A and 4A change.

  On the server terminal 2 side, the facial expression change command means 2a instructs to change the facial expression of the object corresponding to the operator on the screens 3d and 4d of the client terminals 3 and 4 based on the transmitted facial expression data. . In the client terminals 3 and 4 that have received this, the image changing means 3c and 4c change the image of the object on their own screens 3d and 4d in response to a command from the server. However, it is preferable that the client terminal that has sent facial expression data to the server terminal 2 changes the display on its own screen without waiting for a command from the server terminal 2.

  FIG. 2 is a block diagram showing a configuration example of the virtual reality space sharing system of this embodiment. Here, for example, two server terminals 12 and 13 are connected to a network 11 such as the Internet. Storage devices 14 and 15 such as hard disk devices are connected to the server terminals 12 and 13, respectively. The storage devices 14 and 15 store the virtual reality space sharing system program, and can provide the virtual reality space to the client terminals 16, 17, and 18 via the network 11.

  The virtual reality space is displayed as an image on a monitor, which will be described later, of each client terminal 16, 17, 18. In this virtual reality space, an object of the operator who operates each of the client terminals 16, 17 and 18 is displayed, and by operating a keyboard, a mouse, etc., various parts of the body and facial expressions can be freely set in the space. Can be moved to. In addition to the object of the operator, the background building can also be moved.

  A plurality of service provider terminals 19 and 20 are connected to the network 11. The service provider terminals 19 and 20 are provided with a virtual reality space via the network 11 and can provide predetermined services to the client terminals 16, 17 and 18.

Next, a specific configuration of each terminal will be described.
FIG. 3 is a block diagram showing a specific configuration of the server terminal 12. A CPU 21, ROM 22, RAM 23, communication device 24, interface 25, and the like are connected to the bus 30. The CPU 21 controls the operation of the entire server terminal 12 according to a program built in the ROM 22. The RAM 23 appropriately stores data, programs, and the like necessary for the CPU 21 to execute various processes. The communication device 24 exchanges data with the network 11.

  The storage device 14, display device 26, microphone 27, speaker 28, and input device 29 are connected to the interface 25. The storage device 14 is a hard disk, an optical disk, a magneto-optical disk, or the like, and stores programs and data necessary for providing a virtual reality space. The display device 26 includes a CRT, an LCD, and the like, and displays an image of the virtual reality space provided by the server terminal 12.

  Necessary audio signals are input from the microphone 27 and supplied to the client terminals 16, 17, 18 and the service provider terminals 19, 20. On the other hand, audio signals supplied from the client terminals 16, 17, 18 and the service provider terminals 19, 20 are output from the speaker 28.

The input device 29 includes a mouse 29a and a keyboard 29b. Necessary commands are issued by the mouse 29a and the keyboard 29b.
Since the configuration of the server terminal 13 is almost the same as that of the server terminal 12, the description thereof is omitted here.

  FIG. 4 is a block diagram showing a specific configuration of the client terminal 16. A CPU 31, ROM 32, RAM 33, communication device 34, interface 35, etc. are connected to the bus 43. The CPU 31 controls the overall operation of the client terminal 16 according to a program built in the ROM 32. The ROM 32 stores image data of the face of an operator's alternation object, which will be described later. The RAM 33 appropriately stores data, programs, and the like necessary for the CPU 31 to execute various processes. The communication device 34 exchanges data with the network 11.

  A display device 36, a microphone 37, a speaker 38, a storage device 39, and an input device 40 are connected to the interface 35. The display device 36 includes a CRT, an LCD, and the like, and displays a CG (computer graphic) three-dimensional image and an image taken by a normal video camera or the like. A necessary audio signal is input from the microphone 37 and supplied to the server terminals 12 and 13 via the network 11. On the other hand, the audio signal supplied from the server terminals 12 and 13 is output from the speaker 38. The storage device 39 is a hard disk, an optical disk, a magneto-optical disk, or the like, and stores programs and data necessary for an operation of moving each object in the virtual reality space.

  The input device 40 includes a mouse 40a and a keyboard 40b. The mouse 40a is used when a predetermined position is designated. The keyboard 40b is used when inputting predetermined characters and symbols. The input device 40 includes a viewpoint input device 40c and a movement input device 40d. The viewpoint input device 40c is used when inputting a viewpoint in the virtual reality space of an object of the operator of the client terminal 12. Thereby, the viewpoint of the object can be moved up, down, left, and right, and the image ahead of the line of sight of the object can be zoomed. The movement input device 40d is used when the object is moved back and forth and left and right.

  In addition, a CCD camera 41 and a facial expression detection circuit 42 are connected to the interface 35. The CCD camera 41 always captures the face of the operator of the client terminal 16. The photographed video signal is taken into the facial expression detection circuit 42 at regular intervals. The facial expression detection circuit 42 detects the facial expression of the operator based on the video signal by a procedure described later, and sends the result to the server side.

Note that the configuration of the client terminals 17 and 18 is almost the same as that of the client terminal 16, and thus the description thereof is omitted here.
Since the service provider terminals 19 and 20 have substantially the same configuration as the client terminal 16 except for the CCD camera 41 and the expression detection circuit 42, the description thereof is omitted here.

Next, a specific configuration of the facial expression detection circuit 42 will be described.
FIG. 5 is a block diagram showing a specific configuration of the facial expression detection circuit 42. The low pass filter (LPF) 421 removes the high frequency component of the video signal input from the CCD camera 41. Since the human face has pores, fine wrinkles, etc., the video signal from the CCD camera 41 contains a large amount of high frequency components. For this reason, the low pass filter 421 removes this high frequency component. The edge detection circuit 422 converts the analog signal output from the low-pass filter 421 into a digital signal, performs edge detection from the digital signal, and outputs video data with only edges.

  The pattern matching circuit 423 reads out expressionless face data as basic data of the operator's face stored in advance in the memory 424, and performs pattern matching in comparison with the data from the edge detection circuit 422. Adjust the position and size of the face taken. That is, while using the virtual reality space, the operator moves the face position up and down, left and right, or back and forth, and the face coordinate position and size of the basic data differ. This makes it impossible to accurately capture the current facial expression. Therefore, the pattern matching circuit 423 compares the data for one frame from the edge detection circuit 422 with the basic data, and how much the photographed operator's face moves back and forth, left and right, and up and down with respect to the basic data. Detect whether or not The operator's face data photographed based on the movement amount is corrected to the same conditions as the basic data.

  By the way, the memory 424 stores, as basic data, data of an operator's expressionless face, but also stores position data of characteristic portions that characterize facial expressions.

  FIG. 6 is a diagram showing an example of basic data of the operator's face stored in the memory 424. The basic face data 51 is stored in the memory 424. Further, as characteristic portions characterizing facial expressions, for example, left eyebrow left end (point P1), left eyebrow right end (point P2), mouth corner left (point P3), mouth corner right (point P4), mouth center upper end (point P5) ), The positions of the right eye corner (point P6) and the left eye corner (point P7). The points P1 to P7 are stored, for example, in XY coordinates on the frame.

  Returning to FIG. 5, the motion vector detection circuit 425 compares the face data of the operator whose size and position are adjusted by the pattern matching circuit 423 with the basic data 51 of the memory 424, and It is detected how much the points corresponding to the points P1 to P7 shown in FIG. 6 have moved.

  FIG. 7 is a diagram showing an example of operator's face data imaged by the CCD camera 41. Here, an example of a smile state is shown. The operator's face data 52 is detected by the motion vector detection circuit 425 by the left eyebrow left end (point P11), the left eyebrow right end (point P12), the mouth corner left (point P13), the mouth corner right (point P14), and the mouth center upper end (point P14). Point P15), right eye corner (point P16), and left eye corner (point P17) are detected and compared with the corresponding points P1 to P7 on the basic data 51, respectively. Based on these comparisons, the motion vector detection circuit 425 detects a motion vector indicating how much each of the points P11 to P17 has moved from the corresponding points P1 to P7 of the basic data. This is accomplished by performing block matching of a block containing several pixels in the vicinity containing each point.

  Returning to FIG. 5, the facial expression determination circuit 426 compares the motion vector obtained by the motion vector detection circuit 425 with the facial expression determination data in the facial expression table 427 to determine the facial expression of the current operator.

  FIG. 8 is a diagram showing an example of expression determination data in the expression table 427 for determining the facial expression. As shown in the figure, the facial expression judgment data 53 includes “smiling face”, “angry face”, “crying face”, and “surprise face” as facial expressions to be judged. It is given in a moderate range. The facial expression judgment circuit 426 has the facial expression that most closely matches the motion vector of each characteristic part obtained by the motion vector detection circuit 425 in each facial expression of the facial expression judgment data 53 as the facial expression of the current operator. to decide. That is, if it is the data 52 of FIG.

  The facial expression determination circuit 426 that has determined the facial expression in this way outputs the facial expression code indicating the facial expression to the interface 25 shown in FIG. 4 and sends it to the CPU 31 only when the facial expression determined this time is different from the previous determination. When the facial expression code of its own client terminal is output, the CPU 31 reads the image data of the object corresponding to the facial expression code from the ROM 32 and causes the display device 36 to display it. In this case, the facial expression image of the object is stored in the ROM 32 in accordance with each facial expression code, and the corresponding facial expression image is read from the ROM 32 each time the facial expression code is output. It may be pasted and displayed on the part, or the operation of the face and the whole body corresponding to the expression code is stored in the ROM 32, for example, in the case of “crying face”, the operation of wiping tears, etc. The object may be moved according to the expression code.

  Simultaneously with the display of such a screen, the CPU 31 sends a display code to the server terminal 12 via the communication device 34 and the network 11. Receiving this, the server terminal 12 sends the expression code and the ID of the client that has output the display code to the other client terminals 17 and 18 via the network 11.

  Receiving the facial expression code, the client terminals 17 and 18 read the facial expression code and the object corresponding to the ID of the client terminal 16 from the ROM and display them on each display device.

  FIG. 9 is a flowchart showing a processing procedure on the client terminal side regarding the expression code. [S1] It is determined whether a facial expression code is output from the facial expression detection circuit 42 of its own terminal. If it is output, the process proceeds to step S2, and if not, the process proceeds to step S4. [S2] The expression of the object on the screen of the display device 36 is switched to the expression corresponding to the newly output expression code. However, if the operator's own client terminal is set to a mode in which only the scenery of the virtual reality space from the operator's viewpoint is displayed without displaying the operator's alternate object, of course, There is no switching. [S3] The facial expression code is output to the server terminal 12 side. [S4] It is determined whether the expression code and the client ID are sent from the server terminal 12. If sent, the process proceeds to step S5, and if not sent, this flowchart ends. [S5] The facial expression of the alternate object of the client terminal corresponding to the ID sent from the server terminal 12 is switched to the facial expression corresponding to the facial expression code sent simultaneously.

  FIG. 10 is a flowchart showing a processing procedure on the server terminal side regarding the facial expression code. [S11] It is determined whether or not an expression code is input from the client terminal side. If it is input, the process proceeds to step S12. If not input, the process of the flowchart ends. [S12] The inputted facial expression code is transmitted to another client terminal.

  As described above, in this embodiment, the facial expressions of the operators of the client terminals 16, 17, and 18 are photographed by the CCD camera 41 or the like, and the individual of each operator in the virtual reality space is changed according to the change of the facial expressions. Since the facial expression of the object is changed, it is possible to naturally change the facial expression of the object as the operator intends without performing a complicated operation. Therefore, more realistic communication can be achieved.

  In this embodiment, when a facial expression code is sent from the client terminal, the server terminal side sends only the facial expression code to the other client terminal, and corresponds to the facial expression code sent on the client terminal side that received it. The image is read from its own ROM 32 and displayed, but all display data may be transmitted from the server terminal to the client terminal.

It is a block diagram which shows the concept of the function of the virtual reality space sharing system of this form. It is a block diagram which shows the structural example of the virtual reality space sharing system of this form. It is a block diagram which shows the specific structure of a server terminal. It is a block diagram which shows the specific structure of a client terminal. It is a block diagram which shows the specific structure of a facial expression detection circuit. It is a figure which shows an example of the basic data of the operator's face stored in memory. It is a figure which shows an example of the data of the operator's face imaged with the CCD camera. It is a figure which shows an example of the data for facial expression judgment of the facial expression table for judging the facial expression. It is a flowchart which shows the processing procedure by the client terminal side regarding an expression code. It is a flowchart which shows the process procedure by the side of the server terminal regarding an expression code.

Explanation of symbols

22 camera, 23 mechanical capture button, 28 LCD, 131 CPU, 136 display control unit, 201 lens unit, 202 CCD, 210 camera control unit, 211 lens unit drive circuit, 221 EVF area, 227 soft capture button, 260 for spot metering Pointer, 261 AF frame, 361 AE area 1 network, 2 server terminal, 2a expression change command means, 3 client terminal, 3A operator, 3a expression detection means, 3b expression data transmission means, 3c image change means, 3d monitor, 11 Network, 12, 13 server terminal, 14, 15 storage device, 16, 17, 18 client terminal, 31 CPU, 32 ROM, 41 CCD camera, 42 facial expression detection circuit.

Claims (1)

In a virtual reality space sharing system in which a plurality of operators can share a virtual reality space via a network, and a separate object of each operator can be moved in the virtual reality space.
A server for providing the virtual reality space;
A client that receives provision of the virtual reality space from the server via the network;
Facial expression detecting means for detecting facial expressions of the operator of the client;
In the client, facial expression data transmission means for sending the detected facial expression data to the server side;
In the server, based on the transmitted facial expression data, facial expression change command means for giving a command to change the facial expression of the object corresponding to the operator on the screen of each client;
The virtual reality space sharing system, wherein each client includes image changing means for changing an image of the object on its own screen in response to a command from the server.

Images