JP2015005823A - Image processing apparatus and digital mirror system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus for realizing a digital mirror which when displaying an image especially from the back when a user uses the apparatus, reduces discomfort felt by the user.SOLUTION: The image processing apparatus comprises: a moving image input unit (102) which receives a moving image (110) of a user obtained from a visual sensor; an operation input unit (101) which receives user operation for acquiring a moving image of the user obtained from a predetermined orientation; a mirror image or normal image determination unit (103) which determines whether to display the acquired moving image as a mirror image or a normal image, on the basis of the user operation; an image generation unit (104) which generates the mirror image or the normal image from the acquired moving image on the basis of the determination; and an output unit (105) which outputs to a display apparatus (120) the generated mirror image or normal image.

Description

  The present invention relates to an image processing device, a digital mirror system, and an image processing method that digitize an image of a user and display the image on a display device installed at a position where the user can visually recognize the image.

  As one of application destinations of image processing technology, a so-called digital mirror technology in which a system is configured with a sensor and a display device and used instead of a mirror has been proposed in recent years. Research has been carried out with the advantage that information that cannot be superimposed on an actual mirror can be displayed and that information that cannot be displayed with an actual mirror can be displayed.

  The general purpose of using a mirror is to “make the appearance while looking at the mirror”. At this time, it is very convenient if you can see yourself from various directions including angles that cannot be seen with a normal mirror. Therefore, a conventional technique for realizing free-viewpoint mirror image display by installing a plurality of cameras has been proposed.

  Patent Document 1 proposes a digital mirror device that synthesizes images taken by cameras provided on the left and right of an object such as a human face and displays an image that looks like a mirror on a display in front of the user. . By estimating 3D image data of users taken from multiple cameras and rendering from another viewpoint, it is possible to display an image that captures the user from a specific direction.

  Similarly, in Patent Document 2, there is an electronic mirror system that displays a user image viewed from an arbitrary viewpoint as a mirror image by performing view morphing on user images captured by a plurality of cameras and combining intermediate images. Proposed.

JP2002-290964 JP 2004-297734 A

  In the examples and drawings of these patent documents, an image that looks like the user is moved by moving the viewpoint in the horizontal direction several tens of degrees from the front of the user, for example, the camera is installed to the rear. If so, it is possible to move the viewpoint to a viewpoint that reflects the user's back. At this time, since the conventional method always displays a mirror image, it reaches the viewpoint (rear viewpoint) where the back is reflected while the mirror image is displayed. When the mirror image is displayed from the rear viewpoint, the right and left of the user and the left and right of the mirror image are reversed. That is, the user's right hand is displayed on the left side on the mirror image, and the user's left hand is displayed on the right side on the mirror image. Here, in order for the user to smoothly perform the work while looking at the mirror such as adjusting the appearance, the user's top and bottom, left and right must match the top and bottom, and left and right in the mirror image. In other words, when a mirror image is displayed from the rear viewpoint, it is very difficult to work while looking at the mirror.

  Thus, according to the conventional method, a mirror image display from an arbitrary viewpoint can be realized, which is useful, for example, in confirming appearance. However, it has not been examined whether the displayed mirror image is easy to “work to improve the appearance of the confirmed appearance”.

  In other words, the fact that the user can work smoothly without feeling uncomfortable when “working” such as dressing up while looking in the mirror does not take into account the freedom actually realized by the above system. When the user uses the viewpoint digital mirror, there is a point that makes it difficult to use. In other words, the above-mentioned problem occurs because there is no mechanism for appropriately switching and displaying a mirror image and a normal image that is the left and right reverse image in the free viewpoint digital mirror.

  The present invention has been made to solve the above-described problems, and provides a free-viewpoint digital mirror device and method that is easier to use and easier to use when a user performs work while using a free-viewpoint digital mirror. The purpose is to provide.

1st invention of this application is invention which has the following means. In this section, the invention refers to the invention described in the specification at the time of filing.
A video input unit to which a moving image of a user obtained from a visual sensor is input;
An operation input unit for inputting a user operation for acquiring a moving image obtained by capturing the user from a predetermined direction;
A mirror image normal image determination unit that determines whether to display the acquired moving image as a mirror image or a normal image based on the user operation;
An image generating unit that generates the mirror image or the normal image from the acquired moving image based on the determination;
An output unit that outputs the generated mirror image or the normal image to a display device;
An image processing apparatus.

The “visual sensor” is a sensor that detects a moving image and includes not only parts (elements) such as a CCD and a CMOS but also a finished product using the parts. As an example of the visual sensor, there is a camera using a CCD or a finished product.
“User operation” means not only when the user directly operates the input device, but also indirectly transmits the user's intention by detecting and processing the user's movement, line of sight, voice, brain waves, etc. with the detection device. Including cases.
A “mirror image” refers to an image that is reversed left and right or up and down when compared with an actual image. For example, the mirror image includes an image of a user visually recognized by an erect user in front of a mirror, an image obtained by reversing and displaying an image obtained by photographing an erect user with a general camera or the like.
“Normal image” refers to an image that is not reversed right and left or up and down when compared with an actual image. For example, the normal image includes an image obtained by directly displaying an image captured by the user with a general camera or the like, or an image obtained by inverting the mirror image.

  Since up, down, left and right can be relative concepts, FIG. 17 shows a specific example. When a person stands upright in front of the mirror as shown in FIG. 17 (a), the actual image is as shown in FIG. 17 (b), and the mirror image is reversed left and right as shown in FIG. 17 (c). The normal image is the one that is not horizontally reversed as in FIG. 17D (same as the actual image). On the other hand, when a person lies down in front of the mirror, the actual image is as shown in FIG. 17E, and the mirror image is inverted as shown in FIG. As shown in FIG. 17 (d), the image is not reversed upside down (the same as the actual image). Therefore, although it has been described here as up and down or left and right, it is not limited to this description as long as it has the same idea as the example shown in FIG.

The second invention of the present application is an invention having the following means.
The mirror image normal image determination unit further determines whether to display the acquired moving image as a mirror image or a normal image based on the orientation of the user in the acquired moving image. The image processing apparatus described.

  “User orientation” refers to the orientation of the user with respect to the digital mirror device. Examples of the digital mirror device as a reference include a display device and a camera. The user orientation may be the orientation of the user's upper body in the horizontal direction, or the tilt of the user's head or upper body upward.

The third invention of the present application is an invention having the following means.
In the image processing apparatus according to the first invention or the second invention,
One or more visual sensors connected to the video input unit;
The display device connected to the output unit;
A digital mirror system.

The fourth invention of the present application is an invention having the following means.
The digital mirror system according to the third aspect, wherein the one visual sensor is installed in a robot arm that moves around the user.

The fifth invention of the present application is an invention having the following means.
The user operation input to the operation input unit is a swipe operation amount on a touch panel of the user, and the moving images input from the plurality of visual sensors are switched according to the swipe operation amount. A digital mirror system according to any one of the inventions.

The sixth invention of the present application is an invention having the following means.
The mirror image normal image determination unit displays a normal image in the case of the moving image input from the visual sensor installed to face the direction of the display device among the plurality of visual sensors, The digital mirror system according to any one of the third to fifth inventions, wherein the moving image input from the visual sensor is determined to display a mirror image.

The seventh invention of the present application is an invention having the following means.
The mirror image normal image determination unit detects the orientation of the user, displays a mirror image in the case of a moving image input from the visual sensor on the front side of the user with respect to the orientation of the user, and the orientation of the user On the other hand, in the case of a moving image input from the visual sensor on the back side of the user, the digital mirror system according to any one of the third to sixth inventions, which determines to display a normal image.

The eighth invention of the present application is an invention having the following means.
When the moving image from the visual sensor looking down from above the head of the user is switched from the visual sensor on the front side of the user with respect to the direction of the user, the mirror image normal image determination unit The digital mirror system according to any one of the third to seventh inventions, wherein a mirror image is displayed, and when switching from the visual sensor on the back side of the user with respect to the direction of the user, it is determined to display a normal image. .

A ninth invention of the present application is an invention having the following means.
The mirror image normal image determination unit first identifies the user instead of the acquired moving image when the determination in the acquired moving image differs from the determination in the moving image acquired one cycle ago. The digital mirror system according to any one of the third to eighth inventions, wherein it is determined to display the moving image captured from the direction.

The tenth invention of the present application is an invention having the following means.
The pre-operation input unit is configured to temporarily not accept an input in the same direction as the direction of the user input that has been performed so far when switching between the normal image and the mirror image occurs. The digital mirror system according to any one of the ninth invention.

The eleventh invention of the present application is an invention having the following steps.
A video input step in which the moving image of the user obtained from the visual sensor is input;
An operation input step in which a user operation for acquiring a moving image capturing the user from a predetermined direction is input;
A mirror image normal image determination step for determining whether to display the acquired moving image as a mirror image or a normal image based on the user operation;
An image generating step for generating a mirror image or a normal image of the acquired moving image based on the determination;
An output step of outputting the generated mirror image or normal image to a display device;
An image processing method comprising:

The twelfth invention of the present application is the following invention.
A computer program in which a computer is caused to execute the image processing method according to the eleventh invention.

  According to the present invention, it is possible to realize a free-viewpoint digital mirror that allows the user to work more easily when used.

Explanatory drawing which shows the structure of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing which shows the structure of the digital mirror of the 1st Embodiment of this invention. 1 is a block diagram showing the configuration of an image processing apparatus according to a first embodiment of the present invention. 1 is a block diagram showing a configuration of a computer of an image processing apparatus according to a first embodiment of the present invention. 3 is a flowchart showing the operation of the image processing apparatus according to the first embodiment of the present invention. Explanatory drawing explaining operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing explaining operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing explaining operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing explaining operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing explaining operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing explaining other operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing explaining other operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing explaining other operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing explaining other operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing explaining other operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing explaining other operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing explaining other operation | movement of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing which shows the structure of the digital mirror of the 1st Embodiment of this invention. Explanatory drawing which shows the structure of the digital mirror of the 2nd Embodiment of this invention. Explanatory drawing explaining the relationship between the mirror image of this invention, a normal image, and an actual image

Hereinafter, an image processing device and a digital mirror according to one embodiment of the present invention will be specifically described with reference to the drawings.
Note that each of the embodiments described below shows a specific example of the present invention. Numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of constituent elements, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements.

(Embodiment 1)

  FIG. 1 shows a configuration example of an operation input system / display system in the digital mirror system that is the subject of the present embodiment. As shown in FIG. 1 (a), the display system may be a display device installed at a position where the user can visually recognize, and not only can display a mirror image of the user as shown in a normal mirror, As shown in FIG. 1B, a digital mirror system having an operation input system and capable of displaying a mirror moving image with a different viewpoint according to a user operation (touch, gesture, etc.) will be used as an example. A display device provided with an operation input system that can accept a user's operation as shown in FIG. 1 can be realized using a display provided with a touch panel or a gesture sensor, a projector provided with a gesture sensor, or the like. The operation input system for accepting user operations is not limited to the form integrated with the above display device (touch or gesture), but can be operated by a device capable of operating a free viewpoint image, such as a keyboard, mouse, button, remote controller, etc. It may be realized.

  FIG. 2 shows a configuration example in which a sensor (imaging) system is also displayed in the digital mirror system that is the subject of the present embodiment. For example, a plurality of cameras, which are visual sensors, are installed as shown in FIG. 2A so as to surround the subject, and a camera that acquires an image to be displayed is switched according to a user's operation. Freely obtain a mirror image. In particular, by sufficiently increasing the number of cameras, it is possible to realize smooth switching that is not inferior to a free viewpoint image by 3D model and rendering. Further, a more seamless free viewpoint image may be generated using a 3D model or morphing by the methods disclosed in Patent Documents 1 and 2 listed above.

  In addition to the above example, as shown in FIG. 2B, a free-viewpoint digital mirror is provided by providing a visual sensor at the tip of a robot arm that can receive an input from the operation input system and move around the user. It may be realized.

  In addition, a retractable camera set as shown in FIG. 18 may be used. 18A and 18C are views of the camera set as viewed from above, and FIG. 18B is a view of the display device as viewed from the front. In this camera set, as shown in FIG. 18A, a plurality of cameras 402 are installed on each side, vertex, and center of a pantograph-like structure 401. And this is provided horizontally and arranged so as to be housed in the housing section 404 at the top of the display device 403 as shown in FIG. At the time of storage, the structure 401 can be stored in the storage unit 404 by closing as shown in FIG. In the case of such a configuration, since it can be normally stored in the upper part of the display device 403, it does not get in the way, and can be used by being pulled out from the upper part of the display device 403 only during use.

In addition, various installation modes are conceivable, such as installation by embedding a camera in the upper part of the display device, the left and right walls, the rear wall, and the ceiling. As for the camera, it is possible to use a camera as a part using an image sensor such as a CCD or a CMOS in addition to a camera having a completed product form as shown in FIG. It is also possible to use a light field camera composed of a number of microlenses having different focal points. By using a light field camera, it is possible to focus on a portion to be watched and to reduce the number of cameras.
In addition, the example installed in the inside of a motor vehicle is later mentioned as 2nd Embodiment. An example using a PC, a tablet, a smartphone, or the like will be described later as a third embodiment.

  In the following description, the term “moving image” refers to moving image data or a moving image indicated by moving image data. Of course, a set of still images (pictures) constituting the moving image is also included in the moving image. In the present disclosure, a moving image displayed on a display device is referred to as a mirror moving image regardless of a mirror image normal image.

  FIG. 3 is a block diagram illustrating a functional configuration of the image processing apparatus 100 according to the present embodiment. As illustrated in FIG. 3, the image processing apparatus 100 includes an operation input unit 101, a moving image input unit 102, a mirror image normal image determination unit 103, an image generation unit 104, and an output unit 105.

  The image processing apparatus 100 receives the moving image 110 and the user operation 111 as input, performs image processing on the moving image of the user taken from a predetermined direction that is an angle specified by the user, and displays the resulting moving image as a mirror moving image. It outputs to the display apparatus 120 as an image.

  The operation input unit 101 receives a user operation 111 related to a change in the viewpoint angle of the mirror moving image from a terminal used by a user who views the mirror moving image. As described above, the terminal used by the user may be a display device provided with a touch panel, a keyboard, a mouse, a button, a remote control, or the like. For example, the operation input unit 101 may be an interface that reads a user operation 111 such as a gesture or a touch on a touch panel directly or via a communication path.

  The moving image input unit 102 receives an input of a moving image or a plurality of pictures (also referred to as “images”) constituting the moving image. The moving image input unit 102 may be an interface that reads a moving image from a camera, which is a visual sensor, directly or via a communication path.

  The mirror image normal image determination unit 103 determines the display mode of the mirror moving image to be output based on the user operation 111 received by the operation input unit 101 for the moving image 110 received by the moving image input unit 102. In particular, it is determined whether to output a mirror image or a normal image as a mirror moving image. The determination method will be described later.

  The image generation unit 104 performs image conversion on the moving image 110 based on the display state of the mirror moving image determined by the mirror image normal image determination unit 103 to generate a mirror moving image.

  The output unit 105 converts the mirror moving image generated by the image generation unit 104 into a form that can be displayed on the display device 120, and outputs it to the display device 120.

  Note that each component constituting the image processing apparatus 100 may be realized by software such as a program executed on a computer, or may be realized by hardware such as an electronic circuit or an integrated circuit. FIG. 4 is a diagram illustrating a hardware configuration of the image processing apparatus 100 according to the present embodiment configured by a computer.

  In FIG. 4, a moving image 110 acquired by a camera 210a is input to a computer 200 which is an image processing apparatus through an I / F (interface) 201a. The operation unit 210b that receives input from the user outputs the user operation 111 to the I / F 201b, and inputs the user operation 111 to the computer 200 that is an image processing apparatus through the I / F 201b. The computer 200 acquires the moving image 110 and the user operation 111 and performs image generation processing.

  The display device 220 displays a mirror moving image generated by the computer 200.

The computer 200 includes an I / F 201a, an I / F 201b, a CPU 202, a ROM 203, a RAM 204, an HDD 205, and an I / F 206. A program for operating the computer 200 is stored in advance in the ROM 203 or the HDD 205. The program is read out from the ROM 203 or HDD 205 to the RAM 204 and expanded by the CPU 202 as a processor. The CPU 202 executes each coded instruction in the program expanded in the RAM 204. The I / F 201 a and the I / F 201 b capture the moving image 110 and the user operation 111 into the RAM 204 in accordance with the execution of the program. The I / F 206 outputs the generated mirror moving image to the display device 220.
The computer program is not limited to the semiconductor ROM 203 or HDD 205 but may be stored in an external storage device such as a CD-ROM. Alternatively, the data may be transmitted via a wired or wireless network, broadcasting, or the like and taken into the RAM 204 of the computer.

  Next, the operation of the image processing apparatus 100 in the present embodiment will be described with reference to FIG. FIG. 5 is a flowchart showing the operation of the image processing apparatus 100 in the present embodiment. In FIG. 5, the five steps S301 to S305 correspond to the respective processing units 101 to 105 in FIG. That is, the operation input unit 101 is an operation input step S301, the moving image input unit 102 is a moving image input step S302, the mirror image normal image determination unit 103 is a mirror image normal image determination step S303, the image generation unit 104 is an image generation step S304, and the output unit 105 is Each operation of the output step S305 is executed. Note that these steps can be realized by the above-described computer program.

(1) Operation input step S301 (operation input unit 101)
First, the operation input step S301 is executed by the operation input unit 101. The operation input unit 101 acquires a user operation 111 and selects a camera or a camera position as a viewpoint. The selection of the camera or the camera position is performed in order to acquire a moving image that captures the user from a predetermined direction.

  The user operation 111 to be acquired is at least in the X direction (on the ground and the display surface of the mirror) in order to realize free viewpoint movement with respect to the user's image, in particular, as shown in FIG. Information on the moving component in the horizontal direction). In the present embodiment, in order to realize free viewpoint movement in the vertical direction of the user, information on the moving component in the Y direction (direction of the mirror perpendicular to the ground) is also included. As an example, FIG. 6A shows a user operation 111 acquired in the case of input to the touch panel. A difference vector from the touch start point to the end point by the swipe operation is acquired as the user operation 111. The X component of the difference vector corresponds to the rotation of the viewpoint of the mirror image in the X direction, and the Y component corresponds to the rotation of the viewpoint of the mirror image in the Y direction. Then, a camera position to be selected, which will be described later, is determined according to the magnitude of the difference vector that is the operation amount.

  Note that the movement of the user's hand is often inaccurate. For example, even if the hand is moved only with the intention of rotation in the X direction, the resulting vector often contains a Y component. Therefore, in order to realize viewpoint movement with a greater intention of the user, a rule may be adopted in which only one of the X direction and the Y direction can be moved at a time. In this case, only the larger component of the X component and the Y component is acquired as the user operation 111. For example, when the solid line user operation 111 shown in FIG. 6B is obtained, the user operation 111 indicated by the dotted line is acquired because the X component of the difference vector is larger. When the solid line user operation 111 shown in FIG. 6C is obtained, the difference vector has a larger Y component, so the user operation 111 indicated by the dotted line may be acquired. Furthermore, in order to remove the instability of the user operation 111, it is possible to always rotate a certain angle every time there is an operation input regardless of the size of the difference vector. In this case, a fine angle operation cannot be performed, but an angle change caused by an unintended hand movement blur can be removed.

  The moving component may be a value normalized so that the vertical and horizontal value ranges of the screen are 1 instead of the coordinate value.

  These user operations 111 may be performed through a user interface on a device such as a digital mirror provided with the image processing apparatus 100, or a general PC or a portable terminal as a remote controller not provided with the image processing apparatus 100. You may make it receive via a wired or radio | wireless from the operation performed using the user interface on an apparatus.

  In this embodiment, the swipe operation of the touch panel is assigned to the user operation 111, but other operations may be assigned. For example, the above-described difference vector may be generated by selecting two points of a start point and an end point by a tap operation.

  Selection of a camera or camera position based on the acquired user operation 111 will be described with reference to FIG. As an example, consider an example in which twelve cameras surround the user. Although not shown, a display device that presents a mirror moving image to the user or a display device that includes an operation input system is installed in the lower part of the camera 3, for example. Can be captured by the camera 3 and the camera 9 can capture the back of the user.

  When an image taken by the camera 3 from the front of the user is displayed and the operation A in the X direction is input as the user operation 111, the intention of the user is “the direction of the self-image reflected in the mirror image is slightly to the right "I want to see the left side of my image towards". " That is, it is desirable to switch from the camera 3 in front of the user to the image of the camera 4 on the left of the user. Further, when it is desired to look at the left side of the head, the camera 6 is selected by performing the operation A three times from the state of the camera 3, for example.

  When the operation B in the X direction is input as the user operation 111, the intention of the user is “I want to see the right side of the head with the head reflected in the mirror image slightly left”. That is, it is desirable to switch from the camera 3 in front of the user to the image of the camera 2 on the right of the user. Further, when the user wants to see the right side of the head, the camera 12 can be selected by performing the operation B three times from the state of the camera 3, for example. If the camera position obtained in this way, or a list in which the camera number and position are separately associated with each other is stored in the memory in a state that the mirror image normal image determination unit 103 can refer to the camera number, Is transferred to the mirror image normal image determination unit 103.

  In the digital mirror, it is most natural that the image from the front of the user taken by the camera 3 is displayed as a mirror image as an initial state. In such a state, if the user's intention is “I want to see the front of the user”, the user does not need to directly operate the digital mirror. By not doing so, it can be said that the user's intention of “looking at the front of the user” is transmitted.

  The operation of the operation input unit 101 is, for example, when the viewpoint selection of the camera is not discrete but continuous (examples of the conventional documents 1 and 2 and the example of FIG. 2B), for example, A new camera position / viewpoint may be obtained based on movement information (X, Y) of the X component and Y component obtained by the operation and information on the camera position / viewpoint displayed before the operation input.

(2) Moving image input step S302 (moving image input unit 102)
Next, the moving image input step 102 is executed by the moving image input unit 102. The moving image input unit 102 acquires the moving image 110.

  The moving image input unit 102 acquires the moving image 110 from one or a plurality of cameras via a wired or wireless network. The moving image is 30 frames / second. In this embodiment, since the free viewpoint mirror is realized by a plurality of cameras as shown in FIG. 2A, the number of cameras is plural, but the movable camera as shown in FIG. In that case, it may be one.

  Most simply, the moving image input unit 102 acquires a moving image of the camera corresponding to the camera position or number selected by the operation input unit 101. Of course, moving images from all cameras may be acquired, and moving images from specific cameras may be acquired based on user operations. As in the previous example, when the camera viewpoint selection is not discrete but continuous (examples of the conventional documents 1 and 2), it is desirable to acquire moving images of all cameras. Alternatively, as described above, an image corresponding to the camera position / viewpoint obtained by the operation input unit 101 may be generated from a plurality of acquired moving images by image synthesis or the like.

  Note that the operation input step S301 and the moving image input step S302 can be processed independently, and in that case, the order may be reverse to that described above, or the processing may proceed in parallel. .

(3) Mirror image normal image determination step S303 (mirror image normal image determination unit 103)
Next, a mirror image normal image determination unit 103 executes a mirror image normal image determination step S303. The mirror image normal image determination step S303 is performed on the moving image received by the moving image input unit 102 based on the camera position obtained by the operation input unit 101 as the user operation, the selected camera, or the acquired moving image. The display mode of the mirror moving image to be displayed is determined.

  In the simplest case, as shown in FIG. 8, it can be assumed that the user stands in front of the display device in order to visually recognize the mirror moving image. In this case, if it says in FIG. 7, the image of the camera 3 will become right and left opposite to the user who is utilizing if it is not a mirror image, and the operation | work while seeing a mirror moving image is difficult. Similarly, if the image of the camera 9 is not a normal image, the left and right sides are opposite to each other, and it is difficult to work while viewing the mirror moving image. Therefore, it is desirable to determine a mirror image when an image of the camera 3 (user front side) is acquired, and a normal image when an image of the camera 9 (user rear side) is acquired.

  Further, according to the experiments by the inventors, the image of the camera 12 and the image of the camera 6 positioned on the side surface of the user are equally unaffected by either the normal image or the mirror image. It was. It is considered that the boundary between the mirror image and the normal image is determined by the human viewing angle (about 200 degrees). Therefore, the mirror image may be determined as long as the viewpoint is moved within the positions of the cameras 12 and 1 to 6, and the normal image may be determined as long as the viewpoint is moved within the positions of the cameras 6 to 12. Alternatively, a range of up to about 100 degrees on the left and right sides (up to about 200 degrees in total), which is a human viewing angle, centered on the front of the user may be determined as a normal image. Alternatively, either of the positions of the camera 6 and the camera 12 may be displayed. In addition, the cameras 12 and 1 to 6 display normal images (cameras 7 to 11 are mirror images), or the cameras 1 to 5 display normal images (cameras 6 to 12 are mirror images), or the positions of the cameras 6 and 12. The display of the normal image and the mirror image may be switched within the display.

  FIG. 9 is a diagram showing an example of switching between normal image display and mirror image display within the positions of the cameras 6 and 12. Consider the case where the cameras are continuously selected from the camera 3 to the camera 9 via the camera 6 in FIG. Each camera from camera 3 to camera 9 shoots a moving image as shown in FIG. In response to such a moving image input, the mirror image displayed on the display unit is a mirror image for the cameras 3, 4, and 5, and the normal image for the cameras 7, 8, and 9, as shown in FIG. It becomes. The moving image of the camera 6 is initially displayed as a mirror image, but is reversed halfway and a normal image is displayed.

  Although the above description has been given with the example of the camera arrangement in FIG. 7, the application destination of the mirror image normal image determination unit 103 is not limited to the camera arrangement in FIG. In other words, any camera arrangement that can select a viewpoint including the user in the field of view around the user may be used. When the user can be considered to be mainly facing the screen of the display device on which the mirror moving image is displayed, an image obtained from a camera installed so as to face the direction of the display device on which the mirror moving image is displayed May be determined as a normal image, and an image obtained from other cameras may be determined as a mirror image.

  Furthermore, even when the user is not facing the display device, it is easy for the user to work, that is, the mirror image can be determined so that a mirror moving image can be generated in which the mirror moving image always matches the left and right of the user. It is desirable. When the user works while looking in the mirror, the inventor has a great influence on the sense of incongruity when the position of the “hand” performing the work and the orientation of the body as the base are reversed on the mirror image. These experiments have also gained knowledge.

  Therefore, in addition to the user operation, the display mode of the mirror moving image to be finally displayed can be determined based on the orientation of the user in the acquired moving image. The direction of the user here is the direction of the user with respect to the digital mirror device. For example, in the examples of FIGS. 7 and 8, this corresponds to the direction in which the user's body is facing the display device or camera fixed to the actual space or the ground.

  FIG. 10 will be described as an example. When the user turns his / her shoulder along the direction of the cameras 11 to 5, the camera 2 corresponds to the front of the user and the camera 8 corresponds to the back of the user. Therefore, the camera 2 on the front side with respect to the direction of the user (cameras 11, 12, 1 to 5 in a broad sense) is a mirror image, and the camera 8 on the back side with respect to the direction of the user (the camera 6 in a broad sense). 10) to 10) is preferably determined as a normal image. Similarly, according to the experiments by the inventors, the image of the camera 11 and the image of the camera 5 were not both dissimilar, both normal and mirror images. A normal image and a mirror image may be determined for the moving images from the cameras 5 and 11 on the same basis as described.

The above processing can be easily realized if the user's body orientation with respect to the present system is known. For example, when the face and body orientation are the same, or when finding the face orientation in the vertical direction, it is realized by detecting the face or face orientation based on the presence or absence of face parts in the image and how the face parts are visible it can. Many techniques that can estimate the user orientation by determining the user posture from the image have been proposed. For example, as shown in Non-Patent Document 1, a method for estimating a three-dimensional posture by learning an outline as a shape feature has been proposed. If the moving image input unit 102 acquires moving images from a plurality of cameras in advance, it is possible to obtain the user's body orientation with higher accuracy. For example, as in Non-Patent Document 2, there has been proposed a method for accurately obtaining the body orientation based on silhouettes acquired from a plurality of cameras.
(Non-Patent Document 1) Agarwal, A. and Triggs, B .: 3D Human Pose from Silhouettes by
Relevance Vector Regression, IEEE International Conference on Computer Vision
and Pattern Recognition, Vol. 2, pp.882-888 (2004)
(Non-Patent Document 2) Kehl, R., Bray, M. and Gool, LV: Full Body Tracking from Multiple
Views Using Stochastic Sampling, IEEE International Conference on Computer
Vision and Pattern Recognition, Vol.2, pp.129-136 (2005)
Detailed procedures are described in the literature, and the explanation is omitted here.

  In addition, when performing mirror image normal image determination based on the user's posture and body orientation in this way, the mirror image normal image determination unit 103 determines the user's posture every predetermined time and displays a mirror image or a normal image. The determination result may be updated and output to the image generation unit 104. This is because the user may change only the body orientation without performing an operation that changes the display viewpoint. For example, the determination may be performed at a timing of 30 times / second in accordance with the normal video rate. The predetermined time is not limited to a certain time. For example, the posture determination may be performed when any movement is detected in the image.

  With the above procedure, it is possible to determine whether a mirror image or a normal image should be displayed for a moving image from a specific camera position / viewpoint obtained by the operation input unit 101.

(4) Image generation step S304 (image generation unit 104)
Next, the image generation step S304 is executed by the image generation unit 104. If the result determined by the mirror image normal image determination unit 103 is a mirror image, the image generation unit 104 generates a moving image from the viewpoint for the specific camera position / viewpoint obtained by the operation input unit 101, and A mirror image is generated by inverting the left and right sides, and transferred to the output unit 105. If the result determined by the mirror image normal image determination unit 103 is a normal image, a moving image from the viewpoint is generated for the specific camera position / view point obtained by the operation input unit 101, and conversion to a mirror image is performed. Instead, the data is transferred to the output unit 105 as it is.

(5) Output step S305 (output unit 105)
Finally, an output step S305 is executed by the output unit 105. The output unit 105 converts the moving image received from the image generation unit 104 into a format that can be viewed on a display device such as a display, and outputs the converted image to the display device 120 as a mirror moving image.

  As described above, according to the present embodiment, a moving image as intended by a user operation can be acquired, and the acquired moving image can be converted into an image that is easier for the user to work and output. Can do.

(Another operation example 1 of the first embodiment)

  Note that, depending on the arrangement of the cameras, it is possible to adopt a configuration in which the viewpoint from above or below the user can also be selected. In this case, the relationship between natural user operation and viewpoint selection is as shown in FIG. Regarding the user operation 111 described in the operation input unit 101, when the component of the difference vector in the Y direction is large (FIG. 6C), the viewpoint moves in the vertical direction, but when moving the hand upward, the user Since the user wants to see the lower part of the head better, the user selects the camera position / viewpoint looking up from below. Similarly, when moving the hand in the downward direction, the user intends to see the upper part of the head (the top of the head) better, so select a camera position / viewpoint that allows the user to look down from above. .

  When the user performs the viewpoint selection operation in the upward direction (overhead view), the source state depends on whether the viewpoint is the viewpoint from the front side of the user or the viewpoint from the back side of the user. Since a street overhead image is obtained, the mirror image correct image determination step S303 needs to select the mirror image correct image so that the right and left of the user coincide with each other as described above. FIG. 12A shows an example in which the camera viewpoint is moved from the viewpoint (mirror image) on the user's front side to the bird's-eye view when switching from the visual sensor on the user's front side. In this case, by continuing to display the image as a mirror image, a mirror moving image that matches the left and right of the user can be obtained. FIG. 12B shows an example in which the camera viewpoint is moved from a visual sensor on the back side of the user, that is, from a viewpoint (normal image) on the back side of the user to an overhead view state. In this case, by continuing to display the normal image, a mirror moving image that matches the left and right of the user can be obtained. Therefore, when the camera is installed above the user, switching according to the direction of the user may be performed as in the case of the front and rear.

(Other operation example 2 of Embodiment 1)

  Further, when more importance is attached to the interactivity, particularly when the user is continuously changing the viewpoint, for example, in the example of FIG. When is switched, there is a possibility that an uncomfortable feeling of appearance will occur. Furthermore, since the operation direction is reversed between the mirror image and the normal image, as described later, a sudden discontinuity in operation that occurs when switching between the normal image and the mirror image is a continuous operation that has been performed by the user until then. May be adversely affected. That is, when the normal image / mirror image display is switched, a result opposite to the operation intended by the user may occur.

  Here, FIG. 13A illustrates the front, back, left, and right in the mirror moving image 1301 displayed with the display surface (mirror display surface) 1303 of the display device that displays the mirror moving image interposed therebetween for the user 1302. FIG. 10 is a diagram virtually showing a mirror moving image 1301 as an overhead view when viewed from above. Actually, there is no such entity on the other side of the mirror display surface 1303 as viewed from the user, but for the sake of explanation, such an expression is used. As the mirror moving image 1301, a mirror image is shown here. The front, rear, left and right of the mirror image of the user 1302 with the left hand in front are as shown in the figure.

  Here, first, assuming that the display is a mirror image (assuming that the display is not automatically switched to a normal image), a case where the user 1302 rotates and displays a mirror image (mirror moving image) 1301 is considered. . FIG. 13B shows a state in which the direction of the mirror moving image 1301 is changed by the user's operation (the user 1302 omits the display, but similarly to FIG. 13A, the mirror moving images 1304 to 1307 are illustrated on the drawing. Suppose you are watching a mirror video from the bottom of A mirror moving image 1304 shows a front mirror image, 1305 a right mirror image, 1306 a rear mirror image, and 1307 a left mirror image. Here, a thick double arrow indicates the direction of the user operation, and a thin arrow indicates the order in which the display changes. As described above, the user can smoothly see the image of front → right → rear → left → front simply by continuing to operate the mirror surface in the left direction.

  Note that the user operation direction is shown in FIG. A thick double broken line arrow 1308 indicates a user operation in the left direction on the mirror image, a thick double broken line arrow 1309 indicates a user operation in the right direction on the mirror image, and a thick double arrow 1310 indicates a user operation in the left direction on the normal image. An arrow 1311 is a user operation in the right direction on the normal image, and is used in the following drawings in addition to the fact that the thin arrows (including the long chain line) in the figure indicate the order in which the display of the mirror moving image changes.

  Further, even when it is assumed that the display is a normal image, the user operation is similar in smoothness to that of the mirror image. FIG. 14 (a) is a view that virtually shows the mirror moving image 1312 as an overhead view when the display is a normal image, similarly to FIG. 13 (a). Actually, there is no such entity on the other side of the mirror display surface 1303, but it is used for explanation. The front, back, left and right of the normal image of the user 1302 with the left hand in front are as shown in the figure. When the user 1302 rotates and displays the normal image, as shown in FIG. 14B, the user simply performs an operation in the right direction on the mirror surface, and smoothly forward → right → rear → You can see the image from left to front. (The user 1302 omits the display, but it is assumed that the mirror moving image is viewed from the lower part of the mirror moving images 1313 to 1316 in the same manner as in FIG. 14A). The mirror moving image 1313 shows a forward positive image, 1314 shows a right positive image, 1315 shows a rear positive image, and 1316 shows a left positive image.

  On the other hand, the proposal of the present invention is characterized in that a normal image and a mirror image are appropriately switched when a mirror moving image viewed from various viewpoints is displayed. As described above, even in the case of looking forward, rightward, backward, leftward, and forward, the operation direction differs between the mirror image and the normal image, so as shown in FIG. When the left and right images are displayed, the mirror image and the normal image are suddenly switched (in this example, automatic transition from the mirror image right side 1305 to the normal image right side 1314 and the normal image left side 1316 to the mirror image right side 1306). The display direction change from the mirror image front 1304 to the mirror image right 1305 was performed by the user operation in the left direction, but when the display direction change from the normal image right 1314 to the normal image rear 1315 is continuously performed, the display direction is changed to the right. There is a possibility that the user cannot perform the operation smoothly.

  Here, FIG. 15B shows an operation flow for realizing a more desirable relationship between the user operation and the display direction. The part indicated by the long chain line extending from the right side 1314 of the normal image and the left side 1307 of the mirror image forms a loop that continuously displays the mirror moving image in the same direction as the normal image or the mirror image for the user operation in a specific direction. doing. That is, when the transition is made from the mirror image right side 1305, the normal image right side 1314 cannot change the direction any more as long as the user operation to the left is continued as before, and the user operation in the opposite direction is not performed. Otherwise, if the display direction does not change and the transition is from the left side of the normal image 1316, the mirror image left side 1307 cannot change the direction any more as long as the user operation to the right is continued as before. In either case, the display direction may not change unless a user operation is performed in the opposite direction.

  In addition, although specific user operation from specific transition was demonstrated as a trigger which changes to the loop extended from the normal image right side 1314 and mirror image left side 1307 shown with a long chain line here, about the first transition, For example, when another angle reaches a specific angle (≈an angle of about 90 ° with respect to the user), for example, a transition may be made automatically after a certain period of time. By doing as described above, it is possible to easily indicate to the user that the operation direction is changed, and it is possible to reduce the user's uncomfortable feeling.

  Further, FIG. 16A shows another flow for realizing a more desirable relationship between the user operation and the display direction. A mirror image right side 1305, a normal image right side 1314, a normal image left side 1316, and a portion indicated by a long chain line extending from the mirror image left side 1307 form a loop in which the mirror image and the normal image are continuously inverted. That is, in the loop around the right side of the normal image / mirror image, if the user operation in the right direction moving from the right side of the normal image 1314 to the back side of the normal image 1315 is not on the right side of the normal image 1314, as shown in the figure, The inversion of the normal image may be repeated. The same applies to the loop around the left of the normal image / mirror image. In addition, although the case where a specific user operation exists as a trigger for transition to the loop indicated by the long chain line has been described here, for the first transition, what has been a different angle is a specific angle (≈ On the other hand, when the angle reaches 90 °, the transition may be made automatically after a certain period of time. By doing as described above, it is possible to easily indicate to the user that the operation direction is changed, and it is possible to reduce the user's uncomfortable feeling.

  Further, FIG. 16B shows another flow for realizing a more desirable relationship between the user operation and the display direction. That is, as indicated by the long chain line, when the normal image / mirror image is switched, it may be temporarily moved to the front / rear viewpoint of the user. For example, instead of transitioning directly from the mirror image right 1305 to the right normal image 1314, when a leftward operation is accepted, the current image transitions once to the rear normal image 1315. When the operation is further performed in the left direction, the direction is changed from the normal image rear 1315 to the normal image right 1314. During this time, since the user can get used to the operation on the normal image, the user can easily operate rightward thereafter and change the display to the normal image rear 1315 and the normal image left 1316. The same applies to the left side of the normal image / mirror image. Although an example in which there is a specific user operation as a trigger for transitioning to the loop indicated by the long chain line is described here, for the first transition, a different angle until then is a specific angle (≒ It is also possible to automatically transition when the angle reaches about 90 °. By doing so, it is possible to indicate to the user that the mirror image / normal image has changed, that is, the reference of the operation direction has been reversed from the reference so far, and the user's uncomfortable feeling can be reduced.

  Note that the above describes an example in which switching between the normal image and the mirror image occurs in the cameras corresponding to the cameras 6 and 12 in FIG. 7, but in addition to this, the normal image and the mirror image are changed when the selected camera changes. Even when the switching occurs, the same processing can be performed according to the orientation of the user at that time.

  Further, when the normal image / mirror image of each camera is determined each time according to the user's posture, the mirror image normal image determination unit 103 may perform an operation as shown in FIG. That is, if the camera position and mirror / normal image determination result of the previous cycle are different from the camera position and mirror / normal image determination result of the current cycle, if the current cycle determination is a mirror image, for example, For a camera located in front of the user as a camera that captured the user from a particular orientation, if the current cycle determination was a normal image, for example, on the back of the user as a camera that captured the user from a particular orientation Change the camera position to the camera that is located.

  For example, in FIG. 17, the camera position and determination result of the previous cycle are (camera A, mirror image), and the camera position and determination result of the current cycle are (camera B, normal image). Assume that cameras A and B are adjacent or close to each other. In this case, since it is a change from a mirror image to a normal image in a nearby camera, the center of the normal image camera (camera C, normal image) is output to the subsequent image generation unit 104 as the camera position of the current cycle. To do.

  By doing this, it is possible to give the impression that the operation has been reset once, and by resetting the somatic sensation that the user had so far, the camera's viewpoint is continuous, but the appearance and operation It is possible to avoid a situation in which a sudden discontinuity is observed.

  These general or specific aspects may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. The system, method, integrated circuit, computer program And any combination of recording media.

(Embodiment 2)
In Embodiment 1, the example which implements this invention using the hardware assumed mainly to install and use in a residence like FIG.2 and FIG.18 was demonstrated. In the present embodiment, a hardware configuration that is assumed to be installed and used in an automobile interior will be described.

  FIG. 19 is a block diagram showing an example in which the digital mirror system of the present invention is installed in the interior of an automobile. A car navigation device monitor 501, a rearview mirror 502, a windshield 503, a handle 504, a seat belt fulcrum 505, a driver seat 506, a passenger seat 507, a rear seat 508, and a dashboard 509 are provided inside the car 500. .

  The hardware configuration example of this embodiment is the same as that using the computer 200 shown in FIG. A program that implements the functions of the present invention is stored in the ROM 203 or the HDD 205, and is read into the RAM 204 and executed as necessary.

  As the computer 200, for example, a control device constituting a car navigation device can be used. Further, the operation unit 210b may be installed at a position described as the monitor 501 in FIG. 19, and when a car navigation device is already installed here, an input device such as a touch panel or a joystick of the car navigation device. Can be used. Of course, it is also possible to use a motion sensor or a voice recognition device. The display device 220 may be installed at the position indicated as the monitor 501 in FIG. 19, and when a car navigation device is already installed here, a liquid crystal panel of the car navigation device can be used, It is also possible to provide a projector that projects onto the windshield 503 and use it.

  As the camera 210a, a camera A1 is provided on the rearview mirror 502, and a camera A2 is provided on the rear seat 508. A set of these cameras can be used. In some cases, a drive recorder camera or a backward confirmation camera is installed at these positions. In such a case, the camera of the present invention can be obtained by changing the program of the existing camera. You may also use it.

Alternatively, the camera B1 may be provided on the upper portion of the handle and the camera B2 may be provided on the headrest portion of the driver's seat 506, and a set of these cameras may be used. There is a case where a camera for monitoring the driver state is already installed at the position of the camera B1 above the handle. In such a case, the camera of the present invention is changed by changing the program of the existing camera. You may also use as. Since the positions of these camera pairs are the front and back of the user when the user is seated in the driver's seat 506, the user can use the digital mirror system of the present invention with a natural posture. In this case, as the display device 220, a projector that projects onto the windshield 503 is provided and used, whereby display can be performed in front of the user.
Note that when the user of the passenger seat 507 uses the camera C1 on the dashboard 509 and the camera C2 on the headrest portion of the passenger seat 507, these sets may be used. As the display device 220, a projector that projects onto the windshield 503 in front of the passenger seat may be used.

  Furthermore, a camera D1 can be provided near the monitor 501 of the car navigation apparatus, and a camera D2 can be provided at the seat belt fulcrum 505 of the driver's seat 506, and a set of these cameras can be used. When using the monitor 501 of the car navigation device as the display device 220, the user rotates his / her head or body toward the monitor 501, so the position of the camera D1 is just in front of the user and the position of the camera D2 is in the back of the user. . When the user is seated on the passenger seat 507, the camera D3 may be provided on the seat belt fulcrum 505 of the camera D1 and the passenger seat 507, and these sets may be used.

Although Embodiment 2 has been described with a set of cameras before and after the user, three or more cameras may be used. For example, if all the cameras in FIG. 19 are provided and used, moving images from various directions can be obtained by switching the images of these cameras.
If a camera is provided above the user's head and used, the viewpoint from above the user can also be selected.

  As mentioned above, although the example which implement | achieves this invention in the interior of a motor vehicle was demonstrated, the interior of a motor vehicle is narrower than a residence, and it is easy to arrange | position a visual sensor in front and rear, right and left of a user, and an upper part. In addition, since the interior of a car is a private space, it is suitable for adjusting the appearance and can be said to have high affinity with the present invention.

(Embodiment 3)

  In the first embodiment and the second embodiment, the example in which the present invention is realized using dedicated hardware as shown in FIGS. 2, 18 and 19 has been described. In the present embodiment, an example in which the present invention is realized using general-purpose hardware will be described.

  The hardware configuration example of this embodiment is the same as that using the computer 200 shown in FIG. A program that implements the functions of the present invention is stored in the ROM 203 or the HDD 205, and is read into the RAM 204 and executed as necessary.

  As the computer 200, a desktop PC, a notebook PC, a tablet computer, a smartphone, or a mobile phone can be used. These terminals are equipped with an operation unit 210b for inputting a user operation 111, a display device 220, and a camera 210a for photographing the user from the front.

  However, since a camera for photographing the back and sides of the user is not mounted, it is necessary to prepare a camera for photographing the back and sides separately. For example, an external camera such as a CCD camera is connected to a terminal that realizes the functions of the present invention by a wired connection such as USB or wirelessly such as WiFi, and is installed or held by a user behind or on the side. And then shoot. In other words, anything provided with an imaging device and a communication device may be used. For example, a digital camera, a borrowed other person's smartphone, a mobile phone, or the like can also be used. Furthermore, many recent smartphones and mobile phones are equipped with two cameras, an outer camera and an inner camera. However, if the outer camera is configured to be removable so that it can communicate wirelessly, It can be used as a camera for taking pictures.

  As described above, when configured with general-purpose hardware, the digital mirror system of the present invention can be easily realized simply by downloading and executing a program for realizing the functions of the present invention. Furthermore, a portable digital mirror system can be realized.

  The present invention can be used for an apparatus such as a digital mirror, particularly an interactive free viewpoint mirror, in the image processing technology. In addition, the present invention can also be used as a device incorporated in an AV device such as a tablet PC, a smartphone, a PC, a car navigation system, a video camera, or a TV capable of acquiring and browsing user images.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 101 Operation input part 102 Moving image input part 103 Mirror image normal image determination part 104 Image generation part 105 Output part 110 Moving image 111 User operation 120 Display apparatus 200 Computer 201a, 201b, 206 I / F
202 CPU
203 ROM
204 RAM
205 HDD
210a Camera 210b Operation unit 210c Display device

Claims (12)

A video input unit to which a moving image of a user obtained from a visual sensor is input;
An operation input unit for inputting a user operation for acquiring a moving image obtained by capturing the user from a predetermined direction;
A mirror image normal image determination unit that determines whether to display the acquired moving image as a mirror image or a normal image based on the user operation;
An image generating unit that generates the mirror image or the normal image from the acquired moving image based on the determination;
An output unit that outputs the generated mirror image or the normal image to a display device;
An image processing apparatus.   The mirror image normal image determination unit further determines whether to display the acquired moving image as a mirror image or a normal image based on a user orientation in the acquired moving image. Image processing device. The image processing apparatus according to claim 1 or 2,
One or more visual sensors connected to the video input unit;
The display device connected to the output unit;
A digital mirror system.   The digital mirror system according to claim 3, wherein the one visual sensor is installed on a robot arm that moves around the user.   The user operation input to the operation input unit is a swipe operation amount on a touch panel of a user, and the moving images input from the plurality of visual sensors are switched according to the swipe operation amount. 5. The digital mirror system according to any one of 4.   The mirror image normal image determination unit displays a normal image in the case of the moving image input from the visual sensor installed to face the direction of the display device among the plurality of visual sensors, The digital mirror system according to claim 3, wherein the moving image input from the visual sensor is determined to display a mirror image.   The mirror image normal image determination unit detects the orientation of the user, displays a mirror image in the case of a moving image input from the visual sensor on the front side of the user with respect to the orientation of the user, and the orientation of the user The digital mirror system according to claim 3, wherein a moving image input from the visual sensor on the back side of the user is determined to be displayed as a normal image.   When the moving image from the visual sensor looking down from above the head of the user is switched from the visual sensor on the front side of the user with respect to the direction of the user, the mirror image normal image determination unit The digital mirror system according to any one of claims 3 to 7, wherein a mirror image is displayed and a decision is made to display a normal image when switching from the visual sensor on the back side of the user with respect to the orientation of the user.   The mirror image normal image determination unit first identifies the user instead of the acquired moving image when the determination in the acquired moving image differs from the determination in the moving image acquired one cycle ago. The digital mirror system according to claim 3, wherein it is determined to display the moving image captured from the direction.   The pre-operation input unit is configured to temporarily not accept an input in the same direction as the direction of the user input that has been performed so far when switching between the normal image and the mirror image occurs. The digital mirror system in any one of -9. A video input step in which the moving image of the user obtained from the visual sensor is input;
An operation input step in which a user operation for acquiring a moving image capturing the user from a predetermined direction is input;
A mirror image normal image determination step for determining whether to display the acquired moving image as a mirror image or a normal image based on the user operation;
An image generating step for generating a mirror image or a normal image of the acquired moving image based on the determination;
An output step of outputting the generated mirror image or normal image to a display device;
An image processing method comprising: A computer program describing a computer to execute the image processing method according to claim 11.