JP2014096661A - Method for realtime diminishing of moving object in moving image during photographing of moving image, moving image photographing apparatus for the same, and program for mentioned moving image photographing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for, during photographing a moving image, realtime diminishing of a moving object desired to conceal, while a scene being photographed actually is left as it is.SOLUTION: The method comprises: detecting at least one moving object in the moving image; calculating at least one moving direction and speed of the detected moving object; and speculative acquisition of a background image obtained when the moving object moves in the calculated moving direction and the calculated speed. The speculatively acquired background image is used for realtime diminishing of the moving object.

Description

  The present invention relates to a technique for erasing a moving object in a moving image in real time during moving image shooting. More specifically, the present invention relates to a technique for speculatively acquiring a background image when a moving object moves and erasing the moving object in a moving image in real time.

  In recent years, a technology called Augmented Reality (AR) has attracted attention. Augmented reality is also a term for Virtual Reality (VR). In general, augmented reality is the use of information technology to add or add information to the real space. (Ie, “extended”) is defined as real space. An example of augmented reality is the Sekai camera, for example. The Sekai camera can see a text or a picture associated with the balloon when a balloon (that is, an air tag) appears on the screen that is being photographed, and the balloon is tapped. The balloon is displayed on the screen as if it existed in the real space.

  Mixed reality (MR), which is a fusion technology between the real world and the virtual world, has been developed as a field where conventional artificial reality cannot be handled. An example of mixed reality is the virtual Asukakyo project. The virtual Asukakyo project is to project the CG data of a former building on various historical sites or ruins in augmented reality to reproduce the original figure.

  Both augmented reality and mixed reality are technologies that add information to the real world. On the other hand, the opposite concept, Diminished Reality (DR), has developed rapidly in the last few years. The hidden reality is a technique for visually obscuring or erasing an actual object.

  The following Patent Documents 1 to 3 describe techniques for eliminating or changing a virtual object.

  In Patent Document 1, the notification unit 180 notifies the user of the presence of danger when the danger recognition unit 170 recognizes the danger while the AR application is provided to the user. It describes that the display of the application is controlled and the notification unit 180 blinks or transmits the virtual object being displayed (paragraph 0074).

  Patent Document 2 is an example of an output image when the notification is performed by the notification unit 180 as a result of the user Ua running after the image Im32 is displayed, and in the image Im32, the virtual object T1 is It is erased from the screen and the AR application is stopped. Thus, it is described that the notification unit 180 may alert the user by simply interrupting or stopping the AR application ( Paragraph 0091).

  In Patent Document 3, the virtual object image changing unit 114 receives the display information of the display target virtual object from the display target virtual object search unit 110 and the approaching object information from the approaching object detection unit 111, and approaches the approaching object visibility. Even when the approaching object is hidden by the virtual object by using the determination unit 117 and the virtual object display position changing unit 116, the virtual object image is changed and output so as to improve the visibility of the approaching object. (Paragraph 0041).

  Further, Patent Document 4 receives a moving image of the opponent player captured by the camera of the opponent game machine, displays the opponent player in real time on the monitor of the own game machine, and displays the moving image of the own player imaged by the own camera. In a real-time video-compatible game machine that transmits to the machine and displays the player in real time on the monitor of the opponent game machine to play the game, the mask to be attached is selected from a number of masks and selected as the face of the video data And a mask display means for processing and covering the mask.

  The following non-patent documents 1 to 8 describe techniques of concealment reality.

  Non-Patent Document 1 describes that the basic framework of visual concealment / erasure / perspective in concealment reality (DR) is considered to be composed of the components shown in FIG. (Left column of page 240, “2. Technical framework of hidden reality”).

  Non-Patent Document 2 describes a technique for filling in the lossy background and moving foreground of video captured by a still or moving camera (page 63, “Abstract” section).

  Non-Patent Document 3 uses semi-automatically efficient hiding by using the 3D position information of natural feature points registered in the LMDB that is used in the Rehearsal Path Method (RPM) to create a hidden background model. A method for constructing the background surface group model is described (page 318, right column, lines 7 to 10).

  Non-Patent Document 4 describes an examination and solution method for the cause of inconsistency in image quality such as brightness and defocusing regarding optical consistency in Diminished Reality (DR) (page 359, “Summary”). Section).

  Non-Patent Document 5 describes a method of reducing the overlay deviation of a hidden background projection image using image mosaicing when combining the hidden background projection image and the input image (page 87, left column, bottom line to right column, line 4). .

  Non-Patent Document 6 refers to an expression that leaves an object without intentionally erasing it in Diminished Reality (DR) as “half-hidden display (Half-DR)”, and describes a general and effective method thereof ( (Page 322, left column, bottom line to right column, second line).

  Non-Patent Document 7 describes an ideal expression method for half-hide display (Half-DR) (left column of page 174, “1. Introduction”).

  Non-Patent Document 8 describes a technique for acquiring two images with different camera directions and creating a difference image by matching the scenes of the two images by projection conversion (page 9, section “Summary”).

JP 2012-155655 A JP 2012-155654 A JP 2011-128838 A JP 2010-178902 A JP 2005-148253 A JP 2003-177017 A

Naohei Mori, “Technical Framework and Problems of Concealed Reality: ~ Technology for visually concealing, erasing, and seeing through real objects”, [online], Transactions of the Virtual Reality Society of Japan, Vol.16, No.2, pp.239-250, June 2011, [October 22, 2012 search], Internet <URL: http://www.rm.is.ritsumei.ac.jp/ Available from pdf / TVRSJ_MR5_mori.pdf> Yuping Shen et al., “Video Completion for Perspective Camera Under Constrained Motion”, [online], The 18th International Conference on Pattern Recognition (ICPR'06), IEEEE, p. 63-66, 2006, [October 22, 2012 Day search], Internet <URL: http://www.google.co.jp/url?sa=t&rct=j&q=video%20completion%20for%20perspective%20camera%20under%20constrained%20motion&source=web&cd=1&ved=0CCcQFjAA&url= Available from http% 3A% 2F% 2Fcil.eecs.ucf.edu% 2Fpdf% 2Fshen_icpr_video.pdf & ei = ceaEUPauGs3SmAWvjIDQBA & usg = AFQjCNHr_qj0nUv185zD_d9mBoXOTpSp_w> Ippei Sugimoto, “Hidden reality for mixed reality: visual removal of artifacts placed for the purpose of improving tracking performance”, [online], 16th Annual Conference of the Virtual Reality Society of Japan, 14D-1, p.318-321, September 2011, [October 22, 2012 search], Internet <URL: http://www.rm.is.ritsumei.ac.jp/pdf/VRSJ2011_sugimoto.pdf> Possible Takeshima Nakajima et al., “Realization of image quality matching of hidden background restoration in hidden reality”, [online], IEICE, Pattern Recognition / Media Understanding Study Group, IEICE Technical Report, Vol. 110, No 381, PRMU2010-195, p.359-364, January 2011, [October 22, 2012 search], Internet <URL: http://www.rm.is.ritsumei.ac.jp/pdf /sigmr_nakajima.pdf> Hirofumi Komukai, et al., “Reduction of hidden background projection error in hidden reality-Dealing with geometrical inconsistencies using mosaicing method”, Proceedings of the 17th Annual Conference of the Virtual Reality Society of Japan, p.87-90 September 2012, [October 22, 2012 search], Internet <URL: http://www.mclab.ics.ritsumei.ac.jp/pdf/vrsj2012_komukai.pdf> Wataru Koshi et al., “Study on Semi-Hidden Display Model in Hidden Reality”, [online], Proceedings of the 16th Annual Conference of the Virtual Reality Society of Japan, 14D-2, p.322-325, September 2011, [ Retrieved on October 22, 2012], available from the Internet <URL: http://www.rm.is.ritsumei.ac.jp/pdf/VRSJ2011_koshi.pdf> Ai Takahashi et al., “Realization and examination of semi-hidden display method in hidden reality”, [online], 2012 IEICE General Conference Information and Systems Proceedings 2, D-12-80, p.174 , March 2012, [October 22, 2012 search], Internet <URL: http://www.rm.is.ritsumei.ac.jp/pdf/VRSJ2011_koshi.pdf> Tomoaki Harada, “Rejection of visual field interference from dynamic scenes using camera direction change”, [online], Information Processing Society of Japan Research Report, 2004-CVIM-144 (2), p.9-16, 2004 May 6, 2012 [Search October 22, 2012], available from the Internet <URL: http://ci.nii.ac.jp/lognavi?name=nels&lang=jp&type=pdf&id=ART0002932628> David G. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints", [online], the International Journal of Computer Vision, p. 1-28, 2004, [October 25, 2012 search], Internet <URL: Available from http://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf Mustafa Oezuysal et al., “Fast Keypoint Recognition using Random Ferns”, [online], IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol.32, Nr.3, p.448-461, March 2010 Internet <URL: http: //www.google.com/url?sa=t&rct=j&q=fast%20keypoint%20recognition%20using%20random%20ferns&source=web&cd=1&cad=rja&ved=0CCwQFjAA&url=http%3A%2F%2Fcvlab.epfl.ch% 2F ~ lepetit% 2Fpapers% 2Fozuysal_pami10.pdf & ei = K4GIUOAs0IiZBbffgMAG & usg = Available from AFQjCNFl103ERNlTyw89lEW250luPufsSg>

  Video recording equipment (eg, digital video camera and digital movie camera) or various devices equipped with video recording functions (eg, digital camera, mobile phone, smartphone, portable music player, game terminal, tablet) With the widespread use of terminals, car navigation systems, head mounted displays, televisions, and computers (especially notebook computers), there are increasing opportunities for shooting moving images.

  A moving image shooting device or a device equipped with a moving image shooting function includes a monitor that allows a user to view a moving image being shot. Therefore, the user can check the moving image being shot on the monitor in real time while shooting the moving image.

  Human beings can capture things not only by vision but also by other senses. For example, it is an olfactory smell or an auditory sound. The user can capture an event being shot with a sense other than vision by shooting the moving image being shot on the monitor in real time instead of the recorded moving image.

  However, an object that enters the user's vision, particularly a moving object (for example, a passerby, an animal, a vehicle, a train, or an airplane) may become an obstacle, and the user's sense may be hindered.

  SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide a technique for erasing a moving object to be concealed in real time while capturing a scene that is actually captured during moving image capturing.

  In addition, the present invention provides a technique for erasing all moving object objects other than moving object objects that are not desired to be concealed while moving a moving image.

  It is another object of the present invention to provide a technique for automatically obscuring a part or all of a moving object in real time while photographing a moving image without changing a scene actually taken.

  In addition, the present invention provides a technique for displaying a part or all of a moving object in real time while the scene being actually shot is taken during moving image shooting, and obscuring the moving object itself in real time. Objective.

  The present invention provides a technique for concealing a moving object in a moving image in real time during moving image shooting by the moving image shooting device. The technique may include a method for concealing the moving object in real time, a moving image shooting device, a moving image shooting device program, and a moving image shooting device program product.

The above method of the present invention comprises:
Detecting at least one moving object in the video;
Calculating at least one moving direction and speed of the detected moving object;
A step of speculatively acquiring a background image when the moving object moves in the calculated moving direction and speed, wherein the speculatively acquired background image hides the moving object in real time. And obtaining said step, which is used for

  In one embodiment of the present invention, the method displays the speculatively acquired background image superimposed on the moving object in the moving image on the display device, and hides the moving object in real time. A step may further be included.

  In one embodiment of the present invention, the method may further include allowing a user to select at least one moving object among the detected moving objects.

  In one embodiment of the present invention, the method selects whether to hide the selected moving object in real time or to hide a moving object other than the selected moving object in real time. It may further include a step of allowing

  In one embodiment of the present invention, the step of calculating includes a step of calculating a moving direction and a speed of the selected moving object, and the step of obscuring hides the selected moving object in real time. A step of erasing may be included.

  In one embodiment of the present invention, the calculating step includes a step of calculating a moving direction and a speed of a moving object other than the selected moving object, and the step of obscuring is the selected moving object. It may include a step of obscuring other moving object in real time.

In one embodiment of the invention, the method comprises
Recognizing the type or shape of the detected moving object;
Selecting the moving object whose recognized type or shape matches the type or shape designated to hide the moving object.

In addition, the moving image photographing apparatus of the present invention is
Imaging means;
Detecting means for detecting at least one moving object in the moving image during moving image shooting by the imaging means;
Calculating means for calculating at least one moving direction and speed of the detected moving object;
Speculative background image acquisition means for speculatively acquiring a background image when the moving object moves with the calculated moving direction and speed, wherein the speculatively acquired background image The speculative background image acquisition means used for erasing in real time is provided.

  In one embodiment of the present invention, the moving image shooting device displays the speculatively acquired background image superimposed on the moving object in the moving image on a display device, and hides the moving object in real time. It may further comprise a concealing means for erasing.

  In one embodiment of the present invention, the moving image shooting device may further include storage means for storing a moving image in which the moving object is hidden in real time. The storage means may be, for example, a storage medium in the moving image shooting device, or a storage medium accessible by the moving image shooting device via a network. The storage medium accessible by the moving picture photographing device via the network is, for example, NAS; for example, the storage medium in the server, tablet terminal or notebook computer accessible by the moving picture photographing device via the network or connected thereto It can be a storage medium.

  The present invention also provides a moving image shooting device program and a moving image shooting device program product that cause a moving image shooting device to execute the steps of the above method.

  The moving picture shooting apparatus program for executing the functions of the present invention can be embedded in advance in a ROM in the moving picture shooting apparatus. Alternatively, a program for a moving image photographing device for executing the function of the present invention includes a flexible disk, MO, CD-ROM, DVD, BD, hard disk device, memory medium connectable to USB, ROM, MRAM, It can be stored in an arbitrary moving picture photographing device such as a RAM, a computer connectable to the moving picture photographing device, or a readable recording medium connectable to the computer. The moving picture photographing apparatus program can be downloaded from a computer connected via a communication line or copied from another recording medium for storage in the recording medium. In addition, the moving image shooting apparatus program can be compressed or divided into a plurality of pieces and stored in a single or a plurality of recording media. It should also be noted that it is of course possible to provide a program product for a moving image photographing device that implements the present invention in various forms. The moving image shooting device program product may include, for example, a storage medium that records the moving image shooting device program or a transmission medium that transmits the moving image shooting device program.

  It should be noted that the above summary of the present invention does not enumerate all necessary features of the present invention, and that combinations or sub-combinations of these components may also be the present invention.

  Various modifications such as combining each hardware component of the moving image shooting apparatus used in the embodiment of the present invention with a plurality of machines and allocating and executing functions to them can be easily assumed by those skilled in the art. Of course. These modifications are naturally included in the concept of the present invention. However, these constituent elements are examples, and not all the constituent elements are essential constituent elements of the present invention.

  The present invention can be realized as hardware, software, or a combination of hardware and software. A typical example of execution by a combination of hardware and software is execution in an apparatus in which the moving image shooting apparatus program is installed. In such a case, the moving picture photographing apparatus program is loaded into the memory of the apparatus and executed, so that the moving picture photographing apparatus program controls the apparatus to execute the processing according to the present invention. The moving image shooting apparatus program can be composed of a group of instructions that can be expressed in any language, code, or notation. Such a set of instructions can be used by the device to perform a specific function directly, or 1. conversion to other languages, codes or notations; It can be executed after one or both of copying to another medium is performed.

  According to the embodiment of the present invention, the moving image shooting device hides the moving object selected by the user in the moving image or all moving objects other than the moving object selected by the user in the background image of the moving object in real time during moving image shooting. It is possible. In other words, the present invention enables a user to view in real time a moving image in which a moving object that is a concealment target is replaced with a background image that is concealed by the moving object that is a concealment target. Therefore, the user can avoid seeing a disturbing or unnecessary moving object in real time in the screen display during moving image shooting. The technique described above is useful in the following scenes, for example, because it enables the moving object to be hidden in a real-time image instead of a recorded moving image.

-For example, in a scene where a child's athletic meet or play party is being shot, a user who is shooting a movie hides, for example, a passerby (which is a moving object) who blocks his child (which is a moving object). Is possible.
For example, in a scene where a movie is shot at a tourist spot during a trip, the user who is shooting the movie hides a passerby (moving object) that blocks a lover or family member (moving object), for example. Is possible.
・ For example, in a scene where a cityscape is shot (for example, a scene where an electronic map or street view is created) or a scene where a drama, a movie, or a commercial is shot, a user who is shooting a movie has a specific traffic It is possible to hide a person, hide all passers-by other than a specific passer-by, or hide all passers-by.

  In accordance with embodiments of the present invention, security checks may be performed, for example, in public facilities (eg, airport, rail and bus terminals), commercial facilities (eg, museums, zoos, aquariums, and movie theaters), school facilities, etc. When a suspicious person that is a moving object is found in a scene, the surveillance camera monitor hides all moving objects (such as passers-by or spectators) other than the suspicious person in real time with the background image of the moving object. Is possible. Therefore, the supervisor may be able to quickly identify the suspicious person while shooting a moving image.

  According to an embodiment of the present invention, for example, when a criminal is found in a road surveillance camera in a scene where a criminal is searched, the surveillance person of the road surveillance camera can detect a moving object other than the criminal (for example, a passerby or a spectator). ) It is possible to hide everything in real time with the background image of the moving object. Therefore, the monitor may be able to quickly search for the criminal while shooting a moving image.

  According to the embodiment of the present invention, when a lost child, which is a moving object, is found in a scene where a lost child is searched, for example, in a facility such as an airport, a railroad, or a commercial facility, It is possible to hide in real time all moving object objects (for example, passers-by or onlookers) other than children that are considered to be a background image of the moving object object. Therefore, the supervisor may be able to quickly find a lost child while shooting a moving image.

  According to an embodiment of the present invention, for example, when a suspicious person, which is a moving object, possesses a dangerous object (for example, a blade or a firearm), the user can move a moving object other than the approaching dangerous object (for example, a passerby or a spectator). (Or a suspicious person) can be hidden in real time with the background image of the moving object. Therefore, the user may be able to avoid danger by watching a moving image in which only dangerous substances are focused.

It is the figure which showed an example of the hardware constitutions for implement | achieving the moving image imaging device which can be used in the embodiment of this invention. It is the figure which showed an example of the hardware constitutions for implement | achieving the terminal connected to the moving image imaging device and the said moving image imaging device which can be used in the embodiment of this invention via the network. An example in which a background image of a moving object that is an object to be hidden is speculatively acquired according to an embodiment of the present invention, and a moving image in which the moving object is hidden in real time is displayed on the display device of the moving image shooting device Show. According to the embodiment of the present invention, the background image of the moving object that is the object of concealment is speculatively acquired, and a moving image in which all of the moving object is concealed in real time is displayed on the display device of the moving image photographing device. An example is shown. According to the embodiment of the present invention, the background image of the moving object that is the object of concealment is speculatively acquired, and a moving image in which all of the moving object is concealed in real time is displayed on the display device of the moving image photographing device. An example is shown. According to the embodiment of the present invention, a background image of a moving object to be hidden is speculatively acquired, and a moving image in which the moving object selected by the user is hidden in real time is displayed on the display device of the moving image shooting device. An example is shown. According to the embodiment of the present invention, a background image of a moving object to be hidden is speculatively acquired, and a moving object other than the user-selected moving object is hidden in real time on the display device of the video shooting device. An example in which a moving image is displayed is shown. According to the embodiment of the present invention, a background image of a moving object that is a concealment target is speculatively acquired, and a moving object other than the moving object (dangerous object) automatically selected on the display device of the moving image photographing apparatus is displayed. An example is shown in which a moving image that is hidden or partially hidden in real time is displayed. FIG. 6 shows a flowchart for processing to speculatively acquire a background image of a moving object that is a concealment target and hide the moving object in real time on the display device of the moving image shooting device according to an embodiment of the present invention. FIG. FIG. 6 shows a flowchart for processing to speculatively acquire a background image of a moving object that is a concealment target and hide the moving object in real time on the display device of the moving image shooting device according to an embodiment of the present invention. FIG. FIG. 6 shows a flowchart for processing to speculatively acquire a background image of a moving object that is a concealment target and hide the moving object in real time on the display device of the moving image shooting device according to an embodiment of the present invention. FIG. FIG. 6 shows a flowchart for processing to speculatively acquire a background image of a moving object that is a concealment target and hide the moving object in real time on the display device of the moving image shooting device according to an embodiment of the present invention. FIG. FIG. 6 shows a flowchart for processing to speculatively acquire a background image of a moving object that is a concealment target and hide the moving object in real time on the display device of the moving image shooting device according to an embodiment of the present invention. FIG. It is a model figure which shows acquiring the background image of the moving body object which is a concealment object speculatively according to the embodiment of the present invention. FIG. 1B is a diagram illustrating an example of a functional block diagram of a moving image shooting apparatus that preferably includes a hardware configuration according to FIG. 1A and according to an embodiment of the present invention. It is the figure which preferably provided the hardware constitutions according to Drawing 1B, and showed an example of the functional block diagram of the video photography equipment according to the embodiment of the present invention. 1B is a diagram illustrating an example of a functional block diagram of a moving image photographing apparatus with a dangerous substance finding function according to an embodiment of the present invention, preferably having a hardware configuration according to FIG. 1A. FIG.

  Embodiments of the present invention will be described below with reference to the drawings. Throughout the following drawings, the same reference numerals refer to the same objects unless otherwise specified. It should be understood that the embodiments of the present invention are intended to illustrate preferred aspects of the present invention and are not intended to limit the scope of the invention to what is shown here.

FIG. 1 is a diagram showing an example of a hardware configuration for realizing a moving image photographing apparatus (101) that can be used in an embodiment of the present invention.
The moving image shooting device (101) includes various devices equipped with a moving image shooting dedicated device and a moving image shooting function. Examples of the dedicated moving image shooting device include, but are not limited to, a digital video camera and a digital movie camera. For example, digital camera, mobile phone, smartphone, game terminal, tablet terminal, personal digital assistant, car navigation system, head mounted display, television and computer ( For example, it includes, but is not limited to, notebook computers.

  1A is an example of a computer equipped with a moving image shooting function. The computer (101) includes a CPU (102) and a main memory (103), which are connected to a bus (104). The CPU (102) is preferably based on a 32-bit or 64-bit architecture, for example, Intel Core i (TM) series, Core 2 (TM) series, Atom (TM) series, Xeon (TM). Series, Pentium (R) Series or Celeron (R) Series, AMD A Series, Phenom (TM) Series, Athlon (TM) Series, Turion (TM) Series or Sempron (TM), Apple (R) CPU of A series) or Android terminal can be used. A display (106), for example, a liquid crystal display (LCD) (for example, a touch panel display) can be connected to the bus (104) via a display controller (105). The display (106) displays information displayed by operating software running on a computer, for example, software according to the present invention, as well as moving images during moving image shooting and moving images in which moving objects are hidden in real time according to the present invention. Can be used to display with a suitable graphic interface. The bus (104) is also optionally connected via a SATA or IDE controller (107) to a disk (108) such as a hard disk or solid state drive and a drive (109) such as a CD, DVD or BD drive. Can be connected. Further, a keyboard (111) and a mouse (112) may optionally be connected to the bus (104) via a peripheral device controller (110) such as a keyboard / mouse controller or a USB bus. The peripheral device controller (110) can be connected with a moving image photographing means (1116) and a sensor means (117). The moving image shooting means (1116) and the sensor means (117) may be embedded in the moving image shooting device (101) or connected to the moving image shooting device (101) via a cable or via a wireless network. May be.

  The moving image photographing means (116) is not particularly limited as long as it is a device having a function capable of photographing a moving image. The moving image photographing means (116) is, for example, a camera module having a photographing element, such as a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor. sell. The moving image capturing means (116) can be used to generate a series of input images by capturing a real space.

  The sensor means (117) can include, for example, a distance acquisition sensor means (118) and an acceleration acquisition sensor means (119). In addition, the sensor means (117) may include, for example, a gyro sensor, a geomagnetic sensor, and / or a positioning sensor.

  The distance acquisition sensor means (118) is an infrared method, a laser method, or a super method capable of measuring a distance between an object in real space (for example, a moving object) and a moving image photographing device (101) (which may be a user's position). It can be a sonic range sensor. The distance acquisition sensor means (118) may be provided so as to face an arbitrary direction, but may be provided along a direction different from the optical axis of the moving image photographing means (116), for example. In the infrared method, for example, a three-dimensional distance measuring infrared camera can be used. A three-dimensional distance measurement infrared camera is a three-dimensional distance camera that uses a time-of-flight (TOF) distance image sensor that calculates distance from the time of flight of light, and measures the distance of an object in the field of view. It is possible. For example, a laser distance meter may be used in the laser system. The laser distance meter can measure a distance from several centimeters to several tens of meters, and further to about 200 by reflection of laser light. In the ultrasonic method, for example, an ultrasonic wave of about 40 KHz is used, and after the ultrasonic wave is transmitted, the time until it bounces off at the measurement object (object to be measured) is measured and measured with the moving image shooting device (101). It is possible to measure the distance between the object.

  The acceleration acquisition sensor means (119) is a sensor for acquiring acceleration, and can be, for example, a sensor capable of measuring the three-axis acceleration of the user having the moving image photographing device (101). The acceleration acquisition sensor means (119) can also determine whether the user with the moving image shooting device (101) is stationary or moving.

  The gyro sensor (not shown) can be a sensor capable of measuring the tilt angle of the moving image photographing device (101).

  The geomagnetic sensor (not shown) can be a sensor capable of measuring the orientation of the moving image photographing device (101).

  The positioning sensor (not shown) may be a sensor that measures the position of the moving image shooting device (101). The positioning sensor (not shown) can be, for example, a GPS sensor that can receive a Global Positioning System (GPS) signal and measure the latitude, longitude, and altitude of the moving image photographing device (101). . Alternatively, the positioning sensor (not shown) may be, for example, a PlaceEngine (place engine) that measures the position of the moving image shooting device (101) based on the radio field strength of a wireless signal received from a wireless LAN access point. Registered trademark). Alternatively, the positioning sensor (not shown) can be, for example, a hybrid positioning combining a global positioning system (GPS) and PlaceEngine (registered trademark).

  The disk (108) includes an operating system such as Windows (registered trademark), UNIX (registered trademark), MacOS (registered trademark), OS (trademark) for smartphone, Java (registered trademark) processing environment such as J2EE, Java ( (Registered trademark) application, Java (registered trademark) virtual machine (VM), program providing Java (registered trademark) runtime (JIT) compiler, application program for moving image shooting function, enabling implementation of the present invention Application programs and other programs and data (for example, moving image data) can be stored in the main memory (103) so as to be loaded.

  The disc (108) may be built in the moving image shooting device (101), may be connected via a cable so that the moving image shooting device (101) can be accessed, or the moving image shooting device (101). 101) may be connected via a wired or wireless network so that they can be accessed.

  The drive (109) can be used to install a program from a CD-ROM, DVD-ROM or BD to the disk (108), or to store moving image data as required.

  The communication interface (114) follows, for example, the Ethernet (registered trademark) protocol. The communication interface (114) is connected to the bus (104) via the communication controller (113) and plays a role of physically connecting the computer (101) to the communication line (115). A network interface layer is provided for the TCP / IP communication protocol of the communication function of the system. The communication line can be a wired LAN environment or a wireless LAN environment based on a wireless LAN connection standard such as IEEE802.11a / b / g / n.

  The computer may be a personal computer, such as a desktop computer, a notebook computer, or a cloud-use terminal, but is not limited thereto.

FIG. 1B shows an example of a hardware configuration for realizing the moving image shooting device (121) and the terminal (131) connected to the moving image shooting device via the network, which can be used in the embodiment of the present invention. FIG.
In the moving image shooting apparatus (101) shown in FIG. 1A, the technique according to the present invention, that is, the technique for obscuring the moving object in the moving image in real time during moving image shooting can be realized only by the moving image shooting apparatus (101). As shown. However, in the technique according to the present invention, as shown in FIG. 1B, the elements (102 to 119) shown in the moving image photographing apparatus (101) in FIG. 1A can be appropriately separated and mounted on a plurality of apparatuses.

  The moving image shooting device (121) shown in FIG. 1B is capable of collecting at least data for shooting a moving image and determining the distance or acceleration of the moving image shooting device (121) and / or moving object. That's fine. For this purpose, the moving image shooting device (121) only needs to include at least a sensor means (122), a moving image shooting means (125), and a projection means (126), for example. The sensor means (122) may include a distance acquisition sensor means (123), an acceleration acquisition sensor means (124), and other sensor means (for example, a gyro sensor, a geomagnetic sensor, and a positioning sensor). The sensor means (122), moving image photographing means (125) and projection means (126) shown in FIG. 1B correspond to the sensor means (117), moving image photographing means (116) and display (106) shown in FIG. 1A, respectively. .

  The moving image photographing device (121) can be connected to the terminal (131) via a network, for example, a wireless network (WiFi, Bluetooth (registered trademark), wireless LAN). The wireless LAN complies with, for example, IEEE 802.11 a, b, g, n standards.

  The terminal (131) receives image data, distance data, and acceleration data from the moving image shooting device (121), and generates moving image data in which a moving object in the moving image is hidden in real time according to the present invention. Can be done. The hardware configuration of the terminal (131) can correspond to, for example, the hardware configuration illustrated in FIG. 1A.

  The moving image shooting device (121) may be a device with low processing capability such as a head mounted display, a digital camera, a mobile phone, a smartphone, a game terminal, and a tablet terminal. On the other hand, the terminal (131) can be, for example, a device capable of processing a technique for obscuring a moving object in a moving image in real time during moving image shooting according to the present invention. The terminal (131) can be, for example, a computer (including a server) and a tablet terminal.

Examples of usage according to the present invention of the moving image photographing device (121) and terminal (131) shown in FIG. 1B are as follows, for example.
For example, it is assumed that the user is wearing a head mounted display (121) and is shooting a moving image. The head mounted display (121) transmits to the terminal (131) data that makes it possible to determine the captured moving image data, the distance or acceleration of the head mounted display and / or moving object. The terminal (131) may be possessed by the user or may be in a remote place. The head mounted display (121) receives from the terminal (131) image data in which the moving object in the moving image is hidden in real time. The head mounted display (121) projects the image data received from the terminal (131) onto the projection means (126).

  In FIG. 2A to FIG. 4 shown below, first, in order to facilitate understanding of the present invention, according to the embodiment of the present invention, a background image of a moving object to be hidden is speculatively acquired, An example is shown in which a moving image in which a moving object is hidden in real time is displayed on a display device of a photographing apparatus. In the examples of FIGS. 2A to 4, illustrations are used to facilitate the description of the embodiment of the present invention. However, the present invention is not limited to the real space photographed by the moving image photographing device (101, 121). Please note that this is for video data. In the following, it should be noted that when the moving image shooting device (101) is indicated, the moving image shooting device (121) is not excluded.

FIG. 2A shows that a background image of a moving object to be hidden is speculatively acquired according to an embodiment of the present invention, and the moving object is hidden in real time on the display device (106) of the moving image shooting device (101). An example in which the recorded moving image is displayed is shown.
Each frame (201 to 205) of the moving image data is a still image that constitutes a moving image in which an exhibit in the museum is being photographed. By displaying each frame (201 to 205) in succession, a moving image is reproduced on a display device of a moving image photographing device that does not implement the present invention. As shown in each frame (201 to 205), there is one person (moving object) who is viewing the exhibit.
For each frame (211 to 215) of the moving image data, according to the embodiment of the present invention, the background image of the moving object that is the object of concealment is speculatively acquired, and the moving image data (101) is displayed on the display device (106). FIG. 5 shows how a moving object is hidden in real time. By displaying each frame (211 to 215) continuously, a moving image is reproduced on the display device (106) of the moving image photographing device (101) implementing the present invention.

  It is assumed that a user who is shooting a movie wants to delete the onlooker when shooting an exhibit. It is assumed that the user calls the menu of the moving image shooting device (101) and selects the moving object deletion mode at the time of the frame (211). Assume that the moving image shooting device (101) detects a moving object in response to the selection of the deletion mode. The detected moving object can be specified by, for example, displaying the moving object in white or surrounding it with a yellow line, for example, as shown in the frame (212). . Since the moving image object (101) has only one detected moving object (onlooker), the moving image photographing apparatus (101) regards the onlooker as a moving object that is a target to be hidden. The moving image shooting device (101) predicts and calculates the future movement direction and speed of the spectator who is the object of concealment based on the actual movement direction and speed of the spectator, and the calculated movement direction. And speculatively acquiring a background image when the spectator is predicted to move at a speed. Next, the moving image photographing device (101) superimposes the speculatively acquired background image on the spectator (also referred to as superimposed drawing) and hides the spectator in real time. As a result, as shown in the frames (213, 214, and 215), the moving image photographing device (101) displays the moving image in which the spectator is hidden in the speculatively acquired background image on the display device (106). Display above.

FIG. 2B shows that the background image of the moving object to be hidden is speculatively acquired according to the embodiment of the present invention, and all of the moving object objects are displayed in real time on the display device (106) of the moving image photographing device (101). An example in which a hidden video is displayed is shown.
Each frame (221 to 225) of the moving image data is a still image that constitutes a moving image in which an exhibit in the museum is being photographed. By displaying each frame (221 to 225) in succession, a moving image is reproduced on a display device of a moving image photographing device that does not implement the present invention. As shown in each frame (221 to 225), there are a plurality of people (moving object) who are viewing the exhibits.
In each frame (231 to 235) of the moving image data, according to the embodiment of the present invention, the background image of the moving object that is the object of concealment is speculatively acquired, and the moving image data (101) is displayed on the display device (106). FIG. 5 shows how a moving object is hidden in real time. By continuously displaying each frame (231 to 235), a moving image is reproduced on the display device (106) of the moving image photographing device (101) implementing the present invention.

  It is assumed that a user who is shooting a movie wants to delete the plurality of spectators when shooting an exhibit. It is assumed that the user calls the moving image shooting device (101) menu at the time of the frame (231) and selects the moving object deletion mode. It is assumed that the moving image shooting device (101) detects a plurality of moving objects in response to the selection of the deletion mode. As shown in the frame (232), for example, the plurality of detected moving objects are displayed in white, or, for example, a single color ( For example, it may be specified by being surrounded by a line of a different color (for example, yellow, green, blue, etc.) for each moving object. For example, since there are a plurality of detected moving object, the moving image shooting device (101) deletes (hides) a part or all of the moving object, or deletes the moving object selected by the user ( The user is allowed to select whether to delete (hide) all moving objects other than the moving object selected by the user. It is assumed that the user has selected to delete all of the moving object (looker). The moving image shooting device (101) calculates and calculates the future movement direction and speed of all the spectators who are the objects of concealment based on the actual movement direction and speed of each spectator. A speculative background image is obtained when each of the spectators is predicted to have moved in the moving direction and speed. Next, the moving image shooting device (101) superimposes the background image acquired speculatively on each of the above-mentioned spectators and hides all the spectators in real time. As a result, as shown in the frames (233, 234, and 235), the moving image shooting device (101) displays a moving image in which all the spectators are hidden in the speculatively acquired background image on its display device ( 106) is displayed above.

In FIG. 2C, according to an embodiment of the present invention, a background image of a moving object that is a concealment target is speculatively acquired, and all of the moving object objects are displayed in real time on the display device (106) of the moving image photographing device (101). An example in which a hidden video is displayed is shown.
Each frame (241 to 245) of the moving image data is a still image that constitutes a moving image in which the user is photographing a cityscape on the vehicle or in the vehicle. By displaying each frame (241 to 245) continuously, a moving image is reproduced on a display device of a moving image photographing device that does not implement the present invention. As shown in each frame (241 to 245), a pedestrian and a traveling vehicle (moving object) can be seen.
For each frame (251 to 255) of the moving image data, according to the embodiment of the present invention, the background image of the moving object that is the object of concealment is speculatively acquired, and the moving image data (101) is displayed on the display device (106). FIG. 5 shows how a moving object is hidden in real time. By displaying each frame (251 to 255) continuously, a moving image is reproduced on the display device (106) of the moving image photographing device (101) implementing the present invention.

  Assume that a user who is shooting a moving image wants to delete moving objects such as pedestrians, bicycles, and traveling vehicles when shooting a cityscape. Assume that the user calls the menu of the moving image shooting device (101) and selects the moving object deletion mode at the time of the frame (251). It is assumed that the moving image shooting device (101) detects a plurality of moving objects in response to the selection of the deletion mode. The detected plurality of moving objects are, for example, the plurality of moving objects displayed in white, or, for example, a single color (for example, yellow) line or each moving object so as to surround the plurality of moving objects. Can be specified by being surrounded by lines of different colors (for example, yellow, green, blue, etc.). For example, since there are a plurality of detected moving object, the moving image shooting device (101) deletes (hides) a part (for example, only a person) or all of the moving object, or the user selects The user is allowed to select whether to delete (hide) a moving object or to delete (hide) all moving objects other than the moving object selected by the user. It is assumed that the user has selected to delete (hide) all of the moving object. The moving image shooting device (101) calculates and calculates the future moving direction and speed of all the moving objects that are the objects to be hidden based on the actual moving direction and speed of each moving object, and A speculative image is acquired when each moving object is predicted to move with the calculated moving direction and speed. Alternatively, the moving image shooting device (101) is set to a mode in which all moving object objects are deleted by default, and future movement of all moving object objects to be hidden is set as a default. A direction image and a speed are calculated by predicting based on the actual moving direction and speed of each moving object, and a background image when each moving object is predicted to move in the calculated moving direction and speed. Is acquired speculatively. Next, the moving image shooting device (101) superimposes the speculatively acquired background image on each of the moving object objects, and hides all the moving object objects in real time. As a result, as shown in the frames (253, 254, and 255), the moving image photographing device (101) displays a moving image in which all the moving object objects are hidden in the speculatively acquired background image. (106) Display above.

FIG. 3A shows a moving object selected by the user on the display device (106) of the moving image shooting device (101), where a background image of the moving object to be hidden is speculatively acquired according to the embodiment of the present invention. An example in which a moving image hidden in real time is displayed is shown.
Each frame (301 to 305) of the moving image data is a still image constituting a moving image in which a child who is near the exhibit in the museum is being photographed. By displaying each frame (201 to 205) in succession, a moving image is reproduced on a display device of a moving image photographing device that does not implement the present invention. As shown in each frame (301 to 305), there are a plurality of people (moving object) who are viewing the exhibits, including the children.
For each frame (311 to 315) of the moving image data, according to the embodiment of the present invention, the background image of the moving object that is the object of concealment is speculatively acquired, and the moving image data (101) is displayed on the display device (106). Fig. 5 shows how the moving object other than the child is hidden in real time. By displaying each frame (311 to 315) continuously, a moving image is reproduced on the display device (106) of the moving image photographing device (101) implementing the present invention.

  Assume that a user who is shooting a movie wants to delete the above-mentioned spectators other than his / her child when shooting his / her child near the exhibit. It is assumed that the user calls the moving image shooting device (101) menu at the time of the frame (311) and selects the moving object deletion mode. It is assumed that the moving image shooting device (101) detects a plurality of moving objects in response to the selection of the deletion mode. As shown in the frame (312), for example, the detected plurality of moving objects are displayed in white, or, for example, a single color ( For example, it may be specified by being surrounded by a line of a different color (for example, yellow, green, blue, etc.) for each moving object. For example, since there are a plurality of detected moving object, the moving image shooting device (101) deletes (hides) a part or all of the moving object, or deletes the moving object selected by the user ( The user is allowed to select whether to delete (hide) all moving objects other than the moving object selected by the user. It is assumed that the user has selected to delete (hide) all moving objects (onlookers) surrounded by a line selected by tapping the screen on the display device (106). The moving image shooting device (101) predicts and calculates the future movement direction and speed of the spectator who is the object of concealment based on the actual movement direction and speed of each spectator, and the calculated movement A background image is obtained speculatively when it is predicted that each spectator who is the object of concealment has moved in the direction and speed. Next, the moving image photographing device (101) superimposes the speculatively acquired background image on each of the selected viewers to be hidden, and hides all the viewers to be hidden in real time. As a result, as shown in the frames (313, 314, and 315), the moving image shooting device (101) has the background image acquired speculatively, and all the onlookers to be hidden are hidden, A moving image that is not hidden by the child is displayed on the display device (106).

FIG. 3B shows that a background image of a moving object to be hidden is speculatively acquired according to an embodiment of the present invention, and a moving object other than the moving object selected by the user on the display device of the moving image shooting apparatus is displayed in real time. An example in which a hidden video is displayed is shown.
Each frame (321 to 325) of the moving image data is a still image constituting a moving image in which a child who is near the exhibit in the museum is being photographed. By displaying each frame (321 to 325) continuously, a moving image is reproduced on a display device of a moving image photographing device that does not implement the present invention. As shown in each frame (321 to 325), there are a plurality of people (moving object) who are viewing the exhibits, including the above-mentioned children.
For each frame (331 to 335) of the moving image data, according to the embodiment of the present invention, the background image of the moving object to be hidden is speculatively acquired, and the moving image data on the display device (106) of the moving image shooting device (101) is obtained. Fig. 5 shows how the moving object other than the child is hidden in real time. By displaying each frame (331-335) continuously, a moving image is reproduced | regenerated by the display apparatus (106) of the moving image imaging device (101) which implements this invention.

  Assume that a user who is shooting a movie wants to delete the above-mentioned spectators other than his / her child when shooting his / her child near the exhibit. It is assumed that the user calls the menu of the moving image shooting device (101) at the time of the frame (331) and selects the moving object deletion mode. It is assumed that the moving image shooting device (101) detects a plurality of moving objects in response to the selection of the deletion mode. As shown in the frame (332), for example, the plurality of detected moving object objects are displayed in white, or, for example, a single color ( For example, it may be specified by being surrounded by a line of a different color (for example, yellow, green, blue, etc.) for each moving object. For example, since there are a plurality of detected moving object, the moving image shooting device (101) deletes (hides) a part or all of the moving object, or deletes the moving object selected by the user ( The user is allowed to select whether to delete (hide) all moving objects other than the moving object selected by the user. It is assumed that the user has selected to delete (hide) all moving objects (onlookers) other than his child selected by tapping the screen on the display device (106). The moving image shooting device (101) predicts and calculates the future movement direction and speed of the spectator who is the object of concealment based on the actual movement direction and speed of each spectator, and the calculated movement A background image is obtained speculatively when it is predicted that each spectator who is the object of concealment has moved in the direction and speed. Next, the moving image photographing device (101) superimposes the speculatively acquired background image on each of the selected viewers to be hidden, and hides all the viewers to be hidden in real time. As a result, as shown in the frames (233, 234, and 235), the moving image shooting device (101) has the background image acquired speculatively, all the onlookers to be hidden are hidden, A moving image that is not hidden by the child is displayed on the display device (106).

FIG. 4 shows a moving object other than the moving object (hazardous material) automatically selected on the display device of the moving image photographing device, in which the background image of the moving object to be hidden is speculatively acquired according to the embodiment of the present invention. An example in which a moving image in which the moving object is hidden or partially hidden in real time is displayed.
Each frame (401 to 405) of the moving image data is a still image that constitutes a moving image in which an exhibit is being photographed in the museum, and each frame (201 to 205) is continuously displayed, whereby the present invention is A moving image is reproduced on a display device of a moving image photographing device that is not mounted. As shown in each frame (321 to 325), it is assumed that in addition to a plurality of people (moving object) watching the exhibition, a suspicious person possessing a dangerous material appears. The dangerous object is a moving object because it moves with the suspicious person.
In each frame (411 to 415) of the moving image data, according to the embodiment of the present invention, a background image of each moving object of a spectator and a suspicious person (excluding dangerous objects) to be concealed is speculatively acquired. FIG. 5 shows the manner in which the moving object is hidden in real time on the display device (106) of the moving image photographing device (101). By displaying each frame (411-415) continuously, a moving image is reproduced | regenerated by the display apparatus (106) of the moving image imaging device (101) which implements this invention.

  It is assumed that a suspicious person suddenly carrying a dangerous object appears suddenly as shown in the frame (411) when a user who is shooting a movie is shooting an exhibit. It is assumed that the moving image shooting device (101) is in a mode for warning the user when a dangerous object is recognized.

  It is assumed that the moving image photographing device (101) detects a plurality of moving object objects including dangerous substances in the warning mode. As shown in the frame (412), the detected plurality of moving objects are displayed, for example, in a white color, or in a single color (for example, so as to surround the plurality of moving objects). For example, it may be specified by being surrounded by a line of a different color (for example, yellow, green, blue, etc.) for each moving object. The moving image shooting device (101) calculates and calculates the future moving direction and speed of the moving object that is the object of concealment based on the actual moving direction and speed of each moving object. A speculative image is obtained for each of the moving object objects that are to be concealed with the moving direction and speed. Next, the moving image photographing device (101) superimposes the speculatively acquired background image on each of the moving object to be hidden, and hides the moving object to be hidden in real time. However, the moving image shooting device (101) does not hide all the suspicious persons among the moving object, but can partially hide them, that is, intentionally leaving them a little. In addition, when the distance between the suspicious person and the user falls within a predetermined distance, the moving image shooting device (101) releases the suspicious person's concealment process and completely displays the suspicious person. (See frame 415). Furthermore, the moving image photographing device (101) can display the suspicious person in a highlighted manner. The highlighting is performed by, for example, surrounding the suspicious person with a red line, blinking the suspicious person's image, or repeatedly displaying the suspicious person's image intentionally or repeatedly. Can be done. In addition, the moving image shooting device (101) may not intentionally hide a dangerous object possessed by a suspicious person among moving object objects (see frames 413, 414, and 415). Furthermore, the moving image photographing device (101) can display the dangerous article in a highlighted manner to the user. The highlighting is performed by, for example, surrounding the dangerous object with a red line, blinking the image of the dangerous object, or repeatedly displaying the dangerous object image intentionally enlarged or reduced. Can be done.

  Further, the moving image photographing device (101) can emit a warning sound or a warning sound registered in advance in response to detecting a plurality of moving objects including the dangerous substance. The warning sound can be made louder as the suspicious person approaches the user, or the warning content can be emphasized (see balloons in frames 412, 413, 414, and 415). .

  5A to 5E speculatively acquire a background image of a moving object to be hidden according to an embodiment of the present invention, and the moving object is displayed in real time on the display device (106) of the moving image shooting device (101). The flowchart for the process to hide is shown.

  In the flowchart shown below, the operation when the moving image photographing device (101) is a digital video camera will be described as an example. In addition, the moving image photographing device (101) can use various technologies listed in Table 1 described in Non-Patent Document 1 as technologies that are the basis for implementing the present invention. Non-patent documents 1 to 10 and various citations or references cited in these non-patent documents are incorporated herein by reference.

  (See FIG. 5A)

  In step 501, the moving image shooting device (101) starts moving image shooting in response to a moving image shooting start command from the user. An instruction to start moving image shooting from the user can be performed, for example, when the user presses a recording start button provided on the moving image shooting device (101). In response to the recording start button being pressed, the moving image shooting device (101) starts shooting moving images, and the shot moving image data is stored in the built-in memory in the recording and shooting machine (101) or the recording and shooting device (101). ) Start recording in the memory card inserted in The recording format of the moving image data may be, for example, MPEG-4, AVI (Audio Video Interleave), MOV, ASF (Advanced Systems Format), 3 gp, DiVX, FLV, or MKV. The memory card may be various memory media that can be attached to the recording and photographing machine (101) such as an SD card, an SDHC card, an SDXC memory card, a compact flash memory card, and a memory stick. Alternatively, if the recording and shooting device (101) has a wireless communication function, via the wireless communication function (for example, 3G or 4G (for example, LTE), wireless LAN, WiFi, WiMAX), Portable storage owned by the user or storage in the wireless network connection terminal possessed by the user, storage on the network accessible via the wireless network connection terminal possessed by the user (for example, , NAS or storage server) or recording storage device (101) on the network that can be directly accessed via the wireless communication function.

  In step 502, the moving image shooting device (101) responds to the moving image shooting start command and hides a moving object in the moving image in real time during moving image shooting (hereinafter also referred to as “hidden mode”). ) To determine whether or not the moving picture photographing device (101) is set. The user may be able to set the default so that the concealment mode works at the start of recording. Alternatively, the user can select the menu on the display screen (106) attached to the moving picture photographing device (106) or hide it so that the hidden mode works as necessary during moving picture shooting. It may be possible to set the concealment mode on by sliding the mechanical switch that turns the mode on in the on direction. If the movie shooting device (101) is not set to the hide mode, step 502 is repeated until the hide mode is set.

  In step 503, the moving image shooting device (101) performs image recognition processing on the input image data captured by the moving image shooting means (116). For example, the moving image photographing device (101) collates the feature data extracted from the input image data with the feature data stored in advance in the storage device (108). Then, the moving image photographing device (101) recognizes a still body and an area in the input image data. Recognition of a still body and a region by collating feature data can be performed, for example, according to the SIFT method and the Random Ferns method. See, for example, Non-Patent Document 9 (David G. Lowe) for the SIFT method. For the Random Ferns method, see, for example, Non-Patent Document 10 (Mustafa Oezuysal et al.).

  Next, the moving image photographing device (101) detects one or a plurality of moving objects in the moving image from the data subjected to the image recognition process. The moving image photographing device (101) can detect the one or more moving object objects by, for example, a color difference between moving image frames and / or a color change. In the embodiment of the present invention, the moving object is not particularly limited as long as the moving object is an object moving at an arbitrary speed in an arbitrary direction. The moving object is, for example, a moving creature (for example, a person, an animal, a bird, an insect), a moving artifact (for example, a bicycle, a car, a train, a bullet train, an airplane), or a moving natural object ( For example, a star) or an object owned by a moving person, for example. In order to make it easy for the user to recognize the detected moving object on the display screen (106), the moving image shooting device (101) displays the outline of the detected moving object in white. Alternatively, for example, a single color (for example, yellow) line or a different color (for example, yellow, green, blue, etc.) may be displayed for each moving object so as to surround the outer periphery of the moving object.

  In step 503, the moving image shooting device (101) can acquire the contour and / or range data of the detected moving object together with the detection of the moving object in the moving image.

  In step 504, the moving image photographing device (101) determines whether or not the user has newly selected a moving object in the current frame on the display screen (106). During video shooting, the user taps (for example, a double tap or a long tap) on a part of the moving object displayed on the display screen (106) or on the outer line surrounding the moving object, or The moving object can be selected by swapping the screen so as to surround the moving object. The moving image shooting apparatus (101) advances the process to step 505 in response to a new selection by the user in the current frame. On the other hand, the moving image shooting device (101) advances the process to step 506 in response to the user not newly selecting in the current frame. The moving image photographing device (101) can determine that the moving object is not selected by the fact that the user has not selected the moving object even after a predetermined time has elapsed since the detection in step 503.

  In step 505, in response to the user selecting a new moving object in the current frame, the moving image shooting device (101) outlines the moving object (the user-selected moving object) in the selected current frame. And / or the range may be obtained from the input image data. In an alternative embodiment, in step 503, when the moving object in the moving image is detected and the contour and / or range data of the detected moving object is already acquired, the moving image shooting device (101) In response to determining that the user has selected a new moving object in the current frame, in step 505, the outline and / or range of the moving object in the selected current frame is determined from the data acquired in step 503. It can be taken out.

  In step 506, the moving image photographing device (101) recognizes the moving distance, color, and size of the moving object selected by the user in the immediately preceding frame described in step 506.

  In step 507, the moving image photographing device (101) has selected the moving object newly selected by the user in the current frame in step 504 in the frame immediately before the current frame (that is, one frame before). Determine if it is the same as the moving object. By using the moving direction and speed of the moving object calculated in the immediately preceding frame, the accuracy of determination as to whether or not the moving object selected by the user is the same is increased. If the moving image shooting device (101) is the same, the process proceeds to step 508. On the other hand, the moving image shooting device (101) returns the processing to step 502 because the concealment processing in the frame currently being processed ends in response to the fact that they are not the same.

  In step 508, the moving image shooting device (101) determines that the moving object selected by the user in the immediately preceding frame (ie, the moving object newly selected by the user in the current frame in step 504) is the same as above. Is a processing candidate moving object.

  In step 503 to step 508 described above, after the user detects the processing candidate moving object, the moving object newly selected by the user in the current frame in step 504 is the frame immediately before the current frame (that is, 1 The embodiment has been described in which it is possible to increase the accuracy of the determination of whether or not the moving object selected by the user is the same by determining whether or not the moving object selected by the user is the same as before the frame). In an alternative embodiment, in step 503, the moving image shooting device (101) detects one or more moving objects in the moving image from the data subjected to the image recognition process, and the detected one or more moving objects. The user may select an arbitrary moving object among the objects, and the selected moving object may be set as the processing candidate moving object in Step 508. That is, in this alternative embodiment, steps 506-508 are not necessary. Therefore, in this alternative embodiment, in step 503, the moving image shooting device (101) detects a moving object in the moving image, and in step 504, the moving image shooting device (101) detects the above-mentioned detection on the display screen (106). It is determined whether or not any one or more of the moving objects have been selected, and in step 505, the selected moving object is regarded as a processing candidate moving object, and the contour of the selected moving object is determined. And / or the range may be obtained.

  In the flowchart shown in FIG. 5A, as shown in step 507, the moving object newly selected by the user in the current frame in step 504 is the same as the moving object selected by the user in the frame immediately before the current frame. In response to the fact that the moving object newly selected by the user in the current frame in step 504 is the same as the moving object selected by the user in the frame immediately before the current frame, The moving object selected by the user in the immediately preceding frame (that is, the moving object newly selected by the user in the current frame in step 504) is regarded as a processing candidate moving object. This is because the moving object selected by the user is accurately specified. However, as described in the examples of FIGS. 2A to 2C, 3A to 3B, and 4, the moving object selected by the user in a certain frame may be used as the processing candidate moving object as it is. In such a case, steps 506 and 507 shown in FIG. 5A can be omitted.

  (See FIG. 5B)

  In step 511, the moving image photographing device (101) determines whether to hide the processing candidate moving object that is regarded as the processing candidate in step 508.

  The user may be able to set a mode (hereinafter also referred to as “first mode”) that hides the processing candidate moving object by default. Alternatively, it may be possible for the user to turn on the first mode by selecting a menu on the display screen (106) so that the first mode works. In response to whether the first mode is set to default or the first mode is turned on, the moving image shooting device (101) advances the process to step 512.

  In addition, the user may be able to set a mode (hereinafter also referred to as “second mode”) that hides moving object other than the processing candidate moving object by default. Alternatively, it may be possible to set the second mode on by the user or by selecting a menu on the display screen (106) so that the second mode works. The moving image shooting device (101) advances the process to step 513 in response to the second mode being set as default or in response to the second mode being turned on.

  In step 512, the moving image shooting device (101) becomes a process candidate moving object (hereinafter referred to as a moving object that is a hidden object) in response to being in the first mode. A mode for acquiring a background image of “moving object” is also referred to as “moving object to be hidden”) is entered.

  In step 513, the moving image photographing device (101) determines the background of the moving object other than the processing candidate moving object (hereinafter, “moving object to be hidden”) in accordance with the second mode. Enter the image acquisition mode.

  In step 514, the moving image shooting device (101) can recognize the outline and / or range of the moving object that is the concealment target in the second mode. In an alternative embodiment, in step 503, when the moving object in the moving image is detected and the contour and / or range data of the detected moving object is already acquired, the moving image shooting device (101) The contour and / or range of the moving object that is the object of concealment in the second mode may be extracted from the data acquired in step 503.

  (See FIG. 5C)

  In step 521, the moving image photographing device (101) calculates the moving direction and speed of at least one or all of the moving object that is the object of concealment in order to conceal the moving object that is the object of concealment with the background image. To do. The moving image shooting device (101) determines the distance between the moving image shooting device (101) and the moving object to be hidden and / or the hidden object in calculating at least one or all of the moving direction and speed. The distance between the moving object and the background (real background) can be calculated.

  In the moving image shooting device (101), whether or not the moving object to be hidden is moving, for example, whether or not the background of the moving object in the moving image from the moving image shooting means (116) has changed, or an acceleration acquisition sensor. (119).

  The moving image shooting device (101) determines whether the moving image shooting device (101) is moving by, for example, a change in the size of a stationary object in the moving image from the moving image shooting means (116) or a difference from the previous frame. It can be judged by seeking.

  The moving image shooting device (101) can calculate the distance between the moving image shooting device (101) and the moving object to be concealed using, for example, a distance acquisition sensor means (118).

  In step 522, the moving image shooting device (101) speculatively acquires a background image used to hide the object to be hidden from the moving direction and speed of the moving object to be hidden. Determine whether or not. The background image acquired speculatively is stored as speculative background image data in the storage medium (108), for example. In response to the speculatively acquired background image, the moving image shooting device (101) advances the process to step 523. On the other hand, in response to the absence of the speculatively acquired background image, the moving image shooting device (101) advances the process to step 524.

  In step 523, the moving image photographing device (101) retrieves the speculatively acquired background image from the storage medium (108) in response to the speculatively acquired background image.

  In step 524, the moving image shooting device (101) may acquire the background image acquired in a speculative manner by a method of acquiring a background image of the related art. For example, the moving image shooting device (101) goes back to the frame before the current frame of the moving image data being shot. That is, the moving image photographing device (101) searches for a frame having a background image that overlaps the range occupied by the moving object that is the object of concealment, and goes back to the searched frame. Then, the moving image shooting device (101) acquires a background image of the moving object that is the object of concealment from the searched frame. In the acquisition, the moving image shooting device (101) considers the distance between the moving image shooting device (101) and the moving object to be hidden and the distance between the moving object and the background thereof. A portion that can be a background image of the target moving object can be cut out from the searched frame. Alternatively, the moving image photographing device (101) can estimate from the surrounding image of the moving object that is the object of concealment and fill the moving object that is the object of concealment.

  In step 525, the moving image shooting apparatus (101) acquires the brightness, saturation, or combination thereof of the background image acquired in step 523 or step 524 as necessary around the moving object (that is, acquisition) as necessary. In accordance with the amount of light and / or shadow of the surrounding (superimposed on the background image), the lightness and saturation of the surroundings are corrected. That is, the moving image shooting device (101) corrects the optical consistency of the background image according to the amount of light and / or shadow around the moving object that is the object of concealment. In addition, the moving image photographing device (101) corrects the enlargement or reduction of the background image acquired in step 523 or step 524 as necessary so as to match the size of the moving object that is the object of concealment. That is, the moving image photographing device (101) corrects the geometric consistency of the background image.

  In step 526, the moving image photographing device (101) superimposes the background image acquired in step 523 or 524 or the background image corrected in step 525 on the moving object that is the object to be hidden. The moving object that is the object of concealment is concealed from the moving image in real time during moving image shooting by the superposition. If there is a speculatively acquired background image in step 522, the concealment process can be performed promptly through the processes in steps 523 to 525 without acquiring the background image in step 524. It becomes possible to increase the speed of real-time processing.

  In step 527, the moving image photographing device (101) determines whether there is a moving object that is an unprocessed object to be hidden. In response to the fact that there is a moving object that is an unprocessed object to be hidden, the video shooting device (101) returns the process to step 521, and steps 521 to 526 (until there are no moving objects that are an unprocessed object to be deleted). In particular, the calculation step, the acquisition step, and the concealment step) are repeated. On the other hand, the moving image shooting device (101) advances the process to step 528 when there is no moving object that is an unprocessed object to be hidden.

  In step 528, the moving image photographing device (101) can store the moving image data in which the moving object to be hidden is hidden in the storage medium (108). The storage medium (108) may be a built-in memory in the recording and photographing machine (101) or a memory card inserted into the recording and photographing apparatus. Alternatively, if the recording and shooting device (101) has a wireless communication function, via the wireless communication function (for example, 3G or 4G (for example, LTE), wireless LAN, WiFi, WiMax), Portable storage owned by the user or storage in the wireless network connection terminal possessed by the user, storage on the network accessible via the wireless network connection terminal possessed by the user (for example, , NAS or storage server) or recording storage device (101) on the network that can be directly accessed via the wireless communication function.

  In step 529, the moving image photographing device (101) displays the moving image data in which the moving object to be hidden is hidden in real time by the background image on the display device (106).

  (See FIG. 5D)

  In step 531, the moving image shooting device (101) determines whether there is a request for calculation of a risk level that may pose a danger to the user during moving image shooting. The degree of danger that may pose a danger to the user may, for example, be that the user who owns the video shooting device (101) collides with a moving object (regardless of whether or not it is an object to be hidden). The degree of danger may be a degree of danger when the moving object has danger. The risk level when the moving object has a danger is, for example, the risk level when the moving object is a person and a dangerous object (for example, a blade, a firearm, etc.), and the moving object is a person. In the case where the person is a suspicious person or a criminal, the risk level when the moving object is a living organism such as an animal (for example, a beast, a toxic snake, etc.) It is not limited to these. In step 531, the moving image photographing device (101) advances the process to step 541 described in FIG. 5E in response to the request for calculating the degree of risk. On the other hand, in step 531, if there is no request for calculating the predetermined risk level, the moving image shooting device (101) advances the process to step 532 without calculating the risk level. The user can set in advance whether or not to calculate a predetermined risk level by selecting a menu on the display screen (106). Alternatively, the motion picture capture device (101) can be operating in the background to always check whether it poses a risk to the user. For example, when a moving object (whether or not to be concealed) is moving toward the moving image shooting device (101), the user who owns the moving image shooting device (101) When moving toward the direction of the moving object, or depending on the distance between the moving object and the moving image shooting device (101), the moving image shooting device (101) calculates the predetermined risk level. Can start to do. Step 531 is an optional step. Therefore, the moving image shooting apparatus (101) may directly proceed to step 532 without performing the process of step 531.

  Steps 532 to 534 are steps for acquiring the “speculatively acquired background image” used in step 523 of FIG. 5C.

  In step 532, the moving image shooting device (101) calculates the predicted moving direction and speed of the moving object that is the object of concealment, and the predicted distance between the moving object and the moving image shooting device (101). When the user who owns the moving image shooting device (101) is moving, the moving image shooting device (101) is predicted to be the user (that is, the moving image shooting device (101) itself). The moving direction and speed can be calculated.

  In step 533, the moving image shooting device (101) determines the subsequent frames (ie, future frames) based on the predicted moving direction and speed of the moving object to be hidden and optionally the distance. The speculative acquisition of the background image of the moving object that is the object of concealment, which would be necessary to conceal the moving object that is the object of concealment in FIG. The moving image shooting device (101) can acquire a speculative background image of the moving object to be hidden from the current and previous frames. FIG. 6 is a model diagram showing speculative acquisition of a background image of a moving object that is an object to be hidden. Further, when the user is moving, the moving image shooting device (101) hides the moving object to be hidden, the predicted movement direction and speed of the user, and optionally the distance. It is possible to speculatively acquire the background of the moving object that is the object to be deleted. In addition, the moving image shooting device (101), if necessary, changes the brightness, saturation, or combination thereof of the background image acquired speculatively around the expected moving destination of the moving object that is the object of concealment. You may correct | amend beforehand according to the light quantity and / or shadow so that it may match the surrounding brightness and saturation.

  In addition, the moving image shooting device (101) may determine whether the moving object and / or background image to be hidden based on the predicted moving direction and speed of the moving object that is to be hidden and optionally the distance. When the background image used in the immediately preceding frame is diverted, the background image used in the immediately preceding frame is diverted to the speculatively acquired background image. It can be diverted instead.

  The background image acquired speculatively at step 533 can be made available to hide the moving object in the future frame at step 523 of FIG. 5C.

  In step 534, the moving image photographing device (101) stores the background image acquired speculatively in step 533 in the storage medium (108). The speculatively acquired background image can be used in step 523 of FIG. 5C. The storage medium (108) may be a built-in memory in the recording and photographing machine (101) or a memory card inserted into the recording and photographing apparatus. Alternatively, if the recording and shooting device (101) has a wireless communication function, via the wireless communication function (for example, 3G or 4G (for example, LTE), wireless LAN, WiFi, WiMAX), Portable storage owned by the user or storage in the wireless network connection terminal possessed by the user, storage on the network accessible via the wireless network connection terminal possessed by the user (for example, , NAS or storage server) or recording storage device (101) on the network that can be directly accessed via the wireless communication function.

  The moving image photographing device (101) can perform step or step 534 for each moving object that is the object of concealment.

  In step 535, the moving image photographing device (101) determines whether to repeat speculative acquisition of the background image. The moving image photographing device (101) repeats Steps 532 to 534 according to the fact that there is an unprocessed moving object to be hidden. On the other hand, in the moving image shooting device (101), in response to the fact that there is no unprocessed moving object to be hidden, that is, steps 532 to 534 are performed for all moving objects to be hidden. In response, the process may proceed to step 536. In an alternative embodiment, in each step from step 532 to step 534, processing of each moving object that is the object of concealment may be performed. In the alternative embodiment, step 535 is not necessary, and the moving image shooting device (101) advances the process to step 536 in accordance with the end of the process of step 534.

  In step 536, the moving image shooting apparatus (101) determines whether or not the user has ended moving image shooting. The end of the moving image shooting can be performed, for example, when the user presses a recording end button provided on the moving image shooting device (101). In response to the end of moving image shooting, the moving image shooting device (101) advances the process to step 537. On the other hand, the moving image shooting device (101) returns the process to step 502 in FIG. 5A in response to the moving image shooting not yet finished.

  In step 537, the moving image shooting apparatus (101) ends the moving image shooting. Accordingly, the moving image photographing device (101) ends the concealment mode according to the embodiment of the present invention. The moving image shooting device (101) can delete the speculatively acquired background image stored in the storage medium (108) in step 534 in response to the end of the concealment mode.

  (See FIG. 5E)

  In step 541, the moving image shooting device (101), in response to the request for calculating the degree of risk, the predicted moving direction and speed of the moving object that is the object of concealment, and the moving object and moving image shooting device (101). The predicted distance is calculated. When the user who owns the moving image shooting device (101) is moving, the moving image shooting device (101) is predicted to be the user (that is, the moving image shooting device (101) itself). The moving direction and speed can be calculated.

  In step 542, the moving image shooting device (101) determines the risk level of the moving object that is the object of concealment based on the predicted moving direction and speed of the moving object that is the object of concealment and optionally the distance. calculate. Further, when the user is moving, the moving image shooting device (101) hides the moving object to be hidden, the predicted movement direction and speed of the user, and optionally the distance. It is possible to calculate the degree of danger due to the moving object to be erased. Further, the moving image photographing device (101) may further calculate the risk level by determining the shape of the moving object (for example, the shape of a blade or a firearm).

  In step 543, in response to the degree of risk calculated in step 542 being greater than or equal to a predetermined threshold, the moving image photographing device (101) advances the process to step 544 to perform a predetermined operation. On the other hand, the moving image photographing device (101) advances the process to step 545 in response to the degree of risk being less than a predetermined threshold.

In step 544, the moving image photographing device (101) may execute the following predetermined operation in order to urge risk avoidance, for example, in response to the risk level being equal to or higher than a predetermined threshold value:
-Change the degree of obscuration of the hidden moving object; for example, make the overlapping of the background image thin so that the moving object can be seen;
Display a movement prediction image in the moving direction of the hidden moving object; for example, display the movement prediction image as a movement prediction trajectory, for example, the moving object is displayed in a dotted line or lightly at an expected movement position of the moving object. To do
-Change the display method of the hidden moving object; changing the display method includes changing the size or color of the moving object, or displaying a dangerous part in the moving object; The size of the object is made larger than the actual size, or the color of the moving object is made darker than the actual size, for example, converted into a primary color and displayed. For example, dangerous parts are dangerous goods,
-Alternately displaying the hidden moving object and the superimposed background image;
・ Display warning characters;
・ Sounds a warning sound; and ・ Contacts emergency calls.

  In step 545, the moving image photographing device (101) advances the process to step 546 to perform a predetermined operation in response to the risk level calculated in step 542 becoming larger. On the other hand, the moving image photographing device (101) advances the process to step 547 according to the fact that the degree of risk is not increased.

In step 546, the moving image photographing device (101) may execute the following predetermined operation, for example, in order to promote danger avoidance in response to the fact that the degree of danger is gradually increasing:
Reduce the degree of occlusion of the hidden moving object; for example, make the overlapping of the background image thin so that the moving object can be seen;
・ Reduce the number of frames that have been hidden by the moving object: Therefore, the moving object and the background image to be hidden will be displayed blinking alternately.
Increase the thickness of the outline of the moving object:
・ Increase the size of the warning character and display: and ・ Increase the warning sound.

  In step 547, in response to the fact that the distance calculated in step 541 is gradually approaching, the moving image photographing device (101) advances the process to step 548 to perform a predetermined operation. On the other hand, the moving image shooting device (101) advances the processing to step 536 in FIG.

In step 548, the moving image photographing device (101) can execute, for example, the following predetermined operation in order to promote danger avoidance in response to the approaching distance.
・ Reduce the degree of disappearance of the hidden moving object;
Reduce the number of frames that are hidden by the moving object:
Increase the thickness of the outline of the moving object:
・ Increase the size of the warning character and display: and ・ Increase the warning sound.

  The order of the determination steps in steps 543, 545, and 547 can be arbitrarily changed. Further, the determination steps of steps 543, 545, and 547 may be arbitrarily combined. Furthermore, a combination of whether the degree of risk is equal to or higher than a predetermined threshold, whether the degree of risk is gradually increasing, and whether the distance is gradually approaching is combined to change the predetermined operation to a moving image shooting device ( 101) may be executed.

  The predetermined actions of steps 544, 546 and 548 are not limited to the above, and announcing the danger to the user through visual or auditory sense, or by causing the video shooting device (101) itself to vibrate. Any possible technique can be used. Further, the predetermined operation is to change the background image to be superimposed and / or the superposition form of the background image according to the degree of risk or according to the distance between the moving object and the moving image shooting device (101). May be included.

FIG. 6 is a model diagram showing speculative acquisition of a background image of a moving object that is an object of concealment according to an embodiment of the present invention.
The moving image shooting device (101) starts shooting and detects the moving object (601) in the current frame. In addition, the moving image shooting device (101) acquires the contour and / or range data of the detected moving object (601). The moving image shooting device (101) predicts the moving direction and speed (602) of the moving object to be hidden, and the predicted distance (603) between the moving object (601) and the moving image shooting device (101). Is calculated. The moving image shooting device (101), based on the predicted moving direction and speed (602) of the moving object that is the object of concealment, is the next frame after the current frame (hereinafter also referred to as "after 1 frame"). The position of the moving object that is the object to be hidden is obtained (611). In addition, the moving image shooting device (101) is a moving object that is to be hidden one frame later and a hidden object in the current frame as a part of the speculative background image acquired from the current frame. A portion (612) where the moving object overlaps is acquired from the current and previous frames. In addition, the moving image shooting device (101), as a part of the speculative background image acquired from the current frame, a portion (613) where the moving object that is the object to be erased one frame later overlaps the current frame (613). Obtain from current and previous frames. Then, the moving image shooting device (101) speculatively combines the background image (614) of the moving object to be hidden by combining the part (612) of the background image and the part (613) of the background image. It becomes possible to get to. Further, the moving image shooting device (101) enlarges or reduces the speculatively acquired background image (614) based on the predicted distance (603) between the moving object (601) and the moving image shooting device (101). Or the brightness, saturation, or combination thereof of the background image (614) acquired speculatively according to the amount of light and / or shadow around the expected movement destination of the moving object to be hidden It may be corrected in advance so as to match the brightness and saturation.

FIG. 7A is a diagram showing an example of a functional block diagram of the moving image photographing apparatus (101) preferably including the hardware configuration according to FIG. 1A and according to the embodiment of the present invention.
The moving image shooting device (101) includes sensor means (not shown) (including distance acquisition sensor means (701) and acceleration acquisition sensor means (702)), moving image shooting means (703), and hidden reality projection means (704). , Object selection means (705), object detection means (706), moving picture storage means (707), user static motion determination means (708), user movement direction / speed calculation means (709), object static motion determination means (710) , Object moving direction / speed calculating means (711), speculative background image acquisition means (712), speculative background image storage means (713), and hidden reality execution means (714). .

  The sensor means (not shown) provided in the moving image photographing device (101) shown in FIG. 7A has the same function as the sensor means (117) shown in FIG. 1A. The sensor means shown in FIG. 7A includes a distance acquisition sensor means (701) and an acceleration acquisition sensor means (702), and may include a gyro sensor, a geomagnetic sensor, and a positioning sensor as necessary. The distance acquisition sensor means (701) and the acceleration acquisition sensor means (702) correspond to the distance acquisition sensor means (118) and the acceleration acquisition sensor means (119) shown in FIG. 1A, respectively, so that description thereof is omitted here. .

  The moving image shooting means (703) corresponds to the moving image shooting means (116) shown in FIG. 1A, and the description thereof is omitted here.

  The hidden reality projection means (704) is a display device for displaying moving image data in which the moving object to be hidden is hidden in real time in the background image according to the embodiment of the present invention. The hidden reality projection means (704) corresponds to the display device (106) shown in FIG. 1A.

  The object selection means (705) allows the user to select a moving object on the hidden reality projection means (704). Specifically, the object selection means (705), for example, when the user selects a new moving object in the current frame (see step 504 shown in FIG. 5A) and the frame immediately before the current frame. When the user selects a moving object (see step 506 in FIG. 5A), the user is allowed to select a moving object on the hidden reality projection means (704).

  The object detection means (706) can execute step 503 shown in FIG. 5A.

  The moving image storage means (707) stores the image data photographed by the moving image photographing means (703). Also, the moving image storage means (707) can store moving image data in which the moving object that is the object of concealment in step 528 shown in FIG. 5C is concealed. The moving image storage means (707) can be the storage medium (108) shown in FIG. 1A.

  The user static determination unit (708) can determine whether the moving image shooting device (101) (that is, the user who owns the moving image shooting device (101)) is stationary or moving. The user static motion determination means (708) can determine whether the moving object is stationary or moving from the moving image data from the moving image capturing means (703) and the acceleration of the moving object.

  The user moving direction / speed calculating means (709) calculates the moving direction (XYZ axis direction) and moving speed of the moving image shooting device (101) (that is, the user who owns the moving image shooting device (101)). sell. The user moving direction / speed calculating means (709) calculates the traveling direction of the user having the moving image shooting device (101) from the moving image shooting means (703), and calculates the above from the moving image data and the acceleration of the moving image shooting device (101). The user's speed can be calculated.

  The object static determination means (710) can determine whether an object in the moving image is stationary or moving. The object static motion determining means (710) can determine whether the object is stationary or moving from the moving image data from the moving image capturing means (703).

  The object moving direction / speed calculating means (711) can calculate the moving direction (XYZ axis direction) of the moving object and its moving speed.

  The speculative background image acquisition unit (712) may execute Steps 532 to 534 of FIG. 5D. Further, the speculative background image acquisition unit (712) may execute Step 524 of FIG. 5C.

  The speculative background image storage means (713) stores the speculatively acquired background image acquired in step 533 of FIG. 5D. The speculative background image storage means (713) may be the storage medium (108) shown in FIG. 1A.

  The hidden reality implementation means (714) may execute step 526 of FIG. 5C.

FIG. 7B is a diagram showing an example of a functional block diagram of the moving image photographing device (121) and the terminal (131) according to the embodiment of the present invention, preferably provided with the hardware configuration according to FIG. 1B.
The moving image shooting device (121) includes at least sensor means (not shown) (distance acquisition sensor means (701) and acceleration acquisition sensor means (702) in the configuration of the moving image shooting device (101) shown in FIG. 7A. ), Moving image photographing means (703), hidden reality projection means (704), and object selection means (705).

  The terminal (131) includes an object detection unit (706), a moving image storage unit (707), a user static motion determination unit (708), a user movement direction / speed calculation, among the configurations of the video shooting device (101) shown in FIG. 7A. Means (709), object static motion determination means (710), object movement direction / speed calculation means (711), speculative background image acquisition means (712), speculative background image storage means (713), and hidden reality implementation Each component of the means (714) may be provided. Alternatively, the terminal (131) can be changed so that the moving image shooting device (121) includes any part of the components.

FIG. 8 is a diagram showing an example of a functional block diagram of a moving image photographing apparatus (101) having a hardware configuration according to the embodiment of the present invention and preferably having a hardware configuration according to FIG. 1A.
In addition to the components shown in FIG. 7A, the moving image photographing device (101) shown in FIG. 8 includes a distance measurement means (801), a risk determination means (802), a risk avoidance measure determination means (803), and a risk avoidance measure. Implementation means (804) may be provided.

  The distance measuring unit (801) is a moving image photographing device (701) based on a moving image from the moving image photographing unit (703) and data from a distance acquisition sensor unit (701), for example, an infrared type, laser type or ultrasonic type distance measuring sensor. 101) (that is, a user who owns the moving image photographing device (101)) and the moving object that is the object of concealment can be calculated. The distance measuring unit (801) measures the dimension (height) of the moving object based on the still image from the moving image shooting unit (703), and the moving image shooting device (101) by a gyro sensor (not shown). ) Can be measured to measure the distance between the moving object and the moving image shooting device (101). Moreover, the distance measuring means (801) can use the distance measuring method as described in Unexamined-Japanese-Patent No. 2005-148253 (patent document 5) and Unexamined-Japanese-Patent No. 2003-177017 (patent document 6), for example. The contents described in Patent Documents 5 and 6 are incorporated herein by reference. The distance measuring means (801) can perform, for example, step 541 shown in FIG. 5E.

  The risk level judging means (802) can calculate the risk level that the moving object that calculates the risk level can give to the user who owns the moving image photographing device (101). For example, the risk determination means (802) may implement step 542 shown in FIG. 5E.

  The danger avoidance measure determination means (803) should execute any predetermined action according to the threshold of the danger, the magnitude of the danger, and the distance between the moving object and the user who owns the moving image photographing device (101). Can be allocated. For example, the danger avoidance measure determination means (803) can execute steps 543, 545, and 547 shown in FIG. 5E.

  In order to take the risk avoidance measure, the risk avoidance measure execution means (804) performs in advance according to the threshold value of the risk level, the magnitude of the risk level, and the distance between the moving object and the user who owns the moving image shooting device (101). Take prescribed risk aversion measures. The danger avoidance measure implementation means (804) may perform, for example, steps 544, 548 and 547 shown in FIG. 5E.

Claims (20)

A method for obscuring a moving object in a movie during movie shooting, wherein the movie shooting device
Detecting at least one moving object in the video;
Calculating at least one moving direction and speed of the detected moving object;
Speculatively acquiring a background image when the moving object moves in the calculated moving direction and speed, the speculatively acquired background image obscuring the moving object in real time Performing the step of obtaining and using the method. The video shooting device is
The method further comprises: executing a step of superimposing and displaying the speculatively acquired background image on the moving object in the moving image on a display device, and concealing the moving object in real time. The method described. The video shooting device is
The method according to claim 2, wherein the calculating step, the obtaining step, and the hiding step are repeatedly executed while the moving object is hidden in real time. The video shooting device is
Storing the moving image data in which the moving object is hidden in real time in a storage medium in the moving image shooting device or in a storage medium accessible to the moving image shooting device via a network, The method of claim 2. The video shooting device is
Calculating a moving direction of the moving object and / or moving image shooting device, and a distance between the moving object and the moving image shooting device;
The method according to claim 2, further comprising: calculating a degree of danger by the moving object according to the calculated moving direction and the distance. The method according to claim 5, further comprising the step of the moving image photographing device following at least one of the following (1) to (12):
(1) a step of changing a degree of concealment of the concealed moving object in response to the degree of risk being equal to or greater than a predetermined threshold;
(2) displaying a movement prediction image in a moving direction of the hidden moving object in response to the risk being equal to or higher than a predetermined threshold;
(3) a step of changing a display method of the hidden moving object in response to the degree of risk being equal to or greater than a predetermined threshold; changing the display method includes changing the size or color of the moving object; Or a display of a dangerous part in the moving object,
(4) A step of alternately displaying the hidden moving object and the superimposed background image in response to the degree of risk being equal to or higher than a predetermined threshold;
(5) displaying a warning character in response to the risk being equal to or greater than a predetermined threshold;
(6) generating a warning sound in response to the risk being equal to or higher than a predetermined threshold;
(7) a step of notifying an emergency call in response to the risk being equal to or greater than a predetermined threshold;
(8) reducing the degree of concealment of the concealed moving object as the degree of risk increases or as the distance approaches.
(9) A step of reducing the number of frames in which the moving object is hidden as the degree of risk increases or as the distance approaches.
(10) increasing the thickness of the outline of the moving object as the risk increases or as the distance approaches,
(11) A step of increasing and displaying the size of a warning character as the degree of risk increases or as the distance approaches.
(12) A step of increasing the warning sound as the degree of danger increases or as the distance approaches.   The method according to claim 5, wherein calculating the risk level further includes calculating a risk level by determining a shape of the moving object.   3. The method of claim 2, wherein the step of obscuring comprises obscuring all or some of the detected moving object objects in real time. The video shooting device is
The method of claim 2, further comprising: allowing a user to select at least one of the detected moving objects. The video shooting device is
Further comprising the step of allowing a user to select whether to hide the selected moving object in real time or to hide moving objects other than the selected moving object in real time. The method of claim 9. The calculating includes calculating a moving direction and a speed of the selected moving object;
The step of obscuring includes obscuring the selected moving object in real time;
The method of claim 9. The calculating includes calculating a moving direction and a speed of a moving object other than the selected moving object;
The step of erasing includes erasing a moving object other than the selected moving object in real time,
The method of claim 9. The video shooting device is
Recognizing the type or shape of the detected moving object;
The method of claim 2, further comprising: selecting a moving object whose recognized type or shape matches a type or shape designated to hide the moving object. The video shooting device is
Further comprising the step of allowing a user to select whether to hide the selected moving object in real time or to hide moving objects other than the selected moving object in real time. The method of claim 13. The calculating includes calculating a moving direction and a speed of the selected moving object;
The step of obscuring includes obscuring the selected moving object in real time;
The method of claim 12. The calculating includes calculating a moving direction and a speed of a moving object other than the selected moving object;
The step of erasing includes erasing a moving object other than the selected moving object in real time,
The method of claim 12. The step of concealing comprises:
Correcting the brightness and / or saturation of the acquired background image;
The method according to claim 2, further comprising: superimposing the corrected background image on the moving object. The video shooting device is
When there is no speculatively acquired background image for concealing the moving object in real time, the frame is traced back to the frame where the detected background image of the moving object exists, and the moving object Further comprising: cutting out a background image to be superimposed on
The step of obscuring includes the step of superimposing and displaying the clipped background image on the moving object on the display device, and obscuring the moving object.
The method of claim 2. A video recording device,
Imaging means;
Detecting means for detecting at least one moving object in the moving image during moving image shooting by the imaging means;
Calculating means for calculating at least one moving direction and speed of the detected moving object;
Speculative background image acquisition means for speculatively acquiring a background image when the moving object moves with the calculated moving direction and speed, wherein the speculatively acquired background image The moving image photographing device comprising: the speculative background image acquisition means used for obscuring in real time.   A moving image shooting apparatus program for concealing a moving object in a moving image during moving image shooting, causing the moving image shooting device to execute each step of the method according to any one of claims 1 to 18. Program for moving image photographing apparatus.