JP2014197803A - Photographing assist system and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a photographing assist system and program which allow for photographing of an image without photographer's operation when the deviation of a photographing terminal is small.SOLUTION: When a capture instruction signal is received, a CPU of an HMD acquires (S13) photographic image data photographed by a camera, performs image analysis processing (S15), and determines (S17) whether an object is reflected in a photographic image. In a case where an object is reflected (S17: YES), the CPU of the HMD acquires sensor data of an acceleration sensor and in the case of determining that the sensor data is less than a regulated value and the blur of a photographing range is small (S21: YES), acquires a captured image (S31) and transmits it to a PC (S37).

Description

  The present invention relates to an imaging assistance system and program capable of transmitting a captured image captured by an imaging terminal to a browsing terminal via a network.

  There is known a system that can capture an image with a camera, transmit the captured image to a terminal device via a network, and display the image on a display unit of the terminal device (see, for example, Patent Document 1). Since time is required for data transfer over the network, the camera of Patent Document 1 transmits a captured image captured at a predetermined resolution with a relatively low resolution to the terminal device.

  In applications such as supporting remote work, a photographer may take a picture by holding the camera with his hand or wearing it on his body. When a viewer who browses a captured image desires to capture an image, the viewing terminal used by the viewer uses the captured image transmitted from the capturing terminal used by the photographer as a captured image. When the viewer desires a high-resolution image having a higher resolution than the captured image as the captured image, the photographing terminal operates the camera and operates the camera to capture the high-resolution image as a high-resolution image on the viewing terminal. Send.

JP 2007-189503 A

  However, there is a possibility that a high-resolution image in which the image is shaken due to capture by the photographer is transmitted to the viewing terminal. In addition, when the photographer is working, the photographer needs to rest his / her hand to work or let go of the tools used for the work in order to capture.

  The present invention has been made to solve the above-described problems, and provides a photographing assistance system and a program that can photograph an image without a photographer's operation when the displacement of the photographing terminal is small. Objective.

  According to the first aspect of the present invention, a photographing assistance system in which a photographing terminal capable of photographing a landscape around a photographer and a browsing terminal capable of browsing an image photographed by the photographing terminal are connected via a network. The photographing terminal includes a photographing unit capable of photographing a photographing target object that is a photographing target, a displacement detecting unit that detects a displacement of the photographing unit, and a magnitude of the displacement detected by the displacement detecting unit is predetermined. First determining means for determining whether or not the value is smaller than the value; and when the first determining means determines that the magnitude of displacement of the photographing means is smaller than a predetermined value, the photographing means has photographed the photographing object. Provided is an imaging assistance system comprising: an acquisition unit that acquires captured image data that is data indicating a captured image; and a first transmission unit that transmits the captured image data acquired by the acquisition unit to the viewing terminal. It is.

  When the displacement of the photographing means of the photographing terminal is settled, the obtaining means can obtain the photographed image data of the photographing means, so that the photographer takes a hand to perform the work in order to photograph the photographed image without blurring. You can shoot without stopping, and you don't have to let go of your work tools.

It is a perspective view which shows the external appearance of HMD1. It is a block diagram which shows the electrical structure of HMD1 and PC80. It is a figure which shows the browsing screen 100 displayed on the display 92 of PC80. It is a flowchart of the photography assistance program for HMD installed in HMD1. It is a flowchart of the photography assistance program for HMD installed in HMD1. It is a flowchart of the imaging | photography assistance program for PC installed in PC80.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The drawings to be referred to are used to explain technical features that can be adopted by the present invention. The configuration of the illustrated apparatus is not intended to be limited only to the form, but merely an illustrative example.

  As shown in FIG. 1, a photographing terminal (hereinafter referred to as “head mounted display” or “HMD”) 1 according to the present invention and a projector 10 (hereinafter referred to as “head display” or “HD”) are controlled. A device (hereinafter referred to as “control box” or “CB”) 50 is provided. The photographer uses the HMD 1 with the HD 10 attached to the head and the CB 50 attached to the waist belt or arm. The browsing terminal according to the present invention is a personal computer (hereinafter referred to as “PC”) that can be used by a viewer and can send an instruction to the HMD 1 to capture an image captured by a camera 15 described later. ) 80. The HMD 1 and the PC 80 are connected via wireless communication or wired communication. When the viewer remotely supports the work performed by the photographer, the photographer takes a photograph so that the image desired by the photographer is an image desired by the viewer. A photographing assistance system capable of assisting In the following description, the upper, lower, right diagonally downward, left diagonally upward, right diagonally upward and left diagonally downward of FIG. 1 are the upper, lower, forward, backward, right and left sides of the HMD 1, respectively.

  The HD 10 is used by being attached to spectacles 5 as a dedicated wearing tool. The HD 10 may be attached to other wearing tools such as glasses, helmets, and headphones that are used daily by the photographer. The HD 10 irradiates the photographer's eyes with image light. The HD 10 is detachably connected to the CB 50 via the harness 7. The CB 50 controls the HD 10.

  The configuration of the HD 10 will be described. The HD 10 includes a housing 2. The housing 2 includes a half mirror 3 on the right end side (left side in FIG. 1) when viewed from the photographer side. The half mirror 3 is arranged in front of the photographer's eyes (for example, the left eye) when the photographer wears the HD 10 on the head. The HD 10 includes a liquid crystal display (hereinafter referred to as “LCD”) 11 (see FIG. 2), an eyepiece optical system (not shown), and an acceleration sensor 18 inside the housing 2. The LCD 11 displays an image based on an image signal transmitted from the CB 50 via the harness 7. The LCD 11 may be a retinal scanning display unit that displays an image by two-dimensionally scanning laser light having an intensity corresponding to an image signal, an organic EL (Organic Electro-Luminescence) display, or the like.

  The eyepiece optical system condenses image light indicating an image displayed on the LCD 11 in order to guide it to the photographer's eyes. At least a part (for example, half) of the image light emitted from the eyepiece optical system is reflected by the half mirror 3 provided on the left side of the housing 2, and is reflected on one (for example, the left) eyeball (not shown) of the photographer. Incident. Since the half mirror 3 transmits at least part of the light from the real image in the outside world, the photographer can see the image superimposed on the real image (outside landscape) in his field of view.

  The glasses 5 are configured to hold the HD 10 on the photographer's head. The eyeglasses 5 include a support portion 4 at the upper right end (upper left end as viewed from the photographer) of the rim portion that supports the left-eye lens in the frame 6. The support unit 4 holds the housing 10 of the HD 10 and is attached to the glasses 5. The support part 4 can adjust the holding position of the housing 2 in the vertical direction and the horizontal direction. The photographer can place the half mirror 3 at a position that matches the position of the eyeball.

  The eyeglasses 5 include a camera 15, a microphone (hereinafter referred to as “microphone”) 16, and an earphone 17 in a temple portion to be hung on the right ear in the frame 6. The camera 15 captures an external scene in front of the photographer's field of view. The shooting direction and angle of view of the camera 15 are set in advance according to the photographer's field of view. More specifically, the shooting direction and angle of view of the camera 15 are based on the display of image light from the LCD 11 (see FIG. 2) in which the shooting range of the external scenery by the camera 15 can be superimposed on a real image that passes through the half mirror 3. It is set to almost match the range. The camera 15 may be configured so that it can be attached to the shoulder or chest, or removed and photographed. The microphone 16 picks up the voice uttered by the photographer. The earphone 17 outputs a viewer's voice received from the PC 80 (see FIG. 2). A photographed image photographed by the camera 15 is transmitted to the PC 80, and a viewer uses the microphone 16 and the earphone 17 to talk with the photographer, and supports the work performed by the photographer while referring to the photographed image. Since the shape of the glasses 5 itself is similar to that of normal glasses, detailed description thereof is omitted.

  The configuration of the CB 50 will be described. The CB 50 has a box-shaped housing. The CB 50 includes an operation unit 62 including a power switch 63 that incorporates a power lamp 64. The photographer can operate the power switch 63 to turn the power of the HMD 1 on or off. The photographer can perform various settings in the HD 10 and various operations during use via the operation unit 62.

  The CB 50 performs known wireless communication or wired communication, is connected to the PC 80 (see FIG. 2) via the network 9, and transmits / receives various data including sound data and photographed image data to / from the PC 80. The CB 50 may include a USB interface and be connected to the PC 80 using a USB cable. Note that the PC 80 described later is a personal computer, but may be other devices such as a smartphone or a tablet-type mobile terminal.

  The electrical configuration of the HMD 1 will be described with reference to FIG. The HD 10 includes an LCD 11, an LCD driver 12, an image signal receiving unit 13, an acceleration sensor 18, and a connection controller 14. The connection controller 14 is connected to the connection controller 55 of the CB 50 via the harness 7 and performs wired communication between the HD 10 and the CB 50. The image signal receiving unit 13 is connected to the connection controller 14 and receives an image signal transmitted from the image signal transmitting unit 60 of the CB 50. The LCD driver 12 causes the LCD 11 to display an image based on the image signal received by the image signal receiving unit 13. The acceleration sensor 18 detects acceleration in the directions of three axes (X axis, Y axis, Z axis).

  The connection controller 14 is connected to the camera 15, the microphone 16, and the earphone 17. The microphone 16 includes an A / D converter (not shown), digitally converts the input sound, and transmits the converted sound to the CB 50 via the harness 7. The earphone 17 incorporates a D / A converter (not shown), and analog-converts and outputs sound data received from the CB 50 via the harness 7. The camera 15 incorporates an encoder (not shown) that encodes the YUV signal output from the CMOS type light receiving cell, and transmits it to the CB 50 via the harness 7. The camera 15 captures images at a predetermined frame rate, and outputs captured image data, which is captured image data, to the CB 50 frame by frame.

  The electrical configuration of the CB 50 will be described. The CB 50 includes a SoC (System on a chip) 67, an operation unit 62, and a communication unit 65. The SoC 67 is an integrated circuit in which various functions necessary for controlling the HMD 1 are integrated. The SoC 67 includes a CPU 51, a RAM 52, a program ROM 53, a flash ROM 54, a connection controller 55, a resolution processing unit 56, an image compression unit 57, an image processing unit 58, a video RAM 59, an image signal transmission unit 60 and an interface 61. The CPU 51 controls the entire HMD 1. The CPU 51 is connected to the RAM 52, the program ROM 53, the flash ROM 54, the connection controller 55, the resolution processing unit 56, the image compression unit 57, the image processing unit 58, the video RAM 59, the image signal transmission unit 60, and the interface 61 via the system bus.

  The RAM 52 temporarily stores various flags, various data, and the like. The program ROM 53 includes a later-described HMD photographing assistance program, and stores various programs executed by the CPU 51, flags used by the various programs, initial values of data, and the like. Various programs including the HMD photographing assistance program are stored in the program ROM 53 when the HMD 1 is shipped. The flash ROM 54 stores various programs installed after the shipment of the CB 50, flags used by the various programs, data setting values, and the like. The CPU 51 can also execute a program stored in the flash ROM 54.

  The connection controller 55 is connected to the connection controller 14 of the HD 10 via the harness 7 and performs wired communication. The resolution processing unit 56 changes the resolution of the captured image data captured by the camera 15 to a predetermined resolution, and performs processing for generating high-resolution captured image data or low-resolution captured image data according to conditions. The high-resolution captured image data is image data having the maximum resolution among the captured image data output from the camera 15. The low-resolution captured image data is image data obtained by changing the captured image data to a resolution lower than that of the high-resolution image data. As an example, if the resolution of high-resolution captured image data is represented as the number of pixels, for example, 3 million pixels, the number of pixels of low-resolution captured image data is a relatively heavy load on transmission to the PC 80 via the network 9. It is preferable that the resolution is low (for example, about 300,000 pixels). The resolution of the high-resolution captured image data and the low-resolution captured image data can be arbitrarily set according to the resolution of the captured image data. The resolution of the high-resolution image data may be changed by the resolution processing unit 56 to a resolution (for example, about 2 million pixels) slightly lower than the resolution of the captured image data (for example, 3 million pixels).

  The image compression unit 57 performs processing for compressing the high-resolution captured image data and the low-resolution captured image data generated by the resolution processing unit 56 into, for example, a JPEG image. The high-resolution image data generated from the high-resolution captured image data by the CPU 51 performing the processing in the resolution processing unit 56 and the image compression unit 57 is used as a captured image in the execution of an HMD photographing auxiliary program described later. . Similarly, the low-resolution image data generated by the CPU 51 from the low-resolution captured image data is used as an image constituting a video that the viewer browses on the PC 80 in the execution of the HMD imaging assistance program described later.

  The image processing unit 58 performs processing for forming an image to be displayed on the LCD 11 of the HD 10. The video RAM 59 forms a virtual screen that virtually displays the image formed by the image processing unit 58 in the storage area. The image signal transmission unit 60 generates an image signal to be displayed on the LCD 11 based on the virtual screen formed by the video RAM 59 and transmits the image signal to the image signal reception unit 13 of the HD 10 via the harness 7. The interface 61 is connected to an operation unit 62 including a power switch 63 and a power lamp 64, and performs input / output of an input signal of a user operation, a lamp lighting signal, and the like. The interface 61 is connected to the communication unit 65 and inputs / outputs various data transmitted / received to / from the PC 80 connected to the network 9. The communication unit 86 connects to the network 9 by wired communication or wireless communication via an access point (not shown). The communication unit 65 may communicate with the PC 80 directly or wirelessly without using the network 9.

  The PC 80 is a known personal computer used by a viewer. The PC 80 includes a CPU 81 that controls the entire PC 80. CPU 81 is electrically connected to ROM 82, RAM 83, and input / output bus (hereinafter referred to as “I / O bus”) 85 via a data bus. The ROM 82 is a read-only storage device that stores programs such as BIOS executed by the CPU 81. The RAM 83 is a readable / writable storage device that temporarily stores data.

  Connected to the I / O bus 85 are a hard disk drive (hereinafter referred to as “HDD”) 84, a communication unit 86, an audio control unit 87, a display control unit 88, and an input device 93 such as a mouse and a keyboard. The HDD 84 is a storage device in which various basic software (OS) programs and programs are installed. A PC shooting assistance program, which will be described later, is installed in the HDD 84. The communication unit 86 connects to the network 9 by wired communication or wireless communication via an access point (not shown). The audio control unit 87 is connected to a headset 89 that incorporates a microphone 90 and an earphone 91, and controls input / output of audio of a viewer who uses the PC 80. The display control unit 88 performs a drawing process for displaying an image or the like on the display 92. The input device 93 is used by the viewer for inputting operations of the PC 80.

  The photographing assistance program will be described with reference to FIGS. The photographing assistance program includes an HMD photographing assistance program installed in the HMD 1 and a PC photographing assistance program installed in the PC 80.

  The HMD shooting assistance program is a program executed by the CPU 51 of the HMD 1. Although details will be described later, the CPU 51 that executes the HMD shooting assistance program generally performs processing of generating low-resolution image data from the shot image data shot by the camera 15 and transmitting it to the PC 80. When receiving the capture instruction signal from the PC 80, the CPU 51 detects an object within the shooting range of the camera 15 based on the object information received together with the capture instruction signal. “Object information” is information from which the CPU 51 detects an object from captured image data by a known image recognition process or a known character recognition process. The “object” is the whole of the photographing target that is the photographing target or a part to be photographed that is a part thereof. The object to be imaged or the object to be imaged is not limited to an object, and includes predetermined characters or character strings. When the object to be imaged or the region to be imaged is an object, the object information is image information to be compared with the imaged image data in order for the CPU 51 to perform image recognition processing and detect the object. When the object to be imaged or the region to be imaged is a character or a character string, the object information is character information indicating text or the like for the CPU 51 to perform character recognition processing and detect an object. When the CPU 51 detects the object, the CPU 51 shoots at a timing at which it is difficult for the camera 15 to shoot based on the output of the acceleration sensor 18, and transmits the captured image to the PC 80.

  Note that the image recognition process is generally similar to the image information shown as object information by performing edge detection processing on the captured image, extracting the pattern from the pixel data string, and comparing it with the image information shown as object information. This is a process for determining whether or not an object (all or a part of an object) is captured in a captured image based on the degree. In general, the character recognition processing is similar to the image recognition processing, in which a feature amount is detected from the photographed image, matching processing is performed with the character information indicated as the object information, and the object (character or character string) appears in the photographed image. This is a process for determining whether or not. Since the image recognition process and the character recognition process are well known, detailed description thereof is omitted.

  The PC shooting assistance program is a program executed by the PC 80 CPU 81. Although details will be described later, the CPU 81 that executes the PC photographing assistance program generally displays a preview image based on the low resolution image data received from the HMD 1 on the display 92. CPU81 transmits a capture instruction | indication signal and object information to HMD1, when a browsing person performs capture operation and designates an object. When the CPU 81 receives the captured image transmitted from the CPU 51 of the HMD 1 based on the capture instruction signal, the CPU 81 displays the captured image on the display 92.

  The browsing screen 100 on which the viewer operates in the execution of the PC photographing assistance program will be described. The browsing screen 100 shown in FIG. 3 is displayed on the display 92 of the PC 80, and the viewer browses a preview image (video) based on the low resolution image data received from the HMD 1 and browses the high resolution captured image 121. This screen allows you to specify the object you want to In the upper right corner of the browsing screen 100, an end button 101 for ending the execution of the application (PC shooting assistance program) is arranged. A display area 110 for displaying a preview image 111 based on low-resolution image data generated from the captured image data of the camera 15 by the CPU 51 of the HMD 1 is arranged in the left part of the browsing screen 100. The preview image 111 is displayed as a video at a predetermined frame rate. A display area 120 for displaying a captured image 121 received from the HMD 1 is arranged on the right side of the browsing screen 100. The captured image 121 is displayed as a still image.

  In the upper left part of the browsing screen 100, an operation button 130 used when the viewer performs an operation on the preview image 111 is arranged. The operation buttons 130 include a play button 131, a pause button 132, an end button 133, a capture button 134, and a text input box 135. The playback button 131 is a button that is clicked when the viewer starts to display the preview image 111. The pause button 132 is a button that is clicked when the viewer designates an object. The display of the preview image 111 as a video is temporarily stopped. The end button 133 is a button to be clicked when the viewer ends the video display of the preview image 111. The capture button 134 is a button that is clicked when the viewer designates an object that appears in the preview image 111 and requests the HMD 1 to capture the captured image 121 where the object appears. The text input box 135 is a button for inputting a character or character string when the viewer requests the HMD 1 to detect a character or character string as an object. When the viewer operates the pause button 132, the cursor changes to the pencil tool 138, and the viewer can draw a line drawing 139 that designates an object on the layer superimposed on the preview image 111. When the viewer draws a line drawing 139 that designates a closed area with respect to the preview image 111 in the execution of the PC browsing assistance program to be described later, the CPU 81 uses the image of the part included in the area as object information. .

  Next, details of processing performed by the CPU 51 of the HMD 1 and the CPU 81 of the PC 80 in accordance with the execution of the photographing assistance program will be described. First, a process performed by the CPU 51 of the HMD 1 by executing the HMD shooting assistance program will be described with reference to FIGS. When the photographer operates the power switch 63 provided on the CB 50 of the HMD 1, the CPU 51 performs a predetermined operation at the time of start-up according to execution of basic software (not shown). The CPU 51 operates the communication unit 65 and connects to the network 9 by wired communication or wireless communication.

  The CPU 51 starts execution of the HMD shooting assistance program in accordance with the basic software, and performs an initial setting process to be executed at startup as shown in FIG. 4 (S1). The CPU 51 initializes the flags and data stored in the RAM 52, and writes the flags and data initial values stored in the program ROM 53 and the flag and data set values stored in the flash ROM 54 to the RAM 52. The CPU 51 is connected to the PC 80 via the network 9. The CPU 51 acquires captured image data frame by frame from the camera 15, saves it in a frame buffer (not shown) secured in the RAM 52, and starts a process for displaying it on the LCD 11. The CPU 51 starts processing for acquiring sound data from the microphone 16 and transmitting it to the PC 80 via the network 9. The CPU 51 starts processing for receiving sound data from the PC 80 via the network 9 and outputting it from the earphone 17. The CPU 51 starts processing for acquiring sensor data (three-axis acceleration measurement value) from the acceleration sensor 18. Further, the CPU 51 transmits an image signal generated based on image data (for example, a work manual) received from the PC 80 to the HD 10 and starts a process of displaying an image based on the image signal on the LCD 11. The CPU 51 shifts to a video transmission mode for transmitting a captured image to the PC 80. The video transmission mode is a mode in which low-resolution image data is transmitted to the PC 80 so that the PC 80 can display a preview image video.

  When the initial setting process is completed, the CPU 51 converts the latest captured image data stored in the frame buffer (not shown) of the RAM 52 (S3) into low resolution image data in the image processing unit 58, and in the image compression unit 57. JPEG compression is performed (S5). The CPU 51 stores the generated low resolution image data in a transmission buffer (not shown), and sequentially transmits it to the PC 80 according to the frame rate (S7). If the CPU 51 does not receive a capture instruction signal from the PC 80 (S9: NO), the process returns to S3, and repeats the process of generating low-resolution image data from the captured image data of the next frame and transmitting it to the PC 80.

  When the CPU 51 receives a capture signal from the PC 80 (S9: YES), the CPU 51 shifts to the capture mode (S11), and a first timer for measuring a predetermined time (for example, 10 seconds) preset as the first specified time. Start (not shown). The capture mode is a mode for capturing a captured image by the camera 15. As shown in FIG. 5, the CPU 51 acquires the latest photographed image data stored in a frame buffer (not shown) of the RAM 52 (S13). The CPU 51 performs the above-described image recognition processing or character recognition processing as image analysis processing on the acquired captured image data (S15). That is, the CPU 51 performs image recognition processing if the object information received together with the capture instruction signal is image information, and performs character recognition processing if the object information is character information. The CPU 51 displays an image or characters based on the object information on the LCD 11 together with a captured image of the camera 15.

  CPU51 advances a process to S19, when the imaging | photography target object or the imaging | photography target site | part (an object, a character, etc.) which is a part is not contained in photographic image data as an object (S17: NO). CPU51 acquires the time-measurement value of a 1st timer, judges whether 1st specified time has passed, and if 1st specified time has not passed (S19: NO), it returns a process to S13, and the next Image analysis processing is performed on the captured image data of the frame.

  If an object is detected in the captured image data before the first specified time has elapsed (S17: YES), the CPU 51 advances the process to S21. The CPU 51 starts a second timer (not shown) for measuring a predetermined time (for example, 5 seconds) preset as the second specified time, and acquires sensor data from the acceleration sensor 18. If the sensor data is greater than or equal to a preset specified value (S21: NO), the CPU 51 assumes that the photographer is moving, the shooting range of the camera 15 is not constant, and the shot image may be blurred. The process proceeds to S23. The CPU 51 obtains the time value of the second timer and determines whether or not the second specified time has elapsed. If the second specified time has not elapsed (S23: NO), the process returns to S21, and the camera 15 Wait for the shooting range to stabilize.

  When sensor data becomes less than a predetermined value (S21: YES), CPU51 advances a process to S27. Although not shown, the CPU 51 can perform an environment setting process in the execution of the HMD photographing assistance program. In the environment setting process, when capturing a captured image of the camera 15, the CPU 51 sets a single shooting setting for obtaining a single image as a captured image, or a plurality of images obtained by continuously shooting a plurality of images (continuous shooting). The continuous shooting setting for obtaining a captured image can be set, or the photographer can set it. When single shooting is set (S27: NO), the CPU 51 advances the process to S31.

  The CPU 51 acquires captured image data stored in a frame buffer (not shown) of the RAM 52 (S31), and displays a captured image based on the captured image data on the LCD 11 (S33). The photographer looks at the photographed image displayed on the LCD 11 to check whether there is any blurring or out of focus, misrecognition of the object, or the like, and if there is a problem, instructs the photographer to retake the photographed image. The reshooting instruction by the photographer is performed by, for example, known voice recognition. As a result of recognizing the voice of the photographer input to the microphone 16, the CPU 51 returns a process to S13 when a keyword such as “NG” is given and an instruction to retake a photographed image is given (S35: NO).

  In S35, when the photographer's voice input to the microphone 16 is recognized, if the keyword is “OK” or the like and there is no problem with the photographed image (S35: YES), the process proceeds to S37. The CPU 51 converts the captured image data into high resolution image data in the image processing unit 58, JPEG compression in the image compression unit 57, and transmits to the PC 80 (S37). The CPU 51 shifts to the video transmission mode (S39), returns the process to S3 in FIG. 4, and repeats the process of generating low-resolution image data from the captured image data of the next frame and transmitting it to the PC 80.

  If the continuous shooting setting has been made in S27 (S27: YES), the CPU 51 acquires the photographic image data to be stored in the frame buffer (not shown) of the RAM 52 for a predetermined number of frames (S29). . The CPU 51 advances the process to S37, converts the acquired plurality of captured image data into high resolution image data in the image processing unit 58, performs JPEG compression in the image compression unit 57, and transmits it to the PC 80 (S37). Henceforth, the process which CPU51 performs is the same as the case where a single setting is made.

  By the way, in S19 shown in FIG. 5, when the first specified time has elapsed in the time counting of the first timer (S19: YES), the CPU 51 determines that the object could not be detected within the first specified time, and proceeds to S25. Proceed. In S23, when the second specified time has elapsed in the time counting by the second timer (S23: YES), the CPU 51 detects the object, but the shooting range is not stable, and it is difficult to shoot without shaking. The process proceeds to S25.

  The CPU 51 shifts to the manual capture mode (S25), advances the process to S27, advances the process to S31 if the continuous shooting setting is not made, and advances the process to S29 if the continuous shooting setting is made. Do. The manual capture mode is a mode in which a captured image is captured by the camera 15 by, for example, voice recognition uttered by the photographer. On the photographer's LCD 11, a photographed image of the camera 15 and an image or text based on the object information are displayed. The photographer utters a keyword such as “capture”, for example, instructing photographing when the photographed image includes an image or a character based on the object information. In S31, the CPU 51 does not detect the object in the manual capture mode, and acquires the latest captured image data to be stored in the frame buffer (not shown) of the RAM 52 based on the photographer's instruction. Thereafter, the processing performed by the CPU 51 is the same as in the capture mode.

  Next, a process performed by the CPU 81 of the PC 80 by executing the PC shooting assistance program will be described with reference to FIG. When the viewer operates the PC 80 to start the PC shooting assistance program, the CPU 81 executes the PC shooting assistance program installed in the HDD 84. The CPU 81 performs an initial setting process that is executed when the PC photographing assistance program is activated (S51). The CPU 81 initializes flags and data stored in the RAM 83, operates the communication unit 86, and connects to the network 9. The CPU 81 starts processing for acquiring sound data from the microphone 90 of the headset 89 and transmitting it to the HMD 1 via the network 9. The CPU 81 starts processing for receiving sound data from the HMD 1 via the network 9 and outputting it from the earphone 91 of the headset 89. The CPU 81 secures a storage area for a reception buffer (not shown) in the RAM 83.

  When the initial setting process is completed, the CPU 81 stores the low resolution image data received from the HMD 1 in the reception buffer (S53). The CPU 81 sequentially displays the preview image based on the low resolution image data stored in the reception buffer as a video on the display area 110 of the browsing screen 100 according to the frame rate (S55). If the viewer does not click the pause button 132 (see FIG. 3), the CPU 81 determines that the capture operation is not started (S57: NO), returns the process to S51, and repeats the video display of the preview image. . When the viewer clicks the pause button 132 (S57: YES), the CPU 81 performs a process of causing the viewer to enter a capture instruction location in the preview image 111 on the browsing screen 100 (see FIG. 3). (S59). The CPU 81 changes the cursor to the pencil tool 138 so that the viewer can draw a line drawing 139 on the layer superimposed on the preview image 111 and set a closed area for designating the object. When the viewer clicks on the text input box 135, the CPU 81 activates the text input box 135 so that characters or character strings can be input.

  When the viewer clicks the pause button 132 again (S61: NO), the CPU 81 returns the process to S53 and repeats the video display of the newly received preview image, assuming that the viewer has stopped the capture operation. When the viewer clicks the capture button 134 (S61: YES), the CPU 81 sets an image of a part included in the closed region designated by the line drawing 139 drawn by the viewer as object information. When the viewer has input a character or character string in the text input box 135, the CPU 81 sets the input character or character string as object information. The CPU 81 transmits a capture instruction signal and object information to the HMD 1 (S63).

  For example, on the browsing screen 100 shown in FIG. 3, the viewer surrounds the fan exhaust port portion, which is a part of the device shown in the preview image 111 by the pencil tool 138, with the line drawing 139. When the fan exhaust port is included in the closed area surrounded by the line drawing 139, the CPU 81 uses the fan exhaust port as an object and transmits an image of the fan exhaust port to the HMD 1 as object information.

  As shown in FIG. 6, as described above, the CPU 81 waits for reception of high-resolution image data based on the object information transmitted from the CPU 51 of the HMD 1 (S65: NO). When the CPU 81 receives the high resolution image data (S65: YES), the process proceeds to S67. When the received high-resolution image data is a plurality of high-resolution image data captured based on the continuous shooting setting in the HMD 1 (S67: YES), the CPU 81 determines a plurality of captured images based on the high-resolution image data. Are displayed in a tiled manner in the display area 120 (S69). The CPU 81 waits for the viewer to click to select a desired captured image (S71: NO). When the viewer selects a desired captured image (S71: YES), the CPU 81 displays the selected captured image in the display area 120 (S73). In S67, when the received high-resolution image data is one high-resolution image data photographed based on the single shooting setting (S67: NO), the CPU 81 is based on the received high-resolution image data. The captured image is displayed in the display area 120 (S73).

  The viewer can operate the input device 93 to scroll, enlarge, reduce, or rotate the captured image displayed in the display area 120. For example, on the browsing screen 100 shown in FIG. 3, a capture image 121 in which a fan exhaust port is captured as an object is displayed in the display area 120. Since the captured image 121 is an image captured at a higher resolution than the preview image 111, the viewer reads, for example, a character string that is recorded in the center of the fan exhaust port and cannot be read in the preview image 111, in the captured image 121. be able to.

  As illustrated in FIG. 6, when the captured image is not the desired image (S75: NO), the viewer can instruct recapture by clicking the capture button 134. In this case, the CPU 81 returns the process to S63 and transmits the capture instruction signal and the object information to the HMD 1 again (S63).

  When the captured image is an image desired by the viewer and the viewer finishes viewing the captured image, the viewer clicks the pause button 132 to end the capture operation (S75: YES). The CPU 81 returns the process to S53, and repeats the video display of the preview image newly received from the HMD 1 as described above.

  As described above, in the photographing assistance system according to the present invention, when the sensor data of the camera 15 of the HMD 1 is less than the specified value and the blur of the photographing range is settled, the CPU 51 captures the photographed image data of the camera 15 from the frame buffer of the RAM 52. Therefore, the photographer can shoot without stopping the hand to perform the work in order to take a photographed image without blurring, and does not have to release the work tool. Further, when the displacement of the camera 15 is settled and the object is included in the captured image, the CPU 51 can acquire the captured image data of the camera 15 from the frame buffer of the RAM 52. There is no capture, and a captured image in which the object is reflected can be obtained.

  In addition, since the viewer can obtain a captured image that the photographer has viewed and confirmed, it is possible to obtain a captured image in which the image is not shaken and the photographing target part is reflected. In addition, by obtaining a plurality of captured images based on the continuous shooting setting, it becomes possible to select a captured image in a better state, so that the viewer can capture a captured image in which the object is not blurred and the object is captured. Can be obtained. In addition, the PC 80 can check the captured image displayed on the LCD 11 by the photographer and obtain high-resolution image data permitted to be transmitted, so that the viewer does not shake the image and the object is captured. A captured image can be obtained. Further, the CPU 81 of the PC 80 generates the object information, so that the CPU 51 of the HMD 1 can shoot a captured image showing an object desired by the viewer.

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention. The CB 50 has been described as an example of a dedicated controller that controls the HD 10, but the device connected to the HD 10 is not limited to the dedicated controller. For example, a dedicated program corresponding to an HMD shooting assistance program is installed in a general-purpose computer device such as a smartphone, a tablet terminal, or a notebook personal computer and connected to the HD 10 to configure the shooting terminal according to the present invention. Good.

  In the present embodiment, the various programs including the photographing assistance program are stored in the program ROM 53 when the HMD 1 is shipped. However, the programs may not necessarily be shipped in the state stored in the program ROM 53. For example, the CB 50 may include a storage medium (optical medium, memory card, etc.) reading unit, and the CPU 51 may read various programs from a storage medium storing various programs and install them in the flash ROM 54. Alternatively, the CPU 51 may download various programs stored in an external storage unit such as an HDD of the download server via the communication unit 65 and install them in the flash ROM 54. If various programs including the main program are installed in the flash ROM 54 and the HMD 1 is driven, the program ROM 53 is not necessary.

  In this embodiment, high-resolution image data is generated from captured image data, and an image based on the high-resolution image data is used as a captured image. However, the captured image data may be used as it is as high-resolution image data.

  In S23 of the HMD shooting assistance program, if the fluctuation of the shooting range does not fall within the second specified time (S23: YES), the CPU 51 does not shift to the manual capture mode, returns the process to S13, and performs a new shooting. Image data may be acquired and object detection may be performed again.

  In S29 of the HMD imaging assistance program, the CPU 51 acquires captured image data for a plurality of frames, and transmits a plurality of high resolution image data based on the plurality of captured image data to the PC 80 in S37. Then, the process may proceed to S33, and a plurality of captured images based on the captured image data may be displayed on the LCD 11. Then, one high resolution image data based on the photographed image selected by the photographer may be transmitted to the PC 80 in S37. The selection of the captured images may be performed, for example, by assigning a number to a plurality of captured images and displaying the tiles on the LCD 11, causing the photographer to utter the desired captured image number, and specifying by voice recognition. In this way, the high-resolution image data transmitted from the HMD 1 to the PC 80 can be made one, and an increase in traffic on the network 9 can be suppressed.

  In S9 of the HMD shooting assistance program, the CPU 51 shifts to the capture mode when receiving the capture instruction signal, but acquires the sensor data from the acceleration sensor 18 before the shift to the capture mode, and the sensor data becomes less than a predetermined value. You may wait until it becomes and then shift to capture mode. In this case, by shifting to the capture mode, the captured image based on the captured image data acquired by the CPU 51 in S13 has already been captured with reduced image blurring, out-of-focus, and the like. Therefore, if the CPU 51 can detect the object from the captured image data (S17: YES), the process of S21 and S23 may be omitted and the process may proceed to S27 to acquire the captured image that is the basis of the captured image.

  When the object information transmitted by the CPU 81 of the PC 80 together with the capture instruction signal is image information, in this embodiment, a part included in the closed region designated by the line drawing 139 drawn on the preview image 111 is set. It was. Not limited to this, arbitrary image data may be set as the object information. The object information is not limited to the case where the object information is transmitted from the PC 80 to the HMD 1 but may be image data stored in advance in the flash ROM 54 of the HMD 1 or the HMD 1 is stored in a storage device (not shown) of the download server. The downloaded image data may be downloaded and used as object information.

  When the object information is character information, the CPU 51 detects the object (character or character string) by character recognition processing. However, the CPU 51 generates image data of the character or character string based on the character information, and performs image recognition. An object (character or character string) may be detected by processing.

  In the present embodiment, the CPU 51 of the HMD 1 detects an object based on the object information, but the CPU 81 of the PC 80 may detect an object. For example, in the HMD imaging assistance program, the CPU 51 transmits the captured image data acquired in S13 or high-resolution image data based on the captured image data to the PC 80 (corresponding to the “second transmission unit” in the present invention). ). In the PC photographing assistance program, the CPU 81 performs image analysis processing (image recognition processing or character recognition processing) similar to S15 on the photographed image data or high-resolution image data received from the HMD 1 (the “first” in the present invention). Corresponds to “three judgment means”). When the CPU 81 detects an object in the captured image data or the high resolution image data, the CPU 81 may acquire an image based on the captured image data or the high resolution image data as a captured image. The PC 80 can obtain high-resolution image data in which an object is included in a captured image by the CPU 81 performing image analysis processing. When the processing capacity of the CPU 81 of the PC 80 is higher than the processing capacity of the CPU 51 of the HMD 1, the viewer can quickly and accurately obtain a captured image in which the image is not blurred and the object is reflected.

  In the present embodiment, communication between the HD 10 and the CB 50 via the harness 7 (transfer of image data displayed on the LCD 11, transfer of captured image data of the camera 15, transfer of audio data of the microphone 16 and the earphone 17, etc.) is performed by digital data communication. However, the data may be A / D / D / A converted in the CB 50 and analog data communication.

  In the present invention, the HMD 1 corresponds to a “photographing terminal”. The PC 80 corresponds to a “browsing terminal”. The camera 15 corresponds to “photographing means”. The acceleration sensor 18 corresponds to “displacement detection means”. The CPU 51 that determines whether or not the sensor data is less than the specified value in S21 corresponds to “first determination means”. The CPU 51 that acquires captured image data in S31 or S29 corresponds to “acquisition means”. The CPU 51 that transmits the high-resolution image data based on the captured image to the PC 80 in S37 corresponds to the “first transmission unit”. The range that appears in the preview image indicated by the low-resolution image data corresponds to the “shooting range”. In S <b> 17, the CPU 51 that determines whether or not an object is included in the captured image data corresponds to a “second determination unit”. In S <b> 33, the CPU 51 that displays the captured image based on the captured image data on the LCD 11 corresponds to the “first display unit”. The transmission mode corresponds to “capture mode”. In S11, the CPU 51 that shifts to the capture mode corresponds to “transition means”. In S59, the CPU 81 that generates object information by performing a process of causing the viewer to enter a capture instruction location in the preview image 111 corresponds to “generating means”.

1 Head mounted display (HMD)
9 Network 11 Liquid crystal display (LCD)
15 Camera 18 Acceleration sensor 51 CPU
80 Personal computer (PC)
81 CPU

Claims (8)

A photographing assistance system in which a photographing terminal capable of photographing a scene around a photographer and a browsing terminal capable of browsing an image photographed by the photographing terminal are connected via a network,
The photographing terminal is
Photographing means capable of photographing a subject to be photographed;
Displacement detecting means for detecting displacement of the photographing means;
First determination means for determining whether the magnitude of the displacement detected by the displacement detection means is smaller than a predetermined value;
When the first determination unit determines that the magnitude of the displacement of the imaging unit is smaller than a predetermined value, the acquisition unit acquires photographic image data that is data indicating a photographic image obtained by imaging the photographic subject. Means,
First transmission means for transmitting the captured image data acquired by the acquisition means to the browsing terminal;
A photographing assistance system characterized by comprising: The imaging terminal further includes second determination means for determining whether or not an imaging target region that is all or a part of the imaging object is included in an imaging range that is an area in which imaging by the imaging means is possible. Prepared,
The acquisition unit determines that the first determination unit determines that the magnitude of the displacement of the imaging unit is smaller than a predetermined value, and the second determination unit determines that the imaging target region is included in the imaging range. 2. The photographing assistance system according to claim 1, wherein the photographed image data is acquired when the photographing is performed.   The photographing terminal displays the photographed image based on the photographed image data acquired by the acquisition unit so that the photographer can view the first transmission unit before the first transmission unit transmits the photographed image data to the viewing terminal. The imaging assistance system according to claim 1, further comprising first display means.   The acquisition unit of the imaging terminal acquires a plurality of the captured image data respectively indicating a plurality of the captured images continuously captured by the imaging unit. Shooting assistance system. The imaging terminal further includes a transition unit that shifts to a transmission mode capable of transmitting the captured image data when the first determination unit determines that the magnitude of the displacement of the imaging unit is smaller than a predetermined value,
The acquisition unit acquires the captured image data when the shift unit shifts to the transmission mode,
The first transmission means specifies the photographed image data specified by the permission signal input by the photographer among the photographed image data obtained by the obtaining means and permitted to be transmitted to the viewing terminal. The imaging assistance system according to any one of claims 1 to 4, wherein the imaging assistance system is transmitted to. The photographing terminal further includes second transmission means for transmitting the photographed image data obtained by photographing the photographing object by the photographing means to the viewing terminal,
The browsing terminal includes a third determination unit that determines whether or not the imaging target region is included in the captured image based on the imaging range data received from the second transmission unit of the imaging terminal. The imaging assistance system according to any one of claims 1 to 5, characterized in that: The browsing terminal further includes generating means for generating specification data for specifying the imaging target region,
The second determination unit of the imaging terminal determines whether or not the imaging target region specified by the specification data generated by the generation unit is included in the imaging range. The photographing auxiliary system according to any one of items 1 to 6. An imaging terminal including an imaging unit capable of capturing an image of a scene around the photographer, a displacement detection unit that detects a displacement of the imaging unit, and a browsing terminal capable of browsing an image captured by the imaging terminal. A program for functioning as the photographing terminal in the photographing auxiliary system connected via
In the computer of the photographing terminal,
A first determination step of determining whether or not the displacement detected by the displacement detector is smaller than a predetermined value;
An acquisition step of acquiring captured image data, which is data indicating a captured image captured by the imaging unit, when it is determined in the first determination step that the magnitude of displacement of the imaging unit is smaller than a predetermined value;
A first transmission step of transmitting the captured image data acquired in the acquisition step to the viewing terminal;
A program that executes

Images