JP2019121991A - Moving image manual preparing system - Google Patents

Abstract

To form a moving image manual on which a photographer intent is accurately reflected by using sight information of the photographer with a simple method.SOLUTION: A moving image manual preparing system 1 comprises an imaging part 2, a sight detection part 3, a determination part 14, a storage target determination part 16 and a storage part 12. The imaging part 2 acquires a visual field image by photographing a visual field of a photographer. The sight detection part 3 detects sight information of the photographer. The determination part 14 determines whether there is gaze of the photographer on the basis of the sight information. The storage target determination part 16 determines at least one part of the visual field image as a storage target on the basis of a determination result of the determination part 14. The storage part 12 stores the storage target determined by the storage target determination part 16 as data of a moving image manual.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a movie manual creation system for creating a movie manual.

  Conventionally, a document manual has been utilized for sharing and efficiency of work, new employee education, etc., while a moving image manual for displaying the manual as a moving image has attracted attention in recent years, and its creation technology is disclosed in, for example, Patent Document 1 It is done. In Patent Document 1, work image data (for example, 3D animation data) indicating work content is created based on input text data, and work image data and text data are associated and output.

  On the other hand, one method of image processing for moving images is disclosed, for example, in Patent Document 2. In patent document 2, each frame which comprises a moving image is scanned for every area | region of a predetermined size, the importance degree for every area | region is calculated, and the said area | region is clustered based on importance, at least 1 area | region Clusters are generated, important area clusters are selected from the at least one area cluster based on importance, partial frames including important area clusters of each frame are extracted, and a partial moving image composed of the partial frames is generated I am trying to do it. Thereby, it is possible to efficiently view a region of high importance included in the moving image.

  For example, in Patent Document 3, an object is detected from a frame included in a moving image, the object is identified based on an image feature obtained from a predetermined area including the detected object, and an operation related to the object in the moving image is The attention area in each frame of the moving image is determined based on the detected and detected position of the object and the detection result of the motion related to the object. This enables the user of the moving image to easily view the region of interest by trimming or enlarging the region of interest.

  Further, for example, in Patent Document 4, image data of a photographed image and information of a main subject detected from line-of-sight information or line-of-sight information are recorded in a recording medium. As a result, information recorded in the recording medium is used to efficiently detect a scene in which the main subject is desired, and to reproduce and edit an image centered on the main subject efficiently, thereby making it possible to obtain valuable image data sufficiently. It is possible to utilize it. Further, in Patent Document 4, only a scene that can identify a main subject (or a predetermined number of frames before and after that) is extracted from image data of a moving image, and this is edited into image data of one moving image, or the same It is also disclosed that a scene in which the same main subject is present from a plurality of videos in which the main subject is present may be compiled into image data of one video.

JP-A-2006-215986 (see claim 1, paragraphs [0027] and [0051], FIG. 1, etc.) JP, 2016-219879, A (refer to claim 1, paragraph [0008], [0009], [0035], etc.) JP 2014-85845 A (see claim 1, paragraphs [0008], [0009], FIG. 1, etc.) JP-A-2004-7158 (see claim 1, paragraphs [0055], [0060], [0063], FIG. 1, etc.)

  However, the method of creating a moving image manual of Patent Document 1 requires an operation and a process of inputting text data, and can not create a moving image manual by a simple method. Further, Patent Documents 2 and 3 both disclose extracting a part of a moving image, but the extraction does not reflect the intention of the moving image photographer at all. Furthermore, according to Patent Document 4, although it is possible to edit an image centering on the main subject from the information recorded on the recording medium, the information (image data) itself recorded on the recording medium is photographed There is no idea to change according to the gaze information of the person at all. For this reason, even if the techniques of Patent Documents 2 to 4 are applied to creation of a moving image manual, it is not possible to create a moving image manual accurately reflecting the photographer's intention.

  The present invention has been made to solve the above-mentioned problems, and its object is to create a video manual, which accurately reflects the photographer's intention, using the photographer's gaze information in a simple method. To provide a video manual creation system that can

  A moving image manual creation system according to one aspect of the present invention includes an imaging unit that captures a view of a photographer as a moving image and acquires a view image, a gaze detection unit that detects gaze information of the photographer, and the gaze information. And a storage target determination unit configured to determine at least a part of the view image as a storage target based on the determination unit that determines the presence or absence of the photographer's gaze based on the determination result of the determination unit; And a storage unit configured to store the storage target determined by the unit as data of a moving image manual.

  In the above-described moving image manual creation system, the determination unit determines a position of the line of sight of the photographer in the view image and a dwell time of the position of the line of sight based on the line-of-sight information; And a gaze determination unit configured to determine the gaze condition of the photographer based on the position of and the staying time.

  In the above-described moving image manual creation system, the gaze determination unit detects a drift amount indicating a blur of the gaze based on the position of the gaze, and the drift amount is equal to or less than a threshold and the staying time is a predetermined time In the case of the above, it may be determined that the photographer gazes.

  In the above-described moving image manual creation system, when the determination unit determines that the photographer is gazing at the storage object determination unit, a subject being gazed by the photographer is present in the view image. The subject region may be specified, the subject region may be extracted from the view image, and the extracted subject region may be determined as the storage target.

  In the above-described moving image manual creation system, the storage target determination unit may enlarge the extracted subject area and determine the image after enlargement as the storage target.

  In the above-described moving image manual creation system, the storage target determination unit may change an extraction range when extracting the subject region from the view image, according to the information acquired by the determination unit.

  In the above-described moving image manual creation system, the storage target determination unit may change the extraction range in accordance with the residence time acquired by the determination unit.

  In the above-described moving image manual creation system, the storage target determination unit may change the extraction range according to a change in the center of gravity of the plurality of positions of the line of sight acquired by the determination unit.

  In the above-described moving image manual creation system, the storage target determination unit may determine the view image as the storage target when the determination unit determines that the photographer is not gazing.

  The above-mentioned moving picture manual creation system further comprises: a voice input unit to which the voice of the photographer is input; and a voice recognition unit to recognize the voice input by the voice input unit and convert it into text data; The object determining unit may combine the text data with at least a part of the view image to determine the storage object.

  In the above-described moving image manual creation system, the storage target determination unit combines the text data with at least a part of the view image so that the text data is positioned outside the area where the subject is present in the storage target. May be

  According to the above-described configuration of the moving image manual creation system, it is possible to create a moving image manual that accurately reflects the photographer's intention by a simple method using the photographer's line of sight information.

BRIEF DESCRIPTION OF THE DRAWINGS It is a perspective view which shows typically the schematic structure of the moving image manual production system which concerns on embodiment of this invention. It is a block diagram which shows the detailed structure of the control box which the said moving image manual production system has. It is a flowchart which shows the flow of the process in the said moving image manual production system. It is explanatory drawing which shows the view image and gaze information in the case of performing a synchronous process. It is explanatory drawing which shows an example of the view image, gaze information, and audio | voice information obtained after synchronous processing. FIG. 6 is an explanatory view schematically showing an XY coordinate plane in which an angle of view in the horizontal direction and an angle of view in the vertical direction of the imaging unit correspond to the X axis and the Y axis, respectively. It is explanatory drawing which shows an example of the image of the memory object memorize | stored in a memory | storage part. It is explanatory drawing which shows the other example of the view image, gaze information, and audio | voice information after synchronous processing. It is explanatory drawing which shows typically the relationship between the residence time of eyes | visual_axis, and the magnifying power of the display at the time of reproduction | regeneration of memory object. It is explanatory drawing which shows the example of the image of the memory | storage object extracted from a view image. It is explanatory drawing which shows the further another example of the visual field image, gaze information, and audio | voice information after synchronous processing. It is explanatory drawing which shows the other example of the image of the memory object extract | extracted from a view image.

  It will be as follows if the animation manual creation system concerning one embodiment of the present invention is explained based on a drawing. The present invention is not limited to the following contents.

[Configuration of movie manual creation system]
FIG. 1 is a perspective view schematically showing a schematic configuration of a moving image manual creation system 1 of the present embodiment. The moving picture manual creation system 1 is configured to include an imaging unit 2, a gaze detection unit 3, a microphone 4, and a control box 10. The imaging unit 2 and the gaze detection unit 3 are mounted on, for example, the glasses 5. The glasses 5 may be ordinary glasses provided with a diopter correction lens, glasses provided with a dummy lens not correcting diopter, or a head mounted virtual image display device ( It may be a head mounted display).

  The imaging unit 2 is a camera (line-of-sight camera) that captures a visual field of a photographer as a moving image and acquires a visual field image, and is mounted on, for example, a central bridge connecting left and right lenses of the glasses 5. Therefore, when the photographer wears the glasses 5, the imaging unit 2 captures an image of the field of view of the photographer from a position close to the eye of the photographer.

  The gaze detection unit 3 is an element that detects gaze information of the photographer, and includes an illumination unit (illuminator) and a light reception unit. The position of the Purkinje image on the surface of the photographer's cornea and the position of the pupil of the photographer relative to each other by irradiating near infrared rays near the cornea of the photographer by the illuminator and detecting the reflected light by the light receiving unit The change in the position of the photographer can be detected, whereby the gaze direction of the photographer can be detected as the gaze information (corneal reflection method). For example, if the pupil is on the staring side of the position of the corneal reflection of the left eye (the position of the Purkinje image), the photographer looks at the left side (the photographer's gaze is facing to the left). If the pupil is closer to the eye than the position, it can be detected that the photographer is looking to the right (the photographer's gaze is pointing to the right). Although the visual axis detection unit 3 according to this embodiment is configured to perform visual axis detection (eye tracking) by the representative corneal reflection method as described above, the visual axis detection is performed by another method such as the scleral reflection method. It may be a configuration. Further, in the present embodiment, the line-of-sight detection unit 3 is provided at each edge of the left and right lenses of the glasses 5 to detect the line of sight for both eyes, but is provided at the edge of one of the lenses The line of sight may be detected for one eye.

  The microphone 4 is a voice input unit to which the voice of the photographer is input.

  The imaging unit 2, the sight line detection unit 3 and the microphone 4 are connected to the control box 10 via the wiring 6. Thereby, the information acquired by the imaging unit 2 (data of the view image of the photographer), the information acquired by the gaze detection unit 3 (the gaze information of the photographer), and the information acquired by the microphone 4 (the photographer Audio information is output to the control box 10 via the wiring 6. The imaging unit 2, the sight line detection unit 3, the microphone 4, and the control box 10 may be configured to wirelessly transmit information.

[Configuration of control box]
FIG. 2 is a block diagram showing the detailed configuration of the control box 10. As shown in FIG. The control box 10 executes creation of a moving image manual based on information output from the imaging unit 2, the sight line detection unit 3 and the microphone 4. The control box 10 includes an operation unit 11, a storage unit 12, and a control unit 13.

  The operation unit 11 includes buttons and switches operated by the photographer to receive various instruction inputs. The storage unit 12 is a memory for storing various information in addition to the operation program of the control unit 13, and includes, for example, a random access memory (RAM), a read only memory (ROM), a non-volatile memory, and the like. . In particular, in the present embodiment, the storage unit 12 is a memory that stores the storage target determined by the storage target determination unit 16 described later as data of a moving image manual.

  In the present embodiment, “reproduction of a moving image manual” is to read out information (data to be stored) stored in the storage unit 12 and to display an image to be stored on a display unit of a reproduction apparatus (not shown). It is referred to as "reproduction of storage target" or simply "reproduction".

  The control unit 13 is configured of, for example, a CPU (Central Processing Unit). The control unit 13 includes a determination unit 14, a speech recognition unit 15, a storage target determination unit 16, and a main control unit 17, and the above-described CPU exerts the functions of these units. The determination unit 14, the speech recognition unit 15, the storage target determination unit 16, and the main control unit 17 may be configured by separate CPUs, or are configured by dedicated processing circuits that exhibit predetermined functions. It is also good.

  The determination unit 14 determines the presence or absence of the gaze of the photographer based on the line-of-sight information detected by the line-of-sight detection unit 3. Such a determination unit 14 is configured to include a line-of-sight analysis unit 14 a and a gaze determination unit 14 b.

  The line-of-sight analysis unit 14a analyzes and obtains the position of the line-of-sight of the photographer in the view image and the staying time of the position of the line-of-sight based on the line-of-sight information. The line-of-sight analysis unit 14a incorporates a clock unit (not shown), and thereby can measure the above-mentioned residence time.

  The gaze determination unit 14b determines the gaze state (presence or absence of gaze) of the photographer based on the position of the line of sight acquired by the line of sight analysis unit 14a and the staying time. For example, when the blur of the position of the line of sight of the photographer is within a predetermined range and the state continues for a predetermined time or more, the gaze determination unit 14b can determine that the photographer gazes at the subject. Otherwise, it can be determined that the photographer is not gazing at the subject.

  The speech recognition unit 15 recognizes speech inputted by the microphone 4 and converts it into text data. The storage target determination unit 16 determines at least a part of the view image as a storage target based on the determination result of the determination unit 14 (the determination result of the presence or absence of gaze). The main control unit 17 controls the operation of each unit in the control box 10.

[Processing flow]
Next, each process in the moving image manual creation system 1 of the present embodiment will be specifically described. FIG. 3 is a flow chart showing the flow of processing in the moving picture manual creation system 1.

  First, the photographer synchronizes the gaze information acquired by the gaze detection unit 3 with the view image acquired by the imaging unit 2 (S1). By this synchronization process, it is possible to match (synchronize) the direction of the line of sight of the photographer with respect to the subject and the position of the subject in the view image.

  Specifically, this synchronization processing is performed by the photographer looking at a marker before moving image shooting for manual creation. That is, as shown in FIG. 4, the photographer gazes at the center Mo of the marker M, and the imaging unit 2 acquires a view image at this time. At the same time, the line-of-sight information when the photographer gazes at the center Mo of the marker M is acquired by the line-of-sight detection unit 3. The acquired line-of-sight information indicates the direction of the line of sight of the photographer looking at the center Mo of the marker M, so that the line-of-sight information and the view image are synchronized in this state. Therefore, the photographer operates the operation unit 11 in this state to establish synchronization, and ends the synchronization process.

  Next, the photographer operates the operation unit 11 to start moving image shooting for manual creation by the imaging unit 2 (S2). As the photographer proceeds with the actual work while explaining the work content, the image of the photographer's field of vision (moving image) at that time is acquired by the imaging unit 2 and the photographer's gaze information is acquired by the gaze detection unit 3 The voice information of the photographer is acquired by the microphone 4 (S3). Note that acquisition of line-of-sight information by the line-of-sight detection unit 3 may be at an arbitrary interval, and in the present embodiment, it is assumed that line-of-sight information is acquired every 0.1 seconds, for example.

  FIG. 5 shows an example of the view image, the line-of-sight information, and the audio information acquired in S3. Here, as an example, it is assumed that the operation to be a target of creation of the moving image manual is an operation of tightening the screw A2 to the housing A3 using the driver A1. From the figure, it can be seen that line-of-sight information is obtained in which line-of-sight videos (video showing the position of the line of sight) P1 to P3 are located at positions corresponding to the driver A1, the screw A2 and the housing A3 included in the view image. Information (visual field image information, visual line information, audio information) acquired by the imaging unit 2, the visual axis detection unit 3, and the microphone 4 is sent to the control box 10 via the wiring 6.

  The determination unit 14 of the control box 10 analyzes the line-of-sight information acquired by the line-of-sight detection unit 3 to determine the gaze state of the photographer (S4). Specifically, the line-of-sight analysis unit 14a of the determination unit 14 obtains the position of the photographer's line of sight in the view image acquired by the imaging unit 2 and the residence time of the position of the line of sight based on the line-of-sight information. . Then, the gaze determination unit 14b determines the presence or absence of the photographer's gaze based on the position of the sight line and the staying time.

  For example, the gaze determination unit 14b determines the latest gaze position of the photographer obtained based on the gaze information on the XY coordinate plane in which the horizontal angle of view and the vertical angle of view respectively correspond to the X axis and Y axis. The amount of movement of the center of gravity of the position coordinates for a total of ten of the nine line-of-sight positions obtained most recently (angle difference between two line-of-sight directions toward each center of gravity before and after movement) is calculated. The X-coordinate of the center of gravity is an average of X-coordinates of ten eye-gaze positions, and the Y-coordinate of the centroid is an average of Y-coordinates of ten eye-gaze positions. Then, the gaze determination unit 14b determines that the drift amount indicating the blur of the sight line is small if the angle difference is, for example, 0.5 ° or less, and the state (retention time) is a predetermined time (for example, 2 seconds) or more If it continues, it will be judged as a gaze. In the present embodiment, the number of eye positions used when determining the center of gravity is ten, the angle difference is 0.5 ° within 1 °, and the predetermined time is 2 seconds. Can be changed appropriately according to the work content.

  Here, whether or not the angle difference is within 0.5 ° depends on the shooting angle of view (horizontal and vertical angle of view) of the imaging unit 2 and the parameter (number of pixels, size of one pixel) of the imaging device. Can be used to determine For example, FIG. 6 schematically shows the XY coordinate plane described above. The X axis corresponds to the horizontal angle of view of the imaging unit 2, and the Y axis corresponds to the vertical angle of view of the imaging unit 2. In the imaging unit 2, when the angle of view a ° corresponds to the number b of pixels, the angle of view 0.5 ° corresponds to the number of pixels (0.5 b / a). Further, assuming that the length of one side of one pixel (square) of the image sensor is c (μm), the length (μm) corresponding to an angle difference of 0.5 ° is (number of pixels) × (one pixel The length of one side) = (0.5 b / a) × c. Therefore, whether the above-mentioned angular difference is within 0.5 ° by judging whether the movement amount T of the center of gravity G of ten eye gaze positions is less than or equal to (0.5 b / a) × c It can be determined whether or not.

  When it is determined by the gaze determination unit 14b that the photographer has gazed (Yes in S5), the storage target determination unit 16 performs an image process using a well-known ridge detection method to obtain an outline of the subject in the view image. Is detected, and an area surrounded by an outline is specified as an object area where an object exists (S6).

  Next, the storage target determining unit 16 extracts (trimms) the subject region specified in S6 from the view image by image processing (S7). Then, the storage target determining unit 16 determines the extracted subject region as an image of a storage target to be stored in the storage unit 12 (S8). At this time, the storage target determination unit 16 may determine the above-mentioned subject region as an image with the same magnification as the storage target image, or enlarge the above-mentioned subject region to generate a magnified image, and this magnified image is stored. It may be determined as an image. Here, it is assumed that the storage target determination unit 16 determines the enlarged image of the subject region as the storage target image as in the latter case. Note that the above storage target is also a display target displayed on the display unit when the created moving image manual is played back.

  On the other hand, when it is determined by the gaze determination unit 14b that the photographer does not gaze (No in S5), the storage target determination unit 16 stores the entire view image acquired in S3 in the storage unit 12 It determines as an image of an object (S9).

  Next, the voice recognition unit 15 performs voice recognition on the voice information acquired in S3 and converts it into text data (S10). Then, the storage target determination unit 16 sets a comment addition area in which the text data acquired in S10 is added as a comment to the above-described storage target image (S11). At this time, in the image to be stored, the comment addition area is set to an area outside the subject being gazed when the photographer gazes, and when the photographer does not gaze, an area where the subject does not exist. It is set to (area that does not overlap with the subject). In addition, in the image to be stored, it may be determined whether the storage object determining unit 16 performs the same processing as S6 (for example, image processing using a ridgeline detection method) whether or not it is an area where a subject is present. it can.

  Then, the storage target determination unit 16 adds the text data acquired in S10 as a comment to the comment addition area of the storage target image acquired in S8 or S9, and combines these (S12). Then, the storage target determining unit 16 stores the image of the storage target after combination in the storage unit 12 in association with the elapsed time (S13). The above processing is repeatedly performed until the photographer operates the operation unit 11 to input work end (S14), and when the work is ended, a series of processes are ended.

  FIG. 7 shows an example of an image to be stored (stored image, display image) stored in the storage unit 12 in S13. Since the storage target is a view image when the photographer does not gaze, an image to which a comment “5 screws are tightened” is added in the view image is stored in the storage unit 12 as data of the moving image manual. Ru. At this time, since there is no gaze on the subject (driver A1, case A2, screw A3), the above comment is added to the area where the subject does not exist in the view image (area not overlapping with driver A1, case A2 and screw A3) Be done.

  Further, when the photographer gazes at the screw A2, the above-mentioned comment is added to the outside of the image of the screw A2 because the storage target is the image (object region) of the screw A2 extracted and enlarged from the view image. The stored image is stored in the storage unit 12 as data of a moving image manual. Further, when the photographer gazes at the housing A3, the storage target is the image (subject region) of the housing A3 extracted and enlarged from the view image, and thus the above-described storage is performed outside the image of the housing A3. The image to which the comment is added is stored in the storage unit 12 as data of the moving image manual.

  When reproducing a moving image manual, data of the moving image manual stored in the storage unit 12 is read by a reproduction device (not shown), and the display unit (not shown) displays the moving image manual (FIG. The same image is displayed. Thus, the viewer can view the displayed moving image manual.

  As described above, in the moving image manual creation system 1 of the present embodiment, the determination unit 14 determines the presence or absence of the photographer's gaze based on the line-of-sight information acquired by the line-of-sight detection unit 3 (S3 to S5). Then, the storage target determination unit 16 determines at least a part of the view image acquired by the imaging unit 2 as a storage target based on the determination result of the determination unit 14 (S6 to S9). The determined storage target is stored in the storage unit 12 as data of a moving image manual (S13).

  The storage target determination unit 16 determines the storage target based on the determination result of the determination unit 14 (presence or absence of the photographer's gaze). The storage target can be made different according to the line-of-sight information of the person. As a result, it is possible to store a different storage target according to the line-of-sight information in the storage unit 12 as data of the moving image manual, and create a moving image manual that accurately reflects the intention of the photographer. Therefore, at the time of reproduction of the moving image manual, it is possible to convey the intention of the photographer's line of sight to the viewer intuitively and easily.

  Further, since the storage target is determined using the visual field image and the line-of-sight information of the photographer, for example, the operation and process of manually inputting data by the photographer operating the operation unit 11 are unnecessary at all. Therefore, it is possible to automatically create a moving image manual by a simple method using the photographer's view image and gaze information.

  Further, since the determination unit 14 includes the above-described line-of-sight analysis unit 14a and the gaze determination unit 14b, the gaze determination unit 14b determines the position of the line of sight of the photographer determined by the line-of-sight analysis unit 14a; The gaze condition (presence or absence of gaze) of the photographer can be accurately determined based on the staying time of the position of the line of sight. In addition, when the photographer concentrates on work and looks at an object (subject), the degree of fixation increases, but it is also possible to accurately determine the degree of fixation based on the position of the line of sight of the photographer and the staying time. Become.

  In addition, the gaze determination unit 14b detects a drift amount indicating a blur of the gaze based on the position of the gaze of the photographer determined by the gaze analysis unit 14a, and the drift amount is equal to or less than the threshold and the residence time is predetermined time If it is above, it is determined that the photographer is watching (S4, S5). If the amount of drift is equal to or less than the threshold and the residence time is equal to or more than the predetermined time, it can be determined that the photographer's gaze degree is high. In this case, the determination accuracy is improved by determining that the photographer gazes. Can.

  Further, when it is determined by the determination unit 14 that the photographer gazes, the storage target determination unit 16 specifies a subject region in which the subject being gazed by the photographer is present in the view image (S6) ) Extract the subject area from the view image (S7), and determine the extracted subject area as a storage target (S8). By extracting only the subject region intended by the photographer and determining it as the storage target, it is possible to present only the information of the necessary region (subject region) to the viewer at the time of playback of the storage target (not necessary at the time of playback Information may not be presented to the viewer). In this way, the viewer can see the presented information and start working on it without getting lost.

  Further, the storage target determination unit 16 enlarges the extracted subject area and determines the image after enlargement as a storage target (S8). In this case, since the necessary area (subject area) is enlarged and displayed at the time of reproduction of the storage target, it is possible and easy for the viewer to confirm details of the subject from the displayed image.

  Further, when it is determined by the determination unit 14 that the photographer does not gaze, the storage target determination unit 16 determines a view image acquired by the imaging unit 2 as a storage target (S9). When the photographer does not gaze, the entire view image is determined as the storage target. Therefore, at the time of reproduction of the storage target, the viewer looks over the entire view of the photographer (understands in a wide view) and Information (for example, work content) can be grasped.

  Further, the voice recognition unit 15 recognizes the voice of the photographer input by the microphone 4 and converts it into text data (S10), and the storage target determination unit 16 converts the above text data into at least a part of the view image. To determine the storage target (S12). As described above, in order to add text data by voice recognition to a storage target, at the time of playback of the storage target, the viewer clearly views the instruction and advice of the photographer as visual information (text data) while viewing the displayed image. It can be recognized. In addition, since the voice recognition unit 15 is provided, as in the present embodiment, a configuration in which voice input is performed at the same time as shooting of a field of view becomes possible. It is also possible to save time and effort.

  In particular, the storage target determination unit 16 combines the text data with at least a part of the view image so that the text data acquired by speech recognition is located outside the area where the subject is present in the storage target (S11) , S12). In the storage target after composition, since the text data does not overlap in the important area where the subject exists, the viewer can clearly recognize both the important subject and the text data when reproducing the storage target. .

[Variation of Extraction of Subject Region (1)]
In the process of S7 described above, the storage target determining unit 16 may change the extraction range when extracting the subject region from the view image according to the information acquired by the determination unit 14. Hereinafter, variations of such extraction will be described.

  FIG. 8 shows another example of the view image, the line-of-sight information and the audio information acquired in S3. The storage target determining unit 16 may change the extraction range according to the staying time of the line of sight acquired by the line-of-sight analysis unit 14a. For example, the storage target determination unit 16 may narrow the extraction range and extract a part of the subject region when the residence time becomes long. In addition, the storage target determination unit 16 may enlarge a part of the subject region extracted from the view image and determine the image after the enlargement as a storage target. In this case, at the time of reproduction of the storage target, the enlarged image is displayed.

  FIG. 9 schematically shows the relationship between the dwell time of the line of sight and the enlargement factor of the display at the time of reproduction of the storage target. Here, a state in which the amount of drift indicating a blur of the line of sight is small (a state in which an angle difference before and after a change in the line of sight is, for example, 0.5 °) is set as the start of the residence time (0 second). Based on the flow shown in FIG. 3, first, if the state with a small drift amount continues for 2 seconds, it is determined to be a gaze (S4, S5), but it is determined to be a storage target since it is a determination without gaze. The image to be displayed is a view image (S9), and at the time of reproduction of the storage target, the view image is displayed.

  After it is determined that there is a gaze, the subject area is identified in the view image until the start of reduction of the extraction range shown next (from 2 seconds to 5 seconds after the start of the small drift amount). The whole is extracted (S6, S7). Therefore, the subject region is displayed at the time of reproduction of the storage target. The upper part of FIG. 10 shows an image (subject image) of the subject region R1 extracted from the view image of FIG. 8 between the gaze determination and the extraction range reduction start.

  If it is determined that the state of small drift continues for 3 seconds after the presence of gaze (after 5 seconds from the start of small drift amount), the storage target determining unit 16 starts the reduction of the extraction range and also falls within the extraction range. Then, a part of the subject area is extracted (S7), and a part of the extracted subject area is enlarged and determined as a storage target (S8). The lower part of FIG. 10 shows an image (subject image) obtained by extracting and enlarging a region R2 of a part of the subject region R1 from the view image of FIG. 8 after reduction of the extraction range is started. Since the region R2 is a partial region of the subject region R1, it is apparent that the extraction range is smaller than the subject region R1.

  According to the reduction of the extraction area, the extracted subject area is expanded and expanded to the target frame size in 3 seconds after the start of expansion (in 8 seconds from the start of the small drift amount). When the standard deviation (variation) of the line-of-sight position is σ, the target frame size includes 3σ of the line-of-sight position that fluctuates between the start of enlargement after the state of small drift amount is started. It is a state. At the time of reproduction of the storage target, a part (enlarged image) of the subject area is displayed at a predetermined enlargement factor. Note that the enlargement ratio L on the vertical axis in FIG. 9 corresponds to the state in which 3σ of the gaze position is included. Further, the center of the display at the time of reproduction of the storage target is the average of the sight line positions during the synchronization (period from the start of enlargement to the end of enlargement).

  The above angle difference, timing of gaze determination (2 seconds after the start of small drift amount), timing of display enlargement (3 seconds after gaze determination), speed of enlargement, target frame size, etc. are examples. It can be changed appropriately according to the situation.

  For example, in a detailed operation such as soldering of a substrate, the dwell time (gaze time) of the photographer's gaze naturally becomes long, and it is often better for the viewer to see the details at the time of reproduction of the memory object. As described above, as described above, the storage target determination unit 16 changes the extraction range when extracting the subject region from the view image according to the residence time acquired by the sight line analysis unit 14a. While narrowing the extraction range according to time, it is possible to store an image obtained by enlarging a part of the extracted subject region. As a result, when reproducing the storage target, it is possible to display the detail of the subject largely to the viewer and transmit detailed information.

  In addition, for example, when the extraction range is greatly narrowed from the initial stage of gaze (while the dwell time of the gaze position is short), a partial area of the view image is extracted, and the extracted area is enlarged and stored as a storage target In the initial stage of reproduction of the storage target, the viewer may have difficulty grasping the subject and the entire work, and it may become difficult to grasp which work or which part of the subject is viewed by viewing the image. . As in the present embodiment, the extraction range is reduced stepwise and the extracted area is expanded stepwise to be the storage target, so that the display magnification (magnification magnification) gradually increases when the storage target is reproduced. In the initial stage of reproduction, the viewer can easily grasp the subject and the entire work, so that the above-mentioned inconvenience can be avoided.

[Variation of Extraction of Subject Region (2)]
FIG. 11 illustrates still another example of the view image, the line-of-sight information, and the audio information acquired in S3. In the process of S7 described above, even if the storage target determining unit 16 changes the extraction range when extracting the subject region from the view image according to the change of the gravity center of the plurality of positions of the line of sight acquired by the determination unit 14 Good. The center of gravity can be determined by the gaze determination unit 14 b of the determination unit 14, but the method of determination is as described above.

  If it is determined based on the flow shown in FIG. 3 that the photographer is gazing at a certain subject in the view image shown in FIG. 11 (S4, S5), the storage target determination unit 16 The subject region R11 in which the subject is present is specified (S6), the subject region R11 is extracted from the view image (S7), and the extracted subject region R11 is determined as a storage target (S8). The upper part of FIG. 12 shows an image to be stored at this time, that is, an image (subject image) of the subject region R11 extracted from the view image of FIG.

  Next, when the photographer gazes at the left side of the subject along with the work of attaching the seal to the subject from the left side, the eye gaze detection unit 14a detects a plurality of eye positions detected by the eye line analysis unit 14a The center of gravity moves. When the gaze determination unit 14b recognizes changes in the center of gravity of the multiple positions of the line of sight, the storage target determination unit 16 determines from the view image an area where the center of gravity of the line of sight of the photographer is located in the subject area R11. That is, the area R12 on the left side where the photographer gazes at is extracted (S7), and this is enlarged and set as a storage target (S8). The middle part of FIG. 12 shows an image to be stored at this time, that is, an image (enlarged image) of the region R12 extracted from the view image of FIG.

  Furthermore, when sticking the right side of the sticker to the subject, if you gaze at the right side of the subject, the centers of gravity of multiple positions of the line of sight detected by the line of sight analysis unit 14a . When the gaze determination unit 14b recognizes changes in the center of gravity of the multiple positions of the line of sight, the storage target determination unit 16 determines from the view image an area where the center of gravity of the line of sight of the photographer is located in the subject area R11. That is, the area R13 on the right side where the photographer gazes at is extracted (S7), and this is enlarged to be a storage target (S8). The lower part of FIG. 12 shows an image to be stored at this time, that is, an image (enlarged image) of the region R13 extracted from the view image of FIG.

  When strict alignment is required in the sealing process, the gaze position often changes in the same object. As described above, the storage target determination unit 16 changes the extraction range according to the change in the gravity center of the plurality of positions of the line of sight acquired by the determination unit 14 so that the gaze position of the photographer moves within the same subject. Even in this case, it is possible to change the extraction range to extract the image of the gaze position and enlarge the extracted image to obtain the storage target. Thus, at the time of reproduction of the storage target, it is possible to provide (display) the image in which the important position is enlarged to the viewer, and to transmit the important information to the viewer with certainty. In addition, since the barycentric position of the line of sight of the photographer can be brought to the center of the display image at the time of reproduction, an image can be displayed which accurately reflects the intention of the photographer, and information can be transmitted to the viewer. .

  As described above, changing the extraction range at the time of extracting the subject region from the view image according to the information (the residence time, the change of the gravity center of the plurality of positions of the gaze) acquired by the determination unit 14 By this, it is possible to extract and enlarge only the important area that accurately reflects the photographer's intention from the view image, and provide it to the viewer at the time of reproduction. As a result, the viewer can immediately recognize and grasp the important area intended by the photographer in the display image, and can appropriately grasp the content of the work based on the display image.

  In the present embodiment, an example in which the imaging unit 2 and the sight line detection unit 3 are mounted on the glasses 5 has been described. However, for example, the sight line detection unit 3 may be incorporated in the imaging unit 2. In this case, the photographer captures a view image while holding the imaging unit 2. However, this is effective in that the moving picture manual creation system 1 can be configured without using the glasses 5, and the number of parts can be reduced.

  As mentioned above, although embodiment of this invention was described, the scope of the present invention is not limited to this, and it can extend or change and carry out within the range which does not deviate from the main point of invention.

  The present invention can be used, for example, in a system for automatically creating a video manual.

1 movie manual creation system 2 imaging unit 3 gaze detection unit 4 microphone (audio input unit)
12 storage unit 14 determination unit 14a gaze analysis unit 14b gaze determination unit 15 speech recognition unit 16 storage target determination unit

Claims (11)

An imaging unit that captures a field of view of a photographer as a moving image and acquires a field of view image;
A gaze detection unit that detects gaze information of the photographer;
A determination unit that determines the presence or absence of the gaze of the photographer based on the line-of-sight information;
A storage target determining unit configured to determine at least a part of the view image as a storage target based on the determination result of the determination unit;
And a storage unit configured to store the storage target determined by the storage target determination unit as data of a moving image manual. The determination unit
A line-of-sight analysis unit which obtains the position of the line-of-sight of the photographer in the view image and the dwell time of the position of the line-of-sight based on the line-of-sight information;
The movie manual creation system according to claim 1, further comprising: a gaze determination unit that determines the gaze condition of the photographer based on the position of the gaze and the staying time.   The gaze determination unit detects a drift amount indicating a blur of the sight line based on the position of the sight line, and when the drift amount is equal to or less than a threshold and the staying time is equal to or more than a predetermined time The moving image manual creation system according to claim 2, wherein it is determined that the person gazes.   The storage target determination unit specifies a subject region in which the subject being gazed by the photographer is present in the view image when the determination unit determines that the photographer is watching. 4. The moving picture manual creation system according to claim 2, wherein the subject area is extracted from the view image, and the extracted subject area is determined as the storage target.   5. The moving picture manual creation system according to claim 4, wherein the storage target determining unit enlarges the extracted subject area and determines the image after enlargement as the storage target.   5. The moving image manual creation method according to claim 4, wherein the storage target determination unit changes an extraction range when extracting the subject region from the view image according to the information acquired by the determination unit. system.   The moving image manual creation system according to claim 6, wherein the storage target determination unit changes the extraction range in accordance with the residence time acquired by the determination unit.   The moving image manual creation system according to claim 6, wherein the storage target determination unit changes the extraction range in accordance with a change in the center of gravity of the plurality of positions of the line of sight acquired by the determination unit.   The storage target determination unit according to any one of claims 4 to 8, wherein the storage target determination unit determines the view image as the storage target when the determination unit determines that the photographer does not gaze. Video manual creation system. A voice input unit to which the voice of the photographer is input;
And a voice recognition unit that recognizes voice input by the voice input unit and converts the voice into text data.
The moving picture manual creation system according to any one of claims 1 to 9, wherein the storage target determination unit combines the text data with at least a part of the view image to determine the storage target. .   10. The image processing apparatus according to claim 10, wherein the storage target determination unit combines the text data with at least a part of the view image such that the text data is located outside an area where a subject is present in the storage target. Video manual creation system described in.

Images