JP2016218594A - Image processor, control method image processor and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor capable of improving user-friendliness by associating a moving image displayed on a display device with an actual object.SOLUTION: The image processor includes: an external scene sensor which picks up at least one object; and an image generation section that generates a virtual image corresponding to at least one moving object in the picked-up image of the object.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique of an image processing apparatus.

  Conventionally, as described in Patent Document 1, an image generated in response to a part of the motion of a video-captured subject is displayed in synchronization with a video of the subject of video-capture The device is known.

Japanese Patent Laid-Open No. 2002-230086

  However, in the technique described in Patent Document 1, although the moving image of the subject displayed on the display device and the generated image are synchronized, the moving image displayed on the display device is displayed on the display device. There is no relationship with no real object. Therefore, there has been a problem that it is desired to improve the convenience and usability of the user by associating the moving image displayed on the display device with the actual object. In addition, there is a problem that it is desirable to easily create a moving image in which a moving image displayed on the display device and an actual target are associated with each other.

  SUMMARY An advantage of some aspects of the invention is to solve at least a part of the problems described above, and the invention can be implemented as the following forms.

(1) According to an aspect of the present invention, an image processing apparatus is provided. The image processing apparatus includes an outside scene sensor that captures at least one target; and an image generation unit that generates a virtual image corresponding to at least one of the captured targets. . According to this form of the image processing apparatus, since the image generation unit generates a virtual image corresponding to the moving object, the user can easily create a moving image including a virtual image such as work support, for example. , User convenience is improved.

(2) In the image processing apparatus according to the above aspect, the image generation unit is configured to select a movement area of the image generation target that is the target on which the virtual image is generated, the image generation target among the captured targets. The virtual image of the image generation target may be generated in association with at least one of the objects to be excluded. According to the image processing apparatus of this aspect, when a moving image including the generated virtual image is reproduced, the virtual image is a position or size of a target that actually exists associated with the virtual image. Is displayed in association with. For example, when the moving image is a support moving image such as work, by superimposing a virtual image on the work target, the workability of the user is further improved and the usability of the user is improved.

(3) In the image processing apparatus according to the aspect described above, the image generation unit may set at least one of the size of the virtual image and the movement region to the size of the target associated with the movement region of the image generation target. The virtual image may be generated in association with each other. According to the image processing apparatus of this aspect, when a moving image including the generated virtual image is reproduced, the virtual image is a position or size of a target that actually exists associated with the virtual image. Are displayed in association with each other, and the usability of the user is further improved.

(4) In the image processing apparatus according to the above aspect, the image generation unit may generate whether or not the virtual image is displayed in correspondence with whether or not the set trigger target is detected. According to the image processing apparatus of this aspect, since the virtual image is created so that the virtual image is displayed by detecting a predetermined condition set in advance, the display timing according to the use of the virtual image is set. Can be set.

(5) In the image processing apparatus according to the aspect described above, the image generation unit; as a trigger target; a moving target that is the moving target among a plurality of the captured targets, and a predetermined target from the moving target An associated stationary object that is determined to be within a distance and is not moving; and is associated with a combination of the presence / absence of the moving object and the presence / absence of the associated stationary object. A virtual image may be generated that is a combination of the movement target corresponding image as the first image and the related still target image as the related still target virtual image. According to this form of the image processing apparatus, even if no particular operation is accepted, a plurality of virtual images including a virtual image related to the movement target are created, and the usability of the image processing apparatus is improved.

(6) The image processing apparatus according to the above aspect may further include an operation receiving unit that receives an operation. Here, the image generation unit may generate the virtual image by deleting unnecessary portions based on an accepted operation. According to this form of the image processing apparatus, a moving object that does not need to be generated or a stationary object that needs to be generated is selected as a virtual image, and an AR scenario or a composite scenario that is easier for the user to use can be created. , User convenience is improved.

(7) In the image processing apparatus according to the above aspect, the image generation unit generates, as the virtual image, a corresponding image while the moving target is moving among the plurality of captured images. May be. According to the image processing apparatus of this aspect, a virtual image of a moving object is automatically generated even when an operation for selecting a target for generating a virtual image is not performed. Therefore, for example, when a moving image including a virtual image is a support moving image such as work, and it is necessary to move some target in the work, a virtual image of a moving object that is an object to be moved is automatically generated. , User convenience is improved.

(8) The image processing apparatus according to the above aspect further includes: a target selection unit; the target selection unit identifies a shape of a human body and a shape other than the human body as the at least one target; The image generation unit may not generate the virtual image corresponding to the shape of a human body among the captured objects. According to this form of the image processing apparatus, when a moving image including a virtual image is being executed, a user's hand, which is a means for moving an object to be moved, is not displayed as a virtual image. This eliminates the need for visually recognizing unnecessary virtual images such as hands and improves the convenience of the user.

(9) The image processing apparatus according to the above aspect further includes: an audio acquisition unit that acquires external audio; the image generation unit; the image generation target that is the target on which the virtual image is generated; and the image The virtual image may be generated by associating the sound acquired while the generation target is moving. According to the image processing apparatus of this aspect, in the moving image including the virtual image to be created, in addition to visual information such as a virtual image generated based on the captured image, auditory information such as sound Since the associated virtual image is also included, the convenience for the user is improved.

(10) In the image processing apparatus of the above aspect, the image generation unit may generate the acquired voice as a character image in association with the virtual image. According to this form of the image processing apparatus, sound can be generated as visual information in parallel with the virtual image, the user can easily recognize the information, and the convenience for the user is further improved.

(11) The image processing apparatus according to the above aspect may further include a distance measurement unit that measures a distance to the target; the image generation unit may generate the virtual image based on the measured distance Good. According to this form of the image processing apparatus, since the generated virtual image can be generated as a three-dimensional model, the user can more easily recognize the virtual image, and the convenience for the user is further improved.

(12) In the image processing device according to the above aspect, the image generation unit may display a specific image at a specific time point of the virtual moving image when the virtual image is a virtual moving image that changes with time. It may be inserted. According to this form of the image processing apparatus, it is possible to cause the user to recognize what is being performed at a specific time point as visual information via the specific image, and the usability of the user is improved.

(13) In the image processing device of the above aspect, further comprising: an audio acquisition unit that acquires external audio; and when the virtual image is a virtual moving image that changes over time, The virtual moving image may be generated by associating a specific time point of the virtual moving image with the acquired sound. According to this form of the image processing apparatus, it is possible to make the user recognize a specific time point by voice, so that the specific image may overlap the virtual image as compared with the case where the specific image is displayed. In addition, the convenience of the user at a specific time is further improved.

  A plurality of constituent elements of each embodiment of the present invention described above are not essential, and some or all of the effects described in the present specification are to be solved to solve part or all of the above-described problems. In order to achieve the above, it is possible to appropriately change, delete, replace with another new component, and partially delete the limited contents of some of the plurality of components. In order to solve part or all of the above-described problems or to achieve part or all of the effects described in this specification, technical features included in one embodiment of the present invention described above. A part or all of the technical features included in the other aspects of the present invention described above may be combined to form an independent form of the present invention.

  For example, one embodiment of the present invention can be realized as an apparatus including one or two of the two elements of the outside scene sensor and the image generation unit. That is, this apparatus may or may not have an outside scene sensor. Further, the apparatus may or may not have an image generation unit. For example, the outside scene sensor may image at least one target. For example, the image generation unit may generate a virtual image corresponding to at least one of the moving objects among the captured objects. Such an apparatus can be realized as an image processing apparatus, for example, but can also be realized as an apparatus other than the image processing apparatus. According to such a form, it is possible to solve at least one of various problems such as improvement and simplification of the operability of the device, integration of the device, and improvement of convenience of the user who uses the device. it can. Any or all of the technical features of the respective forms of the image processing apparatus described above can be applied to this apparatus.

  The present invention can also be realized in various forms other than the image processing apparatus. For example, a control method for an image processing apparatus, a system having the image processing apparatus, a computer program for realizing the control method and system for the image processing apparatus, a recording medium storing the computer program, and a carrier including the computer program It can be realized in the form of a data signal or the like embodied in

1 is a block diagram functionally showing the configuration of an image processing apparatus according to a first embodiment of the present invention. It is explanatory drawing at the time of imaging the external scene containing a to-be-photographed object with a RGB camera and a distance sensor. It is a flowchart of AR scenario creation processing. It is a flowchart of AR scenario creation processing. It is explanatory drawing which shows the captured image imaged before the work start which does not have a to-be-photographed object. It is explanatory drawing which shows the captured image of the outside scene containing the tracked moving object. It is explanatory drawing which shows the image with which additional information was added to the captured image at the specific time contained in AR scenario. It is explanatory drawing which shows the image after the unnecessary object was erase | eliminated from the captured image. It is explanatory drawing which shows the produced | generated AR image and the object matched with AR image. It is a flowchart of a composite scenario creation process. It is explanatory drawing which shows the edit image displayed when the trigger is set. It is explanatory drawing which shows the image displayed when it branches to a branch scenario when a trigger is detected. It is explanatory drawing which shows the external appearance structure of a head mounting type display apparatus (HMD). It is a flowchart of a composite scenario execution process. It is explanatory drawing which shows the visual field which a user visually recognizes when the corresponding | compatible object set to the composite scenario performed is detected. It is explanatory drawing which shows the visual field which a user visually recognizes when the trigger object set to the branch scenario performed is detected. It is a block diagram which shows functionally the structure of the image processing apparatus in 2nd Embodiment. It is a flowchart of a part of AR scenario creation process in 2nd Embodiment. It is a block diagram which shows functionally the structure of the image processing apparatus in 3rd Embodiment. It is a flowchart of a part of AR scenario creation process in 3rd Embodiment. It is a flowchart of AR scenario creation processing in the fourth embodiment. It is a flowchart of AR scenario creation processing in the fourth embodiment. It is explanatory drawing in the case of imaging the outside scene containing a several to-be-photographed object with the RGB camera and distance sensor in 4th Embodiment. It is explanatory drawing in the case of imaging the outside scene containing a several to-be-photographed object with the RGB camera and distance sensor in 4th Embodiment. It is explanatory drawing in the case of imaging the outside scene containing a several to-be-photographed object with the RGB camera and distance sensor in 4th Embodiment. It is explanatory drawing in the case of imaging the outside scene containing a several to-be-photographed object with the RGB camera and distance sensor in 4th Embodiment. It is explanatory drawing in the case of imaging the outside scene containing a several to-be-photographed object with the RGB camera and distance sensor in 4th Embodiment. It is a table | surface which shows an example of the combination of the trigger object produced in the process of step S95 of the AR scenario production process of 4th Embodiment, and the combination of AR scenario. It is a flowchart of the display image determination process in case AR scenario is performed. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when the display image matched with the combination (1) is displayed on the optical image display part. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when the display image matched with the combination (2) is displayed on the optical image display part. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when the display image matched with the combination (3) is displayed on the optical image display part. It is explanatory drawing which shows an example of the visual field which a user visually recognizes when the display image matched with the combination (4) is displayed on the optical image display part.

  The term “outside scene sensor” in the present specification includes at least one of an RGB camera and a distance sensor described below. Therefore, any of the RGB camera, the distance sensor, or a combination thereof is an example of the “outside scene sensor”. Of course, the “outside scene sensor” is not limited to the RGB camera, the distance sensor, or a combination thereof described in the embodiment, but is included in the outside scene or the outside scene (these may be called a real environment or a real object). A device that acquires and outputs information for estimating the two-dimensional coordinates or three-dimensional coordinates.

  The term “target selection unit” in this specification has a function of selecting a real object to be represented as a virtual image (AR image) or a function of providing information serving as a basis for selection. Say. In the embodiment, each of the object tracking units 12a, 12b, and 12c is an example of a “target selection unit”.

A. First embodiment:
A-1. Configuration of image processing device:
FIG. 1 is a block diagram functionally showing the configuration of the image processing apparatus 100 according to the first embodiment of the present invention. The image processing apparatus 100 generates a three-dimensional model of a continuously captured subject, and generates an augmented reality (AR) image based on the generated three-dimensional model of the subject and various received operations. Note that the AR image in the present embodiment refers to an image displayed in association with an actual object recognized by image recognition or the like.

  The image processing apparatus 100 includes a CPU 10, a data storage unit 50, a power source 60, an RGB camera 31, a distance sensor 32, a microphone 33, an operation unit 34, a display unit 35, a ROM 41, and a RAM 42. I have. The data storage unit 50 stores various data and is configured by a hard disk drive or the like. The power supply 60 supplies power to each unit of the image processing apparatus 100. As the power supply 60, for example, a secondary battery can be used.

  The RGB camera 31 is a camera that captures an outside scene in a predetermined range including a subject. In the present embodiment, the RGB camera 31 includes three first cameras 311, second cameras 312, and third cameras 313 arranged at different positions. The RGB camera 31 transmits the captured RGB data of the outside scene to the sensor control unit 15 of the CPU 10 described later. The distance sensor 32 is a depth sensor that measures the distance to the irradiated object by photographing innumerable irradiated points with an infrared camera. In the present embodiment, the distance sensor 32 is arranged next to each camera so as to correspond to the first camera 311, the second camera 312, and the third camera 313 on a one-to-one basis. That is, the distance sensor 32 includes three depth sensors (a first distance sensor 321, a second distance sensor 322, and a third distance sensor 323) arranged at different positions, like the RGB camera 31. . The distance sensor 32 captures data with an infrared camera and transmits data of countless points of infrared reflected light reflected by the surface of the object to the sensor control unit 15 of the CPU 10. In another embodiment, the distance sensor 32 may measure the distance to the target using a TOF (Time-of-Flight) method. In this embodiment, each of the RGB camera 31 and the distance sensor 32 is configured by three cameras 311, 312, 313 and sensors 321, 322, 323, but in another embodiment, three cameras A smaller number of cameras and sensors may be used, or a larger number of cameras and sensors may be used than three. Further, the number of RGB cameras 31 and the number of distance sensors 32 do not have to be the same, and need not correspond one-to-one. The RGB camera 31 and the cameras 311, 312, 313 correspond to an outside scene sensor in the claims. However, when the RGB camera 31 and the distance sensor 32 have a one-to-one correspondence, the outside scene or the real object is detected by the RGB camera 31 and the distance sensor 32 including the measurement of the distance D from the distance sensor 32 to the real object. Capturing may be referred to as “imaging”. At this time, the captured image is represented by RGBD data. The RGBD data is, for example, data having R, G, B, and distance D values for each pixel.

  FIG. 2 is an explanatory diagram in the case where the outside scene SC including the subject OB is imaged by the RGB camera 31 and the distance sensor 32. As shown in FIG. 2, the three cameras 331, 332, and 333 arranged at different positions capture an outside scene including the subject OB. The captured image of the camera 331 corresponds to the captured image of the first camera 311 and the first distance sensor 321, the captured image of the camera 332 corresponds to the captured image of the second camera 312 and the second distance sensor 322, and the camera 333 These captured images correspond to the captured images of the third camera 313 and the third distance sensor 323. As shown in FIG. 2, the right hand of the cook as the subject OB has the scale TL, the left hand of the subject OB holds the head of the fish FS, and the cook takes the scale of the fish FS by the scale TL. The state is imaged. Based on the RGB data and distance data acquired by the cameras 331, 332, and 333 arranged at different positions, the position and color of the target included in the outside scene SC such as the subject OB, the scale removal TL, and the fish FS are specified. .

  As a method for specifying the position of the camera or sensor, other modes can be used. In addition, for example, when the positional relationship and camera parameters of the cameras 331, 332, and 333 are known and the horizontal axis of the distance sensor 32 is parallel to the ground, A light source that intermittently emits light (for example, infrared light) is installed within a common imaging range of the cameras 331, 332, and 333. Each of the cameras 331, 332, and 333 images the light source (the image is represented by RGBD data for each pixel. The distance D is a distance from the distance sensor 32), thereby each of the cameras 331, 332, and 333. The three-dimensional position (Xi, Yi, Zi) of the light source viewed from (i = 0, 1, 2) is estimated. Then, for example, two coordinates of the common light source viewed from each of the cameras 332 and 333 (i = 1, 2) are converted into coordinates viewed from the camera 331 (i = 0). At this time, a transformation matrix is derived for each of the cameras 332 and 333 so that the coordinates obtained by transforming the coordinates viewed from the cameras 332 and 333 coincide with the coordinates viewed from the camera 331. Specifically, a transformation matrix that minimizes the difference between the coordinates of the camera 331 and the transformed coordinates is calculated by iterative calculation. By making such settings, it is possible to fuse the three-dimensional models generated from the viewpoints of the cameras 331, 332, and 333 and generate one three-dimensional model that does not depend on the camera viewpoint. Note that the accuracy may be improved by changing the position of the installed light source and capturing images by the cameras 331, 332, and 333. Further, the number of sets of the RGB camera 31 and the distance sensor 32 may be four or more.

  The microphone 33 (FIG. 1) acquires external sound while the RGB camera 31 and the distance sensor 32 are imaging a predetermined range or when receiving an operation from the user. The microphone 33 transmits an audio signal based on the acquired audio to the UI control unit 16 of the CPU 10 described later. The operation unit 34 is a user interface (UI) that receives input from the user. The operation unit 34 includes a keyboard and a mouse. The operation unit 34 transmits to the UI control unit 16 of the CPU 10 a control signal corresponding to the pressed keyboard key or a control signal based on a change in the position of the mouse pointer. The display unit 35 is a liquid crystal panel that displays an image based on the image signal transmitted from the UI control unit 16. The user can operate the image processing apparatus 100 by operating the operation unit 34 and the microphone 33 while viewing the image displayed on the display unit 35. The operation unit 34 and the microphone 33 correspond to an operation reception unit in the claims, and the microphone 33 corresponds to an audio acquisition unit.

  The CPU 10 controls the image processing apparatus 100 by reading out a computer program stored in the ROM 41, developing it in the RAM 42 and executing it. The CPU 10 includes an AR scenario control unit 11, an object tracking unit 12, an object recognition unit 13, a three-dimensional model generation unit 14 (3D model generation unit 14), a sensor control unit 15, and a user interface control unit 16 (UI). A control unit 16), an AR scenario operation setting unit 17, an additional information acquiring unit 18, an unnecessary image erasing unit 19, and an AR image extracting unit 21.

  The sensor control unit 15 acquires RGB data of the outside scene transmitted from the RGB camera 31 and countless point data captured by the infrared camera transmitted from the distance sensor 32. The sensor control unit 15 transmits the data acquired from the RGB camera 31 and the distance sensor 32 to each of the object tracking unit 12 and the 3D model generation unit 14. Further, the RGB camera 31 and the distance sensor 32 are controlled based on the control signal transmitted from the UI control unit 16.

  Based on the control signal transmitted from the AR scenario control unit 11, the 3D model generation unit 14 uses the RGB data of the RGB camera 31 transmitted from the sensor control unit 15 and the distance data of the distance sensor 32 to capture a predetermined image. A three-dimensional model (3D model) is created for an object existing in the range. As a specific creation of the three-dimensional model, the 3D model generation unit 14 acquires the shape of the object in the imaging range based on the distance data acquired by the distance sensor 32, and based on the acquired distance data, A three-dimensional model is generated by detecting the same boundary in the acquired object shape. The 3D model generation unit 14 colors the generated three-dimensional model based on the RGB data transmitted from the RGB camera 31. The 3D model generation unit 14 transmits the generated colored three-dimensional model and the detected data of the same boundary to the object recognition unit 13.

  The object recognizing unit 13 uses the generated three-dimensional model and the detected same boundary data based on the control signal transmitted from the AR scenario control unit 11 to generate continuous boundary data. Is recognized as one object. In other words, the object recognizing unit 13 separates the three-dimensional model and recognizes the objects as individual objects based on the data of the discontinuous boundaries. In addition, the object recognition unit 13 compares the human body parts (for example, hands and feet) stored in the data storage unit 50 (described later) with the pattern matching or statistical identification method via the AR scenario control unit 11. Extract the human body from the 3D model. The distance sensor 32 and the sensors 321, 322 and 323 correspond to a distance measuring unit.

  Based on the control signal transmitted from the AR scenario control unit 11, the object tracking unit 12 identifies the movement of the recognized object among the recognized objects while moving. The object tracking unit 12 transmits information specifying a moving object (moving object) and a non-moving object (stationary object) to the UI control unit 16 and the AR scenario control unit 11.

  The UI control unit 16 transmits a control signal to each unit included in the display unit 35 and the CPU 10 based on an operation received by the microphone 33 or the operation unit 34. For example, the UI control unit 16 transmits a signal for controlling the RGB camera 31 and the distance sensor 32 to the sensor control unit 15 based on the operation received by the operation unit 34. Further, an image signal for displaying each of the objects on the display unit 35 is transmitted so that the user can select and operate each of the identified objects transmitted from the object tracking unit 12. In addition, the UI control unit 16 includes a text conversion unit 161 that automatically converts voice acquired by the microphone 33 into a character image when creating an AR scenario, which will be described later. The text conversion unit 161 recognizes the acquired voice and converts it into a corresponding character image.

  The AR scenario operation setting unit 17 sets a situation in which an AR scenario created by the image processing apparatus 100 is operated. In the present embodiment, the AR scenario refers to a moving image including an AR image corresponding to at least one moving object, and the AR scenario includes a voice or a character image inserted by a user. The AR scenario operation setting unit 17 sets the AR scenario to be executed when, for example, an object associated with the generated AR image is detected as a real object by image recognition or the like. In addition, the AR scenario operation setting unit 17 actually sets a branch of a plurality of AR scenarios by detecting a preset specific object.

  The additional information acquisition unit 18 acquires information to be added to the AR scenario based on the operation signal received by the UI transmitted from the UI control unit 16 and the control signal transmitted from the AR scenario control unit 11. Information added to the AR scenario includes, for example, a display method setting for displaying an enlarged or reduced AR image set by an operation received by the operation unit 34, and text obtained by converting speech acquired by the microphone 33. There are insertions.

  The AR scenario control unit 11 controls each unit of the CPU 10 in order to create an AR scenario. The AR scenario control unit 11 distinguishes between an object generated as an AR image and an object for which an AR image is not generated based on the moving object and the stationary object specified by the object tracking unit 12 and the operation accepted by the UI. The discrimination result is transmitted to the unnecessary image erasing unit 19 and the AR image extracting unit 21. The AR scenario control unit 11 transmits and receives various data to and from the data storage unit 50 to read and edit the previously created AR scenario, or to store the newly created AR scenario in the data storage unit 50. Or

  The unnecessary image erasing unit 19 erases an image of an object that is not generated as an AR image among the identified objects, based on the control signals transmitted from the AR scenario control unit 11 and the AR scenario operation setting unit 17. In other words, it can be said that the unnecessary image erasing unit 19 selects an object to be generated as an AR image from the captured images. The unnecessary image erasing unit 19 transmits an image signal of an image deleted as an unnecessary object to the AR scenario control unit 11.

  Based on the control signal transmitted from the AR scenario control unit 11, the AR image extraction unit 21 extracts an object to be displayed in the AR scenario as an AR image, and generates an image. The AR image extraction unit 21 generates an AR image as a three-dimensional image based on the distance data acquired by the distance sensor 32. The AR image extracting unit 21 colors the generated AR image based on the RGB data acquired by the RGB camera 31. The AR image extraction unit 21 transmits a signal for specifying an object to be generated as the extracted AR image to the AR scenario control unit 11. In addition, the AR image extraction unit 21 can automatically extract a specific object stored in the data storage unit 50 as an object to be generated as an AR image by receiving a predetermined operation via the operation unit 34. Examples of the extracted object include an object specified by a drawing created by CAD (computer aided design). The AR image extraction unit 21 corresponds to the image generation unit in the claims.

A-2. AR scenario creation process:
3 and 4 are flowcharts of the AR scenario creation process. In the AR scenario creation process, the image processing apparatus 100 creates an AR image such as a moving object included in an outside scene image captured by the RGB camera 31 and the distance sensor 32.

  In the AR scenario creation process, first, the microphone 33 or the operation unit 34 waits for an operation to start creating an AR scenario (step S12). When the microphone 33 receives a predetermined sound set in advance, the image processing apparatus 100 starts an AR scenario creation process. The operation unit 34 receives a predetermined keyboard button operation, and the image processing apparatus 100 starts the AR scenario creation process. In the process of step S12, when the microphone 33 or the operation unit 34 does not accept an operation for starting the AR scenario creation process (step S12: NO), the microphone 33 or the operation unit 34 continues to wait for an operation for starting the AR creation process (step S12: NO). Step S12).

  When an operation for starting the AR scenario creation process is received (step S12: YES), the AR scenario control unit 11 sets an imaging range captured by the RGB camera 31 and the distance sensor 32 (step S14). The AR scenario control unit 11 sets the range and position of the imaging range when the operation unit 34 receives a predetermined operation. In the present embodiment, the imaging range of the RGB camera 31 and the imaging range of the distance sensor 32 are set as the same range. However, in other embodiments, the imaging range of the RGB camera 31 and the imaging range of the distance sensor 32 are set. And may be set separately.

  When the imaging ranges of the RGB camera 31 and the distance sensor 32 are set, the RGB camera 31 acquires RGB data of the imaging range, and the distance sensor 32 determines the distance from the distance sensor 32 to an object existing in the imaging range. Measurement is performed (step S16). The RGB camera 31 transmits the acquired RGB data of the imaging range to the 3D model generation unit 14 and the object tracking unit 12 via the sensor control unit 15. The distance sensor 32 transmits distance data to an object existing in the measured imaging range to the 3D model generation unit 14 and the object tracking unit 12 via the sensor control unit 15.

  The 3D model generation unit 14 generates a three-dimensional model (3D model) based on the RGB data transmitted from the RGB camera 31 and the distance data transmitted from the distance sensor 32 (step S18). The 3D model generation unit 14 generates a three-dimensional model of the shape of the object included in the imaging range based on the distance data. Further, the 3D model generation unit 14 colors the generated three-dimensional model based on the RGB data. In the present embodiment, the 3D model generated by the 3D model generation unit 14 is obtained by merging each 3D model generated from each viewpoint from the cameras 331, 332, and 333 into one. is there. The object recognition unit 13 recognizes individual objects included in the three-dimensional model by using the detected same boundary data for the generated three-dimensional model (step S20).

  FIG. 5 is an explanatory diagram illustrating a captured image captured before the start of work without the subject OB. When the imaging range is set, the RGB camera 31 and the distance sensor 32 start imaging the outside scene SC. Immediately after the start of imaging, the subject OB is not in the imaging range, and as shown in FIG. 5, the captured image does not include the subject OB that is a cook. In the present embodiment, immediately after the imaging is started, the three-dimensional models of the fish FS and the scale removing TL are created without the subject OB. Although the details will be described later, the generated three-dimensional model of the scale removal TL is used as a partial image constituting the AR scenario by being associated with the movement of the scale removal TL specified by the object tracking unit 12. .

  When the process of step S20 in FIG. 3 is performed, the microphone 33 or the operation unit 34 waits for an operation indicating that the initial setting before the start of moving image capturing has been completed (step S21). When the operation indicating that the initial operation has been completed is not accepted, the CPU 10 executes the processes after step S14 again. In the process of step S21, when the microphone 33 or the operation unit 34 receives an operation indicating that the initial operation has been completed (step S21: YES), next, an operation for starting imaging of a moving object is received. Wait (step S22). When the microphone 33 or the operation unit 34 does not accept an operation for starting imaging (step S22: NO), the microphone 33 or the operation unit 34 waits for acceptance of an operation for starting imaging (step S22). In the process of step S22, when an operation for starting imaging is accepted (step S22: YES), the RGB camera 31 and the distance sensor 32 capture the set imaging range as a moving image as time elapses. (Step S22). The object tracking unit 12 tracks the moving object by distinguishing between the moving object and the stationary object among the objects included in the imaging range and generated as the three-dimensional model (step S24). The object tracking unit 12 measures a change amount of the RGB data of the distinguished moving object and a change amount of the measured distance, and specifies a change in position such as a trajectory or posture of the moving object.

  FIG. 6 is an explanatory diagram showing a captured image of the outside scene SC including the tracked moving object. FIG. 6 shows an outside scene SC captured by the RGB camera 31 and the distance sensor 32 when the same range as the imaging range of FIG. 2 is set. In the outside scene SC of FIG. 6, the scale of the scale TL held by the subject OB who is the cook is closer to the head of the fish FS held by the left hand of the subject OB than the outside scene SC of FIG. 2. The point is different. Therefore, the object tracking unit 12 tracks the scale TL and the right hand of the subject OB having the scale TL as moving objects, and distinguishes the left hand of other subjects OB, fish FS, and the like as stationary objects. Note that FIG. 6 shows captured images captured by the cameras 331, 332, and 333, and thus the cameras 331, 332, and 333 are not included in the captured images. In the present embodiment, the right hand of the subject OB and the scale removal TL are tracked as a moving object. In other embodiments, the scale peeled off from the fish FS is tracked as a moving object by the movement of the scale removal TL. May be.

  The AR scenario control unit 11 executes the AR scenario when the created AR scenario is executed on at least one of the tracked moving object or the stationary object that is in contact with the tracked moving object. Set in the AR scenario data as a trigger to start. A trigger for executing an AR scenario is set, and the trigger is detected in a captured image of a camera mounted on a head-mounted display device (Head Mounted Display, HMD) capable of executing AR display. Then, the AR scenario for which the trigger is automatically set is executed. The AR scenario control unit 11 may or may not set a trigger for the AR scenario.

  Next, the additional information acquisition unit 18 acquires additional information to be added to the AR scenario to be created based on the operation received by the microphone 33 or the operation unit 34 (step S26). As the additional information, for example, a character image obtained by converting the voice acquired by the microphone 33 by the text conversion unit 161 while the RGB camera 31 and the distance sensor 32 are imaging the imaging range including the moving object. is there. As another example of additional information, when an AR scenario created in another information processing apparatus is used, it is detected with respect to a scale removal TL that is an object (recommended object) that is desirably detected. A knife may be set as an object that is not desirable to be used (non-recommended object). In this case, when a knife that is a non-recommended object is detected while the AR scenario is being executed, the AR scenario may be stopped or switched to another AR scenario. Details of the execution of the AR scenario will be described in “A-4. Execution of composite scenario” described later.

  FIG. 7 is an explanatory diagram illustrating an image in which additional information is added to a captured image at a specific time point included in the AR scenario. FIG. 7 shows an image obtained by adding the sound acquired by the microphone 33 as a text image TX1 of additional information to the captured image of FIG. 6 captured by the RGB camera 31 and the distance sensor 32. Note that the operation unit 34 may accept a user's operation to change the font size or color of the text image TX1, the position where the text image TX1 is added to the captured image, or the like.

  When the additional information is acquired (step S26 in FIG. 3), in order for the AR scenario control unit 11 to determine which object the AR image is to be created, the operation unit 34 is a target for creating the AR image. An operation for selecting whether or not to automatically delete an unnecessary image of an object that should not be received is accepted (step S28). The AR scenario control unit 11 causes the display unit 35 to display a selection screen for selecting whether to delete unnecessary images automatically or not automatically, and deletes unnecessary images by an operation received by the operation unit 34. Determine the processing. In the process of step S28, when an operation of selection for automatically erasing unnecessary images is accepted (step S28: YES), the unnecessary image erasing unit 19 converts a stationary object and a human body among moving objects to AR. It is deleted as an unnecessary object that is not a target for generating an image (step S38). In other words, the unnecessary image erasing unit 19 does not erase an object excluding a human body among moving objects as a target for generating an AR image. Further, the unnecessary image erasing unit 19 erases human parts extracted by the object recognition unit 13 from the captured image. An object deleted from the captured image is not displayed as an AR image in the created AR scenario. In the present embodiment, even the same object is called a moving object while it is moving, and is called a stationary object while it is not moving. In other embodiments, a moving object and a stationary object may be defined for each object.

  FIG. 8 is an explanatory diagram showing an image after unnecessary objects are erased from the captured image. The moving object that is not erased as an unnecessary object is only the scale removal TL, but FIG. 8 also shows the fish FS (dashed line) and the subject OB (dashed line) erased as an unnecessary object for the sake of explanation. ing. The unnecessary image erasing unit 19 is a cook who erases a stationary object such as the fish FS included in the outside scene SC from the captured image, and determines whether it is a human body regardless of whether it is stationary or moving. The subject OB is also deleted. In the present embodiment, the fish FS has been described as a stationary object. However, whether the fish FS is stationary or moving may be determined based on the amount of movement. For example, there is a case where the tail of the fish FS is lifted by the subject OB. In this case, even if the fish FS or the like is moving due to the amount of movement or the speed of movement, the fish FS is stationary. It may be determined as a moving object.

  When an unnecessary object is deleted from the video (step S38 in FIG. 4), the AR image extraction unit 21 extracts an object other than the unnecessary object selected as a target for generating the AR image by the unnecessary image deletion unit 19. Then, an AR image in which the extracted object (hereinafter also referred to as “extracted object”) is associated with the object in contact with the extracted object (hereinafter referred to as “contact object”) is generated (step S34). Examples of the association between the extracted object and the contact object include an example of associating the size and direction of the contact object and the movement area of the contact object with the movement area of the extraction object and the size of the extraction object. The movement area and size of the target object for generating the AR image are associated with the movement area and size of the object existing within a predetermined distance from the target object, thereby creating the created AR When the scenario is executed, the contact object associated with the AR image is detected, and the AR image generated based on the extracted object is associated with the detected position / shape / size of the contact object. Is displayed. The AR image extraction unit 21 may associate the sound acquired by the microphone 33 as additional information with the AR image while the moving object for generating the AR image is moving during imaging. As a method of associating the sound as additional information with the AR image, for example, the sound associated with the moving object is displayed as a text image only while the AR image of the moving object is displayed. The target for generating the AR image corresponds to the target for generating an image in the claims.

  FIG. 9 is an explanatory diagram illustrating the generated AR image and an object associated with the AR image. FIG. 9 shows an image AR1 (solid line) of the scale removal TL generated as an AR image and a fish FS (broken line) associated with the image AR1. When the AR scenario including the image AR1 is executed and the apparatus executing the AR scenario detects the fish FS, the scale AR image AR1 is displayed in association with the position of the detected fish FS. . Details of the detection of the fish FS and the display of the image AR1 when the AR scenario is executed by the apparatus will be described in “A-4. Execution of composite scenario”.

  When the AR image is generated (step S34 in FIG. 4), the AR scenario control unit 11 creates a moving image as an AR scenario based on the AR image and the additional information, and the created AR scenario data is stored in the data storage unit. 50 (step S36), and the image processing apparatus 100 ends the AR scenario creation process.

  In the process of step S28, when an operation for manually selecting an unnecessary image is accepted instead of automatic (step S28: NO), the AR scenario control unit 11 is included in the imaging range in the display unit 35. A selection image for selecting each of the moving object and the stationary object is displayed (step S30). The unnecessary image erasing unit 19 erases a moving object or a stationary object selected as an unnecessary object to be erased from the captured images by the RGB camera 31 and the distance sensor 32 based on the operation received by the operation unit 34. . When an object to be manually deleted is selected from the captured image, the user arbitrarily generates an AR image as compared to a case where a stationary object or a human body part is automatically deleted from the captured image. The target to be selected can be selected. For example, in addition to the AR image of the scale removal TL, an AR image of the right hand of the subject OB having the scale removal TL, an AR image of the fish FS, or the like may be generated. In other embodiments, an object for generating an AR image may be selected instead of an object to be erased. When an unnecessary object is erased by the user's operation (step S32), the AR image extraction unit 21 and the AR scenario control unit 11 execute the processes after step S34.

A-3. Composite scenario creation process:
FIG. 10 is a flowchart of the composite scenario creation process. In the composite scenario creation process, the CPU 10 creates a composite scenario in which a plurality of AR scenarios are combined. For example, when a trigger is detected in a certain AR scenario, the AR scenario operation setting unit 17 of the CPU 10 creates a composite scenario that branches to another AR scenario. Examples of the trigger include detection of a specific object within the imaging range and an operation received by the microphone 33 or the operation unit 34.

  In the composite scenario creation process, first, the microphone 33 or the operation unit 34 waits for an operation to start creating a composite scenario (step S42). If the operation for starting the composite scenario creation process is not accepted (step S42: NO), the AR scenario operation setting unit 17 ends the composite scenario creation process (step S42).

  In the process of step S42, when an operation for starting the composite scenario creation process is received (step S42: YES), the AR scenario operation setting unit 17 sets one AR scenario (hereinafter referred to as “basic” as a basis of the composite scenario). A screen for selecting “scenario” is also selected (step S44). The AR scenario operation setting unit 17 displays a plurality of AR scenarios stored in the data storage unit 50 on the display unit 35 and allows the user to operate the operation unit 34 in order to allow the user to select a basic scenario. , One basic scenario is selected from a plurality of AR scenarios. Note that the method for selecting the basic scenario is not limited to this, and various modifications can be made.

  Next, the AR scenario operation setting unit 17 sets a trigger for branching to another scenario combined with the basic scenario (hereinafter also referred to as “branch scenario”) (step S46). The AR scenario operation setting unit 17 sets a trigger for the basic scenario based on operations accepted by the microphone 33 and the operation unit 34. When the AR image extraction unit 21 sets a trigger, the AR image extraction unit 21 displays a preset image on the display unit 35 so that the user can visually recognize that the basic scenario in which the trigger is set is being edited. . In other words, the AR image extraction unit 21 inserts a preset image into the AR scenario when editing the composite scenario. Note that the branch scenario in this embodiment includes a new AR scenario that is executed after all the basic scenarios are completed.

  FIG. 11 is an explanatory diagram showing an edited image KC displayed when a trigger is set. FIG. 11 shows an edited image KC displayed at the time of editing for branching to a branch scenario when the basic scenario is an AR scenario that prompts the fish FS scale to be peeled off. When a trigger is set for the basic scenario, an image AR1 that is an AR image of the scale removal TL is associated with the position of the real fish FS detected in the same manner as when the basic scenario is executed. And the edited image KC are displayed. The edited image KC shows that the basic scenario is an AR scenario of “scale removal”, that “8 minutes and 37 seconds” have passed since the basic scenario started, It is an image showing that there is. The edited image KC can be moved or deleted by an operation received by the operation unit 34. Note that the time of editing corresponds to a specific time point in the claims, and the edited image KC corresponds to a specific image in the claims.

  When the AR scenario operation setting unit 17 sets a trigger (step S46 in FIG. 10), the AR scenario operation setting unit 17 sets a branch scenario to branch when the set trigger is detected when the basic scenario is executed (step S48). . In order to set a branch scenario, the AR scenario operation setting unit 17 displays a plurality of AR scenarios stored in the data storage unit 50 on the display unit 35, and causes the user to operate the operation unit 34, thereby One branch scenario is set from the AR scenario. Note that the method for selecting a branch scenario is not limited to this, and various modifications can be made.

  FIG. 12 is an explanatory diagram showing an image displayed when a branch is made to a branch scenario when a trigger is detected. FIG. 12 shows a text image TX2 and an AR image image AR2 that are displayed when a knife KN set as a trigger is detected from the captured image and the branch scenario is branched. The text image TX2 is a character image for encouraging the fish FS to peel off the scale using the scale removing TL, and is additional information added to be displayed when the branch scenario is branched. The text image TX2 is set so as to be displayed in association with a display range in which an apparatus executing the AR scenario can display an image. The image AR <b> 2 is an AR image representing “×” for indicating that the knife KN is not related to the removal of the scale of the fish FS by being detected as a trigger. The image AR2 is additional information set to be displayed in the branch scenario when the trigger knife KN is detected. The image AR2 is set to be displayed in association with the detected position of the knife KN.

  When the branch scenario is set (step S48 in FIG. 10), the microphone 33 or the operation unit 34 accepts an operation as to whether or not to add another branch scenario to the selected basic scenario (step S48). S50). When an operation for adding another branch scenario is received (step S50: YES), the AR scenario operation setting unit 17 executes the processing after step S46.

  If the operation for adding another branch scenario is not accepted in the process of step S50 (step S50: NO), the AR scenario operation setting unit 17 matches the branch scenario set to the selected basic scenario. The composite scenario is created, the created composite scenario is stored in the data storage unit 50, and the composite scenario creation process is terminated.

A-4. Running a composite scenario:
A composite scenario execution process in which a composite scenario created by the image processing apparatus 100 is executed will be described. In the composite scenario execution process, an apparatus capable of detecting an object in the outside scene and displaying an AR image on the image display unit executes a specific composite scenario based on the specific object detected in the outside scene. is there. An apparatus that can execute a composite scenario includes, for example, an HMD equipped with an imaging camera for detecting an object in the outside scene.

  FIG. 13 is an explanatory diagram showing an external configuration of the head-mounted display device 200 (HMD 200). The HMD 200 is an optically transmissive head-mounted display device that allows a user to visually recognize a virtual image and at the same time directly view an outside scene. HMD200 is provided with the image display part 80 which makes a user visually recognize a virtual image in the state with which the user's head was mounted | worn, and the control part 70 (controller 70) which controls the image display part 80. FIG.

  The image display unit 80 is a wearing body attached to the user's head and has a glasses shape. The image display unit 80 includes a right display drive unit 82, a left display drive unit 84, a right optical image display unit 86, a left optical image display unit 88, a camera 89, a depth sensor 91, and a 9-axis sensor 87. , Including. The right optical image display unit 86 and the left optical image display unit 88 are arranged so as to be positioned in front of the right and left eyes of the user when the user wears the image display unit 80, respectively. The right display drive unit 82 and the left display drive unit 84 are arranged on the side facing the user's head when the user wears the image display unit 80.

  The display driving units 82 and 84 are formed of a liquid crystal display. The optical image display units 86 and 88 as optical members include a light guide plate and a light control plate. The light guide plate is formed of a light transmissive resin material or the like, and guides the image light output from the display drive units 82 and 84 to the user's eyes. The light control plate is a thin plate-like optical element, and is disposed so as to cover the front side of the image display unit 80 which is the side opposite to the user's eye side.

  The camera 89 is disposed at a position corresponding to the user's eyebrow when the user wears the image display unit 80. Therefore, the camera 89 captures an outside scene that is an external scenery in the direction of the user's line of sight in a state where the user wears the image display unit 20 on the head, and acquires a captured image that is the captured image. The depth sensor 91 is a distance sensor that measures the distance to an object included in the imaging range.

  The 9-axis sensor 87 is disposed at a position corresponding to the temple on the right side of the user. The 9-axis sensor 87 is a motion sensor that detects acceleration (3 axes), angular velocity (3 axes), and geomagnetism (3 axes). Since the 9-axis sensor 87 is provided in the image display unit 80, when the image display unit 80 is attached to the user's head, the 9-axis sensor 87 serves as a motion detection unit that detects the movement of the user's head of the HMD 200. Function. Here, the movement of the head includes changes in the speed, acceleration, angular velocity, direction, and direction of the head.

  The image display unit 80 further includes a connection unit 85 for connecting the image display unit 80 to the control unit 70. A part of the connecting portion 85 extends to the right earphone 81 and the left earphone 83. For example, a metal cable or an optical fiber can be used as the cord constituting the connecting portion. The image display unit 80 and the control unit 70 transmit various signals via the connection unit 85.

  The control unit 70 is a device for controlling the HMD 200. The control unit 70 is an operation unit configured by a plurality of keys, a track pad, and the like. The plurality of keys of the control unit 70 detects a pressing operation and transmits a control signal corresponding to the pressed key to the image display unit 80. The track pad of the control unit 70 detects the operation of the user's finger on the operation surface of the track pad, and outputs a signal corresponding to the detected content.

  The control unit 70 has a CPU 75 (not shown) that controls the image display unit 80. The CPU 75 executes a composite scenario stored in the data storage unit 50 received via wireless communication or the like. When the control unit 70 receives a predetermined key operation, the CPU 75 selects an object associated with an AR image included in the basic scenario in the composite scenario (hereinafter, “corresponding object”) from the captured images captured by the camera 89. Also called). The CPU 75 displays the AR image included in the basic scenario on the optical image display units 86 and 88 of the image display unit 80 in association with the detected position of the corresponding object. Further, the CPU 75 detects an image of a target (hereinafter also referred to as “trigger target”) as a trigger for branching from the basic scenario to the branch scenario from the captured images captured by the camera 89. When the CPU 75 detects the trigger target image from the captured images of the camera 89, the CPU 75 branches from the basic scenario to the branch scenario, and displays the AR image based on the branch scenario on the optical image display units 86 and 88.

  FIG. 14 is a flowchart of the composite scenario execution process. In the composite scenario execution process, first, it is determined whether or not the control unit 70 of the HMD 200 mounted on the user's head has received an operation for executing the composite scenario process (step S61). When the control unit 70 does not accept the operation for executing the composite scenario (step S61: NO), the HMD 200 ends the composite scenario execution process.

  In the process of step S61, when the control unit 70 receives an operation for executing a composite scenario (step S61: YES), the control unit 70 selects an image for causing the user to select the composite scenario to be executed. The image is displayed on the image display units 86 and 88 (step S63). The user can select one composite scenario to be executed by visually recognizing the images displayed on the optical image display units 86 and 88 and operating the keys of the control unit 70. When the composite scenario is selected, the CPU 75 of the control unit 70 detects a corresponding object associated with the AR image included in the basic scenario in the selected composite scenario from the captured images captured by the camera 89. (Step S65). The CPU 75 detects the corresponding object of the AR image from the captured image using pattern matching or statistical identification. Further, the CPU 75 measures the distance to the corresponding object measured by the depth sensor 91. In the present embodiment, one composite scenario is selected. However, in another embodiment, a plurality of composite scenarios may be selected to determine a composite scenario to be executed by the detected corresponding object. The number of composite scenarios to be executed and the method for selecting composite scenarios can be variously modified.

  When the CPU 75 detects the corresponding object from the captured image, the CPU 75 identifies the position of the corresponding object in the captured image and displays the AR image and additional information included in the composite scenario in association with the identified corresponding object position. (Step S67). In the HMD 200, since the imaging range of the camera 89 and the positions of the pixels displayed on the optical image display units 86 and 88 visually recognized by the user are set in advance, the CPU 75 associates the AR with the corresponding object. When the image is displayed on the optical image display units 86 and 88, the user can visually recognize the AR image associated with the position of the actual corresponding object. The CPU 75 can also stereoscopically display the AR image in correspondence with the distance to the corresponding object measured by the depth sensor 91 (the stereoscopic display here refers to the interval between two AR images for the left and right eyes). Display with parallax). Note that the CPU 75 does not display the AR image included in the composite scenario when no corresponding object is detected. Further, for example, the additional information includes not only an image displayed on the optical image display units 86 and 88 such as a text image but also a sound output via the earphones 81 and 83.

  FIG. 15 is an explanatory diagram showing a visual field VR visually recognized by the user when a corresponding object set in the composite scenario to be executed is detected. As shown in FIG. 15, the visual field VR visually recognized by the user is displayed on the external scene SC transmitted through the optical image display units 86 and 88 of the image display unit 80 mounted on the head and on the optical image display units 86 and 88. An image AR1 of the scale removal TL and a text image TX1 are included. The outside scene SC includes a fish FS placed on a cutting board. The image AR1 and the text image TX1 of the scale removal TL are displayed on the optical image display units 86 and 88 in association with the position of the fish FS detected from the captured image by the CPU 75. The image AR1 is not a still image but a moving image that reciprocates from the head to the tail of the fish FS.

  When the CPU 75 displays the AR image included in the composite scenario on the optical image display units 86 and 88 (step S67 in FIG. 14), the CPU 75 displays a trigger target image for branching from the captured image of the camera 89 to the branch scenario. The detection is monitored (step S69). When detecting the trigger target image from the captured image (step S69: YES), the CPU 75 branches to the branch scenario associated with the detected trigger target and executes it (step S73). When the CPU 75 branches the AR scenario to be executed into the branch scenario, the CPU 75 detects a corresponding object associated with the AR image included in the branch scenario (step S65). The CPU 75 specifies the position of the detected corresponding object, and displays the AR image included in the branch scenario in association with the specified position of the corresponding object (step S67). Note that the corresponding object associated with the AR image included in the branch scenario and the trigger target may be the same object or different objects.

  FIG. 16 is an explanatory diagram showing a visual field VR visually recognized by the user when a trigger target set in the executed branch scenario is detected. As shown in FIG. 16, the visual field VR visually recognized by the user is displayed on the external scene SC transmitted through the optical image display units 86 and 88 of the image display unit 80 mounted on the head and on the optical image display units 86 and 88. The image AR2 and the text image TX2 representing “x” are included. The outside scene SC includes a fish FS placed on a cutting board and a knife KN held by the user on the right hand. The text image TX2 is displayed on the optical image display units 86 and 88 in association with the position of the fish FS detected from the captured image by the CPU 75. The image AR2 is displayed on the optical image display units 86 and 88 in association with the position of the knife KN detected from the captured image.

  In the process of step S69 in FIG. 14, when the CPU 75 does not detect the trigger target image from the captured images (step S69: NO), the AR scenario being executed in the composite scenario (hereinafter, referred to as “NO”) is described. It is determined whether or not the “execution scenario” is terminated (step S71). The CPU 75 ends the execution scenario when the control unit 70 receives an operation for ending the execution scenario or when it is determined that it is not necessary to display the execution scenario. As an example in which the CPU 75 determines that it is not necessary to display the execution scenario, when the AR scenario that prompts the fish FS to be removed is being executed, the scale of the fish FS is placed on the surface of the fish FS in the imaging range of the camera 89. There is a case where it is determined that does not exist.

  In the process of step S71, when the CPU 75 determines not to end the execution scenario (step S71: NO), the AR image and the additional information included in the execution scenario are continuously displayed on the optical image display units 86 and 88. Display. If it is determined in step S71 that the execution scenario is to be ended (step S71: YES), the HMD 200 ends the combined scenario execution process.

  As described above, in the image processing apparatus 100 according to the present embodiment, the AR image extraction unit 21 is an AR image of a moving object among individual objects recognized by the object recognition unit 13 and the 3D model generation unit 14. Is generated. Therefore, in the image processing apparatus 100 of the present embodiment, an AR image corresponding to the identified object is generated simply by identifying the moving object, so that the user can easily perform an AR scenario such as work support, for example. The user-friendliness is improved.

  Further, in the image processing apparatus 100 of the present embodiment, the AR image extraction unit 21 generates an AR image by associating the moving area to be generated with the AR image with the object deleted by the unnecessary image deletion unit 19. Therefore, in the image processing apparatus 100 according to the present embodiment, when an AR scenario including the generated AR image is executed, the AR image is the position and size of an actually existing object associated with the AR image. Is displayed in association with. For example, when the AR scenario is a support moving image such as work, the workability of the user is further improved by superimposing the AR image on the work target, and the user's usability is improved.

  In the image processing apparatus 100 according to the present embodiment, the AR image extraction unit 21 associates the size of the target to be generated with the AR image to be generated with the size of the target to generate the AR image and the moving area, and converts the AR image into the AR image. Generate. Therefore, in the image processing apparatus 100 according to the present embodiment, when an AR scenario including the generated AR image is executed, the AR image is the position and size of an actually existing object associated with the AR image. Are displayed in association with each other, and the usability of the user is further improved.

  As described above, in the image processing apparatus 100 of this embodiment, the distance sensor 32 measures the distance to the target surface, and the object recognition unit 13 uses the 3D model generated by the 3D model generation unit 14. It is used to recognize each object included in the captured image. The AR image extraction unit 21 generates an AR image of a moving moving object other than the stationary object erased by the unnecessary image erasing unit 19. Therefore, in the image processing apparatus 100 according to the present embodiment, an AR scenario including an AR image of a moving object whose distance is measured can be created simply by capturing a predetermined range, so that the user can easily create an AR scenario. This improves user convenience.

  Further, in the image processing apparatus 100 of the present embodiment, the unnecessary image erasing unit 19 does not generate an AR image for a moving object or a stationary object selected from the captured image based on the operation received by the operation unit 34. Erase as target. That is, the unnecessary image erasing unit 19 selects the target of the generated AR image. Therefore, in the image processing apparatus 100 of this embodiment, a moving object that does not need to be generated or a stationary object that needs to be generated is selected as an AR image, and an AR scenario or a composite scenario that is easier for the user to use is created. This improves user convenience.

  Further, in the image processing apparatus 100 of the present embodiment, the unnecessary image erasure unit 19 erases a stationary object other than the moving object included in the captured image as a target that is not generated as an AR image, and the AR image extraction unit 21 performs an unnecessary image. An AR image of the moving object included in the captured image that has not been erased by the erasure unit 19 is generated. For this reason, the image processing apparatus 100 according to the present embodiment automatically generates an AR image of a moving object without performing an operation of selecting a target for generating an AR image. Therefore, for example, when the AR scenario is a support moving image such as work and it is necessary to move some target in the work, an AR image of the moving object that is the object to be moved is automatically generated, and the user's convenience Will improve.

  Further, in the image processing apparatus 100 of the present embodiment, the unnecessary image erasing unit 19 erases human body parts extracted by the object recognition unit 13 from the captured image, and the AR image extracting unit 21 captures the captured image. The human body part erased from the inside is not generated as an AR image. Therefore, in the image processing apparatus 100 according to the present embodiment, when the AR scenario is executed, the user's hand, which is a means for moving the object to be moved, is not displayed. Therefore, it is not necessary to visually recognize a new AR image, and convenience for the user is improved.

  In the image processing apparatus 100 of the present embodiment, the AR image extraction unit 21 associates the sound acquired by the microphone 33 while the moving object being imaged is moving, and targets the moving object. An AR image is generated. Therefore, in the image processing apparatus 100 according to the present embodiment, the created AR scenario supports not only visual information such as an AR image generated based on a captured image but also auditory information such as sound. Since the attached AR image is included, the convenience for the user is improved.

  Further, in the image processing apparatus 100 of the present embodiment, the voice acquired by the microphone 33 is used as a text image, and an AR image is generated in association with a moving object. Therefore, the voice is used as visual information in parallel with the AR image. It can be generated, and the user can easily recognize the information, and the convenience for the user is further improved.

  In the image processing apparatus 100 of the present embodiment, the AR image extraction unit 21 colors the generated AR image using the RGB data of the imaging range acquired by the RGB camera 31. Therefore, in the image processing apparatus 100 according to the present embodiment, the generated AR image is more similar to the target included in the captured image as compared with the AR image that is not colored, and thus the user can view the AR image. It is easier to recognize and user convenience is improved.

  Further, in the image processing apparatus 100 of the present embodiment, when editing a composite scenario such as a trigger setting for branching to a branch scenario, the AR image extraction unit 21 displays an edited image KC indicating an editing state as an AR scenario. Insert into. Therefore, in the image processing apparatus 100 according to the present embodiment, when an AR scenario is edited, an image for editing the image for allowing the user to recognize that the editing is performed as visual information is edited. Since it is inserted into the scenario, the user convenience is improved.

  In the image processing apparatus 100 according to the present embodiment, since the 3D model is generated as an AR image so that it can be viewed from any direction of 360 degrees, when the AR scenario is executed, the user can , The 3D model can be confirmed from any direction, improving user convenience.

B. Second embodiment:
FIG. 17 is a block diagram functionally showing the configuration of the image processing apparatus 100a in the second embodiment. In the second embodiment, the RGBD data transmitted from the sensor control unit 15a is output as streaming data, so that the 3D model generation unit 14a, the object recognition unit 13a, and the object tracking unit 12a are included in the imaging range. Is different from the first embodiment in that all the three-dimensional models are generated and the generated three-dimensional model is transmitted as streaming data to the AR scenario control unit 11a.

  FIG. 18 is a flowchart of a part of the AR scenario creation process in the second embodiment. In the AR scenario creation process of the second embodiment, while the sensor control unit 15a captures an outside scene and outputs RGBD data for each pixel as streaming data, a desired three-dimensional model is output as streaming data. The process of step S24a of FIG. 18 is different from the process of step S24 of the AR creation scenario process (FIG. 3) of the first embodiment. Therefore, in the second embodiment, step S24a in FIG. 18 will be described, and description of other processing will be omitted. In the process of step S24a in FIG. 18, the 3D model generation unit 14a includes all the objects of the real object and the real environment existing in the imaging range based on the streaming data of the RGBD data for each pixel from the sensor control unit 15a. A three-dimensional model (hereinafter simply referred to as an “entire three-dimensional model”) is generated. In the present embodiment, specifically, the 3D model generation unit 14a generates each three-dimensional model from each viewpoint from the cameras 331, 332, and 332, and fuses these three-dimensional models into one. The whole three-dimensional model that does not depend on the viewpoint from the cameras 331, 332, and 333 is obtained. In the present embodiment, the entire three-dimensional model is represented by data of a polygon mesh (for example, a rendered texture triangle mesh). Then, the 3D model generation unit 14a outputs the data of the entire three-dimensional model as streaming data. Hereinafter, outputting as streaming data is also simply referred to as streaming output.

  Next, the object recognizing unit 13a is a three-dimensional model that is an individual element included in all three-dimensional models based on RGB data streaming data among the RGBD data for each pixel transmitted from the sensor control unit 15a. (Hereinafter also simply referred to as “element three-dimensional model”). In the second embodiment, as a method of distinguishing a three-dimensional model, for example, the object recognition unit 13a distinguishes a real object by detecting an edge in RGB data, and the distinguished real object is represented by RGB data. The area occupying the space is associated with the area in the space of the entire three-dimensional model. If it does so, the part (element 3D model) contained in the said area | region among the whole three-dimensional model will be distinguished from another part. The object recognition unit 13a corrects the elemental 3D model included in the overall 3D model according to the result of the 3D model discrimination.

  The object tracking unit 12a performs image processing on the streaming data of the RGB data transmitted from the sensor control unit 15a, thereby identifying a moving real object (moving object) and a stationary real object. . The object tracking unit 12a tracks the identified moving object in an image space represented by RGB data.

  The AR scenario control unit 11a performs streaming output of the element 3D model corresponding to the moving object tracked by the object tracking unit 12a among the element 3D models in the overall 3D model specified by the object recognition unit 13a. The element three-dimensional model output by streaming from the object tracking unit 12 includes not only the movement of the three-dimensional model but also a change in posture including the direction (for example, rotation). Note that the elemental 3D model output by streaming is represented by polygon mesh data in this embodiment.

  The AR image extraction unit 21 deletes an unnecessary part from the element three-dimensional model when the element three-dimensional model stream-output by the AR scenario control unit 11a includes an unnecessary part. The unnecessary part determined by the AR image extraction unit 21 covers one element three-dimensional model of a plurality of element three-dimensional models or a part of the element three-dimensional model, for example, a target real object. There is a part of an element three-dimensional model that represents a part of a human body (for example, a hand). The AR image extraction unit 21 performs streaming output of the element 3D model remaining after deleting unnecessary portions from the element 3D model.

  The data storage unit 50 records the element three-dimensional model stream-output by the AR image extraction unit 21 as the extracted three-dimensional model. Thereafter, the AR scenario control unit 11a creates an AR scenario using the recorded extracted three-dimensional model. The AR image included in the AR scenario may be an image representing the extracted three-dimensional model, or may be an image obtained by correcting the appearance of the extracted three-dimensional model. The extracted three-dimensional model may be a three-dimensional model (for example, a three-dimensional model of a scale removal TL as shown in FIG. 5) captured at the start of imaging in the AR scenario creation process, or CAD data It may be replaced with other three-dimensional models such as In this case, even after deleting unnecessary portions from the element three-dimensional model, for example, an AR image that does not lack a portion hidden by a hand or the like is realized. The AR image of the present embodiment is represented by stream data of a three-dimensional model (for example, polygon mesh stream data). Therefore, it is possible to arbitrarily change the viewpoint of the AR image and change the direction of the displayed AR image at an arbitrary time point within the time period defined between the start point and the end point of the stream data. Note that the viewpoint from any one of the cameras 331 to 333 may be included in the AR scenario as default viewpoint information by the AR scenario control unit 11a.

  The created AR scenario may include an AR image including continuous motion that is captured while the moving object that is streamed out is actually moving. Note that the length of time during which the AR image continuously moves may be the same as or different from the time during which the captured moving object that is the basis of the AR image is actually moving. Further, the AR scenario may be configured by an AR image of discrete motion instead of an AR image of continuous motion. As an AR image of discrete motion, for example, in an AR scenario, a state of at least a temporary point from a state at the time when a captured moving object starts moving to a state at the time when movement is completed may be generated as an AR image. In addition, the AR image may be generated in which a time point at which the movement starts, a time point at which the movement starts, and a point in time from the time point when the movement starts to the time point at which the movement ends.

C. Third embodiment:
FIG. 19 is a block diagram functionally showing the configuration of the image processing apparatus 100b according to the third embodiment. FIG. 20 is a flowchart illustrating a part of the AR scenario creation process according to the third embodiment. The third embodiment is different from the configuration of the second embodiment in that the CPU 10a of the image processing apparatus 100a does not have the object recognition unit 13a, and the other configurations are the same. As shown in FIG. 19, in the third embodiment, since there is no object recognition unit 13a, the process of step S24b of the AR creation process is different from the process of step S24a of the AR creation process (FIG. 18) of the second embodiment. In the third embodiment, differences from the second embodiment will be described, and descriptions of the same points will be omitted.

  In the process of step S24b in FIG. 19, the object tracking unit 12b receives the entire three-dimensional model output by streaming from the 3D model generation unit 14b. Then, the object tracking unit 12b identifies (distinguishes) a moving three-dimensional model (element three-dimensional model) and a non-moving three-dimensional model in the entire three-dimensional model. Then, the object tracking 12b performs streaming output of the identified (differentiated) element three-dimensional model.

D. Fourth embodiment:
In the fourth embodiment, a plurality of AR scenarios in which an AR image of a moving object and an AR image of a related stationary object that is determined to be related to the moving object and are stationary are automatically obtained from the captured image data. It is mainly different from the first embodiment and the second embodiment. In the fourth embodiment, the AR scenario control unit 11a sets each of the moving object and the related stationary object as trigger targets for executing the AR scenario. The AR scenario control unit 11a has the following combinations (1) to (4) as predetermined combinations of a moving object as a trigger target and a related stationary object that are detected when the AR scenario is executed. Create an AR scenario.
(1) When only a moving object as a trigger target is detected (2) When only a related stationary object as a trigger target is detected (3) When both a moving object and a related stationary object as a trigger target are detected ( 4) When neither a moving object as a trigger object nor a related stationary object is detected Note that in the fourth embodiment, an AR scenario corresponding to each of the detection of the conditions (1) to (4) is created. In other embodiments, AR scenarios corresponding to three or less of the four cases (1) to (4) may be created.

  21 and 22 are flowcharts of the AR scenario creation process in the fourth embodiment. The AR scenario creation process of the fourth embodiment differs from the AR scenario creation process of the second embodiment in the following two points. One is that an AR scenario can be created based on already captured image data, and the other is that a plurality of AR scenarios in the cases (1) to (4) are automatically created. It is. The other processes in the fourth embodiment are the same as the AR scenario creation process in the second embodiment. Therefore, in the AR scenario creation process of the fourth embodiment, the description of the processes from step S14 to step S22 that are the same as those of the second embodiment and the third embodiment is omitted.

  In the AR scenario creation process of the fourth embodiment, first, the microphone 33 or the operation unit 34 receives a predetermined operation for starting creation of an AR scenario (step S81). When the microphone 33 or the operation unit 34 receives a predetermined operation for starting creation of an AR scenario (step S81: YES), the microphone 33 or the operation unit 34 receives a predetermined operation as to whether or not the AR scenario to be created is created based on imaging data ( Step S83). When a predetermined operation that does not create an AR scenario based on the imaging data is received (step S83: NO), the AR scenario control unit 11a performs the same processing from step S14 to step S22 as in the second embodiment. In the process of step S83, when a predetermined operation for creating an AR scenario based on the imaging data is received (step S83: YES), the 3D model generation unit 14a converts the entire three-dimensional model into the same manner as in the second embodiment. Generate (step S24a in FIG. 22). Specifically, the 3D model generation unit 14a generates an entire three-dimensional model based on streaming data of RGDB data for each pixel from the sensor control unit 15a.

  FIG. 23 to FIG. 27 are explanatory diagrams when an external scene SC including a plurality of subjects is imaged by the RGB camera 31 and the distance sensor 32 in the fourth embodiment. FIG. 23 shows an outside scene SC including a housing BX, a cover CV, a tool driver DV, and four bolts BT. 23 to FIG. 26 show changes until the separate housing BX and the cover CV are attached as separate components by the driver DV and the bolt BT. The housing BX is formed with four female screw portions Bh for fitting the male screw portions of the bolt BT. Further, in order to fix the cover CV to the housing BX, when the housing BX and the cover CV are combined, the cover CV has a circular shape at a position corresponding to the female thread Bh of the housing BX. A hole Ch is formed.

  FIG. 24 shows the outside scene SC in which the operator's left hand LH holds the cover CV at a position fixed to the casing BX that has not moved, as compared to the outside scene SC shown in FIG. ing. In the state shown in FIG. 24, the housing BX and the cover CV are not fixed by the bolt BT, and the position of the cover CV is temporarily fixed by the left hand LH. In the state where the outside scene SC shown in FIG. 23 changes to the outside scene SC shown in FIG. 24, the cover CV is a moving object because it moves. Further, the housing BX is a related stationary object because it is a stationary object that is in contact with the cover CV of the moving object.

  FIG. 25 shows the outside scene SC in which the operator's right hand RH grips the driver DV and one bolt BT is attached to the tip of the driver DV as compared to the outside scene SC shown in FIG. In FIG. 25, the positions of the housing BX, the cover CV, and the left hand LH are not changed. In the state changed from the outside scene SC shown in FIG. 24 to the outside scene SC shown in FIG. 25, the driver DV and the bolt BT attached to the tip of the driver DV are moving objects. Note that the driver DV and the bolt BT can be regarded as one moving object when moving together. The three bolts BT that are not attached to the driver DV are stationary objects that are not related to the moving object. Although not shown, in a state before the driver DV and the bolt BT are integrated, the driver DV or the bolt BT is a moving object, and the other is a related stationary object.

  In FIG. 26, as compared with the outside scene SC shown in FIG. 25, one bolt BT attached to the tip of the driver DV held by the right hand RH is inserted into one female thread Bh of the housing BX and rotated. The outside scene SC is shown. Note that the housing BX and the cover CV are not moved in a state where the outside scene SC shown in FIG. 25 changes to the outside scene SC shown in FIG. Therefore, the driver DV and the bolt BT are moving objects, and the housing BX and the cover CV are related stationary objects.

  In FIG. 27, as compared with the outside scene SC shown in FIG. 26, one bolt BT is fitted to one female thread Bh of the housing BX, so that the housing BX and the cover CV are fixed. The outside scene SC with the DV tip separated from the bolt BT is shown. In a state where the outside scene SC shown in FIG. 26 changes to the outside scene SC shown in FIG. 27, the driver DV is a moving object, and the housing BX, the cover CV, and the bolt BT are related stationary objects. In other embodiments, the driver DV may be a moving object, and the housing BX, the cover CV, and the bolt BT may be treated as being independent of the driver DV and not an associated stationary object.

  In step S24a of FIG. 22, 3D models of all targets including moving objects and related stationary objects are created from captured images of changes in the outside scene SC shown in FIGS. Thereafter, the AR scenario control unit 11a selects one moving object from among the one or more moving objects specified by the object tracking unit 12a from the generated entire three-dimensional model (step S85). The AR scenario control unit 11a selects, for example, the cover CV as a moving object in the change of the outside scene SC from FIG. 23 to FIG. 24 in the change of the outside scene SC from FIG. 23 to FIG.

  The AR image extraction unit 21 generates a cover image IMC described later as an AR image of the cover CV as the selected moving object (step S87). Thereafter, the AR scenario control unit 11a determines whether or not there is an associated stationary object that is identified as being in contact with the cover CV as the selected moving object based on the measurement distance by the object recognition unit 13a (step S89). ). The AR scenario control unit 11a identifies a stationary object that exists within a predetermined distance from the moving object as an associated stationary object that is in contact with the moving object. In the change of the outside scene SC from FIG. 23 to FIG. 24, the AR scenario control unit 11a specifies the housing BX as the related stationary object of the cover CV as the moving object. Therefore, it is determined that there is a related stationary object for the cover CV (step S89: YES), and the AR image extraction unit 21 generates AR images of all the related stationary objects (step S91). The AR image extraction unit 21 generates a case image IMX, which will be described later, as an AR image of the case BX that is a related stationary object.

  Thereafter, the unnecessary image erasing unit 19 erases other moving objects and stationary objects excluding the selected moving object and related stationary objects as unnecessary objects (step S93). In the change of the outside scene SC from FIG. 23 to FIG. 24, the unnecessary image erasing unit 19 erases the cover CV, the bolt BT, and the left hand LH as unnecessary objects. Thereafter, the AR scenario control unit 11a sets each of the moving object cover CV and the related stationary object casing BX on which the AR image is generated as a trigger target for executing the AR scenario. The AR scenario control unit 11a creates an AR scenario corresponding to the combination of the presence / absence of a moving object as a trigger target and the presence / absence of a related stationary object (step S95). Details of the AR scenario created corresponding to the combination of trigger targets will be described later.

  When creating each AR scenario corresponding to a predetermined combination, the AR scenario control unit 11a determines whether or not an AR scenario has been created by selecting all moving objects included in the imaging data (step S97). Since the AR scenario control unit 11a selects only the cover CV as the moving object and does not select all the moving objects (step S97: NO), the AR scenario control unit 11a performs the processing after step S85. In the process of step S97, if the AR scenario control unit 11a determines that all moving objects in the imaging data other than the cover CV have been selected (step S97: YES), the AR scenario control unit 11a stores all the created AR scenarios in the data storage unit. 50, and the AR scenario creation process is terminated.

  FIG. 28 is a list showing an example of combinations of trigger targets and AR scenarios created in the process of step S95 of the AR scenario creation process of the fourth embodiment. FIG. 28 shows display images that are displayed in correspondence with the predetermined combinations (1) to (4) to be detected when the created AR scenario is executed. FIG. 28 shows display images corresponding to combinations (1) to (4) when the moving object is the cover CV and the related stationary object is the housing BX. In the following, a display image that is displayed in association with the detected trigger target when the AR scenario is being executed will be described.

  FIG. 29 is a flowchart of display image determination processing when an AR scenario is being executed. In the display image determination process, when an AR scenario is being executed, the HMD 200 as a device that is executing the AR scenario is displayed on the optical image display units 86 and 88 in accordance with the detected combination of trigger targets. This is a process for determining an image.

  In the display image determination process, first, the camera 89 of the HMD 200 captures an outside scene (step S101). The CPU 75 of the HMD 200 determines whether or not the trigger target of the combination (1) shown in FIG. 28 is detected in the captured image of the camera 89 (step S103). When the CPU 75 determines that the trigger target of the combination (1) has been detected (step S103: YES), the CPU 75 displays the display image associated with the combination (1) as an image to be displayed on the optical image display units 86 and 88. Determine (step S111). Thereafter, the CPU 75 ends the display image determination process.

  FIG. 30 is an explanatory diagram illustrating an example of the visual field VR visually recognized by the user when the display image associated with the combination (1) is displayed on the optical image display units 86 and 88. FIG. 30 shows an example of a visual field VR visually recognized by a user who wears the image display unit 80 of the HMD 200 on the head when a cover CV that is a moving object as a trigger target is detected. As shown in FIG. 30, in addition to the actual cover CV, driver DV, and bolt BT included in the outside scene SC, the user visually recognizes the housing image IMX displayed as an image. In other words, as in the combination (1) shown in FIG. 28, the AR scenario control unit 11a executes the AR scenario, and when only the cover CV that is a moving object as a trigger target is detected, the related stationary object An AR scenario in which only the housing image IMX is set as a display image is created. In FIG. 30, an actual driver DV and a plurality of bolts BT are detected. However, in the combination shown in FIG. 28, the actual driver DV and the bolt BT are irrelevant to the presence or absence of a display image. 31 to 33 described later, the actual driver DV and the bolt BT are irrelevant to the presence or absence of the display image.

  In the process of step S103 of FIG. 29, when the CPU 75 determines that the trigger target of the combination (1) has not been detected (step S103: NO), the trigger target of the combination (2) shown in FIG. 28 is detected. It is determined whether or not (step S105). When the CPU 75 determines that the trigger target of the combination (2) has been detected (step S105: YES), the CPU 75 displays the display image associated with the combination (2) on the optical image display units 86 and 88. Determine (step S113). Thereafter, the CPU 75 ends the display image determination process.

  FIG. 31 is an explanatory diagram illustrating an example of the visual field VR visually recognized by the user when the display image associated with the combination (2) is displayed on the optical image display units 86 and 88. FIG. 31 shows an example of a VR visually recognized by the user of the HMD 200 when a casing BX that is a related stationary object as a trigger target is detected. As illustrated in FIG. 31, the CPU 75 executes the AR scenario, and when only the casing BX that is the related object as the trigger target is detected, the image illustrated in the combination (2) in FIG. 28 is an optical image. It is displayed on the display units 86 and 88. The CPU 75 displays the cover image IMC of the cover CV, which is a moving object, as the display image at the position where the cover CV was last stopped in the imaging data when the AR scenario was created, that is, the position where the cover CV was mounted on the housing BX. To display. In other words, the AR scenario control unit 11a creates an AR scenario that displays a cover image IMC that is an AR image of the cover CV at a position integrated with the casing BX that is a related stationary object.

  In the process of step S105 in FIG. 29, when the CPU 75 determines that the trigger target of the combination (2) is not detected (step S105: NO), the CPU 75 detects the trigger target of the combination (3) shown in FIG. It is determined whether or not (step S107). When the CPU 75 determines that the trigger target of the combination (3) has been detected (step S107: YES), the CPU 75 displays the display image associated with the combination (3) as an image to be displayed on the optical image display units 86 and 88. Determine (step S115). Thereafter, the CPU 75 ends the display image determination process.

  FIG. 32 is an explanatory diagram illustrating an example of the visual field VR visually recognized by the user when the display image associated with the combination (3) is displayed on the optical image display units 86 and 88. FIG. 32 shows an example of a visual field VR visually recognized by the user of the HMD 200 when a cover CV that is a moving object as a trigger target and a housing BX that is a related stationary object are detected. As shown in FIG. 32, the CPU 75 executes the AR scenario and detects the image shown in the combination (3) in FIG. 28 when the casing BX and the cover CV as trigger targets are detected. 86,88. The CPU 75 causes the optical image display units 86 and 88 to display the cover image IMC of the cover CV as a display image at a position integrated with the housing BX that is a related stationary object, as with the cover image IMC shown in FIG. . In other words, the AR scenario control unit 11a creates an AR scenario that displays a cover image IMC that is an AR image of the cover CV at a position integrated with the casing BX that is a related stationary object.

  In the process of step S107 of FIG. 29, when the CPU 75 determines that the trigger target of the combination (3) has not been detected (step S107: NO), the CPU 75 detects the trigger target of the combination (4) shown in FIG. It is determined whether or not (step S109). When the CPU 75 determines that the trigger target of the combination (4) has been detected (step S109: YES), the CPU 75 displays the display image associated with the combination (4) as an image to be displayed on the optical image display units 86 and 88. Determination is made (step S117). Thereafter, the CPU 75 ends the display image determination process.

  FIG. 33 is an explanatory diagram illustrating an example of the visual field VR visually recognized by the user when the display image associated with the combination (4) is displayed on the optical image display units 86 and 88. FIG. 33 shows a visual field VR visually recognized by the user of the HMD 200 when the cover CV that is a moving object as a trigger target and the casing BX that is a related stationary object are not detected. As shown in FIG. 33, the CPU 75 executes the AR scenario and displays the image shown in the combination (4) in FIG. 28 on the optical image display units 86 and 88 when no trigger target is detected. . The CPU 75 displays, as display images, a cover image IMC representing a state in which the cover CV that is a moving object and the case BX that is a related stationary object are integrated, and the case image IMX on the optical image display units 86 and 88. Display. In other words, the AR scenario control unit 11a displays the case image IMX and the cover image IMC as an AR image in which the case BX and the cover CV are integrated when no trigger target is detected. Create a scenario.

  In the process of step S109 of FIG. 29, when the CPU 75 determines that the trigger of the combination (4) has not been detected (step S109: NO), any AR image is displayed on the optical image display units 86 and 88. The display image determination process is terminated without displaying. In other embodiments, after the display related process of the fourth embodiment, a display related process using different moving objects and related stationary objects as trigger targets may be performed. As described above, the AR scenario control unit 11a of the fourth embodiment automatically creates an AR scenario including an AR image corresponding to each combination of trigger targets, as illustrated in FIG. Note that the cover CV, which is a moving object, corresponds to the moving object in the claims, and the housing BX, which is the related stationary object, corresponds to the related stationary object in the claims. Further, the cover image IMC of the cover CV corresponds to the movement target corresponding image in the claims, and the casing image IMX of the casing BX that is the stationary related object corresponds to the related still target image in the claims.

  As described above, in the image processing apparatus 100a of the fourth embodiment, the AR scenario control unit 11a sets the cover CV and the housing BX as trigger targets for executing the AR scenario. Therefore, in the image processing apparatus 100a according to the fourth embodiment, the AR scenario is created so as to be executed by detecting a preset specific condition, so that the execution timing can be set according to the use of the AR scenario.

  In the image processing apparatus 100a of the fourth embodiment, the AR scenario control unit 11a combines the presence / absence of the cover CV of the moving object and the presence / absence of the casing BX of the related stationary object detected when the AR scenario is executed. An AR scenario including the presence / absence of display of the AR image of the cover image IMC and the housing image IMX corresponding to is created. Therefore, in the image processing apparatus 100a of the fourth embodiment, a plurality of AR scenarios including an AR image related to the moving object are created based on the imaging data without any particular operation, and the image processing apparatus 100a. Improved usability.

E. Variations:
In addition, this invention is not limited to the said embodiment, It can implement in a various aspect in the range which does not deviate from the summary, For example, the following deformation | transformation is also possible.

E-1. Modification 1:
In the first embodiment, a composite scenario composed of a basic scenario that prompts the fish FS to take the scale and a branch scenario that branches when the kitchen knife KN is detected from the basic scenario has been described. However, the present invention is not limited to this and can be variously modified. For example, it is not necessary to be a composite scenario composed of two AR scenarios, and may be a single AR scenario or a composite scenario in which three or more AR scenarios are combined. Further, unlike the basic scenario and the branch scenario, the AR scenario is not distinguished between the upper and lower scenarios, and a composite scenario may be created by a plurality of AR scenarios handled in parallel.

  In the above embodiment, the generation of the AR image and the like is performed based on the data acquired by the RGB camera 31 and the microphone 33. However, these devices are not necessarily indispensable configurations, and these devices are variously modified. Is possible. For example, the image processing apparatus 100 does not need to include the RGB camera 31, the microphone 33, the operation unit 34, and the display unit 35, but only creates an AR image of a moving object in the imaging range that has been automatically imaged. Scenarios may be created. Further, the CPU 10 does not need to include the additional information acquisition unit 18, and may simply create an AR scenario based on image data of only the captured image.

  In the first embodiment, the AR scenario operation setting unit 17 sets a trigger for branching from a basic scenario to a branch scenario when creating a composite scenario composed of a basic scenario and a branch scenario. Although an image such as the edited image KC is displayed, it is not always necessary to display such an image. Instead of displaying the edited image KC image during editing, the AR scenario operation setting unit 17 displays the image by changing the color of the AR image or the like so that the user can recognize that the trigger has been set. Good. Further, the AR scenario operation setting unit 17 may cause the user to recognize that the trigger is set by outputting sound instead of displaying the image of the edited image KC. In this modification, since the user can recognize the state of editing or the like by voice, the edited image KC is not overlapped with the AR image as compared with the case where an image such as the edited image KC is displayed. The user-friendliness during editing is improved.

  In the above embodiment, the images are taken by the three cameras 311, 312, and 313 and the three distance sensors 321, 322, and 323, but may be taken by one camera or a pair of cameras. And a distance sensor. For example, the outside scene SC is imaged by a pair of cameras 311 and a distance sensor 321 mounted on the HMD, and the wearer of the HMD becomes a subject OB (demonstrator), and the behavior or work of the subject OB is tracked. Thus, an AR scenario may be created. In this case, even if the imaging range changes due to the movement of the head of the subject OB, the correspondence between the real space and the space of the three-dimensional model can be corrected by the 9-axis sensor 87 mounted on the HMD. Moreover, even with one camera, an AR scenario including an AR image of a two-dimensional model or a three-dimensional model can be easily created. In this case, the image processing apparatus 100 that generates the AR image or the AR scenario may be realized by a processor and a storage device on the HMD, or may be an external device capable of bidirectional communication with the HMD via a network (for example, a wireless LAN). It may be realized by a processor and a storage device included in the computer.

  In this case, in addition to the above method, there is the following method as one of methods for preventing the AR image from being blocked by a hand or the like. For example, at the time of creating an AR scenario, a message that appeals to the visual or auditory sense, such as “show the object you have at a different angle or orientation,” is presented to the performer wearing the HMD. When the camera 311 and the distance sensor 321 acquire images with different angles or orientations, it is possible to combine images from different angles or orientations into one three-dimensional model.

  In the above-described embodiment, the unnecessary image erasing unit 19 has deleted the unnecessary object so as not to generate the AR image. However, the processing of the unnecessary object can be variously modified. For example, an unnecessary object may be generated as an AR image similarly to the moving object, or may be generated as a translucent AR image or a two-dimensional image by changing RGB data. In this modification, for example, when the user executing the created AR scenario does not know how to hold the scale TL, the image of the hand portion of the subject OB is associated with the AR image of the scale TL. By displaying the information, the convenience for the user is improved.

E-2. Modification 2:
Further, the CPU 10 may have an automatic learning unit. The automatic learning unit can recognize a moving object or an indeterminate object whose standard is not determined by learning the posture and position of the image of the object included in the captured image. In addition, the automatic learning unit automatically selects the optimal combination of shape, color, and pattern using a multi-recognition algorithm, so that the learning type recognition is highly adaptable to environmental changes (for example, process changes and lighting changes). Can be realized.

  In the above embodiment, the positional relationship between the target for which the AR image is generated and the corresponding object detected within a predetermined range from the target is associated, but the distance is not necessarily close to the target for which the AR image is generated. The AR image may not be generated in association with the positional relationship with the corresponding object. For example, an AR image may be displayed at a preset position when a corresponding object is detected regardless of the position of the corresponding object detected from the captured image. Further, the positional relationship between the corresponding object and the AR image may be set as appropriate by an operation received by the operation unit 34.

  In the above embodiment, the unnecessary image erasing unit 19 generates an AR image in association with the detected position of the corresponding object. However, the unnecessary image erasure unit 19 necessarily generates an AR image in association with the detected position of the corresponding object. do not have to. For example, the AR image may be generated in association with the user's voice acquired by the microphone 33.

  In the first embodiment and the second embodiment, the embodiment has been described by exemplifying the work of cooking as the action or work performed by the subject OB. However, another aspect of the present invention can be applied to an embodiment that creates an AR image that is displayed when a machine part is installed in a factory, a machine is inspected, or other operations performed on a mechanical real object. Still another aspect of the present invention is an embodiment for creating an AR image displayed during work (leisure or game) using toys, such as how to assemble a LEGO (registered trademark) block of LEGO Corporation. Applicable.

E-3. Modification 3:
As described above, in the image processing apparatus 100a according to the fourth embodiment, the AR scenario control unit 11a automatically creates an AR scenario corresponding to each combination of trigger targets detected when the AR is executed. In addition, the AR scenario control unit 11a may add the additional information such as voice to the created AR scenario by accepting a predetermined operation as in the first embodiment. .

  In the image processing apparatus 100a according to the fourth embodiment, the AR scenario control unit 11a creates AR scenarios corresponding to all combinations of the presence / absence of a moving object and the presence / absence of a stationary related object. Only the corresponding AR scenario may be created. In addition, the AR scenario control unit 11a creates an AR scenario corresponding to a combination in which all moving objects in the imaging data are selected in the process of step S97 in FIG. 22, but only some selected moving objects. Only the AR scenario corresponding to can be created.

  Further, in the display image determination process of the fourth embodiment shown in FIG. 29, the display image is determined by determining whether or not the combination trigger object shown in FIG. 28 matches, but each trigger object is detected. The display image may be determined by determining whether or not. For example, when there are cover CV, casing BX, driver DV, and bolt BT as four parts as all trigger targets included in the AR scenario, the flowchart branches depending on whether or not each part is detected, A display image may be determined. Specifically, first, detection of the cover CV is determined, next detection of the housing BX is determined, next detection of the driver DV is determined, and then detection of the bolt BT is determined. The display image may be determined based on all the determination results. In the display image determination process of this modification, even when a plurality of trigger targets are set in the AR scenario, it is possible to display an AR image included in the corresponding AR scenario.

  The present invention is not limited to the above-described embodiments and modifications, and can be realized with various configurations without departing from the spirit of the present invention. For example, the technical features in the embodiments and the modifications corresponding to the technical features in each form described in the summary section of the invention are to solve some or all of the above-described problems, or In order to achieve part or all of the effects, replacement or combination can be performed as appropriate. Further, if the technical feature is not described as essential in the present specification, it can be deleted as appropriate.

10 ... CPU
11 ... AR scenario control unit 12 ... Object tracking unit (target selection unit)
DESCRIPTION OF SYMBOLS 13 ... Object recognition part 14 ... 3D model production | generation part 15 ... Sensor control part 16 ... UI control part 17 ... AR scenario operation setting part 18 ... Additional information acquisition part 19 ... Unnecessary image deletion part 20 ... Image display part 21 ... AR image extraction (Image generator)
31 ... RGB camera (imaging part)
32 ... Distance sensor (distance measuring unit)
33 ... Microphone (operation reception unit, voice acquisition unit)
34. Operation unit (operation reception unit)
35 ... Display unit 50 ... Data storage unit 60 ... Power source 70 ... Control unit 75 ... CPU
DESCRIPTION OF SYMBOLS 80 ... Image display part 81 ... Right earphone 82 ... Right display drive part 83 ... Left earphone 84 ... Left display drive part 85 ... Connection part 86 ... Right optical image display part 87 ... 9 axis sensor 88 ... Left optical image display part 89 ... Camera 91 ... Depth sensor 100 ... Image processing device 161 ... Text converter 200 ... HMD
311 ... 1st camera (imaging part)
312 ... Second camera (imaging unit)
313 ... Third camera (imaging unit)
321 ... 1st distance sensor (distance measuring part)
322 ... Second distance sensor (distance measuring unit)
323 ... Third distance sensor (distance measuring unit)
OB ... Subject SC ... Outside view KC ... Edited image (specific image)
TL ... scale removal KN ... knife VR ... visual field FS ... fish AR1, AR2 ... image TX1, TX2 ... text image CV ... cover (moving target)
BX ... Case (Related stationary object)
DV ... Screwdriver BT ... Bolt Bh ... Housing female thread Ch ... Cover hole LH ... Left hand RH ... Right hand IMX ... Housing image (Related still image)
IMC ... Cover image (image to be moved)

Claims (15)

An image processing apparatus,
An outside scene sensor for imaging at least one object;
An image generation device, comprising: an image generation unit that generates a virtual image corresponding to at least one of the moving objects among the captured objects. The image processing apparatus according to claim 1,
The image generation unit associates a moving area of an image generation target that is the target on which the virtual image is generated with at least one of the targets excluding the image generation target among the captured targets. An image processing apparatus that generates the virtual image to be generated. The image processing apparatus according to claim 2,
The image generation unit generates the virtual image in which at least one of the size of the virtual image and the moving region is associated with the size of the target associated with the moving region of the image generation target. , Image processing device. The image processing apparatus according to claim 2, wherein:
The image generation unit is configured to determine whether or not to display the virtual image according to whether or not the set trigger target is detected. The image processing apparatus according to claim 4,
The image generation unit
As a trigger target, among the plurality of captured targets, the moving target that is the moving target, and the relationship that is determined to be within a predetermined distance from the moving target and the target that is not moving Set the stationary object and
In association with a combination of the presence / absence of the moving target and the presence / absence of the related stationary target, a moving target corresponding image as a virtual image of the moving target, and a related still target image as a virtual image of the related stationary target, An image processing apparatus that generates a virtual image combining the above. The image processing apparatus according to any one of claims 1 to 5, further comprising:
An operation reception unit for receiving operations is provided.
The image generation unit is an image processing apparatus that deletes unnecessary portions and generates the virtual image based on the received operation. The image processing apparatus according to any one of claims 1 to 6, wherein
The image generation unit is an image processing device that generates, as the virtual image, a corresponding image among a plurality of captured objects while the moving object is moving. The image processing apparatus according to any one of claims 1 to 7, further comprising:
With a target selector
The target selecting unit identifies a shape of a human body and a shape other than the human body as the at least one target,
The image generation unit, wherein the image generation unit does not generate the virtual image corresponding to the shape of a human body among the captured objects. The image processing apparatus according to any one of claims 1 to 8, further comprising:
It has an audio acquisition unit that acquires external audio,
The image generation unit generates the virtual image by associating the image generation target, which is the target on which the virtual image is generated, with the sound acquired while the image generation target is moving. , Image processing device. The image processing apparatus according to claim 9,
The image generation unit is an image processing device that generates the acquired voice as a character image in association with the virtual image. The image processing apparatus according to any one of claims 1 to 10, further comprising:
A distance measuring unit for measuring the distance to the object,
The image generation unit is an image processing device that generates the virtual image based on a measured distance. An image processing apparatus according to any one of claims 1 to 11, comprising:
The image generation device is configured to insert a specific image at a specific time point of the virtual moving image when the virtual image is a virtual moving image that changes with time. The image processing apparatus according to any one of claims 1 to 12, further comprising:
It has an audio acquisition unit that acquires external audio,
When the virtual image is a virtual moving image that changes over time, the image generating unit associates a specific time point of the virtual moving image with the acquired sound and associates the virtual moving image with the virtual moving image. An image processing apparatus that generates A control method for an image processing apparatus, comprising:
Imaging at least one object;
Generating a virtual image corresponding to at least one of the moving objects among the imaged objects. A computer program for an image processing apparatus,
An object imaging function for imaging at least one object;
A computer program that causes a computer to realize an image generation function that generates a virtual image corresponding to at least one of the moving objects among the captured objects.

Images