WO2022070603A1 - 情報処理装置、情報処理方法、及びプログラム - Google Patents

情報処理装置、情報処理方法、及びプログラム Download PDF

Info

Publication number
WO2022070603A1
WO2022070603A1 PCT/JP2021/028992 JP2021028992W WO2022070603A1 WO 2022070603 A1 WO2022070603 A1 WO 2022070603A1 JP 2021028992 W JP2021028992 W JP 2021028992W WO 2022070603 A1 WO2022070603 A1 WO 2022070603A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
dimensional region
information processing
cpu
designated
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/028992
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
史憲 入江
貴嗣 青木
一紀 田村
真彦 宮田
泰規 村上
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Corp
Original Assignee
Fujifilm Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujifilm Corp filed Critical Fujifilm Corp
Priority to JP2022553511A priority Critical patent/JP7571151B2/ja
Publication of WO2022070603A1 publication Critical patent/WO2022070603A1/ja
Priority to US18/184,667 priority patent/US20230224451A1/en
Anticipated expiration legal-status Critical
Priority to JP2024177257A priority patent/JP7767550B2/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/349—Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—Three-dimensional [3D] image rendering
    • G06T15/10—Geometric effects
    • G06T15/20—Perspective computation
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • H04N13/117—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30—Image reproducers
    • H04N13/366—Image reproducers using viewer tracking
    • H04N13/383—Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/62—Control of parameters via user interfaces
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H04N23/632—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Definitions

  • the technology of this disclosure relates to an information processing device, an information processing method, and a program.
  • Japanese Patent Application Laid-Open No. 2019-133309 includes a step of setting a virtual space for providing a virtual experience to a user, a step of setting a plurality of moving areas in the virtual space, and a step of setting a virtual viewpoint in the virtual space. , When the distance between the step of instructing a predetermined moving area among a plurality of moving areas and the distance between the virtual viewpoint and the predetermined moving area is equal to or less than the first threshold value according to the movement of a part of the user's body, the virtual viewpoint is set.
  • a program for causing a computer to execute a step of moving to a predetermined moving area and a step of not moving the virtual viewpoint to a predetermined moving area when the distance between the virtual viewpoint and the predetermined moving area exceeds the first threshold value is disclosed. There is.
  • One embodiment according to the technique of the present disclosure provides an information processing device, an information processing method, and a program capable of allowing a user to observe a state in a three-dimensional region from various positions.
  • a first aspect of the technique of the present disclosure comprises a processor and memory built into or connected to the processor, wherein the processor is directed or directed within a three-dimensional region being observed. From the viewpoint position determined based on the coordinates in the three-dimensional area corresponding to the designated position in the reference image showing the aspect in the three-dimensional area when observing the inside of the three-dimensional area from the reference position. It is an information processing device that acquires a subject image showing a subject existing in the three-dimensional region when observing the inside of the three-dimensional region.
  • the second aspect according to the technique of the present disclosure is the information processing apparatus according to the first aspect in which the processor derives the coordinates based on the observation mode in which the processor is observing in the three-dimensional region and the indicated position.
  • the third aspect according to the technique of the present disclosure is the information processing apparatus according to the second aspect, in which the observation mode is determined according to the observation position for observing in the three-dimensional region.
  • the fourth aspect according to the technique of the present disclosure is the information processing apparatus according to the third aspect, in which the processor determines the observation position instruction range in which the observation position can be instructed according to the attribute of the instruction source.
  • a fifth aspect according to the technique of the present disclosure is that the processor acquires an in-three-dimensional region aspect image showing an in-three-dimensional region aspect when the three-dimensional region is observed in the observation mode, and in the three-dimensional region.
  • the aspect image is an image showing an aspect in which an observation position indicating range in a three-dimensional region and a range other than the observation position indicating range can be distinguished, and is an information processing apparatus according to a fourth aspect.
  • the sixth aspect according to the technique of the present disclosure is the information processing apparatus according to the fifth aspect, wherein the reference image is an image based on the aspect image in the three-dimensional region.
  • a seventh aspect according to the technique of the present disclosure is an image showing an aspect in the three-dimensional region when the processor is observed in the observation mode in the three-dimensional region, and the three-dimensional region is shown in coordinates.
  • the information processing apparatus according to any one of the second to sixth aspects, in which the coordinates are derived based on the correspondence with the three-dimensional region image whose position can be specified.
  • An eighth aspect according to the technique of the present disclosure is a virtual viewpoint image or a tertiary image generated based on a plurality of images obtained by capturing a reference image in a three-dimensional region by a plurality of image pickup devices.
  • the information processing apparatus according to any one of the first to seventh aspects, which is an image based on the captured image obtained by capturing the inside of the original region.
  • a ninth aspect of the technique of the present disclosure is the information processing apparatus according to the eighth aspect, wherein the designated position designated with respect to the reference image is a specific position in the virtual viewpoint image or the captured image. ..
  • a tenth aspect according to the technique of the present disclosure is any one of the first to ninth aspects, wherein the reference image is an image including a first mark capable of specifying a designated position in the reference image. It is an information processing device according to the embodiment.
  • the eleventh aspect according to the technique of the present disclosure is any one of the first to tenth aspects in which the subject image includes a second mark capable of specifying a designated designated position with respect to the reference image. It is an information processing device according to the embodiment.
  • a twelfth aspect according to the technique of the present disclosure is an object image showing an object existing in a three-dimensional region when the processor observes the inside of the three-dimensional region from a position within a range where the distance from the indicated position is equal to or less than a threshold value. Is stored in the storage area, it is an information processing apparatus according to any one of the first to eleventh aspects of acquiring an object image instead of the subject image.
  • a thirteenth aspect according to the technique of the present disclosure is a coordinate indicating a position in which the coordinates relating to a specific area in the three-dimensional area are higher than the actual position in the three-dimensional area of the specific area, from the first aspect to the first aspect.
  • the information processing apparatus according to any one of the twelve aspects.
  • the designated position in the three-dimensional region is the designated position on the first line from the viewpoint where the three-dimensional region is observed to the gazing point.
  • the designated position in the reference image is the designated position on the second line from the reference position to the designated point in the reference image. It is an information processing apparatus which concerns on any one aspect of.
  • a fifteenth aspect according to the technique of the present disclosure is that the designated position designated with respect to the three-dimensional region is a position selected from at least one first candidate position, and is designated with respect to the reference image.
  • the indicated position is a position selected from at least one second candidate position, and the processor reduces the subject image when observing the inside of the three-dimensional region from the first candidate position to at least one first candidate position.
  • the first reduced image is associated with at least one second candidate position
  • the second reduced image obtained by reducing the subject image when observing the inside of the three-dimensional region from the second candidate position is associated with the second candidate position. It is an information processing apparatus which concerns on any one aspect.
  • a sixteenth aspect according to the technique of the present disclosure is any one of the first to fifteenth aspects in which the processor detects a designated position based on a designated area image indicating a designated area in a three-dimensional area. It is an information processing device according to one aspect.
  • a seventeenth aspect according to the technique of the present disclosure is a virtual viewpoint image generated based on a plurality of images obtained by capturing a subject image in a three-dimensional region by a plurality of image pickup devices. It is an information processing apparatus according to any one aspect from 1 to 16.
  • the eighteenth aspect according to the technique of the present disclosure is an aspect in the three-dimensional region when the inside of the three-dimensional region to be observed is instructed or the inside of the three-dimensional region is observed from the reference position. Indicates a subject existing in the three-dimensional region when observing the inside of the three-dimensional region from a viewpoint position determined based on the coordinates in the three-dimensional region corresponding to the indicated designated position with respect to the indicated reference image. It is an information processing method including acquiring a subject image.
  • the nineteenth aspect according to the technique of the present disclosure is instructed by a computer to the inside of the three-dimensional region to be observed, or in the three-dimensional region when the inside of the three-dimensional region is observed from the reference position.
  • It is a program for executing a process including acquiring a subject image showing a subject.
  • FIG. 28 It is a flowchart which shows an example of the flow of the image generation processing which concerns on 2nd Embodiment. It is a conceptual diagram which shows the modification of the content of the image generation processing which concerns on 2nd Embodiment. It is a schematic perspective view which shows an example of the mode in which a user's finger is imaged by a plurality of image pickup devices.
  • CPU is an abbreviation for "Central Processing Unit”.
  • NVM is an abbreviation for "Non-Volatile Memory”.
  • RAM is an abbreviation for "RandomAccessMemory”.
  • SSD is an abbreviation for "Solid State Drive”.
  • HDD is an abbreviation for "Hard Disk Drive”.
  • EEPROM refers to the abbreviation of "Electrically Erasable and Programmable Read Only Memory”.
  • I / F refers to the abbreviation of "Interface”.
  • ASIC is an abbreviation for "Application Specific Integrated Circuit”.
  • PLD is an abbreviation for "Programmable Logic Device”.
  • FPGA refers to the abbreviation of "Field-Programmable Gate Array”.
  • SoC is an abbreviation for "System-on-a-chip".
  • CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor”.
  • CCD refers to the abbreviation of "Charge Coupled Device”.
  • EL refers to the abbreviation of "Electro-Luminescence”.
  • GPU refers to the abbreviation of "Graphics Processing Unit”.
  • LAN is an abbreviation for "Local Area Network”.
  • 3D refers to the abbreviation of "3 Dimensions”.
  • USB is an abbreviation for "Universal Serial Bus”.
  • HMD refers to the abbreviation of "Head Mounted Display”.
  • LTE is an abbreviation for "Long Term Evolution”.
  • 5G is an abbreviation for "5th generation (wireless technology for digital cellular networks)”.
  • TDM is an abbreviation for "Time-Division Multiplexing”.
  • HMD is an abbreviation for "Head Mounted Display”.
  • a CPU is illustrated as an example of the “processor” according to the technique of the present disclosure, but the “processor” according to the technique of the present disclosure includes a plurality of processing devices such as a CPU and a GPU. It may be a combination of.
  • the GPU operates under the control of the CPU and is responsible for executing image processing.
  • match is, in addition to perfect match, an error that is generally acceptable in the technical field to which the technology of the present disclosure belongs and is to the extent that it does not contradict the gist of the technology of the present disclosure. Refers to a match in the sense of including.
  • the information processing system 2 includes an information processing device 10 and a user device 12.
  • the server is applied as an example of the information processing apparatus 10.
  • this is only an example, and it may be a personal computer, a plurality of personal computers, a plurality of servers, a device combining a personal computer and a server, or the like. It may be.
  • a smartphone is applied as an example of the user device 12.
  • the smartphone is merely an example, and may be, for example, a personal computer, a tablet terminal, or a portable multifunctional terminal such as an HMD.
  • the information processing apparatus 10 and the user device 12 are connected so as to be communicable via, for example, a base station (not shown).
  • Communication standards used in base stations include wireless communication standards including 5G standards, LTE standards, etc., wireless communication standards including WiFi (802.11) standards and / or Bluetooth® standards, TDM standards, and /.
  • it includes a wired communication standard including an Ethernet (registered trademark) standard.
  • the information processing device 10 acquires an image and transmits the acquired image to the user device 12.
  • the image refers to, for example, an captured image obtained by being imaged, an image generated based on the captured image, and the like.
  • An example of an image generated based on a captured image is a virtual viewpoint image.
  • the user device 12 is used by the user 13.
  • the user device 12 includes a touch panel display 16.
  • the touch panel display 16 is realized by the display 18 and the touch panel 20.
  • An example of the display 18 is an EL display (for example, an organic EL display or an inorganic EL display).
  • the display is not limited to the EL display, and may be another type of display such as a liquid crystal display.
  • the touch panel display 16 is formed by superimposing the touch panel 20 on the display area of the display 18, or by forming an in-cell type in which the touch panel function is built in the display 18.
  • the in-cell type is merely an example, and may be an out-cell type or an on-cell type.
  • the user device 12 executes a process (for example, a process on the user device side described later) according to an instruction received from the user by the touch panel 20 or the like.
  • a process for example, a process on the user device side described later
  • the user device 12 exchanges various information with and from the information processing apparatus 10 according to an instruction received from the user by the touch panel 20 or the like.
  • the user device 12 receives the image transmitted from the information processing device 10 and displays the received image on the display 18.
  • the user 13 appreciates the image displayed on the display 18.
  • the information processing device 10 includes a computer 22, a transmission / reception device 24, a communication I / F 26, and a bus 28.
  • the computer 22 includes a CPU 22A, an NVM 22B, and a RAM 22C, and the CPU 22A, the NVM 22B, and the RAM 22 are connected via a bus 28.
  • one bus is shown as the bus 28 for convenience of illustration, but a plurality of buses may be used.
  • the bus 28 may include a serial bus or a parallel bus composed of a data bus, an address bus, a control bus, and the like.
  • the CPU 22A is an example of a "processor" according to the technique of the present disclosure.
  • the CPU 22A controls the entire information processing apparatus 10.
  • the NVM22B stores various parameters, various programs, and the like. Examples of the NVM22B include EEPROM, SSD, and / or HDD.
  • the RAM 22C is an example of a "memory" according to the technique of the present disclosure. Various information is temporarily stored in the RAM 22C.
  • the RAM 22C is used as a working memory by the CPU 22A.
  • the transmission / reception device 24 is connected to the bus 28.
  • the transmission / reception device 24 is a device including a communication processor (not shown), an antenna, and the like, and transmits and receives various information to and from the user device 12 via a base station (not shown) under the control of the CPU 22A. That is, the CPU 22A exchanges various information with and from the user device 12 via the transmission / reception device 24.
  • Communication I / F26 is realized by, for example, a device having an FPGA.
  • the communication I / F 26 is connected to a plurality of image pickup devices 30 via a LAN cable (not shown).
  • the image pickup device 30 is an image pickup device having a CMOS image sensor, and is equipped with an optical zoom function and / or a digital zoom function.
  • CMOS image sensor another type of image sensor such as a CCD image sensor may be adopted.
  • the plurality of image pickup devices 30 are installed in the soccer stadium 36 (see FIG. 2) and image the subject in the soccer stadium 36.
  • the captured image obtained by capturing the subject by the imaging device 30 is used, for example, to generate a virtual viewpoint image. Therefore, each of the plurality of image pickup devices 30 is installed in a soccer stadium 36 (see FIG. 2) at a place different from each other, that is, a place where a plurality of captured images capable of generating a virtual viewpoint image can be obtained.
  • the communication I / F 26 is connected to the bus 28 and controls the exchange of various information between the CPU 22A and the plurality of image pickup devices 30.
  • the communication I / F 26 controls a plurality of image pickup devices 30 according to the request of the CPU 22A.
  • the communication I / F 26 outputs an image captured by being imaged by each of the plurality of image pickup devices 30 (hereinafter, also simply referred to as “captured image”) to the CPU 22A.
  • the communication I / F 26 is exemplified here as a wired communication I / F, it may be a wireless communication I / F such as a high-speed wireless LAN.
  • the NVM 22B stores the three-dimensional area image 32 and the image generation processing program 34.
  • the three-dimensional region image 32 is a three-dimensional image showing an aspect of the three-dimensional region, and the three-dimensional region image 32 is provided with coordinates capable of specifying a position in the three-dimensional region. There is.
  • the image generation processing program 34 is an example of a "program" related to the technique of the present disclosure.
  • the CPU 22A reads the image generation processing program 34 from the NVM 22B and executes the image generation processing program 34 on the RAM 22C to perform the image generation processing (see FIG. 12).
  • the three-dimensional area image 32 is a three-dimensional image showing the soccer stadium 36.
  • the soccer stadium 36 is an example of a "three-dimensional area" according to the technique of the present disclosure.
  • the soccer stadium 36 is a three-dimensional area including a soccer field 36A and a spectator seat 36B constructed so as to surround the soccer field 36A, and is an observation target of the user 13.
  • an observer that is, a user 13 is observing the inside of the soccer stadium 36 from the spectator seat 36B.
  • the three-dimensional area image 32 is given coordinates that can specify the position in the soccer stadium 36.
  • coordinates that can specify the position in the soccer stadium 36 three-dimensional coordinates that can specify the position in the rectangular parallelepiped 38 with one vertex of the rectangular parallelepiped 38 surrounding the soccer stadium 36 as the origin are applied. ..
  • the coordinates related to the position of the soccer field 36A shown by the three-dimensional area image 32 are the coordinates indicating a position higher than the actual position of the soccer field 36A.
  • the coordinates related to the position of the soccer field 36A refer to the coordinates given to the position of the soccer field 36A among the coordinates given to the three-dimensional area image 32.
  • the coordinates indicating a position higher than the actual position are coordinates indicating a position higher than the actual position by a distance corresponding to the average height of a general adult, for example.
  • the soccer field 36A is an example of a "specific area" according to the technique of the present disclosure.
  • the user device 12 includes a display 18, a computer 40, an image pickup device 42, a transmission / reception device 44, a speaker 46, a microphone 48, a reception device 50, and a bus 52.
  • the computer 40 includes a CPU 40A, an NVM 40B, and a RAM 40C, and the CPU 40A, the NVM 40B, and the RAM 40C are connected via a bus 52.
  • a bus 52 In the example shown in FIG. 3, one bus is shown as the bus 52 for convenience of illustration, but a plurality of buses may be used.
  • the bus 52 may include a serial bus or a parallel bus composed of a data bus, an address bus, a control bus, and the like.
  • the CPU 40A controls the entire user device 12.
  • the NVM40B stores various parameters, various programs, and the like.
  • An example of the NVM40B is EEPROM.
  • Various information is temporarily stored in the RAM 40C.
  • the RAM 40C is used as a working memory by the CPU 40A.
  • the image pickup device 42 is an image pickup device having a CMOS image sensor, and is equipped with an optical zoom function and / or a digital zoom function. Instead of the CMOS image sensor, another type of image sensor such as a CCD image sensor may be adopted.
  • the image pickup device 42 is connected to the bus 52, and the CPU 40A controls the image pickup device 42.
  • the captured image obtained by being captured by the image pickup device 42 is acquired by the CPU 40A via the bus 52.
  • the transmission / reception device 44 is connected to the bus 52.
  • the transmission / reception device 44 is a device including a communication processor (not shown), an antenna, and the like, and transmits / receives various information to / from the information processing device 10 via a base station (not shown) under the control of the CPU 40A. .. That is, the CPU 40A exchanges various information with and from the information processing device 10 via the transmission / reception device 44.
  • the speaker 46 converts an electric signal into sound.
  • the speaker 46 is connected to the bus 52.
  • the speaker 46 receives the electric signal output from the CPU 40A via the bus 52, converts the received electric signal into sound, and outputs the sound obtained by converting the electric signal to the outside of the user device 12.
  • the microphone 48 converts the collected sound into an electric signal.
  • the microphone 48 is connected to the bus 52.
  • the electric signal obtained by converting the sound collected by the microphone 48 is acquired by the CPU 40A via the bus 52.
  • the reception device 50 receives instructions from the user 13 and the like. Examples of the reception device 50 include a touch panel 20 and hard keys (not shown). The reception device 50 is connected to the bus 52, and the instruction received by the reception device 50 is acquired by the CPU 40A.
  • the NVM 40B stores the user device side processing program 54.
  • the CPU 40A reads the user device side processing program 54 from the NVM 40B and executes the user device side processing program 54 on the RAM 40C to perform user device side processing (see FIG. 11).
  • the observation mode in which the user 13 is observing the inside of the soccer stadium 36 is the viewpoint position 56, the line-of-sight direction 58, and the viewing angle of the user 13. Specified by ⁇ .
  • the viewpoint position 56 corresponds to a position where the user 13 observes the inside of the soccer stadium 36 in the real space, and is an example of the “reference position” and the “observation position” according to the technique of the present disclosure.
  • the viewpoint position 56 corresponds to the position of the image pickup device 42 mounted on the user device 12, and the line-of-sight direction 58 corresponds to the direction of the optical axis of the image pickup optical system (not shown) included in the image pickup device 42.
  • the viewing angle ⁇ is an angle corresponding to the angle of view of the image pickup apparatus 42.
  • the region observed by the user 13 in the real space is specified from the captured image obtained by capturing the inside of the soccer stadium 36 by the imaging device 42.
  • a viewpoint position different from the viewpoint position 56 is set by using the captured image, and the subject existing in the soccer stadium 36 when observing the inside of the soccer stadium 36 from the set viewpoint position is shown.
  • the subject image is viewed by the user 13.
  • the information processing system performs the following user device side processing and image generation processing.
  • the CPU 40A acquires a live view image obtained by being imaged by the image pickup device 42.
  • the live view image is an image showing a designated area in the soccer stadium 36.
  • the region designated in the soccer stadium 36 refers to, for example, a region defined by the viewpoint position 56, the line-of-sight direction 58, and the viewing angle ⁇ .
  • the live view image is an example of "an image showing an aspect in a three-dimensional region when the inside of the three-dimensional region is observed in an observation mode" according to the technique of the present disclosure.
  • the CPU 40A generates a reference image 60 using the acquired live view image.
  • the reference image 60 is an example of a "designated area image" according to the technique of the present disclosure.
  • the reference image 60 is an image showing an aspect of the soccer stadium 36 when the inside of the soccer stadium 36 is observed from the viewpoint position 56 (see FIG. 4).
  • the reference image 60 is an image based on the live view image.
  • FIG. 5 as an example of the reference image 60, an image in which the cross-shaped target mark 60A is superimposed on the live view image is shown.
  • the target mark 60A is a mark that is displaced in the reference image 60 according to an instruction given by the user 13, and indicates a position designated by the user 13 as a viewpoint position for the user 13 to observe the inside of the soccer stadium 36.
  • the target mark 60A is a mark that can specify the position designated by the user 13 in the reference image 60.
  • the position of the target mark 60A in the reference image 60 that is, the position of the target mark 60A superimposed on the live view image is the "instructed position" and the "specific position in the captured image” according to the technique of the present disclosure.
  • the home position of the target mark 60A is the center of the reference image 60.
  • the center of the target mark 60A is located at the center of the reference image 60.
  • the CPU 40A causes the display 18 to display the reference image 60.
  • the CPU 40A receives a change instruction, which is an instruction to change the position of the target mark 60A, by the touch panel 20 while the reference image 60 is displayed on the display 18. Then, the position of the target mark 60A in the reference image 60 is changed according to the change instruction.
  • the change instruction is a swipe given to the touch panel 20 on the target mark 60A displayed on the display 18. That is, the user 13 touches the target mark 60A via the touch panel 20 and slides the touched position on the touch panel 20 to indicate the change destination of the position of the target mark 60A.
  • the CPU 40 updates the reference image 60 displayed on the display 18 to the reference image 60 in which the position of the target mark 60A has been changed.
  • the reference image 60 is a live view image, instead of touching and changing the position of the target mark 60A on the display 18, the user 13 moves the user device 12 to move the target mark with respect to the reference image 60.
  • the position of 60A may be moved.
  • the CPU 40A is an instruction to determine the position of the target mark 60A in the reference image 60 while the reference image 60 is displayed on the display 18.
  • the reference image 62 with the instruction position is generated.
  • the reference image 62 with a designated position is an image to which the designated position specifying information 62A is added to the reference image 60.
  • the designated position specifying information 62A is information that can specify the position designated by the user 13 as the viewpoint position for the user 13 to observe in the soccer stadium 36, that is, the position of the target mark 60A in the reference image 60 can be specified.
  • the CPU 40A transmits the reference image 62 with the indicated position to the information processing device 10 via the transmission / reception device 44 (see FIG. 3).
  • the CPU 40A acquires the virtual viewpoint image 64 generated by the information processing apparatus 10.
  • the virtual viewpoint image 64 is a moving image. However, this is only an example, and the virtual viewpoint image 64 may be a still image.
  • the CPU 40 displays the acquired virtual viewpoint image 64 on the display 18. Here, the CPU 40 does not simply display the virtual viewpoint image 64 on the display 18, but displays it on the display 18 as a new reference image 60.
  • the new reference image 60 is an image based on the virtual viewpoint image 64. That is, here, the new reference image 60 refers to an image in which the target mark 60A is superimposed on the virtual viewpoint image 64.
  • the virtual viewpoint image 64 includes a mark 61 (see FIG. 10) capable of specifying a position corresponding to the designated position used for generating the virtual viewpoint image 64 in the virtual viewpoint image 64.
  • the mark 61 is also included in the new reference image 60.
  • the new reference image 60 is an example of a "designated area image" according to the technique of the present disclosure.
  • the virtual viewpoint image 64 is an example of "an image showing an aspect in a three-dimensional region when the inside of the three-dimensional region is observed in an observation mode" according to the technique of the present disclosure. Further, the position of the target mark 60A superimposed on the virtual viewpoint image 64 is an example of the "instructed position" and the "specific position in the virtual viewpoint image” according to the technique of the present disclosure.
  • the CPU 22A acquires a reference image 62 with an instruction position from the user device 12. Further, the CPU 22A acquires the three-dimensional region image 32 from the NVM 22B.
  • the CPU 22A inside the soccer stadium 36 when observing the inside of the soccer stadium 36 from a viewpoint position determined based on the coordinates in the soccer stadium 36 corresponding to the position of the target mark 60A in the reference image 60. Acquires a subject image showing a subject existing in.
  • the CPU 22A compares the reference image 62 with the indicated position acquired from the user device 12 with the three-dimensional region image 32 acquired from the NVM 22B.
  • a matching feature point is specified between the reference image 62 with the designated position and the three-dimensional area image 32.
  • which pixel in the reference position image 62 with the designated position corresponds to which pixel in the three-dimensional region image 32 is specified.
  • the CPU 22A has an observation mode in which the user 13 is observing the inside of the soccer stadium 36 (in the example shown in FIG. 4, the viewpoint position 56, the line-of-sight direction 58, and the viewing angle ⁇ ) and the position of the target mark 60A in the reference image 60 (
  • the coordinates in the soccer stadium 36 corresponding to the designated position are derived from the three-dimensional region image 32 based on this position (also referred to simply as the “designated position”).
  • the observation mode in which the user 13 is observing the inside of the soccer stadium 36 is determined according to the viewpoint position 56, and changes according to the displacement of the viewpoint position 56. Since the observation mode in which the user 13 is observing the inside of the soccer stadium 36 is represented by the reference position image 62, the CPU 22A is based on the correspondence between the reference image 62 with the instruction position and the three-dimensional area image 32. Then, the coordinates in the soccer stadium 36 corresponding to the designated position are derived from the three-dimensional area image 32.
  • the CPU 22A corresponds to the comparison result between the reference image 62 with the designated position and the three-dimensional region image 32 (for example, which pixel in the reference image 62 with the designated position corresponds to which pixel in the three-dimensional region image 32.
  • the coordinates of the position corresponding to the indicated position are extracted from the three-dimensional region image 32 by using the comparison result).
  • the CPU 22A generates a virtual viewpoint image 64 using a viewpoint position determined based on the coordinates extracted from the three-dimensional region image 32.
  • the viewpoint position determined based on the coordinates extracted from the three-dimensional area image 32 refers to, for example, a position in the soccer stadium 36 specified from the coordinates extracted from the three-dimensional area image 32.
  • the virtual viewpoint image 64 is a kind of subject image showing a subject existing in the soccer stadium 36 when observing the inside of the soccer stadium 36 from a viewpoint position determined based on the coordinates extracted from the three-dimensional area image 32.
  • the observation mode observation mode in which the user 13 virtually observes the inside of the soccer stadium 36
  • the viewpoint position the viewpoint position
  • the line-of-sight direction the viewing angle used to generate the virtual viewpoint image 64
  • It is defined by a viewpoint position determined based on the coordinates extracted from the original region image 32, a line-of-sight direction previously specified by the user 13 and the like, and a viewing angle predetermined by the user 13 and the like.
  • the CPU 22A sets the designated position in the virtual viewpoint image 64 with respect to the designated position.
  • a identifiable mark 61 in the example shown in FIG. 8, a broken line cross mark is added.
  • the mark 61 is an example of the "second mark” according to the technique of the present disclosure.
  • the viewpoint position, the line-of-sight direction, and the viewing angle used to generate the virtual viewpoint image 64 are examples of the "observation mode" according to the technique of the present disclosure.
  • the region defined by the viewpoint position, the line-of-sight direction, and the viewing angle used to generate the virtual viewpoint image 64 is an example of the "region designated within the three-dimensional region" according to the technique of the present disclosure.
  • the viewpoint position used for generating the virtual viewpoint image 64 corresponds to a position where the user 13 virtually observes the inside of the soccer stadium 36, and is an example of the “observation position” according to the technique of the present disclosure.
  • a moving image using a 3D polygon generated based on a plurality of captured images obtained by imaging the inside of the soccer stadium 36 by a plurality of imaging devices 30 is applied.
  • the moving image is only an example, and may be a still image.
  • the CPU 22A transmits the virtual viewpoint image 64 to the user device 12 via the transmission / reception device 24 (see FIG. 1).
  • the virtual viewpoint image 64 transmitted in this way is received by the user device 12 and displayed on the display 18 as a new reference image 60 (see FIG. 8).
  • step ST10 the CPU 40A acquires a live view image from the image pickup device 42, and then the user device side processing shifts to step ST12.
  • step ST12 the CPU 40A generates a reference image 60 based on the live view image acquired in step ST10, and then the processing on the user device side shifts to step ST14.
  • step ST14 the CPU 40A displays the reference image 60 generated in step ST12 on the display 18, and then the processing on the user device side shifts to step ST16.
  • step ST16 the CPU 40A determines whether or not the indicated position has been determined.
  • the confirmation instruction is received by the touch panel 20, it is determined that the instruction position is confirmed, and when the confirmation instruction is not received by the touch panel 20, it is determined that the instruction position is not confirmed. If the designated position is not fixed in step ST16, the determination is denied and the user device side processing proceeds to step ST28. If the designated position is confirmed in step ST16, the determination is affirmed, and the user device-side processing proceeds to step ST18.
  • step ST28 the CPU 40A determines whether or not the condition for ending the user device side processing (hereinafter referred to as "user device side processing end condition") is satisfied.
  • the first example of the user device side processing end condition there is a condition that the instruction to end the user device side processing is accepted by the receiving device 50.
  • the first predetermined time for example, 60 minutes
  • the processing end condition on the user device side there is a condition that the processing capacity of the CPU 40A has decreased to less than the reference level.
  • step ST28 If the user device side processing end condition is not satisfied in step ST28, the determination is denied and the user device side processing proceeds to step ST10. If the user device side processing end condition is satisfied in step ST28, the determination is affirmed and the user device side processing ends.
  • step ST18 the CPU 40A is based on the reference image 60 generated in step ST12 or the reference image 60 to which the target mark 60A is given to the virtual viewpoint image 64 received by the transmission / reception device 44 in step ST20 described later.
  • a reference image 62 with a designated position is generated.
  • the CPU 40A transmits the generated reference image 62 with an instruction position to the information processing device 10 via the transmission / reception device 44.
  • the process on the user device side shifts to step ST20.
  • step ST20 the CPU 40A determines whether or not the virtual viewpoint image 64 transmitted from the information processing device 10 is received by the transmission / reception device 44 by executing the process of step ST60 of the image generation process shown in FIG. .. If the virtual viewpoint image 64 is not received by the transmission / reception device 44 in step ST20, the determination is denied and the determination in step ST20 is performed again. When the virtual viewpoint image 64 is received by the transmission / reception device 44 in step ST20, the determination is affirmed, and the processing on the user device side shifts to step ST22.
  • step ST22 the CPU 40A displays the virtual viewpoint image 64 received by the transmission / reception device 44 in step ST20 as a new reference image 60 on the display 18, and then the user device side processing shifts to step ST24.
  • step ST24 the CPU 40A determines whether or not the indicated position has been determined. If the designated position is not fixed in step ST24, the determination is denied and the user device side processing proceeds to step ST26. If the designated position is confirmed in step ST24, the determination is affirmed, and the user device side processing proceeds to step ST18.
  • step ST26 the CPU 40A determines whether or not the processing end condition on the user device side is satisfied. If the user device side processing end condition is not satisfied in step ST26, the determination is denied and the user device side processing proceeds to step ST24. If the user device side processing end condition is satisfied in step ST26, the determination is affirmed and the user device side processing ends.
  • FIG. 12 is an example of the "information processing method" according to the technique of the present disclosure.
  • step ST50 the CPU 22A uses the transmission / reception device 24 to transmit the reference image 62 with an instruction position transmitted by executing the process of step ST18 of the user device side process shown in FIG. Determine if it was received by. If the reference image 62 with the indicated position is not received by the transmission / reception device 24 in step ST50, the determination is denied and the image generation process proceeds to step ST62. When the reference image 62 with the indicated position is received by the transmission / reception device 24 in step ST50, the determination is affirmed, and the image generation process proceeds to step ST52.
  • step ST52 the CPU 22A acquires the three-dimensional area image 32 from the NMV 22B, and then the image generation process shifts to step ST54.
  • step ST54 the CPU 22A compares the reference image 62 with the indicated position received by the transmission / reception device 24 in step ST50 with the three-dimensional area image 32 acquired in step ST52, and then the image generation process shifts to step ST56. do.
  • step ST56 the CPU 22A corresponds to the designated position specified from the designated position specifying information 62A of the designated image 62 with the designated position by using the comparison result between the reference image 62 with the designated position and the three-dimensional region image in step ST54.
  • the coordinates to be performed are extracted from the three-dimensional area image 32, and then the image generation process shifts to step ST58.
  • step ST58 the CPU 22A generates a virtual viewpoint image 64 using the viewpoint position determined based on the coordinates extracted in step ST56, and then the image generation process shifts to step ST60.
  • step ST60 the CPU 22A transmits the virtual viewpoint image 64 generated in step ST58 to the user device 12 via the transmission / reception device 24, and then the image generation process shifts to step ST62.
  • step ST62 the CPU 22A determines whether or not the condition for ending the image generation process (hereinafter referred to as "image generation process end condition") is satisfied.
  • image generation process end condition there is a condition that the information processing apparatus 10 is instructed to end the image generation processing by the administrator of the information processing apparatus 10 or the like.
  • image generation processing end condition there is a condition that a second predetermined time (for example, 10 hours) has elapsed since the execution of the image generation processing was started.
  • a third example of the image generation processing end condition there is a condition that the processing capacity of the CPU 22A is reduced to less than the reference level.
  • step ST62 If the condition for ending the image generation process is not satisfied in step ST62, the determination is denied and the image generation process proceeds to step ST50. If the condition for ending the image generation process is satisfied, the determination is affirmed and the image generation process ends.
  • the reference image 60 (FIG. 4) showing the mode in the soccer stadium 36 when the user 13 observes the soccer stadium 36 (see FIG. 4) from the viewpoint position 56 (see FIG. 4). 6)
  • the virtual viewpoint image 64 showing the subject existing in the stadium is acquired by the CPU 22A.
  • the virtual viewpoint image 64 acquired by the CPU 22A is displayed on the display 18 of the user device 12. Therefore, according to this configuration, the user 13 can observe the inside of the soccer stadium 36 from various positions.
  • the coordinates in the soccer stadium 36 corresponding to the instructed position are derived by the CPU 22A based on the observation mode and the instructed position in which the user 13 is observing the inside of the soccer stadium 36. Therefore, according to this configuration, it is possible to specify which position in the soccer stadium 36 the position instructed by the user 13 is.
  • the observation mode in which the user 13 is observing the inside of the soccer stadium 36 is determined according to the position where the user 13 is observing the inside of the soccer stadium 36 in the real space or the virtual space. ing. Therefore, when the position where the user 13 is observing the inside of the soccer stadium 36 is displaced, the observation mode in which the user 13 is observing the inside of the soccer stadium 36 also changes accordingly. Also in this case, the coordinates in the soccer stadium 36 corresponding to the instructed position are derived by the CPU 22A based on the observation mode and the instructed position in which the user 13 is observing in the soccer stadium 36.
  • the position specified by the user 13 is any of the positions in the soccer stadium 36. It is possible to identify whether it is the position of.
  • the coordinates in the soccer stadium 36 corresponding to the instructed position are derived by the CPU 22A based on the correspondence between the live view image or the virtual viewpoint image 64 and the three-dimensional area image 32. Therefore, according to this configuration, it is estimated which position in the soccer stadium 36 is the position instructed by the user 13 only by the information visually obtained from the live view image or the virtual viewpoint image 64 and human intuition. Compared to the case, it is possible to specify with higher accuracy which position in the soccer stadium 36 the position instructed by the user 13 is.
  • an image based on the live view image and an image based on the virtual viewpoint image 64 are used as the reference image 60. Therefore, according to this configuration, the user 13 can instruct the viewpoint position while checking the state in the real space, or can instruct the viewpoint position while checking the state in the virtual space.
  • the position of the target mark 60A in the live view image or the virtual viewpoint image 64 is the position instructed by the user 13.
  • the position of the target mark 60A is changed according to the change instruction given by the user 13 (see FIG. 6), and is fixed according to the confirmation instruction given by the user 13 (see FIG. 7). Therefore, according to this configuration, the position intended by the user 13 can be set as the viewpoint position.
  • the target mark 60A is used as a mark capable of specifying the designated position in the reference image 60, and the target mark 60A is included in the reference image 60 on the display 18 of the user device 12. It is displayed in the state of. Therefore, according to this configuration, the user 13 can visually recognize the designated position on the reference image 60.
  • the mark 61 is included in the virtual viewpoint image 64.
  • the mark 61 is a mark in which the designated position used for generating the virtual viewpoint image 64 can be specified in the virtual viewpoint image 64. Therefore, according to this configuration, the user 13 can be made to guess the designated position used for generating the virtual viewpoint image 64 from the virtual viewpoint image 64.
  • the coordinates related to the position of the soccer field 36A shown by the three-dimensional area image 32 are the coordinates indicating a position higher than the actual position of the soccer field 36A. Therefore, according to this configuration, it is possible to prevent the viewpoint position from being set on the ground of the soccer field 36A.
  • the designated position is detected by the CPU 22A based on the reference image 60. That is, the position of the target mark 60A is detected by the CPU 22A as the designated position. Therefore, according to this configuration, the designated position can be specified with higher accuracy than in the case where the designated position is estimated only by the information visually obtained from the live view image or the virtual viewpoint image 64 and human intuition.
  • the CPU 22A generates a virtual viewpoint image 64 and transmits it to the user device 12 in the image generation process, but the technique of the present disclosure is not limited to this.
  • the image pickup device 30 when the image pickup device 30 is installed at a position that matches the viewpoint position determined based on the coordinates extracted from the three-dimensional area image 32, it is determined based on the coordinates extracted from the three-dimensional area image 32.
  • the captured image obtained by being imaged by the image pickup device 30 installed at a position corresponding to the obtained viewpoint position is transmitted from the information processing device 10 to the user device 12 as a substitute image for the virtual viewpoint image 64. You may do it.
  • a new virtual viewpoint image 64 is generated and transmitted to the user device 12.
  • the disclosed technology is not limited to this.
  • an object existing in the soccer stadium 36 when the inside of the soccer stadium 36 is observed from a position within the range where the distance from the designated position specifying information 62A of the reference image 62 with the designated position is equal to or less than the threshold value.
  • the CPU 22A may acquire the object image from the storage area and transmit the acquired object image to the user device 12.
  • the CPU 22A sets the coordinates corresponding to the designated position specified from the designated position specifying information 62A of the reference image 62 with the designated position in the three-dimensional region, as in the first embodiment. Extract from image 32. Then, the CPU 22A determines whether or not the virtual viewpoint image 64 associated with the coordinates within the vicinity of the coordinates extracted from the three-dimensional region image 32 is stored in the NVM 22B.
  • the neighborhood range refers to, for example, a range within a radius of 2 meters. Further, the radius of 2 meters is an example of the "threshold value" according to the technique of the present disclosure.
  • the NVM22B is an example of a "storage area" according to the technique of the present disclosure.
  • the virtual viewpoint image 64 is an example of the "object image” according to the technique of the present disclosure, and the object shown by the virtual viewpoint image 64 is an example of the "object” according to the technique of the present disclosure.
  • the CPU 22A When the virtual viewpoint image 64 associated with the coordinates in the vicinity of the coordinates extracted from the three-dimensional area image 32 by the CPU 22A is not stored in the NVM 22B, the CPU 22A is in the same manner as in the first embodiment. A virtual viewpoint image 64 is generated, and the generated virtual viewpoint image 64 is transmitted to the user device 12 via the transmission / reception device 24. Further, the CPU 22A associates the generated virtual viewpoint image 64 with the coordinates extracted from the three-dimensional region image 32, and stores the virtual viewpoint image 64 to which the coordinates are associated in the NVM 22B.
  • the CPU 22A has the CPU 22A.
  • a virtual viewpoint image 64 associated with the coordinates closest to the coordinates extracted from the three-dimensional region image 32 is acquired from the NVM 22B.
  • the CPU 22A transmits the virtual viewpoint image 64 acquired from the NVM 22B to the user device 12 via the transmission / reception device 24.
  • the virtual viewpoint image 64 can be provided to the user 13 more quickly than in the case of generating a new virtual viewpoint image 64 each time the CPU 22A acquires the reference image 62 with the designated position.
  • the virtual viewpoint image 64 is generated and transmitted to the user device 12 or stored in the NVM 22B, but the technique of the present disclosure is limited to this. Instead, the captured image may be transmitted to the user device 12 or stored in the NVM 22B together with the virtual viewpoint image 64 or in place of the virtual viewpoint image 64.
  • the user 13 virtually observes the inside of the soccer stadium 36 from a viewpoint position determined based on the coordinates in the soccer stadium 36 corresponding to the position designated with respect to the reference image 60.
  • the user is from a viewpoint position determined based on the coordinates in the soccer stadium 36 corresponding to the position instructed with respect to the soccer stadium 36 to be observed.
  • a case where 13 virtually observes the inside of the soccer stadium 36 will be described.
  • the same components as those in the first embodiment are designated by the same reference numerals, the description thereof will be omitted, and the parts different from those in the first embodiment will be described.
  • the information processing system 66 includes an information processing device 10, a user device 12, and an HMD 68.
  • the HMD 68 includes an HMD main body 70 and a band 72.
  • the band 72 is a stretchable member formed in a band shape from one end to the other end of the HMD main body 70.
  • the HMD 68 has an outer shape formed in an annular shape by the HMD main body 70 and the band 72, and is fixed in close contact with the upper half of the head of the user 13.
  • the HMD main body 70 has a display 74, an HMD camera 76, a computer 78, a reception device 80, and a transmission / reception device 82.
  • the display 74 has a screen (not shown) and a projection unit (not shown).
  • the screen is made of a transparent material, and the user 13 visually recognizes the real space through the screen. That is, the HMD 68 is a transmissive HMD.
  • the HMD main body 70 does not necessarily have to be provided with the computer 78, and the computer 78 may be provided separately from the HMD main body 70.
  • the HMD main body 70 has only a function of displaying the data received from the computer 78 via the transmission / reception device 82 and transmitting the data related to the image obtained by being captured by the HMD camera 76 to the computer 78. It may have. Further, the HMD camera 76 may also be provided separately from the HMD main body 70. For example, the HMD camera 76 may be a camera that can be attached to and detached from the HMD main body 70.
  • the screen is located at a position facing the eyes of the user 13, and an image is projected on the inner surface of the screen (the surface on the user 13 side) by the projection unit.
  • the projection unit is a well-known device, detailed description thereof will be omitted, but a display element such as a liquid crystal display for displaying an image and a projection optical system for projecting an image displayed on the display element toward the inner surface of the screen. It is a device to have.
  • the screen is realized by a half mirror that reflects the image projected by the projection unit and transmits the light in the real space.
  • the projection unit projects an image on the inner surface of the screen at a predetermined frame rate (for example, 60 frames / sec). The image is reflected by the inner surface of the screen and is incident on the eyes of the user 13. As a result, the user 13 visually recognizes the image.
  • the HMD camera 76 is an imaging device having a CMOS image sensor, and is equipped with an optical zoom function and / or a digital zoom function. Instead of the CMOS image sensor, another type of image sensor such as a CCD image sensor may be adopted.
  • the HMD camera 76 is located in front of the forehead of the user 13 and images the front of the user 13.
  • the computer 78 includes a CPU 78A, an NVM 78B, and a RAM 78C, and the CPU 78A, the NVM 78B, and the RAM 78 C are connected via a bus 84.
  • a bus 84 In the example shown in FIG. 15, for convenience of illustration, one bus is shown as the bus 84, but a plurality of buses may be used.
  • the bus 84 may include a serial bus or a parallel bus composed of a data bus, an address bus, a control bus, and the like.
  • the CPU 78A controls the entire HMD main body 70.
  • the NVM78B stores various parameters, various programs, and the like.
  • An example of the NVM78B is EEPROM.
  • Various information is temporarily stored in the RAM 78C.
  • the RAM 78C is used as a working memory by the CPU 78A.
  • the display 74 is connected to the bus 84. Specifically, the projection unit described above is connected to the bus 84. The display 74 displays various information under the control of the CPU 78A.
  • the HMD camera 76 is connected to the bus 84, and the CPU 78A controls the HMD camera 76.
  • the captured image obtained by being imaged by the HMD camera 76 is acquired by the CPU 78A via the bus 84.
  • the transmission / reception device 82 is connected to the bus 84.
  • the transmission / reception device 82 is a device including a communication processor (not shown), an antenna, and the like, and transmits / receives various information to / from the information processing device 10 via a base station (not shown) under the control of the CPU 78A. .. That is, the CPU 78A exchanges various information with and from the information processing device 10 via the transmission / reception device 82.
  • the reception device 80 is a device including at least one hard key, and receives an instruction from the user 13.
  • the reception device 80 is connected to the bus 84, and the CPU 78A acquires the instruction received by the reception device 80.
  • the NVM78B stores the HMD side processing program 85.
  • the CPU 78A reads the HMD side processing program 85 from the NVM 78B and executes the HMD side processing program 85 on the RAM 78C to perform the HMD side processing (see FIG. 28).
  • the CPU 78A acquires the HMD image 86 (see FIG. 17) from the HMD camera 76.
  • the HMD image 86 is, for example, a live view image.
  • the CPU 78A causes the display 74 to display the HMD image 86.
  • the user 13 can observe the real space through the entire display 74.
  • the HMD image 86 is displayed on the display 74.
  • the HMD image 86 is displayed in a state of being superimposed on a part of the real space area by the user 13 via the display 74. Since the light in the real space also passes through the display area of the HMD image 86, the user 13 can observe the real space through the display area of the HMD image 86.
  • the finger of the user 13 when the finger of the user 13 enters the angle of view of the HMD camera 76, the finger of the user is imaged by the HMD camera 76 and projected as an image in the HMD image 86.
  • the user 13 tentatively indicates a position pointed by the finger as a viewpoint position for the user 13 to observe the inside of the soccer stadium 36. Since the direction in which the finger is actually pointed is deviated from the line-of-sight direction of the user 13, the point pointed by the finger is not the position intended by the user 13 as the viewpoint position.
  • the position intended by the user 13 as the viewpoint position is the tip of the line of sight passing through the fingertip of the user 13.
  • the direction of the optical axis OA of the image pickup optical system of the HMD camera 76 substantially coincides with the line-of-sight direction of the user 13. Therefore, the information processing system 66 is observing a position in the soccer stadium 36 in contact with the optical axis OA when the center of the angle of view and the position of the fingertip of the user 13 coincide with each other, that is, the inside of the soccer stadium 36 at the present time.
  • One point (gaze point) that the user 13 is gazing at is a position tentatively instructed by the user 13 as a viewpoint position for the user 13 to observe the inside of the soccer stadium 36 (hereinafter, also referred to as a "temporary instruction position").
  • the CPU 78A detects the finger of the user 13 using the HMD image 86 obtained by being imaged by the HMD camera 76.
  • the CPU 78A generates an HMD image 88 with a tentatively indicated position (see FIG. 20) when the fingertip of the user 13 is stationary at the center of the angle of view.
  • the state where the fingertip of the user 13 is stationary at the center of the angle of view continues for a predetermined time (for example, 3 seconds). Means that.
  • the CPU 78A transmits the generated HMD image 88 with a temporary instruction position to the information processing device 10 via the transmission / reception device 82 (see FIG. 15).
  • the HMD image 88 with the provisional instruction position is an image in which the provisional instruction position identification information 88A is added to the HMD image 86.
  • the temporary instruction position specifying information 88A is information that can specify the temporary instruction position in the HMD image 86 (for example, the position of the pixel corresponding to the temporary instruction position in the HMD image 86, that is, the image is reflected in the HMD image 86. Information that can identify the position of the pixel corresponding to the position of the fingertip that is crowded).
  • the CPU 22A acquires the HMD image 88 with the provisionally indicated position from the HMD 68.
  • the CPU 22A displays a captured image showing a subject including the temporary designated position specified from the temporary designated position specifying information 88A of the HMD image 88 with the temporary designated position in the soccer stadium 36 from a viewpoint position different from the current viewpoint position of the user 13. Is acquired from the image pickup apparatus 30 as another viewpoint position image 90 (see FIG. 22) showing an aspect in the soccer stadium 36 when the soccer stadium 36 is observed.
  • the CPU 22A acquires a three-dimensional area image 32 from the NVM 22B, and generates another viewpoint position image 92 with a designated position candidate (see FIG. 22) with reference to the acquired three-dimensional area image 32. Specifically, first, the CPU 22A specifies the position of the optical axis OA in the soccer stadium 36 by specifying the feature points that match between the HMD image 88 with the provisionally indicated position and the three-dimensional area image 32. do. Next, the CPU 22A compares the HMD image 88 with the temporary designated position with the three-dimensional area image 32, and extracts the coordinates of the temporary designated position in the soccer stadium 36 from the three-dimensional area image 32 based on the comparison result. .. Next, the CPU 22A generates a plurality of designated position candidates.
  • the plurality of designated position candidates are positions defined by a predetermined interval (for example, an interval of 5 meters on a real space scale) on the optical axis OA including the temporary designated position. Coordinates obtained from the three-dimensional region image 32 are associated with each of the plurality of designated position candidates. Then, the CPU 22A generates the other viewpoint position image 92 (see FIG. 22) with the designated position candidate by adding information such as a plurality of designated position candidates to the other viewpoint position image 90. The CPU 22A transmits the other viewpoint position image 92 with the designated position candidate to the HMD 68 via the transmission / reception device 24 (see FIG. 1).
  • a predetermined interval for example, an interval of 5 meters on a real space scale
  • the optical axis OA is an example of the "first line” according to the technique of the present disclosure
  • the provisional indication position is an example of the "viewpoint” according to the technique of the present disclosure.
  • the other viewpoint position image 92 with the designated position candidate is an image in which a plurality of dot marks 92A and messages 92B are superimposed on the other viewpoint position image 90.
  • a plurality of dot marks 92A are arranged at predetermined intervals on an image showing the optical axis OA, and each dot mark 92A is a mark capable of specifying a designated position candidate. It is not essential to display the image showing the optical axis OA, and the image showing the optical axis OA may not be displayed.
  • Each dot mark 92A is associated with the coordinates obtained from the three-dimensional area image 32 as the coordinates that can specify the position of the designated position candidate.
  • the message 92B is a message prompting the user 13 to select a designated position candidate, and in the example shown in FIG. 22, the message "Please specify any dot (position)" as an example of the message 92B. It is shown.
  • the CPU 78A acquires another viewpoint position image 92 with a designated position candidate from the information processing device 10. Then, as shown in FIG. 24 as an example, the CPU 78A causes the display 74 to display the other viewpoint position image 92 with the designated position candidate.
  • the user 13 intends to move the fingertip to the dot mark 92A of the plurality of dot marks 92A included in the other viewpoint position image 92 with the designated position candidate with respect to the inside of the soccer stadium 36.
  • the observation position intended by the user 13 is an example of the "instructed position", the "observation position", and the "instructed position on the first line” according to the technique of the present disclosure.
  • the CPU 78A acquires the HMD image 86 from the HMD camera 76, and detects the finger of the user 13 using the acquired HMD image 86.
  • the CPU 78A uses information including the coordinates associated with the dot mark 92A at which the fingertip of the user 13 is located as the designated position specifying information 94. It is transmitted to the information processing device 10 via the transmission / reception device 82 (see FIG. 15).
  • the designated position specifying information 94 is information that can identify the designated position candidate selected by the user 13 via the dot mark 92A, that is, a position designated by the user 13 as a viewpoint position for the user 13 to observe the inside of the soccer stadium 36. Is identifiable information.
  • the CPU 22A acquires the designated position specifying information 94 from the HMD 68.
  • the CPU 22A extracts coordinates from the designated position specifying information 94, and generates a virtual viewpoint image 64 (see FIG. 10) using the viewpoint position determined based on the extracted coordinates.
  • the CPU 22A transmits the generated virtual viewpoint image 64 to the user device 12 via the transmission / reception device 24.
  • the virtual viewpoint image 64 is displayed on the display 18 as in the first embodiment.
  • the technique of the present disclosure is not limited to this, and the virtual viewpoint image 64 is transmitted to the HMD 68, and the HMD 68 is used.
  • the virtual viewpoint image 64 may be displayed on the display 74.
  • step ST100 whether or not the virtual viewpoint image 64 transmitted by executing the process of step 214 of the image generation process shown in FIG. 30 is received by the transmission / reception device 44. Is determined. If the virtual viewpoint image 64 is not received by the transmission / reception device 44 in step ST100, the determination is denied and the processing on the user device side shifts to step ST104. When the virtual viewpoint image 64 is received by the transmission / reception device 44 in step ST100, the determination is affirmed, and the processing on the user device side shifts to step ST102.
  • step ST102 the CPU 40A displays the virtual viewpoint image 64 received by the transmission / reception device 44 in step ST100 on the display 18, and then the user device side processing shifts to step ST104.
  • step ST104 the CPU 40A determines whether or not the processing end condition on the user device side is satisfied. If the user device side processing end condition is not satisfied in step ST104, the determination is denied and the user device side processing proceeds to step ST100. If the user device side processing end condition is satisfied in step ST104, the determination is affirmed and the user device side processing ends.
  • step ST150 the CPU 78A acquires the HMD image 86 from the HMD camera 76, and then the HMD side processing shifts to step ST152.
  • step ST152 the CPU 78A displays the HMD image 86 acquired in step 150 on the display 74, and then the HMD side processing shifts to step ST154.
  • step ST154 the CPU 78A executes the finger detection process using the HMD image 86 acquired in step ST152.
  • the finger detection process refers to a process of detecting the finger of the user 13 using the HMD image 86.
  • the HMD side process shifts to step ST156.
  • step ST156 the CPU 78A determines whether or not the finger of the user 13 is detected by the finger detection process in step ST154. If the finger of the user 13 is not detected by the finger detection process of step ST154 in step ST156, the determination is denied and the process proceeds to step ST178 shown in FIG. 29. When the finger of the user 13 is detected by the finger detection process of step ST154 in step ST156, the determination is affirmed, and the HMD side process shifts to step ST158.
  • step ST158 the CPU 78A determines whether or not the fingertip of the user 13 is stationary at the center of the angle of view of the HMD camera 76. In step ST158, if the fingertip of the user 13 is not stationary at the center of the angle of view of the HMD camera 76, the determination is denied and the HMD side processing shifts to step ST150. In step ST158, when the fingertip of the user 13 is stationary at the center of the angle of view of the HMD camera 76, the determination is affirmed and the HMD side processing shifts to step ST160.
  • step ST160 the CPU 78A generates an HMD image 88 with a provisionally indicated position based on the HMD image 86 acquired in step ST150, and then the HMD side processing shifts to step ST162.
  • step ST162 the CPU 78A transmits the HMD image 88 with the temporary instruction position generated in step ST160 to the information processing device 10 via the transmission / reception device 82, and then the HMD side processing shifts to step ST164.
  • step ST164 the CPU 78A determines whether or not the other viewpoint position image 92 with the indicated position candidate transmitted by executing the process of step ST206 shown in FIG. 30 is received by the transmission / reception device 82. If the other viewpoint position image 92 with the designated position candidate is not received by the transmission / reception device 82 in step ST164, the determination is denied and the determination in step ST164 is performed again. In step ST164, when the other viewpoint position image 92 with the designated position candidate is received by the transmission / reception device 82, the determination is affirmed, and the HMD side processing shifts to step ST166.
  • step ST166 the CPU 78A causes the display 74 to display the other viewpoint position image 92 with the indicated position candidate received by the transmission / reception device 82 in step ST164, and then the HMD side processing proceeds to step ST168 shown in FIG. Transition.
  • step ST168 shown in FIG. 29 the CPU 78A acquires the HMD image 86 from the HMD camera 76, and then the HMD side processing shifts to step ST170.
  • step ST170 the CPU 78A executes the finger detection process using the HMD image 86 acquired in step ST168, and then the HMD side process shifts to step ST172.
  • step ST172 the CPU 78A determines whether or not the finger of the user 13 is detected by the finger detection process of step ST170. In step ST172, if the finger of the user 13 is not detected by the finger detection process of step ST170, the determination is denied and the process proceeds to step ST180. In step ST172, when the finger of the user 13 is detected by the finger detection process of step ST170, the determination is affirmed, and the HMD side process shifts to step ST174.
  • step ST180 the CPU 78A determines whether or not the condition for ending the HMD-side processing (hereinafter referred to as "HMD-side processing end condition") is satisfied.
  • HMD-side processing end condition there is a condition that the instruction to end the HMD side processing is accepted by the reception device 80.
  • a third predetermined time for example, 60 minutes
  • a third predetermined time for example, 60 minutes
  • step ST180 If the HMD side processing end condition is not satisfied in step ST180, the determination is denied and the HMD side processing proceeds to step ST168. If the HMD side processing end condition is satisfied in step ST180, the determination is affirmed and the HMD side processing ends.
  • step ST174 the CPU 78A determines whether or not the fingertip (fingertip of the user 13) detected in step ST172 is located at the dot mark 92A on the other viewpoint position image 92 with the indicated position candidate displayed on the display 74. Is determined. If the fingertip of the user 13 is not located at the dot mark 92A on the other viewpoint position image 92 with the designated position candidate in step ST174, the determination is denied and the HMD side processing shifts to step ST168. In step ST174, if the fingertip of the user 13 is located at the dot mark 92A on the other viewpoint position image 92 with the designated position candidate, the determination is denied and the HMD side processing shifts to step ST176.
  • step ST176 the CPU 78A transmits information including the coordinates associated with the dot mark 92A on which the fingertip of the user 13 is located to the information processing device 10 via the transmission / reception device 82 as the instruction position identification information 94. After that, the processing on the HMD side shifts to step ST178.
  • step ST178 the CPU 78A determines whether or not the HMD side processing end condition is satisfied. If the HMD side processing end condition is not satisfied in step ST178, the determination is denied and the HMD side processing proceeds to step ST150 shown in FIG. 28. If the HMD side processing end condition is satisfied in step ST178, the determination is affirmed and the HMD side processing ends.
  • step ST200 the CPU 22A sends the HMD image 88 with the provisional instruction position transmitted by executing the process of step ST162 of the HMD side process shown in FIG. 28 to the transmission / reception device 24. Determine if it was received by. If the HMD image 88 with the provisional instruction position is not received by the transmission / reception device 24 in step ST200, the determination is denied and the image generation process proceeds to step ST208. When the HMD image 88 with the provisional instruction position is received by the transmission / reception device 24 in step ST200, the determination is affirmed, and the image generation process proceeds to step ST202.
  • step ST202 the CPU 22A captures a captured image showing a subject including the temporary instruction position specified from the temporary instruction position identification information 88A of the HMD image 88 with the temporary instruction position received by the transmission / reception device 24 in step ST200. It is acquired from the image pickup apparatus 30 as another viewpoint position image 90 showing an aspect in the soccer stadium 36 when the inside of the soccer stadium 36 is observed from a viewpoint position different from the viewpoint position of 13. After the process of step ST202 is executed, the image generation process shifts to step ST204.
  • step ST204 the CPU 22A acquires a three-dimensional area image 32 from the NVM 22B, generates another viewpoint position image 92 with a designated position candidate by referring to the acquired three-dimensional area image 32, and then the image generation process is performed in step ST206. Move to.
  • step ST206 the CPU 22A transmits the other viewpoint position image 92 with the indicated position candidate generated in step ST202 to the HMD 68 via the transmission / reception device 24, and then the image generation process shifts to step ST208.
  • step ST208 the CPU 22A determines whether or not the instruction position specifying information 94 transmitted by executing the process of step ST176 shown in FIG. 29 is received by the transmission / reception device 24. If the instruction position specifying information 94 is not received by the transmission / reception device 24 in step ST208, the determination is denied and the image generation process proceeds to step ST216. When the instruction position specifying information 94 is received by the transmission / reception device 24 in step ST208, the determination is affirmed and the image generation process proceeds to step ST210.
  • step ST210 the CPU 22A extracts coordinates from the instruction position specifying information 94 received by the transmission / reception device 24 in step ST208, and then the image generation process shifts to step ST212.
  • step ST212 the CPU 22A generates a virtual viewpoint image 64 using the viewpoint position determined based on the coordinates extracted from the designated position specifying information 94 in step ST210, and then the image generation process shifts to step ST214.
  • step ST214 the CPU 22A transmits the virtual viewpoint image 64 generated in step ST212 to the user device 12 via the transmission / reception device 24, and then the image generation process shifts to step ST216.
  • step ST216 the CPU 22A determines whether or not the image generation processing end condition is satisfied. If the condition for ending the image generation process is not satisfied in step ST216, the determination is denied and the image generation process proceeds to step ST200. If the condition for ending the image generation process is satisfied, the determination is affirmed and the image generation process ends.
  • the user 13 enters the soccer stadium 36 from the viewpoint position determined based on the coordinates in the soccer stadium 36 corresponding to the position instructed with respect to the soccer stadium 36.
  • the CPU 22A acquires a virtual viewpoint image 64 showing a subject existing in the soccer stadium 36 when virtually observed.
  • the virtual viewpoint image 64 acquired by the CPU 22A is displayed on the display 18 of the user device 12. Therefore, according to this configuration, the user 13 can observe the inside of the soccer stadium 36 from various positions.
  • the gaze point for example, provisional
  • the viewpoint for example, the position of the HMD camera 76 in the soccer stadium 36
  • the specified position for example, any one of a plurality of designated position candidates
  • the specified position for example, any one of a plurality of designated position candidates
  • the viewpoint position in which the intention of the user 13 is reflected with high accuracy is compared with the case where there is no room to select the designated position corresponding to the viewpoint position used for generating the virtual viewpoint image 64 from a plurality of candidates. It is possible to generate a virtual viewpoint image 64 based on the above.
  • the other viewpoint position image 92 with the indicated position candidate displayed on the display 74 is a two-dimensional image, but the technique of the present disclosure is not limited to this, and the plurality of dot marks 92A, Alternatively, the mark alternative to these (for example, a star mark, a triangle mark, a quadrangle mark, etc.) may be a stereoscopic image visually recognizable via the display 74.
  • the stereoscopic image may be generated based on, for example, a parallax image obtained by using a plurality of phase difference pixels, or may be based on a shake image obtained by applying vibration to the HMD camera 76. May be generated.
  • the CPU 22A of the information processing apparatus 10 has been described with reference to an example of a form in which the HMD image 88 with a provisionally indicated position is acquired from the HMD 68, but the technique of the present disclosure is limited to this. Not done.
  • the CPU 22A of the information processing apparatus 10 may acquire the reference image 96 with a provisional instruction position from the user device 12.
  • the reference image 96 with a tentative instruction position is an image corresponding to the reference image 62 with a tentative instruction position described in the first embodiment, and includes the tentative instruction position identification information 96A.
  • the provisional designated position specifying information 96A is information corresponding to the designated position specifying information 62A described in the first embodiment. That is, the reference image 96 with a provisional instruction position is an image obtained by adding the instruction position identification information 62A described in the first embodiment to the reference image 60 as the provisional instruction position identification information 96A.
  • the CPU 22A is a captured image showing a subject including a temporary designated position (a position corresponding to the designated position described in the first embodiment) specified from the temporary designated position specifying information 96A of the reference image 96 with a temporary designated position.
  • the CPU 22A acquires a three-dimensional area image 32 from the NVM 22B, and generates another viewpoint position image 92 with a designated position candidate by referring to the acquired three-dimensional area image 32. Specifically, first, the CPU 22A specifies the position of the optical axis OA in the soccer stadium 36 by specifying the feature points that match between the reference image 96 with the provisionally indicated position and the three-dimensional area image 32. do. Next, the CPU 22A compares the reference image 96 with the temporary designated position and the three-dimensional area image 32, and extracts the coordinates of the temporary designated position in the soccer stadium 36 from the three-dimensional area image 32 based on the comparison result. .. Next, the CPU 22A generates a plurality of designated position candidates by the method described in the second embodiment.
  • the CPU 22A generates the other viewpoint position image 92 with the designated position candidate by giving information such as a plurality of designated position candidates to the other viewpoint position image 90.
  • the CPU 22A transmits the other viewpoint position image 92 with the designated position candidate to the user device 12 via the transmission / reception device 24.
  • the other viewpoint position image 92 with the designated position candidate is displayed on the display 18 of the user device 12.
  • Information 94 is transmitted to the information processing apparatus 10.
  • the finger of the user 13 is detected based on the HMD image 86 obtained by imaging the finger of the user 13 by the HMD camera 76 .
  • the disclosed technology is not limited to this.
  • the finger of the user 13 may be detected from a plurality of captured images captured by a plurality of image pickup devices 30 installed in the soccer stadium 36.
  • the finger of the user 13 may be detected by the CPU 22A, the CPU 78A, or the like based on the HMD image 86 and the plurality of captured images.
  • the method of detecting the finger of the user 13 is not limited to the above, and the finger of the user 13 may be detected by attaching a known device whose position and direction can be specified to the finger of the user 13.
  • the user 13 can point to the viewpoint position using the finger attached to the device, so that the viewpoint position can be determined in the same manner as in the above embodiment.
  • the user 13 may hold a known device whose position and direction can be specified and point to a specific direction so that the viewpoint position is determined as in the above embodiment.
  • the designated position on the line from the viewpoint where the soccer stadium 36 is observed toward the gazing point that is, the designated position on the optical axis OA is used for generating the virtual viewpoint image 64.
  • the technique of the present disclosure is not limited to this.
  • the designated position on the line from the viewpoint position 56 (see FIG. 4) toward the specified point in the image corresponding to the reference image 60 (for example, on the line of sight of the user 13) is used to generate the virtual viewpoint image 64. It may be the viewpoint position to be used.
  • the other viewpoint position image 98 with the indicated position candidate may be displayed on the display 18 of the user device 12.
  • the other viewpoint position image 98 with the designated position candidate is an image corresponding to the other viewpoint position image 92 with the designated position candidate, and the plurality of dot marks 92A indicate the line of sight 58A of the user 13 not on the image showing the optical axis OA.
  • the point located on the image is different from the other viewpoint position image 92 with the designated position candidate. It is not essential to display the image showing the line of sight 58A, and the image showing the line of sight 58A may not be displayed.
  • the line of sight 58A corresponds to, for example, the optical axis of the image pickup optical system of the image pickup apparatus 42 in the example shown in FIG.
  • the position specified from the coordinates associated with the touched dot mark 92A is generated as the virtual viewpoint image 64. It is the viewpoint position used for. Therefore, according to this configuration, the viewpoint position in which the intention of the user 13 is reflected with high accuracy is compared with the case where there is no room to select the designated position corresponding to the viewpoint position used for generating the virtual viewpoint image 64 from a plurality of candidates. It is possible to generate a virtual viewpoint image 64 based on the above.
  • the CPU 22A has described by giving an example of a form in which the dot mark 92A is associated with the designated position candidate, but the technique of the present disclosure is not limited to this.
  • the CPU 22A may associate the designated position candidate with the thumbnail image 100B (see FIG. 36), which is a reduced version of the virtual viewpoint image 64 when observing the inside of the soccer stadium 36.
  • the thumbnail image 100B in association with the designated position candidate will be described with reference to FIGS. 34 to 38.
  • the CPU 22A in the image generation process of the information processing apparatus 10, the CPU 22A generates another viewpoint position image 100 (see FIG. 36) with a designated position candidate by the method described in the second embodiment. ..
  • the CPU 22A has a plurality of virtual viewpoints using each of the plurality of viewpoint positions determined based on each of the plurality of coordinates associated with the plurality of designated position candidates included in the other viewpoint position image 100 with the designated position candidate.
  • Generate image 64 Then, the CPU 22A stores each of the generated plurality of virtual viewpoint images 64 in the NVM 64 in a state of being associated with the related designated position candidate.
  • the CPU 22A acquires a plurality of virtual viewpoint images 64 from the NVM 64 and generates a plurality of thumbnail images 100B (see FIG. 36) corresponding to the plurality of virtual viewpoint images 64.
  • the CPU 22A associates the thumbnail image 100B with each of the plurality of designated position candidates included in the other viewpoint position image 100 with the designated position candidate.
  • the CPU 22A transmits the other viewpoint position image 100 with the designated position candidate associated with the thumbnail image 100B to the HMD 68 via the transmission / reception device 24 (see FIG. 1).
  • the display 74 of the HMD 68 displays the other viewpoint position image 100 with the designated position candidate.
  • a plurality of dot marks 92A are arranged at predetermined intervals along the image showing the optical axis OA.
  • the display of the image showing the optical axis OA is not essential, and the image showing the optical axis OA may not be displayed.
  • the thumbnail image 100B associated with each of the plurality of dot marks 92A is arranged along the image showing the optical axis OA.
  • the message 100C is attached to the other viewpoint position image 100 with the designated position candidate.
  • the message 100C is a message prompting the user 13 to select a designated position candidate, and in the example shown in FIG. 36, the message "Please specify one of the thumbnail images" is shown as an example of the message 100C. Has been done.
  • the fingertip of the user 13 is positioned on any thumbnail image 100B in a state where the other viewpoint position image 100 with a designated position candidate is displayed on the display 74 of the HMD 68, the fingertip is The positioned thumbnail image 100B is transmitted by the CPU 78A to the information processing device 10 via the transmission / reception device 82 (see FIG. 15).
  • the CPU 22A acquires the thumbnail image 100B from the HMD 68, and acquires the virtual viewpoint image 64 corresponding to the acquired thumbnail image 100B from the NVM 22B. Then, the CPU 22A transmits the virtual viewpoint image 64 acquired from the NVM 22B to the user device 12 via the transmission / reception device 24 (see FIG. 1). As a result, the virtual viewpoint image 64 is displayed on the display 18 of the user device 12.
  • the thumbnail image 100B is associated with each of the plurality of designated position candidates included in the other viewpoint position image 100 with the designated position candidate, and is selected by the user 13.
  • the virtual viewpoint image 64 corresponding to the thumbnail image 100B is displayed on the display 18 of the user device 12. Therefore, according to this configuration, the viewpoint position in which the intention of the user 13 is reflected with high accuracy is compared with the case where there is no room to select the designated position corresponding to the viewpoint position used for generating the virtual viewpoint image 64 from a plurality of candidates. It is possible to generate a virtual viewpoint image 64 based on the above. Further, the user 13 can predict what kind of virtual viewpoint image 64 will be provided from the information processing apparatus 10 through the thumbnail image 100B.
  • one of the thumbnail images 100B is selected by the user 13 while the other viewpoint position image 100 with the indicated position candidate is displayed on the display 74 of the HMD 68.
  • the techniques of the present disclosure are not limited to this.
  • a designated position candidate on a line for example, on an image showing the line of sight of the user 13
  • a designated point for example, a provisional designated position
  • the thumbnail image 100B associated with the user 13 may be selected by the user 13.
  • the other viewpoint position image 102 with the designated position candidate may be displayed on the display 18 of the user device 12.
  • the other viewpoint position image 102 with the designated position candidate is an image corresponding to the other viewpoint position image 100 with the designated position candidate, and the plurality of thumbnail images 100B show the line of sight 58A of the user 13 not on the image showing the optical axis OA.
  • the point located on the image is different from the other viewpoint position image 100 with the designated position candidate.
  • the line of sight 58A corresponds to, for example, the optical axis of the image pickup optical system of the image pickup apparatus 42 in the example shown in FIG.
  • the thumbnail image 100B displayed on the display 18 of the user device 12 is an example of the "second reduced image" according to the technique of the present disclosure.
  • the touched thumbnail images The virtual viewpoint image 64 corresponding to 100B is transmitted to the user device 12 and displayed on the display 18 of the user device 12. Therefore, according to this configuration, the viewpoint position in which the intention of the user 13 is reflected with high accuracy is compared with the case where there is no room to select the designated position corresponding to the viewpoint position used for generating the virtual viewpoint image 64 from a plurality of candidates. It is possible to generate a virtual viewpoint image 64 based on the above. Further, the user 13 can predict what kind of virtual viewpoint image 64 will be provided from the information processing apparatus 10 through the thumbnail image 100B.
  • the information processing system 105 includes an information processing device 10 and a user device 12.
  • the NVM 22B stores an observation range limiting processing program 104 and a three-dimensional area image 106 with spectator seat information.
  • the CPU 22A reads the observation range limiting processing program 104 from the NVM 22B and executes the observation range limiting processing program 104 on the RAM 22C to perform the observation range limiting processing (see FIG. 47).
  • the three-dimensional area image 106 with spectator seat information is an image in which spectator seat information 106A (see FIG. 41) is added to the three-dimensional area image 32 described in the first embodiment.
  • the NVM 22B stores a plurality of three-dimensional area images 106 with spectator seat information.
  • the three-dimensional area image 106 with a plurality of spectator seat information is an image determined for each spectator venue, and is used properly for each spectator venue.
  • the spectator seat 36B (see Fig. 2) is classified by grade.
  • the grade is determined for each purchase amount of the spectator ticket, and in the example shown in FIG. 41, the area of the spectator seat 36B is differentiated by the grades of S seat, A seat, and B seat.
  • the grade of the spectator seat 36B is reflected as the spectator seat information 106A in the three-dimensional area image 106 with the spectator seat information.
  • the spectator seat information 106A is information including grade identification information that can specify the grade of the spectator seat 36B and coordinates that can specify each area divided by grade in the soccer stadium 36.
  • grade identification information that can specify the grade of the spectator seat 36B and coordinates that can specify each area divided by grade in the soccer stadium 36.
  • the position where the user 13 is watching is also given a grade, and the user 13 can watch the game only in an area of the same class. That is, in the information processing system 105, the viewpoint position can be set by the user 13 for the area in the soccer stadium 36 where the user 13 can watch the game, but for the other areas, the viewpoint position can be set. Setting the viewpoint position by the user 13 is prohibited.
  • the user 13 is an example of the “instructor” related to the technology of the present disclosure
  • the grade of the spectator seat 36B is an example of the “attribute” related to the technology of the present disclosure.
  • the grade given to the position where the user 13 is watching the game in the spectator seat 36B is specified based on the live view image obtained by being imaged by the user device 12.
  • the CPU 40A acquires a live view image from the image pickup device 42. Then, the CPU 40A transmits the acquired live view image to the information processing device 10 via the transmission / reception device 44 (see FIG. 3).
  • the CPU 22A acquires a live view image from the user device 12.
  • the CPU 22A acquires a three-dimensional area image 106 with spectator seat information by referring to the live view image acquired from the user device 12.
  • the CPU 22A calculates the degree of matching of the feature points between the live view image and the three-dimensional area image 32 included in the three-dimensional area image 106 with spectator seat information, and based on the calculated degree of matching, the CPU 22A calculates the degree of matching.
  • One three-dimensional area image 106 with spectator seat information is selected and acquired from a plurality of three-dimensional area images 106 with spectator seat information.
  • the CPU 22A acquires the three-dimensional area image 106 with spectator seat information including the three-dimensional area image 32 having the maximum degree of coincidence of the feature points with the live view image.
  • the CPU 22A adds the same grade area information to the three-dimensional area image 106 with spectator seat information.
  • the same grade area information is information (for example, coordinates) that can identify the same grade area 110 (see FIG. 45), which is an area of the same grade as the area that the user 13 is watching.
  • the same grade area information is generated by the CPU 22A based on the live view image and the three-dimensional area image 106 with spectator seat information.
  • the CPU 22A indicates a user watching an area in which the user 13 is watching the image having the highest degree of matching with the live view image among the three-dimensional area images 32 included in the acquired three-dimensional area image 106 with spectator seat information. Specify as an area image.
  • the CPU 22A specifies a grade corresponding to the user spectator area image with reference to the three-dimensional area image 106 with spectator seat information.
  • the CPU 22A identifies an area of the same grade as the specified grade, that is, the same grade area 110 with reference to the three-dimensional area image 106 with spectator seat information.
  • the CPU 22A adds the same-grade area information, which is information that can identify the specified same-grade area 110, to the three-dimensional area image 106 with spectator seat information. Then, the CPU 22A transmits the three-dimensional area image 106 with the battle seat information to which the same grade area information is added to the user device 12 via the transmission / reception device 24 (see FIG. 40).
  • the CPU 40A acquires a three-dimensional area image 106 with spectator seat information to which the same grade area information is added from the information processing device 10 and images the image. Acquire a live view image from the device 42.
  • the CPU 40A generates a reference image 108 (see FIG. 45) which is an image based on the three-dimensional area image 106 with spectator seat information to which the same grade area information is added.
  • the CPU 40A refers to the three-dimensional area image 106 with the same grade area information and the spectator seat information, and generates the reference image 108 using the live view image.
  • the CPU 40A causes the display 18 to display the reference image 108.
  • the reference image 108 includes the target mark 60A and the same grade area 110.
  • the same grade area 110 is shown in a manner distinguishable from other areas in the reference image 108.
  • the same grade area 110 is bordered by a thick line.
  • the viewpoint position can be set by the user 13 only for the same grade area 110. Therefore, even if the user device 12 requests the information processing apparatus 10 to set the viewpoint position in an area other than the same grade area 110 by the same method as in the first embodiment, the information processing apparatus 10 still has the information processing apparatus 10. , Does not respond to the request for setting the viewpoint position from the user device 12.
  • the three-dimensional area image 106 with spectator seat information to which the same grade area information is added is an example of the "three-dimensional area in-mode image" according to the technique of the present disclosure. Further, the same grade area 110 is an example of the "observation position indicating range” according to the technique of the present disclosure.
  • FIG. 46 An example of the flow of processing on the user device side performed by the CPU 40A of the user device 12 will be described with reference to FIG. 46.
  • the flowchart shown in FIG. 46 is different from the flowchart shown in FIG. 11 in that steps ST250 to ST262 are provided in front of step ST10 of the flowchart shown in FIG. In the following, only the steps different from the flowchart shown in FIG. 11 will be described.
  • step ST250 the CPU 40A acquires a live view image from the image pickup device 42, and then the user device side processing shifts to step ST252.
  • step ST252 the CPU 40A transmits the live view image acquired in step ST250 to the information processing apparatus 10 via the transmission / reception device 44 (see FIG. 3), and then the processing on the user device side shifts to step ST254.
  • step ST254 the CPU 40A receives the three-dimensional area image 106 with the same grade area information and the spectator seat information transmitted by executing the process of step ST304 shown in FIG. 47 by the transmission / reception device 44 (see FIG. 3). Determine if it is. If the three-dimensional area image 106 with spectator seat information is not received by the transmission / reception device 44 in step ST254, the determination is denied and the processing on the user device side shifts to step ST262. In step ST254, when the three-dimensional area image 106 with spectator seat information is received by the transmission / reception device 44, the determination is affirmed, and the processing on the user device side shifts to step ST256.
  • step ST262 the CPU 40A determines whether or not the processing end condition on the user device side is satisfied. If the user device side processing end condition is not satisfied in step ST262, the determination is denied and the user device side processing proceeds to step ST254. If the user device side processing end condition is satisfied in step ST262, the determination is affirmed, and the user device side processing proceeds to step ST256.
  • step ST256 the CPU 40A refers to the three-dimensional area image 106 with the same grade area information and the spectator seat information received by the transmission / reception device 44 in step ST254, and uses the live view image acquired in step ST256 as the reference image 108. Is generated, and then the processing on the user device side shifts to step ST260.
  • step ST260 the CPU 40A displays the reference image 108 generated in step ST258 on the display 18, and then the user device side processing shifts to step ST10 (see FIG. 11).
  • step ST300 the CPU 22A receives the live view image transmitted by executing the process of step ST252 shown in FIG. 46 by the transmission / reception device 24 (see FIG. 40). Determine if it has been done. If the live view image is not received by the transmission / reception device 24 in step ST300, the determination is denied and the observation range limiting process proceeds to step ST306. When the live view image is received by the transmission / reception device 24 in step ST300, the determination is affirmed, and the observation range limiting process proceeds to step ST302.
  • step ST302 the CPU 22A acquires a three-dimensional area image 106 with spectator seat information from the NVM22B with reference to the live view image received by the transmission / reception device 24 in step ST300. Further, the CPU 22A generates the same grade area information based on the live view image received by the transmission / reception device 24 in step ST300 and the three-dimensional area image 106 with spectator seat information acquired from the NVM 22B. Then, the CPU 22A adds the same grade area information to the generated three-dimensional area image 106 with spectator seat information. After the process of step ST302 is executed, the observation range limiting process shifts to step ST304.
  • step ST304 the CPU 22A uses the user device 12 to obtain the three-dimensional area image 106 with the same grade area information and the spectator seat information obtained in step ST304, that is, the three-dimensional area image 106 with the spectator seat information to which the same grade area information is added. After that, the observation range limiting process proceeds to step ST306.
  • the CPU 22A determines whether or not the condition for ending the observation range limiting process (hereinafter referred to as "observation range limiting process end condition") is satisfied.
  • observation range control processing end condition there is a condition that the administrator of the information processing apparatus 10 gives an instruction to end the observation range limiting process to the information processing apparatus 10.
  • a fourth predetermined time for example, 10 hours
  • the processing capacity of the CPU 22A has decreased to less than the reference level.
  • step ST306 If the observation range limiting process end condition is not satisfied in step ST306, the determination is denied and the observation range limiting process shifts to step ST300. If the observation range limiting process end condition is satisfied, the determination is affirmed and the observation range limiting process ends.
  • the viewpoint position can be set according to the grade of the spectator seat 36B by adding the same grade area information to the three-dimensional area image 106 with the spectator seat information.
  • the same grade area 110 is determined.
  • the same grade area 110 is reflected in the reference image 108 (see FIG. 45).
  • the reference image 108 reflecting the same grade area 110 is displayed on the display 18 of the user device 12. Therefore, according to this configuration, the user 13 can visually recognize the area where the viewpoint position can be set.
  • the CPU 22A generates a three-dimensional area image 106 with spectator seat information to which the same grade area information is added.
  • the same-grade area information is information that can identify the same-grade area 110, which is an area of the same grade as the area that the user 13 is watching. That is, the three-dimensional area image 106 with spectator seat information to which the same grade area information is added is an image that can distinguish the same grade area 110 from the other areas. Therefore, according to this configuration, as compared with the case where an image in which the same grade area 110 and other areas cannot be distinguished is used, the user 13 has an area in which the viewpoint position can be set and an area in which the viewpoint position cannot be set. Can be easily grasped.
  • the reference image 108 (see FIG. 45) is an image based on the three-dimensional area image 106 with the same grade area information and the spectator seat information. That is, the reference image 108 is an image in which the same grade area 110 can be identified, and is displayed on the display 18 of the user device 12. Therefore, according to this configuration, the user 13 can visually recognize the area where the viewpoint position can be set.
  • the example in which the range in which the observation position can be instructed according to the grade of the spectator seat 36B (in the example shown in FIG. 45, the same grade area 110) is determined by the CPU 22A has been described.
  • the technique of the present disclosure is not limited to this.
  • the team color supported by the user 13 the color that the user 13 likes most, the gender of the user 13, the age group of the user 13, and the user.
  • the range in which the observation position can be instructed may be determined by the CPU 22A according to the attribute of the user 13, such as the clothes of the thirteen.
  • the NVM 22B may store in advance a three-dimensional area image with attribute information to which information that can identify the attribute of the user 13 is added to the three-dimensional area image 32.
  • the range in which the observation position can be instructed according to the grade of the spectator seat 36B is not limited to the same grade area and may be any range. Further, the range in which the observation position can be instructed according to the grade of the spectator seat 36B does not need to be divided by the range of the spectator seat, and may be divided inside, for example, the soccer field 36A. More specifically, for example, the higher the grade of the spectator seat 36B, the closer the observation position to the goal may be instructed.
  • the reference image 108 is displayed on the display 18 of the user device 12 by way of example.
  • the technique of the present disclosure is not limited to this, and the reference image is displayed on the display 74 of the HMD 68. 108 may be displayed.
  • the soccer stadium 36 has been exemplified, but the technique of the present disclosure is not limited to this, and the baseball field, rugby field, curling field, athletic field, swimming field, concert hall, outdoor music field, etc. And, as long as it is a place where a plurality of image pickup devices 30 can be installed, such as a theater venue or the like, it may be any place.
  • the computer 22 is exemplified, but the technique of the present disclosure is not limited to this.
  • a device including an ASIC, FPGA, and / or PLD may be applied.
  • a combination of a hardware configuration and a software configuration may be used. The same applies to the computers 40 and 78.
  • the image generation processing program 34 and the observation range limiting processing program 104 are stored in the NVM 22B, but the present disclosure.
  • the program may be stored in an arbitrary portable storage medium 200 such as an SSD or a USB memory which is a non-temporary storage medium.
  • the program stored in the storage medium 200 is installed in the computer 22, and the CPU 22A determines the image generation process and the observation range control process according to the program (hereinafter, when it is not necessary to distinguish between them, "specification”. "Processing”) is executed.
  • the program is stored in a storage unit such as another computer or a server device connected to the computer 22 via a communication network (not shown), and the program is stored in the information processing device 10 in response to a request from the information processing device 10. It may be downloaded to. In this case, the specific process based on the downloaded program is executed by the CPU 22A of the computer 22.
  • the CPU 22A is exemplified, but the technique of the present disclosure is not limited to this, and a GPU may be adopted. Further, a plurality of CPUs may be adopted instead of the CPU 22A. That is, the specific process may be executed by one processor or a plurality of physically separated processors.
  • processors can be used as hardware resources for executing specific processing.
  • the processor include, as described above, software, that is, a CPU, which is a general-purpose processor that functions as a hardware resource that executes a specific process according to a program.
  • a dedicated electric circuit which is a processor having a circuit configuration specially designed for executing a dedicated process such as FPGA, PLD, or ASIC can be mentioned.
  • a memory is built in or connected to any processor, and each processor executes a specific process by using the memory.
  • a hardware resource that performs a particular process may consist of one of these various processors, or a combination of two or more processors of the same type or dissimilarity (eg, a combination of multiple FPGAs, or a CPU). And FPGA). Further, the hardware resource for executing the specific process may be one processor.
  • one processor is configured by a combination of one or more CPUs and software, and this processor performs specific processing.
  • this processor performs specific processing.
  • SoC there is a form of using a processor that realizes the functions of the entire system including a plurality of hardware resources for executing specific processing with one IC chip.
  • the specific processing is realized by using one or more of the above-mentioned various processors as a hardware resource.
  • a and / or B is synonymous with "at least one of A and B". That is, “A and / or B” means that it may be only A, it may be only B, or it may be a combination of A and B. Further, in the present specification, when three or more matters are connected and expressed by "and / or", the same concept as “A and / or B" is applied.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • Software Systems (AREA)
  • Geometry (AREA)
  • Computing Systems (AREA)
  • Processing Or Creating Images (AREA)
  • User Interface Of Digital Computer (AREA)
PCT/JP2021/028992 2020-09-30 2021-08-04 情報処理装置、情報処理方法、及びプログラム Ceased WO2022070603A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2022553511A JP7571151B2 (ja) 2020-09-30 2021-08-04 情報処理装置、情報処理方法、及びプログラム
US18/184,667 US20230224451A1 (en) 2020-09-30 2023-03-16 Information processing apparatus, information processing method, and program
JP2024177257A JP7767550B2 (ja) 2020-09-30 2024-10-09 情報処理装置、情報処理方法、及びプログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-166413 2020-09-30
JP2020166413 2020-09-30

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/184,667 Continuation US20230224451A1 (en) 2020-09-30 2023-03-16 Information processing apparatus, information processing method, and program

Publications (1)

Publication Number Publication Date
WO2022070603A1 true WO2022070603A1 (ja) 2022-04-07

Family

ID=80951330

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/028992 Ceased WO2022070603A1 (ja) 2020-09-30 2021-08-04 情報処理装置、情報処理方法、及びプログラム

Country Status (3)

Country Link
US (1) US20230224451A1 (https=)
JP (2) JP7571151B2 (https=)
WO (1) WO2022070603A1 (https=)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2022126206A (ja) * 2021-02-18 2022-08-30 キヤノン株式会社 画像処理装置、画像処理方法及びプログラム
US12519924B2 (en) * 2022-08-31 2026-01-06 Snap Inc. Multi-perspective augmented reality experience

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017212592A (ja) * 2016-05-25 2017-11-30 キヤノン株式会社 制御装置、制御方法、及び、プログラム
WO2018030206A1 (ja) * 2016-08-10 2018-02-15 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ カメラワーク生成方法及び映像処理装置
JP2020024619A (ja) * 2018-08-08 2020-02-13 キヤノン株式会社 情報処理装置、情報処理方法、及びプログラム
JP2020135130A (ja) * 2019-02-14 2020-08-31 キヤノン株式会社 画像表示装置、画像表示装置の制御方法及びプログラム

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2005100367A (ja) 2003-09-02 2005-04-14 Fuji Photo Film Co Ltd 画像生成装置、画像生成方法、及び画像生成プログラム
JP7341674B2 (ja) 2019-02-27 2023-09-11 キヤノン株式会社 情報処理装置、情報処理方法及びプログラム

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017212592A (ja) * 2016-05-25 2017-11-30 キヤノン株式会社 制御装置、制御方法、及び、プログラム
WO2018030206A1 (ja) * 2016-08-10 2018-02-15 パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ カメラワーク生成方法及び映像処理装置
JP2020024619A (ja) * 2018-08-08 2020-02-13 キヤノン株式会社 情報処理装置、情報処理方法、及びプログラム
JP2020135130A (ja) * 2019-02-14 2020-08-31 キヤノン株式会社 画像表示装置、画像表示装置の制御方法及びプログラム

Also Published As

Publication number Publication date
US20230224451A1 (en) 2023-07-13
JP7571151B2 (ja) 2024-10-22
JP2025004184A (ja) 2025-01-14
JPWO2022070603A1 (https=) 2022-04-07
JP7767550B2 (ja) 2025-11-11

Similar Documents

Publication Publication Date Title
JP7767550B2 (ja) 情報処理装置、情報処理方法、及びプログラム
KR102506480B1 (ko) 영상 처리 장치 및 그 영상 처리 방법
JP7335335B2 (ja) 情報処理装置、情報処理方法、及びプログラム
JP6253127B2 (ja) 情報提供装置
KR20160061133A (ko) 이미지 표시 방법 및 그 전자 장치
JP7488285B2 (ja) 情報処理装置、情報処理装置の作動方法、及びプログラム
US11670022B2 (en) Electronic device and method for displaying and generating panoramic image
JP7163498B2 (ja) 表示制御装置、表示制御方法、及びプログラム
EP4436159A1 (en) Information processing apparatus, image processing method, and program
US12260544B2 (en) Image restriction method, recording medium, information processing apparatus, and support system
EP3639523B1 (en) Method and apparatus for managing a wide view content in a virtual reality environment
US20230085590A1 (en) Image processing apparatus, image processing method, and program
US20250337854A1 (en) Display terminal, communication system, and display method
JP7754917B2 (ja) 画像処理装置、画像処理方法、及びプログラム
JP2016033800A (ja) 情報処理装置、情報処理システム、その制御方法及びプログラム
US8699797B2 (en) Image processing apparatus, image processing method thereof, and computer-readable storage medium
WO2017096566A1 (zh) 一种显示的方法、装置及系统
KR102176805B1 (ko) 뷰 방향이 표시되는 vr 컨텐츠 제공 시스템 및 방법
US20240015274A1 (en) Image processing apparatus, image processing method, and program
JP7389140B2 (ja) 情報処理装置、情報処理方法、及びプログラム
JP7317119B2 (ja) 情報処理装置、情報処理方法、及びプログラム
JP2018190447A (ja) 情報処理装置、情報処理システム、その制御方法及びプログラム
US12542882B2 (en) Image processing apparatus, image processing method, and program
CN113950609B (zh) 使用信号指纹和会话特定标识符的粗略重定位
CN107833265A (zh) 一种图像切换展示方法和虚拟现实设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21874906

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022553511

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21874906

Country of ref document: EP

Kind code of ref document: A1