US20230079285A1 - Display control device, display control method, and program - Google Patents
Display control device, display control method, and program Download PDFInfo
- Publication number
- US20230079285A1 US20230079285A1 US17/792,564 US202117792564A US2023079285A1 US 20230079285 A1 US20230079285 A1 US 20230079285A1 US 202117792564 A US202117792564 A US 202117792564A US 2023079285 A1 US2023079285 A1 US 2023079285A1
- Authority
- US
- United States
- Prior art keywords
- satellite
- image
- volume data
- small volume
- live view
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 108
- 238000003384 imaging method Methods 0.000 claims abstract description 126
- 230000008569 process Effects 0.000 claims abstract description 96
- 230000000295 complement effect Effects 0.000 claims description 21
- 238000011038 discontinuous diafiltration by volume reduction Methods 0.000 claims description 11
- 238000009966 trimming Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 abstract description 32
- 238000005516 engineering process Methods 0.000 abstract description 20
- 238000004891 communication Methods 0.000 description 79
- 230000005540 biological transmission Effects 0.000 description 21
- 230000008859 change Effects 0.000 description 18
- 238000010191 image analysis Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 7
- 230000006870 function Effects 0.000 description 6
- 230000004044 response Effects 0.000 description 6
- 230000008520 organization Effects 0.000 description 5
- 230000015572 biosynthetic process Effects 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 3
- 239000000470 constituent Substances 0.000 description 2
- 238000012937 correction Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005401 electroluminescence Methods 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000007123 defense Effects 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000000763 evoking effect Effects 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/50—Control of the SSIS exposure
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
- H04N23/661—Transmitting camera control signals through networks, e.g. control via the Internet
-
- G06T5/009—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/90—Dynamic range modification of images or parts thereof
- G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/63—Control of cameras or camera modules by using electronic viewfinders
- H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
- H04N23/632—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/63—Control of cameras or camera modules by using electronic viewfinders
- H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/681—Motion detection
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2625—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects for obtaining an image which is composed of images from a temporal image sequence, e.g. for a stroboscopic effect
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/268—Signal distribution or switching
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/183—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a single remote source
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B64—AIRCRAFT; AVIATION; COSMONAUTICS
- B64G—COSMONAUTICS; VEHICLES OR EQUIPMENT THEREFOR
- B64G1/00—Cosmonautic vehicles
- B64G1/10—Artificial satellites; Systems of such satellites; Interplanetary vehicles
- B64G1/1021—Earth observation satellites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10032—Satellite or aerial image; Remote sensing
- G06T2207/10041—Panchromatic image
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/2187—Live feed
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
- H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
- H04N21/2343—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements
- H04N21/234363—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements by altering the spatial resolution, e.g. for clients with a lower screen resolution
Definitions
- the present technology relates to a display control device, a display control method, and a program, and more particularly, to a display control device, a display control method, and a program enabling images to be checked in real time.
- Remote sensing is performed by imaging the ground with an observation satellite equipped with an imaging device and observing the state of a target area or object (see, for example, PTL 1 and 2).
- an observation satellite equipped with an imaging device and observing the state of a target area or object (see, for example, PTL 1 and 2).
- PTL 1 and 2 the number of small observation satellites orbiting in low orbits is increasing.
- Satellite images captured by observation satellites are downlinked as the artificial satellite passes over a ground station. If the transmission volume of the image data is large, a communication delay will occur, making it difficult to check the image in real time.
- the present technology has been made in view of such a situation, and enables an image to be checked in real time.
- a display control device provides a display control device including: a receiving unit that receives small volume data that is information on current imaging by an artificial satellite; and a control unit that displays a live view image based on the small volume data on a display unit, wherein the receiving unit receives a satellite image corresponding to the live view image as large volume data at a timing different from that of the small volume data.
- a display control method provides a display control method for allowing a display control device to execute: receiving small volume data that is information on current imaging by an artificial satellite; displaying a live view image based on the small volume data on a display unit; and receiving a satellite image corresponding to the live view image as large volume data at a timing different from that of the small volume data.
- a program provides a program for causing a computer to execute: receiving small volume data that is information on current imaging by an artificial satellite; displaying a live view image based on the small volume data on a display unit; and receiving a satellite image corresponding to the live view image as large volume data at a timing different from that of the small volume data.
- small volume data that is information on current imaging by an artificial satellite is received, a live view image based on the small volume data is displayed on a display unit, and a satellite image corresponding to the live view image is received as large volume data at a timing different from that of the small volume data.
- the display control device can be realized by causing a computer to execute a program.
- the program to be executed by the computer can be provided by transmitting through a transmission medium or by recording on a recording medium.
- the display control device may be an independent device or an internal block constituting one device.
- FIG. 1 is a block diagram showing a configuration example of a satellite image processing system according to an embodiment to which the present technology is applied.
- FIG. 2 is a block diagram showing a functional configuration of a satellite management device and an artificial satellite.
- FIG. 3 is a flowchart of a first live view image display process.
- FIG. 4 is a diagram illustrating a process according to a user's control instruction.
- FIG. 5 is a flowchart of a second live view image display process.
- FIG. 6 is a block diagram showing a configuration example of an embodiment of a computer to which the present technology is applied.
- FIG. 1 is a block diagram showing a configuration example of a satellite image processing system according to an embodiment to which the present technology is applied.
- a satellite image processing system 1 of FIG. 1 is a system that enables satellite images captured by one or more artificial satellites 21 (hereinafter, simply referred to as satellite 21 ) to be checked by a device on the ground in real time.
- the satellite 21 is equipped with an imaging device and has at least a function of imaging the ground.
- a satellite operating company has a satellite management device 11 that manages a plurality of satellites 21 , and a plurality of communication devices 13 that communicate with the satellites 21 .
- a part of the satellite management device 11 and the plurality of communication devices 13 may be a device owned by a company other than the satellite operating company.
- the satellite management device 11 and the plurality of communication devices 13 are connected to each other via a predetermined network 12 .
- the communication device 13 is arranged in a ground station (ground base station) 15 . Note that FIG. 1 shows an example in which the number of communication devices 13 is three, that is, communication devices 13 A to 13 C, but the number of communication devices 13 is arbitrary.
- the satellite management device 11 manages a plurality of satellites 21 owned by the satellite operating company. Specifically, the satellite management device 11 acquires related information from an information providing servers 41 of one or more external organizations as necessary, and determines the operation plan of the plurality of satellites 21 owned by the satellite management device 11 . Then, the satellite management device 11 causes a predetermined satellite 21 to perform imaging by transmitting an imaging start instruction to the predetermined satellite 21 via the communication device 13 in response to a customer's request. The satellite management device 11 acquires, displays, or stores a satellite image transmitted from the satellite 21 via the communication device 13 . The acquired satellite image may be provided (transmitted) to the customer after performing predetermined image processing thereon as necessary. The acquired satellite image may be provided (transmitted) to an image analysis server 42 of an image analysis company, and may be provided to a customer after performing predetermined image processing thereon.
- the information providing server 41 provided in the external organization supplies predetermined related information to the satellite management device 11 via a predetermined network in response to a request from the satellite management device 11 or periodically.
- the related information provided from the information providing server 41 includes, for example, the following.
- satellite orbit information hereinafter referred to as TLE information
- NORAD North American Aerospace Defense Command
- meteorological information such as the weather at a predetermined point on the earth and the amount of clouds may be acquired from a meteorological information providing company as an external organization.
- the image analysis server 42 performs predetermined image processing on the satellite image captured by the satellite 21 supplied from the satellite management device 11 via a predetermined network.
- the processed image is provided to a customer of the image analysis company or supplied to the satellite management device 11 of the satellite operating company.
- the image analysis server 42 performs metadata generation processing for adding predetermined metadata to the satellite image captured by the satellite 21 , correction processing such as distortion correction of the satellite image, and image composition processing such as color composition processing.
- the image processing of the satellite image may be performed by the satellite operating company, and in this case, the satellite operating company and the image analysis company are the same.
- the satellite management device 11 and the image analysis server 42 may be realized by one device.
- the communication device 13 communicates with a predetermined satellite 21 designated by the satellite management device 11 via an antenna 14 according to the control of the satellite management device 11 .
- the communication device 13 transmits an imaging start instruction to start imaging of a predetermined area on the ground to a predetermined satellite 21 .
- the communication device 13 receives the satellite image transmitted from the satellite 21 and supplies the same to the satellite management device 11 via the network 12 .
- the transmission from the communication device 13 of the ground station 15 to the satellite 21 is also referred to as an uplink, and the transmission from the satellite 21 to the communication device 13 is also referred to as a downlink.
- the communication device 13 can directly communicate with the satellite 21 and can also communicate with a relay satellite 22 .
- a relay satellite 22 for example, a geostationary satellite is used.
- the network 12 or the network between the information providing server 41 or the image analysis server 42 and the satellite management device 11 is an arbitrary communication network, which may be a wired communication network or a wireless communication network or may be configured by both of them.
- the network 12 and the network between the information providing server 41 or the image analysis server 42 and the satellite management device 11 may be configured by one communication network, or may be configured by a plurality of communication networks.
- These networks may be communication networks or communication channels of any communication standard, such as, for example, the Internet, a public telephone network, a wide area communication network for wireless mobiles such as so-called 4G line and 5G line, a wireless communication network that performs communication compliant with WAN (Wide Area Network), LAN (Local Area Network), and Bluetooth (registered trademark) standards, a communication channel for short-range wireless communication such as NFC (Near Field Communication), a communication channel for Infrared communication, a communication network for wired communication compliant with standards such as HDMI (registered trademark) (High-Definition Multimedia Interface) and USB (Universal Serial Bus).
- any communication standard such as, for example, the Internet, a public telephone network, a wide area communication network for wireless mobiles such as so-called 4G line and 5G line, a wireless communication network that performs communication compliant with WAN (Wide Area Network), LAN (Local Area Network), and Bluetooth (registered trademark) standards, a communication channel for short-range wireless communication such
- Each satellite 21 may be operated in a single unit or a plurality of units.
- a plurality of satellites 21 operated in a plurality of units constitute one satellite group 31 .
- satellites 21 A and 21 B are operated as a single unit, and satellites 21 C and 21 D form one satellite group 31 A.
- FIG. 1 for the sake of simplicity, an example in which one satellite group 31 is composed of two satellites 21 is shown, but the number of satellites 21 constituting one satellite group 31 is not limited to two.
- Systems that operate a plurality of satellites 21 as one unit include constellation and formation flight.
- Constellation is a system that deploys services mainly globally by launching a large number of satellites 21 into a single or a plurality of orbital planes. Even a single satellite has a predetermined function, and a plurality of satellites 21 are operated for the purpose of improving the observation frequency and the like.
- the formation flight is a system that deploys a plurality of satellites 21 in a narrow area of about several kilometers while maintaining a relative positional relationship. Formation flight can provide services that cannot be realized by a single satellite, such as high-precision 3 D measurement and speed detection of moving objects. In the present embodiment, it does not matter whether the operation of the satellite group is a constellation or a formation flight.
- a method of directly communicating with the satellite 21 as in the satellite 21 A and the satellite 21 B, and a method of indirectly communicating with the communication device 13 by performing inter-satellite communication with a satellite 21 C which is the other satellite 21 , as in a satellite 21 D may be used.
- the method of indirectly communicating includes communication via the relay satellite 22 . Which method is used to communicate with the ground station 15 (the communication device 13 thereof) may be determined in advance by the satellite 21 or may be appropriately selected according to the content of the communication.
- the satellite 21 as an observation satellite images a predetermined point on the ground based on the imaging start instruction from the satellite management device 11 . If the data of the image captured by the satellite 21 is transmitted as it is, a delay occurs due to a large data volume, and it is difficult to observe the image as a live view image in real time.
- the satellite image processing system 1 is configured so that an image similar to the image captured by the satellite 21 can be observed in real time by the satellite management device 11 on the ground.
- FIG. 2 is a block diagram showing a functional configuration of the satellite management device 11 and the satellite 21 regarding the observation of the image captured by the satellite 21 .
- the satellite 21 includes an antenna AT, a satellite communication unit 61 , an imaging device 62 , a control unit 63 , and a storage unit 64 .
- the configuration of the satellite 21 is mainly for functions related to images only, and although not shown, the satellite 21 is also equipped with propulsion devices such as solid-state motors and ion engines regarding attitude control, sensors such as a GPS receiver, a star tracker (attitude sensor), an accelerometer, and a gyro sensor regarding position control, and a power supply such as batteries and solar panels.
- propulsion devices such as solid-state motors and ion engines regarding attitude control
- sensors such as a GPS receiver, a star tracker (attitude sensor), an accelerometer, and a gyro sensor regarding position control
- a power supply such as batteries and solar panels.
- the satellite communication unit 61 Based on the control of the control unit 63 , the satellite communication unit 61 transmits image data of the image captured by the imaging device 62 , state data indicating the state of the satellite 21 at the time of imaging, and the like to the communication device 13 of the ground station 15 via the antenna AT.
- the data transmitted from the satellite communication unit 61 to the communication device 13 is supplied from the communication device 13 to the satellite management device 11 .
- the imaging device 62 is configured of, for example, a camera module including an image sensor, and images an object based on the control of the control unit 63 .
- the imaging device 62 is configured of a radar device.
- the control unit 63 controls the operation of the entire satellite 21 .
- the control unit 63 causes the imaging device 62 to perform imaging based on the imaging start instruction from the satellite management device 11 .
- the control unit 63 stores the satellite image obtained by imaging as a large volume data in a storage unit 64 , and also performs a volume reduction process for generating a small volume data having a volume smaller than that of the satellite image.
- the small volume data is, for example, an image obtained by converting the satellite image captured by the imaging device 62 into a small volume data, and state data indicating the state of the satellite 21 when the imaging device 62 performs imaging.
- the storage unit 64 stores control programs executed by the control unit 63 and parameters.
- the storage unit 64 stores the image data (large volume data) captured by the imaging device 62 and the small volume data generated by the control unit 63 , and supplies the same to the satellite communication unit 61 or the control unit 63 as necessary.
- the satellite management device 11 includes a control unit 81 , a communication unit 82 , a storage unit 83 , an operation unit 84 , and a display unit 85 .
- the control unit 81 manages a plurality of satellites 21 owned by the satellite operating company by executing a satellite management application program stored in the storage unit 83 .
- the control unit 81 determines the operation plan of the plurality of satellites 21 using the related information acquired from the information providing server 41 as necessary, and sends an attitude control signal and an imaging instruction to each satellite 21 via the communication device 13 .
- the control unit 81 performs processing such as displaying an observation image on the display unit 85 based on the large volume data and the small volume data of the satellite image transmitted from the satellite 21 via the communication device 13 .
- the communication unit 82 performs predetermined communication with the communication device 13 via the network 12 and performs predetermined communication with the image analysis server 42 according to the instruction of the control unit 81 .
- the communication unit 82 receives large volume data and small volume data related to satellite images transmitted from the satellite 21 .
- the storage unit 83 stores large volume data and small volume data related to the satellite image transmitted from the satellite 21 in accordance with the instruction of the control unit 81 .
- the operation unit 84 is configured of, for example, a keyboard, a mouse, a touch panel, or the like, and receives commands and data inputs based on user (operator) operations and supplies the same to the control unit 81 .
- the display unit 85 is configured of, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.
- the display unit 85 displays the screen of the satellite management application program, displays a satellite image based on a large volume data transmitted from the satellite 21 , a live view image based on a small volume data transmitted from the satellite 21 , and the like.
- the satellite management device 11 functions as a display control device that displays a satellite image captured by the imaging device 62 of the satellite 21 on a predetermined display unit (the display unit 85 or an external display device) based on a user's operation.
- This process starts, for example, when the user who operates the satellite management device 11 performs a live view image display start operation.
- step S 11 the control unit 81 of the satellite management device 11 transmits an imaging start instruction to the satellite 21 via the communication unit 82 .
- step S 41 the control unit 63 of the satellite 21 receives the imaging start instruction transmitted from the satellite management device 11 via the satellite communication unit 61 . Then, in step S 42 , the control unit 63 performs imaging to generate a high resolution image having a resolution of the normal performance of the imaging device 62 .
- the imaging device 62 performs imaging to generate a 4K resolution when it can perform imaging at a maximum of 4K resolution, and performs imaging to generate an HD resolution when it can perform imaging at a maximum of HD resolution.
- the image data of the high resolution image obtained by imaging is stored in the storage unit 64 .
- the image data of the high resolution image may be stored in the storage unit 64 as RAW data, or may be stored in the storage unit 64 as coded data encoded by predetermined coding.
- step S 43 the control unit 63 executes a volume reduction process for generating small volume data having a volume smaller than that of the high resolution image based on the high resolution image obtained by imaging.
- the information indicating the satellite state when the satellite 21 performed imaging is generated as small volume data.
- the information indicating the satellite state includes, for example, attitude information indicating the attitude of the satellite 21 when the satellite 21 performed imaging, and setting information when the imaging device 62 performs imaging (hereinafter referred to as camera setting information).
- the camera setting information is, for example, information related to camera setting values such as resolution, zoom magnification, shutter speed, sensitivity, and aperture.
- the imaging range (angle of view) of the imaging device 62 can be specified based on the attitude of the satellite 21 and the zoom setting value of the imaging device 62 .
- the attitude information of the satellite 21 can be also regarded as a part of the camera setting information.
- a low volume image obtained by converting a high resolution image obtained by imaging into a low volume image may be generated as small volume data.
- a low resolution image obtained by converting a high resolution image to a low resolution such as VGA a panchromatic image obtained by converting a color high resolution image into a monochrome image
- image feature amount information obtained by extracting a feature amount of a characteristic subject included in a high resolution image a partial image obtained by trimming only a part of a high resolution image, and the like can be generated as small volume data.
- Examples of image feature amount information obtained by extracting the feature amount of a high resolution image include, for example, information indicating the characteristics of the shape of clouds, information on dynamic subjects such as airplanes, information on the tint of light rays such as red sunset, and information on the reflection position of the ground surface due to sunlight such as the reflection state of the sea or lake.
- a low frame rate image obtained by converting the frame rate of a high resolution image obtained by imaging to a lower frame rate may be generated as small volume data.
- the control unit 63 generates a low frame rate image obtained by converting the frame rate of a high resolution image of 30 fps into a frame rate of 1 fps as small volume data.
- the small volume data is data generated by performing the volume reduction process on information regarding the current imaging by the satellite 21 .
- step S 44 the control unit 63 transmits the small volume data generated by the volume reduction process to the satellite management device 11 via the satellite communication unit 61 .
- the satellite 21 repeatedly executes the processes of steps S 42 to S 44 described above until an imaging end instruction is transmitted from the satellite management device 11 .
- steps S 42 to S 44 small volume data is transmitted to the satellite management device 11 at a predetermined frame rate. Since the small volume data has a small volume, it can be transmitted in frame units without delay.
- step S 12 the control unit 81 of the satellite management device 11 receives the small volume data transmitted from the satellite 21 via the communication unit 82 .
- step S 13 a complementary process for complementing the small volume data is executed to generate a live view image.
- step S 14 the control unit 81 causes the display unit 85 to display the generated live view image.
- the control unit 81 acquires related information from one or more information providing servers 41 of the external organization as necessary.
- Related information includes, for example, the following.
- the control unit 81 can acquire the TLE information as related information and specify the position of the satellite 21 at the imaging time.
- the control unit 81 acquires, as the related information, archive images (baseline images) which are satellite images the same imaging point of the past (for example, a few days ago or a month ago), accumulated by operating companies that operate other satellite observation services, and quasi-real-time images that are satellite images of the same imaging point at a relatively close time of several tens of minutes to several hours before the current time.
- archive images baseline images
- quasi-real-time images that are satellite images of the same imaging point at a relatively close time of several tens of minutes to several hours before the current time.
- the archive image or the quasi-real-time image to be acquired has the same season, imaging time, weather conditions, and the like as when the small volume data is generated.
- the control unit 81 can acquire meteorological information such as the weather, the cloud distribution, the cloud amount, and the position information of the sun at the same imaging point when the small volume data is generated from the meteorological information providing company as an external organization.
- the control unit 81 can acquire AIS (Automatic Identification System) information indicating the positions of ships and airplanes at the imaging time as the related information from the information providing server 41 of an operation information providing company that provides the operation information of ships and airplanes.
- AIS Automatic Identification System
- the control unit 81 executes a complementary process using the above-mentioned related information as necessary, and generates a live view image in which an image (satellite image) captured by the satellite 21 is estimated from the small volume data.
- the control unit 81 uses the TLE information, the baseline image, the quasi-real-time image, and the like as related information, and reproduces the image captured by the imaging device 62 by CG (computer graphics) or the like at the angle of view captured by the imaging device 62 to generate a live view image.
- the control unit 81 when the small volume data is a low resolution image obtained by converting a high resolution image to a low resolution such as VGA, the control unit 81 generates a high resolution image from the low resolution image using super-resolution technology to obtain a live view image.
- a baseline image or a quasi-real-time image may be acquired as related information and interpolation processing or the like may be performed thereon.
- the control unit 81 when the small volume data is a panchromatic image obtained by converting a color high resolution image into a monochrome image, the control unit 81 generates a color high resolution image from the panchromatic image using machine learning to obtain a live view image. At this time, a colorization process may be performed using the image feature amount information extracted from the feature amount included in the high resolution image.
- the control unit 81 reproduces a characteristic subject from Information on cloud shape, dynamic subject, and tint of light rays and information on reflection position of sunlight as image feature amount information and superimposes the characteristic subject on an archive image or a quasi-real-time image to generate a live view image.
- a panchromatic image and image feature amount information are acquired as small volume data, for example, a process of adding the tint of light rays to the panchromatic image or a process of superimposing a cloud shape and a dynamic subject on the panchromatic image may be performed.
- External information can be also used for meteorological information and the position of dynamic subjects.
- the control unit 81 when the small volume data is a low frame rate image, the control unit 81 generates a live view image by interpolating and generating an image between the frames of the acquired low frame rate image to increase the frame rate.
- the frame rate of the live view image does not have to be the same as the frame rate of the high resolution image.
- the complementary process of step S 13 may be omitted.
- the complementary process may not be performed and the low resolution image or the panchromatic image may be displayed on the display unit 85 as it is as a live view image.
- the control unit 81 may display the live view image displayed on the display unit 85 so that the user can understand that the live view image is an image estimated from the small volume data. For example, information (characters) indicating that it is an estimated image may be superimposed on the live view image, or an outer frame image indicating that it is an estimated image may be added and displayed on the outer periphery of the live view image.
- step S 15 the control unit 81 determines whether the user's operation on the satellite 21 has been performed by the operation unit 84 . When it is determined in step S 15 that the user's operation has not been performed, the process returns to step S 12 , and the processes of steps S 12 to S 15 described above are repeated.
- step S 15 when it is determined in step S 15 that the user's operation has been performed, the process proceeds to step S 16 , and the control unit 81 transmits a control instruction corresponding to the user's operation to the satellite 21 via the communication unit 82 .
- Examples of the user's operation in step S 15 include the following operations.
- the user may perform an operation of instructing a change of camera setting information such as a resolution, a zoom magnification, a shutter speed, a sensitivity, and an aperture.
- an operation of changing the imaging direction (imaging point) of the imaging device 62 may be performed
- the imaging device 62 is attached to the satellite 21 at a fixed position
- the attitude of the satellite 21 itself is changed in response to an instruction to change the imaging direction.
- the imaging device 62 has a movable portion capable of changing the relative position with respect to the satellite 21
- the attitude of the satellite 21 itself may be changed in response to an instruction to change the imaging direction, or the relative position of the imaging device 62 may change.
- the zoom magnification may be either optical zoom or digital zoom.
- the user may give a live view image transmission instruction requesting a large volume data which is a high resolution image corresponding to the live view image displayed on the display unit 85 .
- control unit 63 of the satellite 21 receives the control instruction transmitted from the satellite management device 11 via the satellite communication unit 61 in step S 45 .
- step S 46 the control unit 63 determines whether the control instruction from the satellite management device 11 is a large volume data transmission instruction.
- step S 46 When it is determined in step S 46 that the control instruction from the satellite management device 11 is not a large volume data transmission instruction, the process proceeds to step S 47 , and the control unit 63 performs control based on the control instruction.
- the control instruction is a change of camera setting information such as resolution and zoom magnification
- the control unit 63 performs control to change a part of the camera setting information of the imaging device 62 to a designated setting value.
- the camera setting value of the imaging device 62 for capturing a high resolution image which is repeatedly executed in steps S 42 to S 44 described above, is changed.
- step S 42 when the imaging device 62 changes a predetermined camera setting value to a plurality of values, and performs bracket shooting in which imaging is performed with a plurality of different camera setting values, that is, when the imaging device 62 performs bracket shooting with the camera setting information instructed to be changed by the user, small volume data based on a high resolution image captured with the camera setting value instructed to be changed by the user can be immediately transmitted to the satellite management device 11 .
- the data transmitted as small volume data is partial image data in which only a partial region of a high resolution image is trimmed
- the user's camera setting information change instruction is to change the zoom magnification or the angle of view
- a partial image having a zoom magnification and an angle of view after the change can be generated using the high resolution image before trimming, and can be immediately transmitted to the satellite management device 11 as small volume data.
- the user issues a camera setting information change instruction to move the angle of view to the right while looking at the partial image C 1 while observing the partial image C 1 obtained by trimming only the central portion of the high resolution image B 1 on the display unit 85 .
- the control unit 63 of the satellite 21 receives the camera setting information change instruction, and starts the attitude control to change the attitude of the imaging device 62 in the instructed direction.
- the angle of view of the high resolution image B 2 captured by the imaging device 62 at time t 2 is substantially the same as the angle of view of the high resolution image B 1 at time t 1 .
- the control unit 63 trims the partial image C 1 corresponding to the angle of view instructed by the user from the high resolution image B 2 captured by the imaging device 62 at time t 2 , and immediately transmits the same to the satellite management device 11 as small volume data.
- the partial image C 3 obtained by trimming only the central portion of the high resolution image B 3 is transmitted to the satellite management device 11 as small volume data.
- the partial image having the angle of view designated by the user can be displayed on the display unit 85 before the attitude of the imaging device 62 reaches the state instructed to be changed by the user, it is possible to reduce the delay felt by the user checking the live view image.
- step S 46 when it is determined in step S 46 that the control instruction from the satellite management device 11 is a large volume data transmission instruction, the process proceeds to step S 48 , and the control unit 63 transmits the instructed large volume data to the satellite management device 11 via the satellite communication unit 61 . That is, the data of the high resolution image corresponding to the live view image displayed on the display unit 85 in the process of step S 14 is transmitted from the satellite 21 to the satellite management device 11 .
- step S 16 After the control instruction corresponding to the user's operation is transmitted to the satellite 21 in step S 16 , the process proceeds to step S 17 , and the control unit 81 determines whether the control instruction transmitted to the satellite 21 is a large volume data transmission instruction.
- step S 17 When it is determined in step S 17 that the control instruction transmitted to the satellite 21 is not a large volume data transmission instruction, the process returns to step S 12 , and the processes after step S 12 described above are executed again.
- step S 17 when it is determined in step S 17 that the control instruction transmitted to the satellite 21 is a large volume data transmission instruction, the process proceeds to step S 18 , and the control unit 81 receives the large volume data transmitted from the satellite 21 in response to the transmitted control instruction and stores the same in the storage unit 83 . After step S 18 , the process returns to step S 12 , and the processes after step S 12 described above are executed again.
- the first live view image display process is continuously executed until the user performs a live view image display end operation, and when the live view image display end operation is performed, the first live view image display process ends.
- the satellite management device 11 receives small volume data having a volume smaller than that of a high resolution image having the resolution of the normal performance of the imaging device 62 from the satellite 21 . Since the small volume data has a small volume, it can be transmitted in frame units without delay, so that the live view image based on the small volume data can be displayed in real time. Then, based on the instruction of the user who has checked the live view image displayed on the display unit 85 , large volume data which is a high resolution image corresponding to the live view image is received and stored.
- the transmission and reception of large volume data which is the process of steps S 18 and S 48 described above, may be executed at a timing other than during the execution of the first live view image display process described above, in other words, at a timing other than when the live view image is observed in real time.
- the transmission and reception of large volume data may be performed in the background different from the transmission and reception of small volume data.
- the number of times the large volume data is transmitted is smaller than that of small volume data.
- the small volume data is data of 30 fps
- the large volume data is data of 1 fps, or data transmitted once every 10 minutes.
- the large volume data transmitted and received in the background can be used as a quasi-real-time image in the complementary process of step S 13 described above.
- the large volume data can be used as teacher data when machine learning for estimating a live view image is performed as the complementary process. Using machine learning to generate the live view image, the accuracy of the live view image generated by estimation can be improved.
- the satellite 21 captures a high resolution image based on an imaging start instruction from the satellite management device 11 , generates small volume data based on the high resolution image, and transmits the same to the satellite management device 11 .
- the high resolution image is stored in the storage unit 64 , and is transmitted to the satellite management device 11 as large volume data at another timing as necessary.
- the satellite 21 does not capture a high resolution image, but performs imaging for generating small volume data. Then, the satellite 21 captures the high resolution image corresponding to the live view image and transmits the same as large volume data only when the user observing the live view image gives an instruction to capture the high resolution image.
- the second live view image display process will be described with reference to the flowchart of FIG. 5 .
- This process starts, for example, when the user who operates the satellite management device 11 performs a live view image display start operation.
- step S 61 the control unit 81 of the satellite management device 11 transmits an imaging start instruction to the satellite 21 via the communication unit 82 .
- step S 81 the control unit 63 of the satellite 21 receives the imaging start instruction transmitted from the satellite management device 11 via the satellite communication unit 61 . Then, in step S 82 , the control unit 63 executes a small volume data generation process.
- the same small volume data as the first live view image display process described above is generated.
- information indicating the satellite state when the satellite 21 performed imaging is generated as small volume data.
- a low volume image having a smaller volume than the high resolution image captured with the normal performance of the imaging device 62 for example, a low resolution image having a lower resolution than a color high resolution image, a panchromatic image obtained by converting a color high resolution image into a monochrome image, image feature amount information obtained by extracting the feature amount of a characteristic subject included in a high resolution image, a partial image corresponding to a partial region of a high resolution image, and a low frame rate image whose frame rate is lower than that of a high resolution image, is generated as the small volume data.
- step S 83 the control unit 63 transmits the small volume data generated by the small volume data generation process to the satellite management device 11 via the satellite communication unit 61 .
- the satellite 21 repeatedly executes the processes of steps S 82 and S 83 until the imaging end instruction is transmitted from the satellite management device 11 .
- steps S 82 and S 83 small volume data is transmitted to the satellite management device 11 at a predetermined frame rate. Since the small volume data has a small volume, it can be transmitted in frame units without delay.
- step S 62 the control unit 81 of the satellite management device 11 receives the small volume data transmitted from the satellite 21 via the communication unit 82 .
- step S 63 a complementary process for complementing the small volume data is executed to generate a live view image.
- step S 64 the control unit 81 causes the display unit 85 to display the generated live view image.
- step S 63 The details of the complementary process in step S 63 and the display of the live view image in step S 64 are the same as those of the first live view image display process described above, and thus the description thereof will be omitted. Moreover, the complementary process of step S 63 may be omitted.
- step S 65 the control unit 81 determines whether the user's operation on the satellite 21 has been performed by the operation unit 84 . When it is determined in step S 65 that the user's operation has not been performed, the process returns to step S 62 , and the processes of steps S 62 to S 65 described above are repeated.
- step S 65 when it is determined in step S 65 that the user's operation has been performed, the process proceeds to step S 66 , and the control unit 81 transmits a control instruction corresponding to the user's operation to the satellite 21 via the communication unit 82 .
- the operation that the user can perform while observing the live view image is the same as that of the first live view image display process described above.
- control unit 63 of the satellite 21 receives the control instruction transmitted from the satellite management device 11 via the satellite communication unit 61 in step S 84 .
- step S 85 the control unit 63 determines whether the control instruction from the satellite management device 11 is a large volume data transmission instruction.
- step S 85 When it is determined in step S 85 that the control instruction from the satellite management device 11 is not a large volume data transmission instruction, the process proceeds to step S 86 , and the control unit 63 performs control based on the control instruction.
- the control instruction is a change of camera setting information such as resolution and zoom magnification
- the control unit 63 performs control to change a part of the camera setting information of the imaging device 62 to a designated setting value.
- the camera setting value of the imaging device 62 for generating small volume data which is repeatedly executed in steps S 82 and S 83 described above, is changed.
- step S 85 when it is determined in step S 85 that the control instruction from the satellite management device 11 is a large volume data transmission instruction, the process proceeds to step S 87 , and the control unit 63 performs imaging to generate a high resolution image having the resolution of the normal performance of the imaging device 62 .
- the generated high resolution image is stored in the storage unit 64 .
- step S 88 the control unit 63 transmits the image data of the generated high resolution image as large volume data to the satellite management device 11 via the satellite communication unit 61 .
- step S 66 After the control instruction corresponding to the user's operation is transmitted to the satellite 21 in step S 66 , the process proceeds to step S 67 , and the control unit 81 determines whether the control instruction transmitted to the satellite 21 is a large volume data transmission instruction.
- step S 67 When it is determined in step S 67 that the control instruction transmitted to the satellite 21 is not a large volume data transmission instruction, the process returns to step S 62 , and the processes after step S 62 described above are executed again.
- step S 67 when it is determined in step S 67 that the control instruction transmitted to the satellite 21 is a large volume data transmission instruction, the process proceeds to step S 68 , and the control unit 81 receives the large volume data transmitted from the satellite 21 according to the transmitted control instruction and stores the same in the storage unit 83 . After step S 68 , the process returns to step S 62 , and the processes after step S 62 described above are executed again.
- the second live view image display process is continuously executed until the user performs a live view image display end operation, and when the live view image display end operation is performed, the second live view image display process ends.
- transmission and reception of large volume data which is the process of steps S 68 and S 88 described above, may be executed at a timing other than during the execution of the second live view image display process described above, in other words, at a timing other than when the live view image is observed in real time.
- the satellite image processing system 1 According to the first and second live view image display processes executed by the satellite image processing system 1 , small volume data is transmitted when observing a live view image, and a high resolution image is transmitted to the satellite management device 11 as large volume data at another timing as necessary. In this way, the user can check the image on the ground in real time.
- the present technology can be applied not only to satellite image processing systems that remotely instruct imaging by artificial satellites, but also to remote control and remote monitoring of mobile systems operated in remote areas.
- the mobile system is a system including, for example, a robot, particularly a remote-existing robot (telexistence robot), an autonomous traveling vehicle, an autonomous aircraft (drone), a manned traveling vehicle, or a manned aircraft as a mobile device.
- a robot particularly a remote-existing robot (telexistence robot), an autonomous traveling vehicle, an autonomous aircraft (drone), a manned traveling vehicle, or a manned aircraft as a mobile device.
- a user who operates and monitors the mobile device from a remote location can check the minimum situation using a live view image with small volume data.
- the live view image can be generated by executing the complementary process for complementing the small volume data, as in the case of the satellite image processing system 1 described above.
- an assumed image can be reproduced (estimated) based on the attitude information, position information, and the like of the mobile device.
- the movement route and existence environment of the mobile device are known, or when an archive image of the same point in the past or a quasi-real-time image of the same point taken at a relatively close time is accumulated as information on the movement route and existence environment, they can be used for complementation.
- a live view image having a high resolution may be generated using super-resolution technology based on VGA images, a color high resolution image may be generated from a panchromatic image, and low frame rate images may be interpolated to generate a live view image having a high frame rate.
- a live view image based on the feature amount information can be also generated. For example, when the movement route and existence environment of the mobile device are known, or when information on the movement route and existence environment is accumulated, only the real-time feature amount of a subject unique in the movement route and existence environment may be extracted from an archive image to generate an estimated image.
- the above-described series of processing executed by the satellite management device 11 can also be performed by hardware or software.
- a program including the software is installed in a computer.
- the computer includes a computer which is embedded in dedicated hardware or, for example, a general-purpose personal computer capable of executing various functions by installing various programs.
- FIG. 6 is a block diagram showing a configuration example of hardware of a computer that performs, by means of a program, a series of processing executed by the satellite management device 11 .
- a central processing unit (CPU) 301 In the computer, a central processing unit (CPU) 301 , a readonly memory (ROM) 302 , and a random access memory (RAM) 303 are connected to each other by a bus 304 .
- CPU central processing unit
- ROM readonly memory
- RAM random access memory
- An input/output interface 305 is further connected to the bus 304 .
- An input unit 306 , an output unit 307 , a storage unit 308 , a communication unit 309 , and a drive 310 are connected to the input/output interface 305 .
- the input unit 306 is, for example, a keyboard, a mouse, a microphone, a touch panel, or an input terminal.
- the output unit 307 is, for example, a display, a speaker, or an output terminal.
- the storage unit 308 is, for example, a hard disk, a RAM disc, or a nonvolatile memory.
- the communication unit 309 is a network interface or the like.
- the drive 310 drives a removable recording medium 311 such as a magnetic disk, an optical disc, a magneto-optical disc, or a semiconductor memory.
- the CPU 301 performs the above-described series of processing, for example, by loading a program stored in the storage unit 308 on the RAM 303 via the input/output interface 305 and the bus 304 and executing the program.
- the RAM 303 data or the like necessary for the CPU 301 to perform various kinds of processing is also appropriately stored.
- the program executed by the computer can be recorded on, for example, the removable recording medium 311 serving as a package medium for supply.
- the program can be supplied via a wired or wireless transfer medium such as a local area network, the Internet, or digital satellite broadcasting.
- the computer by mounting the removable recording medium 311 on the drive 310 , it is possible to install the program in the storage unit 308 via the input/output interface 305 .
- the program can be received by the communication unit 309 via a wired or wireless transfer medium to be installed in the storage unit 308 .
- the program can be installed in advance in the ROM 302 or the storage unit 308 .
- the steps having been described in the flowcharts may be carried out in parallel or with necessary timing, for example, when evoked, even if the steps are not executed in time series along the order having been described therein, as well as when the steps are executed in time series.
- the system means a set of a plurality of constituent elements (devices, modules (components), or the like) and all the constituent elements may be included or not included in the same casing. Accordingly, a plurality of devices accommodated in separate casings and connected via a network and one device in which a plurality of modules are accommodated in one casing may all be a system.
- the present technology may have a configuration of clouding computing in which a plurality of devices share and process one function together via a network.
- each step described in the above flowchart can be executed by one device or shared by a plurality of devices.
- one step includes a plurality of processes
- the plurality of processes included in the one step can be executed by one device or shared and executed by a plurality of devices.
- the present technology can employ the following configurations.
- a display control device including: a receiving unit that receives small volume data that is information on current imaging by an artificial satellite; and a control unit that displays a live view image based on the small volume data on a display unit, wherein the receiving unit receives a satellite image corresponding to the live view image as large volume data at a timing different from that of the small volume data.
- the display control device wherein the artificial satellite generates the small volume data by performing a volume reduction process of reducing a volume of the captured satellite image, and the large volume data is the satellite image before the volume reduction process is performed.
- the receiving unit receives a satellite image captured by the artificial satellite based on an instruction of a user who has checked the live view image displayed on the display unit as the large volume data.
- the small volume data is information indicating a satellite state of the artificial satellite.
- the information indicating the satellite state of the artificial satellite is attitude information indicating an attitude of the artificial satellite and camera setting information when the artificial satellite performed imaging.
- the display control device wherein the small volume data is a low volume image obtained by converting a satellite image obtained by the imaging of the artificial satellite into an image having a small volume.
- the control unit displays the low volume image as it is as the live view image on the display unit.
- the display control device according to (1), (7), or (8), wherein the small volume data is a low resolution image obtained by converting a satellite image obtained by the imaging of the artificial satellite into an image having a low resolution.
- the display control device according to (1), (7), or (8), wherein the small volume data is a panchromatic image obtained by converting a satellite image obtained by the imaging of the artificial satellite into a monochrome image.
- the display control device (11) The display control device according to (1), wherein the small volume data is image feature amount information obtained by extracting a feature amount of a characteristic subject included in a satellite image obtained by the imaging of the artificial satellite. (12) The display control device according to (1), (7), or (8), wherein the small volume data is a partial image obtained by trimming only a partial region of a satellite image obtained by the imaging of the artificial satellite. (13) The display control device according to (1), (7), or (8), wherein the small volume data is a low frame rate image obtained by converting a frame rate of a satellite image obtained by the imaging of the artificial satellite into a frame rate lower than that of the satellite image.
- the display control device according to any one of (1) to (13), wherein the control unit executes a complementary process for complementing the small volume data, generates the live view image, and displays the live view image on the display unit.
- the control unit executes a complementary process for complementing the small volume data, generates the live view image, and displays the live view image on the display unit.
- the control unit generates the live view image using a past image obtained by imaging the same point when performing the complementary process.
- the display control device wherein the small volume data is a low resolution image obtained by converting a satellite image obtained by the imaging of the artificial satellite to an image having a low resolution, or a low frame rate image obtained by converting a frame rate of the satellite image obtained by the imaging of the artificial satellite to a frame rate lower than that of the satellite image, and the control unit improves a resolution or a frame rate to generate the live view image as the complementary process.
- the small volume data is image feature amount information obtained by extracting a feature amount of a characteristic subject included in a satellite image obtained by the imaging of the artificial satellite, and the control unit estimates and generates the live view image using the image feature amount information as the complementary process.
- the display control device wherein the small volume data is a panchromatic image obtained by converting a satellite image obtained by the imaging of the artificial satellite into a monochrome image, and the control unit generates a color live view image from the panchromatic image as the complementary process.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Studio Devices (AREA)
- Closed-Circuit Television Systems (AREA)
Abstract
Description
- The present technology relates to a display control device, a display control method, and a program, and more particularly, to a display control device, a display control method, and a program enabling images to be checked in real time.
- Remote sensing is performed by imaging the ground with an observation satellite equipped with an imaging device and observing the state of a target area or object (see, for example,
PTL 1 and 2). Particularly in recent years, the number of small observation satellites orbiting in low orbits is increasing. -
- JP 2000-111359 A
-
- JP 2006-115283 A
- Satellite images captured by observation satellites are downlinked as the artificial satellite passes over a ground station. If the transmission volume of the image data is large, a communication delay will occur, making it difficult to check the image in real time.
- The present technology has been made in view of such a situation, and enables an image to be checked in real time.
- A display control device according to one aspect of the present technology provides a display control device including: a receiving unit that receives small volume data that is information on current imaging by an artificial satellite; and a control unit that displays a live view image based on the small volume data on a display unit, wherein the receiving unit receives a satellite image corresponding to the live view image as large volume data at a timing different from that of the small volume data.
- A display control method according to one aspect of the present technology provides a display control method for allowing a display control device to execute: receiving small volume data that is information on current imaging by an artificial satellite; displaying a live view image based on the small volume data on a display unit; and receiving a satellite image corresponding to the live view image as large volume data at a timing different from that of the small volume data.
- A program according to one aspect of the present technology provides a program for causing a computer to execute: receiving small volume data that is information on current imaging by an artificial satellite; displaying a live view image based on the small volume data on a display unit; and receiving a satellite image corresponding to the live view image as large volume data at a timing different from that of the small volume data.
- In one aspect of the present technology, small volume data that is information on current imaging by an artificial satellite is received, a live view image based on the small volume data is displayed on a display unit, and a satellite image corresponding to the live view image is received as large volume data at a timing different from that of the small volume data.
- The display control device according to one aspect of the present technology can be realized by causing a computer to execute a program. The program to be executed by the computer can be provided by transmitting through a transmission medium or by recording on a recording medium.
- The display control device may be an independent device or an internal block constituting one device.
-
FIG. 1 is a block diagram showing a configuration example of a satellite image processing system according to an embodiment to which the present technology is applied. -
FIG. 2 is a block diagram showing a functional configuration of a satellite management device and an artificial satellite. -
FIG. 3 is a flowchart of a first live view image display process. -
FIG. 4 is a diagram illustrating a process according to a user's control instruction. -
FIG. 5 is a flowchart of a second live view image display process. -
FIG. 6 is a block diagram showing a configuration example of an embodiment of a computer to which the present technology is applied. - Modes for embodying the present technology (hereinafter referred to as embodiments) will be described below with reference to the accompanying drawings. Also, in the present specification and the figures, components having substantially the same functional configuration will be denoted by the same reference numerals, and thus repeated descriptions thereof will be omitted. The description will be made in the following order.
- 1. Configuration example of satellite image processing system
2. First live view image display process
3. Second live view image display process
4. Application example
5. Computer configuration example -
FIG. 1 is a block diagram showing a configuration example of a satellite image processing system according to an embodiment to which the present technology is applied. - A satellite
image processing system 1 ofFIG. 1 is a system that enables satellite images captured by one or more artificial satellites 21 (hereinafter, simply referred to as satellite 21) to be checked by a device on the ground in real time. In the present embodiment, thesatellite 21 is equipped with an imaging device and has at least a function of imaging the ground. - A satellite operating company has a
satellite management device 11 that manages a plurality ofsatellites 21, and a plurality ofcommunication devices 13 that communicate with thesatellites 21. A part of thesatellite management device 11 and the plurality ofcommunication devices 13 may be a device owned by a company other than the satellite operating company. Thesatellite management device 11 and the plurality ofcommunication devices 13 are connected to each other via apredetermined network 12. Thecommunication device 13 is arranged in a ground station (ground base station) 15. Note thatFIG. 1 shows an example in which the number ofcommunication devices 13 is three, that is,communication devices 13A to 13C, but the number ofcommunication devices 13 is arbitrary. - The
satellite management device 11 manages a plurality ofsatellites 21 owned by the satellite operating company. Specifically, thesatellite management device 11 acquires related information from aninformation providing servers 41 of one or more external organizations as necessary, and determines the operation plan of the plurality ofsatellites 21 owned by thesatellite management device 11. Then, thesatellite management device 11 causes apredetermined satellite 21 to perform imaging by transmitting an imaging start instruction to thepredetermined satellite 21 via thecommunication device 13 in response to a customer's request. Thesatellite management device 11 acquires, displays, or stores a satellite image transmitted from thesatellite 21 via thecommunication device 13. The acquired satellite image may be provided (transmitted) to the customer after performing predetermined image processing thereon as necessary. The acquired satellite image may be provided (transmitted) to animage analysis server 42 of an image analysis company, and may be provided to a customer after performing predetermined image processing thereon. - The
information providing server 41 provided in the external organization supplies predetermined related information to thesatellite management device 11 via a predetermined network in response to a request from thesatellite management device 11 or periodically. The related information provided from theinformation providing server 41 includes, for example, the following. For example, satellite orbit information (hereinafter referred to as TLE information) described in TLE (Two Line Elements) format may be acquired from NORAD (North American Aerospace Defense Command) as the external organization as related information. For example, meteorological information such as the weather at a predetermined point on the earth and the amount of clouds may be acquired from a meteorological information providing company as an external organization. - The
image analysis server 42 performs predetermined image processing on the satellite image captured by thesatellite 21 supplied from thesatellite management device 11 via a predetermined network. The processed image is provided to a customer of the image analysis company or supplied to thesatellite management device 11 of the satellite operating company. For example, theimage analysis server 42 performs metadata generation processing for adding predetermined metadata to the satellite image captured by thesatellite 21, correction processing such as distortion correction of the satellite image, and image composition processing such as color composition processing. The image processing of the satellite image may be performed by the satellite operating company, and in this case, the satellite operating company and the image analysis company are the same. Thesatellite management device 11 and theimage analysis server 42 may be realized by one device. - The
communication device 13 communicates with apredetermined satellite 21 designated by thesatellite management device 11 via anantenna 14 according to the control of thesatellite management device 11. For example, thecommunication device 13 transmits an imaging start instruction to start imaging of a predetermined area on the ground to apredetermined satellite 21. Thecommunication device 13 receives the satellite image transmitted from thesatellite 21 and supplies the same to thesatellite management device 11 via thenetwork 12. The transmission from thecommunication device 13 of theground station 15 to thesatellite 21 is also referred to as an uplink, and the transmission from thesatellite 21 to thecommunication device 13 is also referred to as a downlink. Thecommunication device 13 can directly communicate with thesatellite 21 and can also communicate with arelay satellite 22. As therelay satellite 22, for example, a geostationary satellite is used. - The
network 12 or the network between theinformation providing server 41 or theimage analysis server 42 and thesatellite management device 11 is an arbitrary communication network, which may be a wired communication network or a wireless communication network or may be configured by both of them. Thenetwork 12 and the network between theinformation providing server 41 or theimage analysis server 42 and thesatellite management device 11 may be configured by one communication network, or may be configured by a plurality of communication networks. These networks may be communication networks or communication channels of any communication standard, such as, for example, the Internet, a public telephone network, a wide area communication network for wireless mobiles such as so-called 4G line and 5G line, a wireless communication network that performs communication compliant with WAN (Wide Area Network), LAN (Local Area Network), and Bluetooth (registered trademark) standards, a communication channel for short-range wireless communication such as NFC (Near Field Communication), a communication channel for Infrared communication, a communication network for wired communication compliant with standards such as HDMI (registered trademark) (High-Definition Multimedia Interface) and USB (Universal Serial Bus). - Each
satellite 21 may be operated in a single unit or a plurality of units. A plurality ofsatellites 21 operated in a plurality of units constitute one satellite group 31. InFIG. 1 ,satellites satellites satellite group 31A. In the example ofFIG. 1 , for the sake of simplicity, an example in which one satellite group 31 is composed of twosatellites 21 is shown, but the number ofsatellites 21 constituting one satellite group 31 is not limited to two. - Systems that operate a plurality of
satellites 21 as one unit (satellite group 31) include constellation and formation flight. Constellation is a system that deploys services mainly globally by launching a large number ofsatellites 21 into a single or a plurality of orbital planes. Even a single satellite has a predetermined function, and a plurality ofsatellites 21 are operated for the purpose of improving the observation frequency and the like. On the other hand, the formation flight is a system that deploys a plurality ofsatellites 21 in a narrow area of about several kilometers while maintaining a relative positional relationship. Formation flight can provide services that cannot be realized by a single satellite, such as high-precision 3D measurement and speed detection of moving objects. In the present embodiment, it does not matter whether the operation of the satellite group is a constellation or a formation flight. - When the
communication device 13 communicates with eachsatellite 21, a method of directly communicating with thesatellite 21 as in thesatellite 21A and thesatellite 21B, and a method of indirectly communicating with thecommunication device 13 by performing inter-satellite communication with asatellite 21C which is theother satellite 21, as in asatellite 21D may be used. The method of indirectly communicating includes communication via therelay satellite 22. Which method is used to communicate with the ground station 15 (thecommunication device 13 thereof) may be determined in advance by thesatellite 21 or may be appropriately selected according to the content of the communication. - In the satellite
image processing system 1 configured as described above, thesatellite 21 as an observation satellite images a predetermined point on the ground based on the imaging start instruction from thesatellite management device 11. If the data of the image captured by thesatellite 21 is transmitted as it is, a delay occurs due to a large data volume, and it is difficult to observe the image as a live view image in real time. - Therefore, the satellite
image processing system 1 is configured so that an image similar to the image captured by thesatellite 21 can be observed in real time by thesatellite management device 11 on the ground. -
FIG. 2 is a block diagram showing a functional configuration of thesatellite management device 11 and thesatellite 21 regarding the observation of the image captured by thesatellite 21. - The
satellite 21 includes an antenna AT, asatellite communication unit 61, animaging device 62, acontrol unit 63, and astorage unit 64. The configuration of thesatellite 21 is mainly for functions related to images only, and although not shown, thesatellite 21 is also equipped with propulsion devices such as solid-state motors and ion engines regarding attitude control, sensors such as a GPS receiver, a star tracker (attitude sensor), an accelerometer, and a gyro sensor regarding position control, and a power supply such as batteries and solar panels. - Based on the control of the
control unit 63, thesatellite communication unit 61 transmits image data of the image captured by theimaging device 62, state data indicating the state of thesatellite 21 at the time of imaging, and the like to thecommunication device 13 of theground station 15 via the antenna AT. The data transmitted from thesatellite communication unit 61 to thecommunication device 13 is supplied from thecommunication device 13 to thesatellite management device 11. - The
imaging device 62 is configured of, for example, a camera module including an image sensor, and images an object based on the control of thecontrol unit 63. When thesatellite 21 is a synthetic aperture radar (SAR) satellite, theimaging device 62 is configured of a radar device. - The
control unit 63 controls the operation of theentire satellite 21. For example, thecontrol unit 63 causes theimaging device 62 to perform imaging based on the imaging start instruction from thesatellite management device 11. Thecontrol unit 63 stores the satellite image obtained by imaging as a large volume data in astorage unit 64, and also performs a volume reduction process for generating a small volume data having a volume smaller than that of the satellite image. The small volume data is, for example, an image obtained by converting the satellite image captured by theimaging device 62 into a small volume data, and state data indicating the state of thesatellite 21 when theimaging device 62 performs imaging. - The
storage unit 64 stores control programs executed by thecontrol unit 63 and parameters. Thestorage unit 64 stores the image data (large volume data) captured by theimaging device 62 and the small volume data generated by thecontrol unit 63, and supplies the same to thesatellite communication unit 61 or thecontrol unit 63 as necessary. - The
satellite management device 11 includes acontrol unit 81, acommunication unit 82, astorage unit 83, anoperation unit 84, and adisplay unit 85. - The
control unit 81 manages a plurality ofsatellites 21 owned by the satellite operating company by executing a satellite management application program stored in thestorage unit 83. For example, thecontrol unit 81 determines the operation plan of the plurality ofsatellites 21 using the related information acquired from theinformation providing server 41 as necessary, and sends an attitude control signal and an imaging instruction to eachsatellite 21 via thecommunication device 13. Thecontrol unit 81 performs processing such as displaying an observation image on thedisplay unit 85 based on the large volume data and the small volume data of the satellite image transmitted from thesatellite 21 via thecommunication device 13. - The
communication unit 82 performs predetermined communication with thecommunication device 13 via thenetwork 12 and performs predetermined communication with theimage analysis server 42 according to the instruction of thecontrol unit 81. For example, thecommunication unit 82 receives large volume data and small volume data related to satellite images transmitted from thesatellite 21. - The
storage unit 83 stores large volume data and small volume data related to the satellite image transmitted from thesatellite 21 in accordance with the instruction of thecontrol unit 81. - The
operation unit 84 is configured of, for example, a keyboard, a mouse, a touch panel, or the like, and receives commands and data inputs based on user (operator) operations and supplies the same to thecontrol unit 81. - The
display unit 85 is configured of, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. Thedisplay unit 85 displays the screen of the satellite management application program, displays a satellite image based on a large volume data transmitted from thesatellite 21, a live view image based on a small volume data transmitted from thesatellite 21, and the like. - The
satellite management device 11 functions as a display control device that displays a satellite image captured by theimaging device 62 of thesatellite 21 on a predetermined display unit (thedisplay unit 85 or an external display device) based on a user's operation. - Next, a first live view image display process for displaying a live view image for real-time observation will be described with reference to the flowchart of
FIG. 3 . This process starts, for example, when the user who operates thesatellite management device 11 performs a live view image display start operation. - First, in step S11, the
control unit 81 of thesatellite management device 11 transmits an imaging start instruction to thesatellite 21 via thecommunication unit 82. - In step S41, the
control unit 63 of thesatellite 21 receives the imaging start instruction transmitted from thesatellite management device 11 via thesatellite communication unit 61. Then, in step S42, thecontrol unit 63 performs imaging to generate a high resolution image having a resolution of the normal performance of theimaging device 62. For example, theimaging device 62 performs imaging to generate a 4K resolution when it can perform imaging at a maximum of 4K resolution, and performs imaging to generate an HD resolution when it can perform imaging at a maximum of HD resolution. The image data of the high resolution image obtained by imaging is stored in thestorage unit 64. The image data of the high resolution image may be stored in thestorage unit 64 as RAW data, or may be stored in thestorage unit 64 as coded data encoded by predetermined coding. - In step S43, the
control unit 63 executes a volume reduction process for generating small volume data having a volume smaller than that of the high resolution image based on the high resolution image obtained by imaging. - In the volume reduction process, for example, information indicating the satellite state when the
satellite 21 performed imaging is generated as small volume data. The information indicating the satellite state includes, for example, attitude information indicating the attitude of thesatellite 21 when thesatellite 21 performed imaging, and setting information when theimaging device 62 performs imaging (hereinafter referred to as camera setting information). The camera setting information is, for example, information related to camera setting values such as resolution, zoom magnification, shutter speed, sensitivity, and aperture. When theimaging device 62 is attached to thesatellite 21 at a fixed position, the imaging range (angle of view) of theimaging device 62 can be specified based on the attitude of thesatellite 21 and the zoom setting value of theimaging device 62. Thus, the attitude information of thesatellite 21 can be also regarded as a part of the camera setting information. - In the volume reduction process, a low volume image obtained by converting a high resolution image obtained by imaging into a low volume image may be generated as small volume data. For example, a low resolution image obtained by converting a high resolution image to a low resolution such as VGA, a panchromatic image obtained by converting a color high resolution image into a monochrome image, image feature amount information obtained by extracting a feature amount of a characteristic subject included in a high resolution image, a partial image obtained by trimming only a part of a high resolution image, and the like can be generated as small volume data. Examples of image feature amount information obtained by extracting the feature amount of a high resolution image include, for example, information indicating the characteristics of the shape of clouds, information on dynamic subjects such as airplanes, information on the tint of light rays such as red sunset, and information on the reflection position of the ground surface due to sunlight such as the reflection state of the sea or lake.
- In the volume reduction process, a low frame rate image obtained by converting the frame rate of a high resolution image obtained by imaging to a lower frame rate may be generated as small volume data. For example, the
control unit 63 generates a low frame rate image obtained by converting the frame rate of a high resolution image of 30 fps into a frame rate of 1 fps as small volume data. - The small volume data is data generated by performing the volume reduction process on information regarding the current imaging by the
satellite 21. - In step S44, the
control unit 63 transmits the small volume data generated by the volume reduction process to thesatellite management device 11 via thesatellite communication unit 61. Thesatellite 21 repeatedly executes the processes of steps S42 to S44 described above until an imaging end instruction is transmitted from thesatellite management device 11. By repeating steps S42 to S44, small volume data is transmitted to thesatellite management device 11 at a predetermined frame rate. Since the small volume data has a small volume, it can be transmitted in frame units without delay. - In step S12, the
control unit 81 of thesatellite management device 11 receives the small volume data transmitted from thesatellite 21 via thecommunication unit 82. In step S13, a complementary process for complementing the small volume data is executed to generate a live view image. Then, in step S14, thecontrol unit 81 causes thedisplay unit 85 to display the generated live view image. - When performing the complementary process, the
control unit 81 acquires related information from one or moreinformation providing servers 41 of the external organization as necessary. Related information includes, for example, the following. For example, thecontrol unit 81 can acquire the TLE information as related information and specify the position of thesatellite 21 at the imaging time. For example, thecontrol unit 81 acquires, as the related information, archive images (baseline images) which are satellite images the same imaging point of the past (for example, a few days ago or a month ago), accumulated by operating companies that operate other satellite observation services, and quasi-real-time images that are satellite images of the same imaging point at a relatively close time of several tens of minutes to several hours before the current time. Naturally, if a similar image is stored in thestorage unit 83 of itself (the satellite management device 11) instead of the operating company that operates another satellite observation service, it can be used. It is preferable that the archive image or the quasi-real-time image to be acquired has the same season, imaging time, weather conditions, and the like as when the small volume data is generated. For example, thecontrol unit 81 can acquire meteorological information such as the weather, the cloud distribution, the cloud amount, and the position information of the sun at the same imaging point when the small volume data is generated from the meteorological information providing company as an external organization. For example, thecontrol unit 81 can acquire AIS (Automatic Identification System) information indicating the positions of ships and airplanes at the imaging time as the related information from theinformation providing server 41 of an operation information providing company that provides the operation information of ships and airplanes. - The
control unit 81 executes a complementary process using the above-mentioned related information as necessary, and generates a live view image in which an image (satellite image) captured by thesatellite 21 is estimated from the small volume data. - For example, when the small volume data is information indicating the satellite state such as the attitude information of the
satellite 21 and the camera setting information, thecontrol unit 81 uses the TLE information, the baseline image, the quasi-real-time image, and the like as related information, and reproduces the image captured by theimaging device 62 by CG (computer graphics) or the like at the angle of view captured by theimaging device 62 to generate a live view image. - For example, when the small volume data is a low resolution image obtained by converting a high resolution image to a low resolution such as VGA, the
control unit 81 generates a high resolution image from the low resolution image using super-resolution technology to obtain a live view image. At this time, a baseline image or a quasi-real-time image may be acquired as related information and interpolation processing or the like may be performed thereon. - For example, when the small volume data is a panchromatic image obtained by converting a color high resolution image into a monochrome image, the
control unit 81 generates a color high resolution image from the panchromatic image using machine learning to obtain a live view image. At this time, a colorization process may be performed using the image feature amount information extracted from the feature amount included in the high resolution image. - For example, when the small volume data is image feature amount information obtained by extracting the feature amount included in a high resolution image, the
control unit 81 reproduces a characteristic subject from Information on cloud shape, dynamic subject, and tint of light rays and information on reflection position of sunlight as image feature amount information and superimposes the characteristic subject on an archive image or a quasi-real-time image to generate a live view image. When a panchromatic image and image feature amount information are acquired as small volume data, for example, a process of adding the tint of light rays to the panchromatic image or a process of superimposing a cloud shape and a dynamic subject on the panchromatic image may be performed. External information can be also used for meteorological information and the position of dynamic subjects. - For example, when the small volume data is a low frame rate image, the
control unit 81 generates a live view image by interpolating and generating an image between the frames of the acquired low frame rate image to increase the frame rate. The frame rate of the live view image does not have to be the same as the frame rate of the high resolution image. - The complementary process of step S13 may be omitted. For example, when the small volume data is a low resolution image obtained by converting a high resolution image to a low resolution, or a panchromatic image obtained by converting a high resolution image into a monochrome image, the complementary process may not be performed and the low resolution image or the panchromatic image may be displayed on the
display unit 85 as it is as a live view image. - In the display of the live view image in step S14, the
control unit 81 may display the live view image displayed on thedisplay unit 85 so that the user can understand that the live view image is an image estimated from the small volume data. For example, information (characters) indicating that it is an estimated image may be superimposed on the live view image, or an outer frame image indicating that it is an estimated image may be added and displayed on the outer periphery of the live view image. - In step S15, the
control unit 81 determines whether the user's operation on thesatellite 21 has been performed by theoperation unit 84. When it is determined in step S15 that the user's operation has not been performed, the process returns to step S12, and the processes of steps S12 to S15 described above are repeated. - On the other hand, when it is determined in step S15 that the user's operation has been performed, the process proceeds to step S16, and the
control unit 81 transmits a control instruction corresponding to the user's operation to thesatellite 21 via thecommunication unit 82. - Examples of the user's operation in step S15 include the following operations. For example, the user may perform an operation of instructing a change of camera setting information such as a resolution, a zoom magnification, a shutter speed, a sensitivity, and an aperture. For example, an operation of changing the imaging direction (imaging point) of the
imaging device 62 may be performed When theimaging device 62 is attached to thesatellite 21 at a fixed position, the attitude of thesatellite 21 itself is changed in response to an instruction to change the imaging direction. When theimaging device 62 has a movable portion capable of changing the relative position with respect to thesatellite 21, the attitude of thesatellite 21 itself may be changed in response to an instruction to change the imaging direction, or the relative position of theimaging device 62 may change. The zoom magnification may be either optical zoom or digital zoom. - For example, as a user operation in step S15, the user may give a live view image transmission instruction requesting a large volume data which is a high resolution image corresponding to the live view image displayed on the
display unit 85. - In line with the processing of step S16 of the
satellite management device 11, thecontrol unit 63 of thesatellite 21 receives the control instruction transmitted from thesatellite management device 11 via thesatellite communication unit 61 in step S45. - Subsequently, in step S46, the
control unit 63 determines whether the control instruction from thesatellite management device 11 is a large volume data transmission instruction. - When it is determined in step S46 that the control instruction from the
satellite management device 11 is not a large volume data transmission instruction, the process proceeds to step S47, and thecontrol unit 63 performs control based on the control instruction. For example, when the control instruction is a change of camera setting information such as resolution and zoom magnification, thecontrol unit 63 performs control to change a part of the camera setting information of theimaging device 62 to a designated setting value. After the process of step S47, the camera setting value of theimaging device 62 for capturing a high resolution image, which is repeatedly executed in steps S42 to S44 described above, is changed. - As a general control flow, after the camera setting value of the
imaging device 62 is changed in response to a control instruction from thesatellite management device 11, capturing of high resolution image starts with the changed camera setting value. Therefore, there is a slight delay before the user's camera setting information change instruction is reflected in the live view image. - However, in the case of the following combination of specific imaging conditions and the camera setting information change instruction, it is possible to reduce the delay felt by the user when checking the live view image.
- For example, in the case of capturing the high resolution image in step S42 described above, when the
imaging device 62 changes a predetermined camera setting value to a plurality of values, and performs bracket shooting in which imaging is performed with a plurality of different camera setting values, that is, when theimaging device 62 performs bracket shooting with the camera setting information instructed to be changed by the user, small volume data based on a high resolution image captured with the camera setting value instructed to be changed by the user can be immediately transmitted to thesatellite management device 11. - For example, when the data transmitted as small volume data is partial image data in which only a partial region of a high resolution image is trimmed, and the user's camera setting information change instruction is to change the zoom magnification or the angle of view, a partial image having a zoom magnification and an angle of view after the change can be generated using the high resolution image before trimming, and can be immediately transmitted to the
satellite management device 11 as small volume data. - With reference to
FIG. 4 , an example in which the user's camera setting information change instruction is to change the angle of view will be described. - At time t1, the user issues a camera setting information change instruction to move the angle of view to the right while looking at the partial image C1 while observing the partial image C1 obtained by trimming only the central portion of the high resolution image B1 on the
display unit 85. - At time t2, the
control unit 63 of thesatellite 21 receives the camera setting information change instruction, and starts the attitude control to change the attitude of theimaging device 62 in the instructed direction. However, since the attitude of theimaging device 62 is not immediately changed to the desired attitude, the angle of view of the high resolution image B2 captured by theimaging device 62 at time t2 is substantially the same as the angle of view of the high resolution image B1 at time t1. Thecontrol unit 63 trims the partial image C1 corresponding to the angle of view instructed by the user from the high resolution image B2 captured by theimaging device 62 at time t2, and immediately transmits the same to thesatellite management device 11 as small volume data. - When the attitude of the
imaging device 62 coincides with the imaging direction designated by the user at time t3, the partial image C3 obtained by trimming only the central portion of the high resolution image B3 is transmitted to thesatellite management device 11 as small volume data. - As described above, since the partial image having the angle of view designated by the user can be displayed on the
display unit 85 before the attitude of theimaging device 62 reaches the state instructed to be changed by the user, it is possible to reduce the delay felt by the user checking the live view image. - Returning to the flowchart of
FIG. 3 , when it is determined in step S46 that the control instruction from thesatellite management device 11 is a large volume data transmission instruction, the process proceeds to step S48, and thecontrol unit 63 transmits the instructed large volume data to thesatellite management device 11 via thesatellite communication unit 61. That is, the data of the high resolution image corresponding to the live view image displayed on thedisplay unit 85 in the process of step S14 is transmitted from thesatellite 21 to thesatellite management device 11. - In the
satellite management device 11, after the control instruction corresponding to the user's operation is transmitted to thesatellite 21 in step S16, the process proceeds to step S17, and thecontrol unit 81 determines whether the control instruction transmitted to thesatellite 21 is a large volume data transmission instruction. - When it is determined in step S17 that the control instruction transmitted to the
satellite 21 is not a large volume data transmission instruction, the process returns to step S12, and the processes after step S12 described above are executed again. - On the other hand, when it is determined in step S17 that the control instruction transmitted to the
satellite 21 is a large volume data transmission instruction, the process proceeds to step S18, and thecontrol unit 81 receives the large volume data transmitted from thesatellite 21 in response to the transmitted control instruction and stores the same in thestorage unit 83. After step S18, the process returns to step S12, and the processes after step S12 described above are executed again. - The first live view image display process is continuously executed until the user performs a live view image display end operation, and when the live view image display end operation is performed, the first live view image display process ends.
- In the first live view image display process, the
satellite management device 11 receives small volume data having a volume smaller than that of a high resolution image having the resolution of the normal performance of theimaging device 62 from thesatellite 21. Since the small volume data has a small volume, it can be transmitted in frame units without delay, so that the live view image based on the small volume data can be displayed in real time. Then, based on the instruction of the user who has checked the live view image displayed on thedisplay unit 85, large volume data which is a high resolution image corresponding to the live view image is received and stored. - The transmission and reception of large volume data, which is the process of steps S18 and S48 described above, may be executed at a timing other than during the execution of the first live view image display process described above, in other words, at a timing other than when the live view image is observed in real time.
- When there are two or more communication systems between the
satellite 21 and thecommunication device 13, the transmission and reception of large volume data may be performed in the background different from the transmission and reception of small volume data. At this time, since the large volume data takes a long communication time, the number of times the large volume data is transmitted is smaller than that of small volume data. For example, when the small volume data is data of 30 fps, the large volume data is data of 1 fps, or data transmitted once every 10 minutes. The large volume data transmitted and received in the background can be used as a quasi-real-time image in the complementary process of step S13 described above. - The large volume data can be used as teacher data when machine learning for estimating a live view image is performed as the complementary process. Using machine learning to generate the live view image, the accuracy of the live view image generated by estimation can be improved.
- Next, a second live view image display process executed by the satellite
image processing system 1 will be described. - In the first live view image display process described above, the
satellite 21 captures a high resolution image based on an imaging start instruction from thesatellite management device 11, generates small volume data based on the high resolution image, and transmits the same to thesatellite management device 11. In this case, the high resolution image is stored in thestorage unit 64, and is transmitted to thesatellite management device 11 as large volume data at another timing as necessary. - On the other hand, in the second live view image display processing, the
satellite 21 does not capture a high resolution image, but performs imaging for generating small volume data. Then, thesatellite 21 captures the high resolution image corresponding to the live view image and transmits the same as large volume data only when the user observing the live view image gives an instruction to capture the high resolution image. - The second live view image display process will be described with reference to the flowchart of
FIG. 5 . This process starts, for example, when the user who operates thesatellite management device 11 performs a live view image display start operation. - First, in step S61, the
control unit 81 of thesatellite management device 11 transmits an imaging start instruction to thesatellite 21 via thecommunication unit 82. - In step S81, the
control unit 63 of thesatellite 21 receives the imaging start instruction transmitted from thesatellite management device 11 via thesatellite communication unit 61. Then, in step S82, thecontrol unit 63 executes a small volume data generation process. - In the small volume data generation process, the same small volume data as the first live view image display process described above is generated. For example, information indicating the satellite state when the
satellite 21 performed imaging is generated as small volume data. Alternatively, a low volume image having a smaller volume than the high resolution image captured with the normal performance of theimaging device 62, for example, a low resolution image having a lower resolution than a color high resolution image, a panchromatic image obtained by converting a color high resolution image into a monochrome image, image feature amount information obtained by extracting the feature amount of a characteristic subject included in a high resolution image, a partial image corresponding to a partial region of a high resolution image, and a low frame rate image whose frame rate is lower than that of a high resolution image, is generated as the small volume data. - In step S83, the
control unit 63 transmits the small volume data generated by the small volume data generation process to thesatellite management device 11 via thesatellite communication unit 61. Thesatellite 21 repeatedly executes the processes of steps S82 and S83 until the imaging end instruction is transmitted from thesatellite management device 11. By repeating steps S82 and S83, small volume data is transmitted to thesatellite management device 11 at a predetermined frame rate. Since the small volume data has a small volume, it can be transmitted in frame units without delay. - In step S62, the
control unit 81 of thesatellite management device 11 receives the small volume data transmitted from thesatellite 21 via thecommunication unit 82. In step S63, a complementary process for complementing the small volume data is executed to generate a live view image. Then, in step S64, thecontrol unit 81 causes thedisplay unit 85 to display the generated live view image. - The details of the complementary process in step S63 and the display of the live view image in step S64 are the same as those of the first live view image display process described above, and thus the description thereof will be omitted. Moreover, the complementary process of step S63 may be omitted.
- In step S65, the
control unit 81 determines whether the user's operation on thesatellite 21 has been performed by theoperation unit 84. When it is determined in step S65 that the user's operation has not been performed, the process returns to step S62, and the processes of steps S62 to S65 described above are repeated. - On the other hand, when it is determined in step S65 that the user's operation has been performed, the process proceeds to step S66, and the
control unit 81 transmits a control instruction corresponding to the user's operation to thesatellite 21 via thecommunication unit 82. The operation that the user can perform while observing the live view image is the same as that of the first live view image display process described above. - In line with the processing of step S66 of the
satellite management device 11, thecontrol unit 63 of thesatellite 21 receives the control instruction transmitted from thesatellite management device 11 via thesatellite communication unit 61 in step S84. - Subsequently, in step S85, the
control unit 63 determines whether the control instruction from thesatellite management device 11 is a large volume data transmission instruction. - When it is determined in step S85 that the control instruction from the
satellite management device 11 is not a large volume data transmission instruction, the process proceeds to step S86, and thecontrol unit 63 performs control based on the control instruction. For example, when the control instruction is a change of camera setting information such as resolution and zoom magnification, thecontrol unit 63 performs control to change a part of the camera setting information of theimaging device 62 to a designated setting value. After the process of step S86, the camera setting value of theimaging device 62 for generating small volume data, which is repeatedly executed in steps S82 and S83 described above, is changed. - On the other hand, when it is determined in step S85 that the control instruction from the
satellite management device 11 is a large volume data transmission instruction, the process proceeds to step S87, and thecontrol unit 63 performs imaging to generate a high resolution image having the resolution of the normal performance of theimaging device 62. The generated high resolution image is stored in thestorage unit 64. Subsequently, in step S88, thecontrol unit 63 transmits the image data of the generated high resolution image as large volume data to thesatellite management device 11 via thesatellite communication unit 61. - In the
satellite management device 11, after the control instruction corresponding to the user's operation is transmitted to thesatellite 21 in step S66, the process proceeds to step S67, and thecontrol unit 81 determines whether the control instruction transmitted to thesatellite 21 is a large volume data transmission instruction. - When it is determined in step S67 that the control instruction transmitted to the
satellite 21 is not a large volume data transmission instruction, the process returns to step S62, and the processes after step S62 described above are executed again. - On the other hand, when it is determined in step S67 that the control instruction transmitted to the
satellite 21 is a large volume data transmission instruction, the process proceeds to step S68, and thecontrol unit 81 receives the large volume data transmitted from thesatellite 21 according to the transmitted control instruction and stores the same in thestorage unit 83. After step S68, the process returns to step S62, and the processes after step S62 described above are executed again. - The second live view image display process is continuously executed until the user performs a live view image display end operation, and when the live view image display end operation is performed, the second live view image display process ends.
- Similarly to the first live view image display process, transmission and reception of large volume data, which is the process of steps S68 and S88 described above, may be executed at a timing other than during the execution of the second live view image display process described above, in other words, at a timing other than when the live view image is observed in real time.
- According to the first and second live view image display processes executed by the satellite
image processing system 1, small volume data is transmitted when observing a live view image, and a high resolution image is transmitted to thesatellite management device 11 as large volume data at another timing as necessary. In this way, the user can check the image on the ground in real time. - The present technology can be applied not only to satellite image processing systems that remotely instruct imaging by artificial satellites, but also to remote control and remote monitoring of mobile systems operated in remote areas.
- The mobile system is a system including, for example, a robot, particularly a remote-existing robot (telexistence robot), an autonomous traveling vehicle, an autonomous aircraft (drone), a manned traveling vehicle, or a manned aircraft as a mobile device.
- In these mobile systems, a user who operates and monitors the mobile device from a remote location can check the minimum situation using a live view image with small volume data.
- The live view image can be generated by executing the complementary process for complementing the small volume data, as in the case of the satellite
image processing system 1 described above. - For example, an assumed image can be reproduced (estimated) based on the attitude information, position information, and the like of the mobile device. At this time, when the movement route and existence environment of the mobile device are known, or when an archive image of the same point in the past or a quasi-real-time image of the same point taken at a relatively close time is accumulated as information on the movement route and existence environment, they can be used for complementation.
- A live view image having a high resolution may be generated using super-resolution technology based on VGA images, a color high resolution image may be generated from a panchromatic image, and low frame rate images may be interpolated to generate a live view image having a high frame rate.
- A live view image based on the feature amount information can be also generated. For example, when the movement route and existence environment of the mobile device are known, or when information on the movement route and existence environment is accumulated, only the real-time feature amount of a subject unique in the movement route and existence environment may be extracted from an archive image to generate an estimated image.
- The above-described series of processing executed by the
satellite management device 11 can also be performed by hardware or software. When the series of processing is performed by software, a program including the software is installed in a computer. Here, the computer includes a computer which is embedded in dedicated hardware or, for example, a general-purpose personal computer capable of executing various functions by installing various programs. -
FIG. 6 is a block diagram showing a configuration example of hardware of a computer that performs, by means of a program, a series of processing executed by thesatellite management device 11. - In the computer, a central processing unit (CPU) 301, a readonly memory (ROM) 302, and a random access memory (RAM) 303 are connected to each other by a
bus 304. - An input/
output interface 305 is further connected to thebus 304. Aninput unit 306, anoutput unit 307, astorage unit 308, acommunication unit 309, and adrive 310 are connected to the input/output interface 305. - The
input unit 306 is, for example, a keyboard, a mouse, a microphone, a touch panel, or an input terminal. Theoutput unit 307 is, for example, a display, a speaker, or an output terminal. Thestorage unit 308 is, for example, a hard disk, a RAM disc, or a nonvolatile memory. Thecommunication unit 309 is a network interface or the like. Thedrive 310 drives aremovable recording medium 311 such as a magnetic disk, an optical disc, a magneto-optical disc, or a semiconductor memory. - In the computer that has the foregoing configuration, the
CPU 301 performs the above-described series of processing, for example, by loading a program stored in thestorage unit 308 on theRAM 303 via the input/output interface 305 and thebus 304 and executing the program. In theRAM 303, data or the like necessary for theCPU 301 to perform various kinds of processing is also appropriately stored. - The program executed by the computer (the CPU 301) can be recorded on, for example, the
removable recording medium 311 serving as a package medium for supply. The program can be supplied via a wired or wireless transfer medium such as a local area network, the Internet, or digital satellite broadcasting. - In the computer, by mounting the
removable recording medium 311 on thedrive 310, it is possible to install the program in thestorage unit 308 via the input/output interface 305. The program can be received by thecommunication unit 309 via a wired or wireless transfer medium to be installed in thestorage unit 308. In addition, the program can be installed in advance in theROM 302 or thestorage unit 308. - In the present description, the steps having been described in the flowcharts may be carried out in parallel or with necessary timing, for example, when evoked, even if the steps are not executed in time series along the order having been described therein, as well as when the steps are executed in time series.
- In the present specification, the system means a set of a plurality of constituent elements (devices, modules (components), or the like) and all the constituent elements may be included or not included in the same casing. Accordingly, a plurality of devices accommodated in separate casings and connected via a network and one device in which a plurality of modules are accommodated in one casing may all be a system.
- The embodiments of the present technology are not limited to the above-described embodiments, and various changes can be made without departing from the gist of the present technology.
- For example, a combination of all or part of the above-mentioned plurality of embodiments may be employed.
- For example, the present technology may have a configuration of clouding computing in which a plurality of devices share and process one function together via a network.
- In addition, each step described in the above flowchart can be executed by one device or shared by a plurality of devices.
- Further, in a case in which one step includes a plurality of processes, the plurality of processes included in the one step can be executed by one device or shared and executed by a plurality of devices.
- The effects described in the present specification are merely examples and are not limited, and there may be effects other than those described in the present specification.
- The present technology can employ the following configurations.
- (1) A display control device including: a receiving unit that receives small volume data that is information on current imaging by an artificial satellite; and a control unit that displays a live view image based on the small volume data on a display unit, wherein the receiving unit receives a satellite image corresponding to the live view image as large volume data at a timing different from that of the small volume data.
(2) The display control device according to (1), wherein the receiving unit receives the large volume data transmitted from the artificial satellite based on an instruction of a user who has checked the live view image displayed on the display unit.
(3) The display control device according to (1) or (2), wherein the artificial satellite generates the small volume data by performing a volume reduction process of reducing a volume of the captured satellite image, and the large volume data is the satellite image before the volume reduction process is performed.
(4) The display control device according to (1) or (2), wherein the receiving unit receives a satellite image captured by the artificial satellite based on an instruction of a user who has checked the live view image displayed on the display unit as the large volume data.
(5) The display control device according to (1), wherein the small volume data is information indicating a satellite state of the artificial satellite.
(6) The display control device according to (5), wherein the information indicating the satellite state of the artificial satellite is attitude information indicating an attitude of the artificial satellite and camera setting information when the artificial satellite performed imaging.
(7) The display control device according to (1), wherein the small volume data is a low volume image obtained by converting a satellite image obtained by the imaging of the artificial satellite into an image having a small volume.
(8) The display control device according to (7), wherein the control unit displays the low volume image as it is as the live view image on the display unit.
(9) The display control device according to (1), (7), or (8), wherein the small volume data is a low resolution image obtained by converting a satellite image obtained by the imaging of the artificial satellite into an image having a low resolution.
(10) The display control device according to (1), (7), or (8), wherein the small volume data is a panchromatic image obtained by converting a satellite image obtained by the imaging of the artificial satellite into a monochrome image.
(11) The display control device according to (1), wherein the small volume data is image feature amount information obtained by extracting a feature amount of a characteristic subject included in a satellite image obtained by the imaging of the artificial satellite.
(12) The display control device according to (1), (7), or (8), wherein the small volume data is a partial image obtained by trimming only a partial region of a satellite image obtained by the imaging of the artificial satellite.
(13) The display control device according to (1), (7), or (8), wherein the small volume data is a low frame rate image obtained by converting a frame rate of a satellite image obtained by the imaging of the artificial satellite into a frame rate lower than that of the satellite image.
(14) The display control device according to any one of (1) to (13), wherein the control unit executes a complementary process for complementing the small volume data, generates the live view image, and displays the live view image on the display unit.
(15) The display control device according to (14), wherein the control unit generates the live view image using a past image obtained by imaging the same point when performing the complementary process.
(16) The display control device according to (14), wherein the small volume data is a low resolution image obtained by converting a satellite image obtained by the imaging of the artificial satellite to an image having a low resolution, or a low frame rate image obtained by converting a frame rate of the satellite image obtained by the imaging of the artificial satellite to a frame rate lower than that of the satellite image, and the control unit improves a resolution or a frame rate to generate the live view image as the complementary process.
(17) The display control device according to (14), wherein the small volume data is image feature amount information obtained by extracting a feature amount of a characteristic subject included in a satellite image obtained by the imaging of the artificial satellite, and the control unit estimates and generates the live view image using the image feature amount information as the complementary process.
(18) The display control device according to (14), wherein the small volume data is a panchromatic image obtained by converting a satellite image obtained by the imaging of the artificial satellite into a monochrome image, and the control unit generates a color live view image from the panchromatic image as the complementary process.
(19) A display control method for allowing a display control device to execute: receiving small volume data that is information on current imaging by an artificial satellite; displaying a live view image based on the small volume data on a display unit; and receiving a satellite image corresponding to the live view image as large volume data at a timing different from that of the small volume data.
(20) A program for causing a computer to execute: receiving small volume data that is information on current imaging by an artificial satellite; displaying a live view image based on the small volume data on a display unit; and receiving a satellite image corresponding to the live view image as large volume data at a timing different from that of the small volume data. -
- 1 Satellite image processing system
- 11 Satellite management device
- 13 Communication device
- 15 Ground station
- 21 Satellite
- 41 Information providing server
- 42 Image analysis server
- 61 Satellite communication unit
- 62 Imaging device
- 63 Control unit
- 64 Storage unit
- 81 Control unit
- 82 Communication unit
- 83 Storage unit
- 84 Operation unit
- 85 Display unit
- 301 CPU
- 302 ROM
- 303 RAM
- 306 Input unit
- 307 Output unit
- 308 Storage unit
- 309 Communication unit
- 310 Drive
Claims (20)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2020006727 | 2020-01-20 | ||
JP2020-006727 | 2020-01-20 | ||
PCT/JP2021/000142 WO2021149485A1 (en) | 2020-01-20 | 2021-01-06 | Display control device, display control method, and program |
Publications (2)
Publication Number | Publication Date |
---|---|
US20230079285A1 true US20230079285A1 (en) | 2023-03-16 |
US12041363B2 US12041363B2 (en) | 2024-07-16 |
Family
ID=76991741
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/792,564 Active 2041-03-22 US12041363B2 (en) | 2020-01-20 | 2021-01-06 | Display control device, display control method, and program for displaying images based on satellite imaging |
Country Status (5)
Country | Link |
---|---|
US (1) | US12041363B2 (en) |
EP (1) | EP4096219A4 (en) |
JP (1) | JPWO2021149485A1 (en) |
CN (1) | CN114946176A (en) |
WO (1) | WO2021149485A1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220009631A1 (en) * | 2018-01-30 | 2022-01-13 | Aerovironment, Inc. | Methods and Systems for Cloud-Based Management of Images Captured by Aerial Vehicles |
US20220318971A1 (en) * | 2021-04-02 | 2022-10-06 | Samsung Electronics Co., Ltd. | Electronic device for supporting machine learning-based image processing |
Family Cites Families (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
DE4216828C2 (en) * | 1992-05-21 | 1994-08-18 | Dornier Gmbh | Earth observation procedures |
JPH09153131A (en) | 1995-11-30 | 1997-06-10 | Hitachi Ltd | Method and device for processing picture information and picture information integrating system |
JP2000111359A (en) | 1998-10-05 | 2000-04-18 | Hitachi Ltd | Earth observation system |
DE19950433A1 (en) | 1999-10-19 | 2001-04-26 | Philips Corp Intellectual Pty | Network has nodes for media access checking with test signal generators for providing test signals outside allocated time slots, detectors of faults in associated and/or other node(s) |
SE519700C2 (en) * | 1999-12-23 | 2003-04-01 | Wespot Ab | Image Data Processing |
JP2002218439A (en) | 2001-01-19 | 2002-08-02 | Fujitsu General Ltd | Image monitor system |
JP2003264726A (en) | 2002-03-08 | 2003-09-19 | Hitachi Kokusai Electric Inc | Network imaging apparatus |
JP2004007066A (en) * | 2002-05-30 | 2004-01-08 | Kddi Corp | Picture transmission device |
JP2006115283A (en) | 2004-10-15 | 2006-04-27 | Mitsubishi Electric Corp | Data compressor and data expanding device |
JP4701093B2 (en) | 2006-01-31 | 2011-06-15 | 綜合警備保障株式会社 | Image processing apparatus, monitoring center, monitoring system, image processing method, and image processing program |
JP2012138745A (en) | 2010-12-27 | 2012-07-19 | Canon Inc | Image display system |
US9854209B2 (en) * | 2011-04-19 | 2017-12-26 | Ford Global Technologies, Llc | Display system utilizing vehicle and trailer dynamics |
JP5939124B2 (en) | 2012-10-12 | 2016-06-22 | 株式会社Jvcケンウッド | Imaging apparatus and image processing method |
US10853410B2 (en) * | 2014-03-04 | 2020-12-01 | Orbit Logic, Inc. | Method for providing imaging satellite information on a mobile device |
JP6072100B2 (en) * | 2015-01-30 | 2017-02-01 | キヤノン株式会社 | Radiation imaging system, control method, control method, and program |
AU2017223078B2 (en) | 2016-02-22 | 2022-11-17 | Live Earth Imaging Enterprises, LLC | Image sensor and method for a geostationary orbiting satellite |
-
2021
- 2021-01-06 WO PCT/JP2021/000142 patent/WO2021149485A1/en unknown
- 2021-01-06 JP JP2021573049A patent/JPWO2021149485A1/ja active Pending
- 2021-01-06 EP EP21744996.6A patent/EP4096219A4/en active Pending
- 2021-01-06 CN CN202180009107.0A patent/CN114946176A/en active Pending
- 2021-01-06 US US17/792,564 patent/US12041363B2/en active Active
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20220009631A1 (en) * | 2018-01-30 | 2022-01-13 | Aerovironment, Inc. | Methods and Systems for Cloud-Based Management of Images Captured by Aerial Vehicles |
US11919639B2 (en) * | 2018-01-30 | 2024-03-05 | Aerovironment, Inc. | Methods and systems for cloud-based management of images captured by aerial vehicles |
US20220318971A1 (en) * | 2021-04-02 | 2022-10-06 | Samsung Electronics Co., Ltd. | Electronic device for supporting machine learning-based image processing |
US11954833B2 (en) * | 2021-04-02 | 2024-04-09 | Samsung Electronics Co., Ltd | Electronic device for supporting machine learning-based image processing |
Also Published As
Publication number | Publication date |
---|---|
EP4096219A1 (en) | 2022-11-30 |
WO2021149485A1 (en) | 2021-07-29 |
EP4096219A4 (en) | 2023-06-14 |
US12041363B2 (en) | 2024-07-16 |
CN114946176A (en) | 2022-08-26 |
JPWO2021149485A1 (en) | 2021-07-29 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20200084414A1 (en) | Real-time moving platform management system | |
JP7497727B2 (en) | Imaging method and transmitting device for satellite system | |
US6674391B2 (en) | System and method of simulated image reconstruction | |
US12041363B2 (en) | Display control device, display control method, and program for displaying images based on satellite imaging | |
EP4095809A1 (en) | Image generation device, image generation method, and program | |
WO2020250709A1 (en) | Artificial satellite and control method for same | |
US20240056556A1 (en) | Artificial satellite and ground system | |
US20240048671A1 (en) | Ground system and image processing method thereof | |
WO2020250708A1 (en) | Image management method and metadata data structure | |
US20240029391A1 (en) | Sensor device and data processing method thereof | |
WO2020250706A1 (en) | Image processing method and metadata data structure |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SONY GROUP CORPORATION, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:UMEDA, TETSUO;SHIMIZU, ITARU;KIKUCHI, NAOMICHI;SIGNING DATES FROM 20220601 TO 20220805;REEL/FRAME:060845/0113 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: SENT TO CLASSIFICATION CONTRACTOR |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |