WO2021200212A1

WO2021200212A1 - Information processing device, information processing method, and program

Info

Publication number: WO2021200212A1
Application number: PCT/JP2021/011067
Authority: WO
Inventors: 翼乾; 翔介青合; 浩太堀内
Original assignee: ソニーグループ株式会社
Priority date: 2020-03-30
Filing date: 2021-03-18
Publication date: 2021-10-07

Abstract

This information processing device (20) comprises a control unit (21). The control unit (21) extracts, from image data, important image data in an important area and unimportant image data in an unimportant area. The control unit (21) allocates the important image data and the unimportant image data to respective different network slices.

Description

Information processing equipment, information processing methods and programs

This disclosure relates to information processing devices, information processing methods and programs.

Conventionally, there is a wireless communication technology that exchanges various data using wireless communication. For example, in recent years, an increasing number of applications have been used to access mobile network games from terminal devices such as smartphones via wireless networks. A technique for rendering a three-dimensional model in two dimensions when such an application transmits an image to a terminal device is known. In such a technique, a high-quality image is transmitted to a terminal device while reducing the processing load of the terminal device by allocating a bit rate to the gaze object by the user on the screen.

Japanese Unexamined Patent Publication No. 2007-79664

However, with the above technology, when the amount of image transmission increases, there is a risk that image transmission delay and image quality deterioration may occur, and usability may deteriorate.

Therefore, the present disclosure provides an information processing device, an information processing method, and a program capable of suppressing deterioration of usability.

According to the present disclosure, an information processing device is provided. The information processing device includes a control unit. The control unit extracts important image data of the important region and non-important image data of the non-important region from the image data. The control unit allocates the important image data and the non-important image data to different network slices.

It is a figure which shows the outline of the information processing system to which the proposed technique of this disclosure is applied. It is a figure for demonstrating the outline of the proposed technique of this disclosure. It is a figure which shows an example of the structure of the information processing system which concerns on 1st Embodiment of this disclosure. It is a figure for demonstrating the gaze area which concerns on 1st Embodiment of this disclosure. It is a sequence diagram for demonstrating the communication process which concerns on the 1st Embodiment of this disclosure. It is a block diagram which shows the structure of the information processing system which concerns on the 2nd Embodiment of this disclosure. It is a figure for demonstrating the important area which concerns on the 2nd Embodiment of this disclosure. It is a figure for demonstrating the important area which concerns on the 2nd Embodiment of this disclosure. It is a sequence diagram for demonstrating the communication process which concerns on the 2nd Embodiment of this disclosure. It is a block diagram which shows the structure of the information processing system which concerns on 3rd Embodiment of this disclosure. It is a sequence diagram for demonstrating the communication process which concerns on the 3rd Embodiment of this disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

Further, in the following description, for example, a numerical value may be given as a specific example, but such a numerical value is an example, and another numerical value may be used.

The explanations will be given in the following order.
1. 1. Introduction 1.1. Outline of the proposed technology 2. First Embodiment 2.1. Information processing system 2.2. Setting important areas 2.3. Communication processing 3. Second embodiment 4. Third embodiment 5. Other embodiments 6. supplement

<< 1. Introduction >>
<1.1. Outline of proposed technology ＞
First, the outline of the proposed technique according to the present disclosure will be described with reference to FIGS. 1 and 2. FIG. 1 is a diagram showing an outline of an information processing system 1 to which the proposed technique of the present disclosure is applied. FIG. 2 is a diagram for explaining an outline of the proposed technique of the present disclosure. The proposed technique according to the present disclosure is implemented in the information processing system 1 shown in FIG. As shown in FIG. 1, the information processing system 1 includes an information processing device 20 and a terminal device 10. The information processing device 20 and the terminal device 10 communicate with each other via the base station device 30.

The information processing device 20 is, for example, a server device, and provides services such as game distribution and video distribution to the terminal device 10 via the base station device 30. The information processing device 20 transmits a moving image to, for example, the terminal device 10.

The terminal device 10 displays a moving image received from the information processing device 20 to the user. FIG. 1 shows a case where the terminal device 10 is configured as a so-called head-mounted device that is worn and used on at least a part of the user's head. The terminal device 10 may be configured to be able to detect the line of sight of the user, for example.

The base station device 30 is a wireless communication device that wirelessly communicates with the terminal device 10. The base station device 30 is, for example, a device corresponding to a radio base station (Node B, eNB, gNB, etc.), and provides the terminal device 10 with a cellular communication service such as NR (New Radio). The base station device 30 may be a wireless relay station. The base station device 30 may be a road base station device such as an RSU (Road Side Unit). Further, the base station device 30 may be an optical overhanging device called RRH (Remote Radio Head).

The base station device 30 connects to the information processing device 20 via, for example, a network, and transmits information about services provided by the information processing device 20, such as moving images, to the terminal device 10. Further, the base station device 30 provides the information from the terminal device 10 to the information processing device 20 via the network.

In this way, the information processing device 20 and the terminal device 10 transmit and receive information via the base station device 30, so that the terminal device 10 receives a service such as video distribution from the information processing device 20. Can be done.

Here, when the information processing device 20 transmits the moving image data to be distributed using a transmission line of the same quality, there is a problem that data loss or error occurs depending on the communication environment, and usability is deteriorated.

In recent years, NR (New Radio) is being studied as a next-generation wireless access method for LTE. In NR and next-generation core networks, one type of communication assumes various use cases including eMBB (Enhanced Mobile Broadband), mMTC (Massive Machine Type Communications), and URLLC (Ultra Reliable and Low Latency Communications). Slicing technology (Network Slicing) for accommodating in a network is being studied. According to slicing technology, logical networks called slices or network slices can coexist in one physical network. This makes it possible to efficiently provide the network according to the requirements of the service used by the user.

Therefore, as shown in FIG. 2, in the proposed technique, the information processing device 20 extracts an important region and a non-important region from image data which is one frame of a moving image to be distributed, for example. The information processing device 20 allocates the image data of the important area and the image data of the non-important area to different network slices and transmits the image data. As a result, one image data can be transmitted using transmission lines of different qualities depending on the area.

For example, in FIG. 2, the information processing apparatus 20 extracts, for example, an area that the user is gazing at as an important area, and extracts other areas as non-important areas. The information processing apparatus 20 allocates the image data of the important region to the slice S1 for transmitting the data with high reliability and low delay and transmits the image data, and allocates the non-important region to the slice S2 for transmitting the data with the best effort and transmits the data.

As a result, the information processing device 20 can transmit, for example, image data of an important region that the user is gazing at with high reliability and low delay, and can suppress delay of data in the important region and deterioration of image quality. It is possible to suppress the deterioration of usability.

Further, the information processing device 20 transmits, for example, image data of a non-important area that the user is not paying attention to with normal quality (for example, best effort). As a result, data delay and image quality deterioration may occur in the non-important area, but since such an area is not the area that the user is paying attention to, the usability is not significantly affected.

In this way, the information processing device 20 allocates the important area and the non-important area of the image data to different network slices, so that the data delay and the image quality deterioration of the important area can be suppressed, and the deterioration of usability can be suppressed.

Further, if the information processing device 20 allocates all the image data to a network slice with high reliability and low delay and transmits the data, the radio resource may be significantly occupied and the utilization efficiency of the radio resource may decrease.

Therefore, in the proposed technique of the present disclosure, as described above, the information processing apparatus 20 allocates the important region and the non-important region of the image data to different network slices. As a result, the information processing system 1 can suppress the deterioration of the utilization efficiency of the wireless resource while suppressing the deterioration of the usability.

<< 2. First Embodiment >>
<2.1. Information processing system ＞
FIG. 3 is a diagram showing an example of the configuration of the information processing system 1 according to the first embodiment of the present disclosure. The information processing system 1 includes a terminal device 10 and an information processing device 20.

[Terminal device 10]
The terminal device 10 is a so-called head-mounted device that is worn on the user's head, such as an AR glass or a head-mounted display.

As shown in FIG. 3, the terminal device 10 includes a control unit 11, a storage unit 12, a communication unit 13, and a display unit 14. The configuration of the terminal device 10 is not limited to the configuration shown in FIG. 3, and may be another configuration as long as it is a configuration for performing information processing described later.

(Communication unit 13)
The communication unit 13 is a communication interface (I / F) that communicates with an external device. The communication unit 13 is realized by, for example, a NIC (Network Interface Card) or the like. For example, the communication unit 13 connects to the core network by performing wireless communication with the base station device 30. The communication unit 13 receives image data from the information processing device 20 via the base station device 30. The communication unit 13 transmits the line-of-sight information described later to the information processing device 20 via the base station device 30.

(Display unit 14)
The display unit 14 displays the image data transmitted by the information processing device 20. When the terminal device 10 according to the present embodiment is, for example, an AR glass or a head-mounted display, the display unit 14 corresponds to the display unit of these AR glasses or the head-mounted display. Here, it is assumed that the terminal device 10 is an AR glass or a head-mounted display, but the present invention is not limited to this. For example, the display unit 14 may be a display device (such as the AR glass or head-mounted display described above) different from the terminal device 10, and the terminal device 10 may be an information processing device for displaying image data on the display device. In this case, the terminal device 10 may be, for example, a smartphone or a PC.

(Memory unit 12)
The storage unit 12 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 12 stores programs, calculation parameters, and the like used for processing by the control unit 11.

(Control unit 11)
The control unit 11 is, for example, a controller, and various programs stored in a storage device inside the terminal device 10 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like are stored in a RAM (Random Access Memory). ) Is executed as a work area. For example, the various programs include programs of applications installed in the terminal device 10. Further, the control unit 11 is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 11 has a line-of-sight detection unit 111, and realizes or executes an information processing function or operation described below. The internal configuration of the control unit 11 is not limited to the configuration shown in FIG. 3, and may be another configuration as long as it is a configuration for performing information processing described later.

(Gaze detection unit 111)
The line-of-sight detection unit 111 detects the user's line-of-sight direction by using eye tracking technology based on the detection results of, for example, a camera, an optical sensor, and a motion sensor (all not shown) mounted on the terminal device 10. .. The line-of-sight detection unit 111 determines the gaze area of the display screen that the user is gazing at based on the detected line-of-sight direction. The line-of-sight detection unit 111 transmits the line-of-sight information including the determined gaze area to the information processing device 20. As will be described later, the gaze area is an area used when the information processing apparatus 20 sets an important area, and thus can be said to be information about the important area.

[Information processing device 20]
As shown in FIG. 3, the information processing device 20 is, for example, a server device, and delivers video and data (for example, moving images) to the terminal device 10.
The information processing device 20 includes a control unit 21, a storage unit 22, a communication unit 23, and a display unit 24. The configuration of the information processing device 20 is not limited to the configuration shown in FIG. 3, and may be any other configuration as long as it performs information processing described later.

(Communication unit 23)
The communication unit 23 is a communication interface (I / F) that communicates with an external device. The communication unit 23 is realized by, for example, a NIC (Network Interface Card) or the like. For example, the communication unit 23 connects to the base station device 30 via a network. The communication unit 23 receives line-of-sight information from the terminal device 10 via the base station device 30. The communication unit 23 transmits image data to the terminal device 10 via the base station device 30.

(Display unit 24)
The display unit 24 is a display that displays various information of the information processing device 20. The display unit 24 may be a display such as a liquid crystal display.

(Memory unit 22)
The storage unit 22 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 22 stores programs, calculation parameters, and the like used for processing by the control unit 11. The storage unit 22 may hold the image data to be transmitted to the terminal device 10.

(Control unit 21)
The control unit 21 is, for example, a controller, and various programs stored in a storage device inside the information processing device 20 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like are stored in a RAM (Random Access). It is realized by executing Memory) as a work area. For example, the various programs include programs of applications installed in the information processing apparatus 20. Further, the control unit 21 is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 21 has an important area setting unit 211, and realizes or executes an information processing function or operation described below. The internal configuration of the control unit 21 is not limited to the configuration shown in FIG. 3, and may be another configuration as long as it is a configuration for performing information processing described later.

(Important area setting unit 211)
The important area setting unit 211 sets the important area and the non-important area based on the line-of-sight information acquired from the terminal device 10. The important area setting unit 211 determines the gaze area included in the line-of-sight information as an important area, and determines the other areas as non-important areas.

The important area setting unit 211 extracts the image data corresponding to the determined important area as important information (important image data) and assigns it to a network slice (slice S1 in FIG. 3) that transmits a signal with high reliability and low delay. Slice S1 is, for example, a network slice for URLLC. Further, the important area setting unit 211 extracts the image data corresponding to the determined non-important area as non-important information (non-important image data) and allocates it to the network slice (slice S2 in FIG. 3) to be transmitted with best effort.

The important image data and the non-important image data assigned to the slices S1 and S2 by the important area setting unit 211 are transmitted to the terminal device 10 via the communication unit 23.

<2.2. Setting important areas>
Here, the setting of the important area in the information processing system 1 will be described. First, the gaze area according to the first embodiment of the present disclosure will be described with reference to FIG. FIG. 4 is a diagram for explaining a gaze area according to the first embodiment of the present disclosure.

As described above, the terminal device 10 determines the area in which the user is gazing based on the line-of-sight direction of the user.

As shown in FIG. 4, it is assumed that the user wears the terminal device 10 which is a head mount display, for example, on the head and is watching the object Ob displayed on the screen of the terminal device 10.

At this time, as described above, the terminal device 10 uses eye tracking technology or the like based on the detection results of sensors such as a camera, an optical sensor, or a motion sensor (all of which are not shown) mounted on the own device. Detect the line-of-sight direction.

The terminal device 10 detects at least one feature amount, position, and movement of the user's eye as an eye tracking technique, for example. The terminal device 10 detects at least one feature amount, position, and movement of the user's eye by photographing the user's eye with a camera, for example, and analyzes the captured image, and performs eye tracking (line-of-sight detection). .. Alternatively, the terminal device 10 may perform line-of-sight detection by detecting at least one feature amount, position, and movement of the user's face or head in addition to or instead of the user's eyes.

The terminal device 10 determines the display area of the object Ob displayed in the detected line-of-sight direction as the gaze area. For example, when the cube object Ob is displayed as shown in FIG. 4, the display area of the cube object Ob is set as the gaze area.

Alternatively, a predetermined area R1 including the object Ob displayed in the line-of-sight direction of the terminal device 10 may be determined as the gaze area.

In this way, the terminal device 10 determines the area including the object Ob displayed in the line-of-sight direction as the gaze area, and transmits the line-of-sight information including the information regarding the determined gaze area to the information processing device 20.

When the information processing device 20 acquires the line-of-sight information from the terminal device 10, the information processing device 20 extracts important image data and non-important image data corresponding to each of the important area and the non-important area from the image data based on the line-of-sight information. The information processing device 20 adds slice-related information for using the corresponding slice to each image data so that the extracted image data is transmitted under desired transmission conditions (for example, low delay, best effort, etc.). ..

Here, the slice-related information is information (metadata) for specifying a slice to be assigned according to the use and characteristics of the signal to be transmitted (important image data or non-important image data in this case). The control unit 21 of the information processing device 20 controls the communication unit 23 so as to associate the image data with the slice-related information for specifying the slice to which the image data is to be assigned and transmit the image data to the base station device 30. The base station apparatus 30 assigns each received image data to a slice specified by each slice-related information, and transmits each image data to the terminal apparatus 10 using the assigned slice.

For example, the information processing device 20 determines that important image data is a signal having a high priority and needs to be transmitted with a low delay. The information processing device 20 generates slice-related information for identifying a slice S1 having a high priority so as to ensure transmission with low delay for the important image data. The information processing device 20 associates the generated slice-related information with important image data.

Further, the information processing device 20 determines that the non-important image data is a signal having a lower priority than the important image data. The information processing apparatus 20 generates slice-related information for identifying a slice S2 different from the slice S1 (for example, a slice S2 having a lower priority than the slice S1). The information processing device 20 associates the generated slice-related information with the non-important image data.

In this way, the information processing apparatus 20 can allocate and transmit slices according to the characteristics required for each image data, and the user can view the important image data having a high priority in a low delay state while not transmitting it. Important image data can also be viewed.

Although the control unit 21 of the information processing apparatus 20 has described an example of generating slice-related information for specifying a slice to be assigned to image data to be transmitted, the present invention is not limited to this. The control unit 21 of the information processing apparatus 20 may simply associate the identification information indicating the type (important, non-important, etc.) of the image data to be transmitted with the image data. In this case, the base station apparatus 30 may allocate each image data to the corresponding slices for each type based on the identification information associated with each received image data. The base station apparatus 30 holds in advance the relationship between the type of image data and the slices to be assigned for each type, and assigns slices to each image data by referring to the relationship.

In this way, the base station apparatus 30 can assign slices to the image data, so that each image data can be transmitted in different network slices.

<2.3. Communication processing>
FIG. 5 is a sequence diagram for explaining the communication process according to the first embodiment of the present disclosure.

As shown in FIG. 5, the terminal device 10 detects the line of sight of the user (step S101). Next, the terminal device 10 determines the gaze area to be gazed by the user from the detected line of sight of the user, and transmits the line-of-sight information including the information about the determined gaze area to the information processing device 20 (step S102).

The information processing device 20 sets an important area based on the line-of-sight information (step S103). The information processing device 20 extracts important image data in an important region and non-important image data in an region excluding the important region from the image data, and assigns each of the extracted image data to different slices (step S104).

The information processing device 20 distributes the video to the user by transmitting each image data in each slice (step S105).

The information processing system 1 may execute the above-mentioned communication processing for each frame of the video to be distributed, for example. Alternatively, the cycle of line-of-sight detection and the cycle of video distribution (frame cycle) may be different. In this case, the information processing device 20 updates the important area at the timing when the line-of-sight information is acquired from the terminal device 10.

Here, a case where the terminal device 10 fails to receive non-important image data will be described. In this case, the terminal device 10 shall display to the user the non-important image data that was successfully received at a timing prior to the reception failure. Alternatively, when the terminal device 10 receives the non-important image data later than the important image data, the terminal device 10 displays the non-important image data received before that until the non-important image data is received to the user. It shall be.

Further, when a defect or an error occurs in at least a part of the non-important image data received by the terminal device 10, the terminal device 10 receives the non-important image data received before the non-important image data in which the defect or the error occurs. It shall be displayed to the user. Alternatively, the terminal device 10 may interpolate (correct) using the non-important image data for which the loss or error is correctly received and present it to the user.

As described above, in the first embodiment, the information processing device 20 divides the image data into an important area and a non-important area according to the gaze area detected by the terminal device 10. The information processing device 20 allocates the image data of the important region and the image data of the non-important region to different network slices.

As a result, the information processing system 1 can suppress the deterioration of usability and the deterioration of the utilization efficiency of wireless resources.

<< 3. Second embodiment >>
In the first embodiment described above, the information processing device 20 sets the important area based on the line-of-sight information detected by the terminal device 10, but the information processing device 20 sets the important area based on the image data. You may. Therefore, as a second embodiment, a case where the information processing apparatus 20 sets an important region based on the image data will be described.

FIG. 6 is a block diagram showing the configuration of the information processing system 2 according to the second embodiment of the present disclosure. The terminal device 10 of the information processing system 2 shown in FIG. 6 is different from the information processing system 1 shown in FIG. 3 in that it does not have the line-of-sight detection unit 111 and does not transmit the line-of-sight information to the information processing device 20. ..

Further, the important area setting unit 211 of the information processing device 20 shown in FIG. 6 sets an important area based on image data instead of line-of-sight information.

Here, the setting of the important area by the important area setting unit 211 will be described with reference to FIGS. 7 and 8. 7 and 8 are diagrams for explaining important areas according to the second embodiment of the present disclosure.

The important area setting unit 211 identifies an important area, for example, by performing image recognition of image data in advance. For example, as shown in FIG. 7, when the video of a soccer game is distributed, the important area setting unit 211 identifies the ball and the player holding the ball from the image data by image recognition, and holds the ball. The area including the player and the periphery of the player is set as the important area R1. Further, the important area setting unit 211 sets an area other than the important area R1 as a non-important area.

In this way, the important area setting unit 211 recognizes the image data and sets the important area according to the type of the video to be distributed. The important area setting unit 211 sets a predetermined area including a player in the case of sports and a performer in the case of a drama or a movie as an important area. Alternatively, what the video creator uses to set the important area (for example, an object such as a person or a ball) may be specified. In this case, the important area setting unit 211 performs image recognition, identifies the designated one, and sets the important area.

Alternatively, the important area setting unit 211 may set a predetermined (set) area as the important area. The important area is generally placed in the center of the screen. Therefore, as shown in FIG. 8, the important area setting unit 211 sets, for example, the central area of the screen as the important area and the other peripheral areas as the non-important area.

FIG. 9 is a sequence diagram for explaining the communication process according to the second embodiment of the present disclosure.

As shown in FIG. 9, the information processing device 20 performs image recognition of image data (step S201). Next, the information processing device 20 sets an important region based on the result of image recognition (step S202). The subsequent processing is the same as the communication processing shown in FIG.

As described above, in the second embodiment, the information processing device 20 divides the image data into an important area and a non-important area according to the result of image recognition. The information processing device 20 allocates the image data of the important region and the image data of the non-important region to different network slices.

As a result, the information processing system 2 can suppress the deterioration of usability and the deterioration of the utilization efficiency of wireless resources.

<< 4. Third Embodiment >>
In the first and second embodiments described above, the information processing device 20 delivers the video to the terminal device 10, but the information processing device may receive the video. Therefore, as a third embodiment, a case where the information processing device 60 receives an image from the image pickup device 50 will be described.

FIG. 10 is a block diagram showing the configuration of the information processing system 3 according to the third embodiment of the present disclosure. The information processing system 3 shown in FIG. 10 has an imaging device 50 instead of the terminal device 10 shown in FIG. Further, the information processing device 60 is different from the information processing device 20 shown in FIG. 3 in that it does not have the important area setting unit 211 and the display unit 24.

The imaging device 50 shown in FIG. 10 includes a control unit 51, a storage unit 52, a communication unit 53, and an imaging unit 54.

The image pickup unit 54 takes an image of the image pickup target and generates image data. The imaging unit 54 may support high-resolution shooting such as 4K (horizontal number of pixels 3840 x vertical pixel number 2160) or 8K (horizontal pixel number 7680 x vertical pixel number 4320).

The storage unit 52 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 52 stores programs, calculation parameters, and the like used for processing by the control unit 51.

The communication unit 53 is a communication interface (I / F) that communicates with an external device. The communication unit 53 is realized by, for example, a NIC (Network Interface Card) or the like. For example, the communication unit 53 connects to the core network by performing wireless communication with the base station device 30. The communication unit 53 transmits image data to the information processing device 20 via the base station device 30.

The control unit 51 is, for example, a controller, and various programs stored in a storage device inside the image pickup device 50 by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like are stored in a RAM (Random Access Memory). ) Is executed as a work area. For example, the various programs include programs of applications installed in the image pickup apparatus 50. Further, the control unit 51 is realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 51 has an important area setting unit 511, and realizes or executes the functions and operations of information processing described below. The internal configuration of the control unit 51 is not limited to the configuration shown in FIG. 10, and may be another configuration as long as it is a configuration for performing information processing described later.

The important area setting unit 511 performs image analysis (for example, image recognition) of the image data captured by the imaging unit 54 to set the important area. For example, when the image data is a photograph of a sport such as soccer, the important area setting unit 511 sets the area including the ball and the player holding the ball as the important area. Alternatively, when the image data is a photograph of a marathon, the area including the runner is set as the important area, and the other area including, for example, the spectators along the road is set as the non-important area.

In addition to the method of identifying the person to be included in the important area by image analysis in this way, for example, by attaching a sensor such as an infrared sensor or a position sensor to the person, the important area setting unit 511 identifies the person on the image. It may be.

Alternatively, the image pickup device 50 may be provided with a sensor such as a microphone so that the image pickup device 50 detects the direction of sound using the sensor. The important area setting unit 511 may set the area determined based on the direction of the sound as the important area, assuming that there is an important person (for example, a speaker) in the direction of the detected sound.

Further, when the imaging device 50 shoots a program distributed by TV broadcasting or Internet broadcasting, the performers to be included in the important area from the plurality of performers appearing in the program are voted by, for example, a user who watches the program. You may decide. The important area setting unit 511 sets the area including the performer who was the most popular in the voting by the user as the important area.

Further, the important area setting unit 511 may set the important area according to the scenario of the image to be captured. For example, in a live music program or the like, there is a case where it is decided in advance as a scenario which camera is used at what time and what kind of image is to be shot. In this case, the important area setting unit 511 sets the important area and the non-important area according to a predetermined scenario.

The important area setting unit 511 allocates the image data of the important area and the image data of the non-important area to different network slices among the image data of the captured video. Since such an allocation is the same as the allocation by the information processing apparatus 20 of the information processing system 1, detailed description thereof will be omitted.

FIG. 11 is a sequence diagram for explaining the communication process according to the third embodiment of the present disclosure.

As shown in FIG. 11, the image pickup apparatus 50 performs image recognition of image data (step S301). Next, the image pickup apparatus 50 sets an important region based on the result of image recognition (step S302).

The imaging device 50 extracts important image data in an important region and non-important image data in an region excluding the important region from the image data, and assigns each extracted image data to a different slice (step S303).

The image pickup apparatus 50 transmits an image to the information processing apparatus 20 by transmitting each image data in each slice (step S304).

As described above, in the third embodiment, the image pickup apparatus 50 divides the image data into an important region and a non-important region according to, for example, the result of image recognition. The image pickup apparatus 50 allocates the image data of the important region and the image data of the non-important region to different network slices. In this way, even when the device on the terminal side (here, the image pickup device 50) sends an image to the information processing device 20, the information processing system 3 suppresses the deterioration of usability and the utilization efficiency of wireless resources. Can be suppressed.

<< 5. Other embodiments >>
Each of the above embodiments shows an example, and various modifications and applications are possible.

In each of the above embodiments, the terminal device 10 is a head-mounted device, but the present invention is not limited to this. The terminal device 10 may be a device for displaying an image to a user, and may be a device such as a TV, a personal computer, or a smartphone.

Further, in each of the above embodiments, the information processing systems 1 to 3 transmit the important image data in the slice S1 having low delay or high reliability, and transmit the non-important image data in the slice S2 which transmits the non-important image data with the best effort. , Not limited to this.

For example, the information processing device 20 or the image pickup device 50 may transmit the important image data as data having a higher resolution than the non-important image data. For example, as described above, when the image pickup device 50 captures a 4K or 8K image, the image pickup device 50 transmits important image data with a high resolution of 4K or 8K. On the other hand, the image pickup apparatus 50 performs processing such as down-conversion on the non-important image data to lower the resolution and transmit the data. In other words, it can be said that the image pickup apparatus 50 down-converts the image data in the non-important region to generate the non-important image data. In this way, the information processing systems 1 to 3 may change the resolution according to the importance of the area. As described above, the important image data may have a higher resolution than the non-important image data.

Further, in each of the above embodiments, the case where the important

area setting units

211 and 511 set the important area and the non-important area, in other words, the image data is divided into two levels of importance, has been shown, but the present invention is limited to this. Not done. The important

area setting units

211 and 511 may divide the image data into areas according to the importance of three or more levels. In this case, the important

area setting units

211 and 511 allocate to different network slices according to the importance of the divided areas. For example, it is assumed that the important

area setting units

211 and 511 divide the image data into three stages of the most important, important, and non-important areas. In this case, the important

area setting units

211 and 511 allocate, for example, the image data divided into the most important areas to slices that require low-delay and highly reliable communication. The important

area setting units

211 and 511 allocate the image data divided into important areas to slices that require low-delay communication, and allocate the image data divided into non-important areas to slices to be transmitted with best effort.

As described above, the number of areas divided by the important

area setting units

211 and 511 and the number of slices to be allocated may be plural, and the number is not limited.

The control device for controlling the terminal device 10, the information processing devices 20, 60, the base station device 30, or the image pickup device 50 of each of the above embodiments may be realized by a dedicated computer system or a general-purpose computer system. ..

For example, a communication program for executing the above-mentioned operation (for example, communication processing) is stored and distributed in a computer-readable recording medium such as an optical disk, a semiconductor memory, a magnetic tape, or a flexible disk. Then, for example, the control device is configured by installing the program on a computer and executing the above-mentioned processing. At this time, the control device may be a terminal device 10, information processing devices 20, 60, a base station device 30, or an external device (for example, a personal computer) of the image pickup device 50. Further, the control device is a terminal device 10, an information processing device 20, 60, a base station device 30, or a device inside the image pickup device 50 (for example, a control unit 11, a control unit 21, or a control unit 51). May be good.

Further, the above communication program may be stored in a disk device provided in a server device on a network such as the Internet so that it can be downloaded to a computer or the like. Further, the above-mentioned functions may be realized by collaboration between the OS (Operating System) and the application software. In this case, the part other than the OS may be stored in a medium and distributed, or the part other than the OS may be stored in the server device so that it can be downloaded to a computer or the like.

Further, among the processes described in the above-described embodiment, all or a part of the processes described as being automatically performed can be manually performed, or the processes described as being manually performed can be performed. All or part of it can be done automatically by a known method. In addition, the processing procedure, specific name, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Further, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of distribution / integration of each device is not limited to the one shown in the figure, and all or part of the device is functionally or physically dispersed / physically distributed in arbitrary units according to various loads and usage conditions. Can be integrated and configured.

Further, the above-described embodiments can be appropriately combined in an area where the processing contents do not contradict each other. Further, the order of each step shown in the sequence diagram or the like of the above-described embodiment can be changed as appropriate.

Further, for example, the present embodiment includes a device or any configuration constituting the system, for example, a processor as a system LSI (Large Scale Integration) or the like, a module using a plurality of processors, a unit using a plurality of modules, or a unit. It can also be implemented as a set or the like (that is, a part of the configuration of the device) to which other functions are added.

In the present embodiment, the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and a device in which a plurality of modules are housed in one housing are both systems. ..

Further, for example, the present embodiment can have a cloud computing configuration in which one function is shared and jointly processed by a plurality of devices via a network.

<< 6. Supplement >>
Although the preferred embodiments of the present disclosure have been described in detail with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that anyone with ordinary knowledge in the technical field of the present disclosure may come up with various modifications or modifications within the scope of the technical ideas set forth in the claims. Is, of course, understood to belong to the technical scope of the present disclosure.

The above-described embodiments can be appropriately combined as long as the processing contents do not contradict each other.

Further, the effects described in the present specification are merely explanatory or exemplary and are not limited. That is, the techniques according to the present disclosure may exhibit other effects apparent to those skilled in the art from the description herein, in addition to or in place of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
The important image data of the important area and the non-important image data of the non-important area are extracted from the image data.
A control unit that assigns the important image data and the non-important image data to different network slices.
Information processing device equipped with.
(2)
The information processing device according to (1), wherein the control unit allocates the important image data to a network slice having a high priority.
(3)
The control unit
Obtaining information on the important area according to the line-of-sight direction of the user who views the image data,
The information processing apparatus according to (1) or (2), which extracts the important image data and the non-important image data from the image data according to the acquired information.
(4)
The information processing device according to (1) or (2), wherein the control unit extracts the important image data and the non-important image data from the image data based on the image processing result of the image data.
(5)
The information processing device according to (1) or (2), wherein the control unit extracts the important image data and the non-important image data from the image data based on the preset important area.
(6)
The information processing apparatus according to any one of (1) to (5), wherein the important image data has a higher resolution than the non-important image data.
(7)
The information processing device according to (6), wherein the control unit down-converts the image data in a non-important region to generate the non-important image data.
(8)
The important image data of the important area and the non-important image data of the non-important area are extracted from the image data.
Allocate the important image data and the non-important image data to different network slices.
Information processing method.
(9)
The computer extracts important image data in the important area and non-important image data in the non-important area from the image data.
A control unit that assigns the important image data and the non-important image data to different network slices.
A program that functions as.

1 Information processing system 10 Terminal equipment 11, 21, 51 Control unit 12, 22, 52

Storage unit

13, 23, 53 Communication unit 14, 24 Display unit 20, 60 Information processing equipment 30 Base station equipment 50 Imaging device 64 Imaging unit 111 Line-of-sight detection unit 211,511 Important area setting unit

Claims

The important image data of the important area and the non-important image data of the non-important area are extracted from the image data.
A control unit that assigns the important image data and the non-important image data to different network slices.
Information processing device equipped with.
The information processing device according to claim 1, wherein the control unit allocates the important image data to a network slice having a high priority.
The control unit
Obtaining information on the important area according to the line-of-sight direction of the user who views the image data,
The information processing apparatus according to claim 1, wherein the important image data and the non-important image data are extracted from the image data according to the acquired information.
The information processing device according to claim 1, wherein the control unit extracts the important image data and the non-important image data from the image data based on the image recognition result of the image data.
The information processing device according to claim 1, wherein the control unit extracts the important image data and the non-important image data from the image data based on the preset important area.
The information processing device according to claim 1, wherein the important image data has a higher resolution than the non-important image data.
The information processing device according to claim 6, wherein the control unit down-converts the image data in a non-important region to generate the non-important image data.
The important image data of the important area and the non-important image data of the non-important area are extracted from the image data.
Allocate the important image data and the non-important image data to different network slices.
Information processing method.
The computer extracts important image data in the important area and non-important image data in the non-important area from the image data.
A control unit that assigns the important image data and the non-important image data to different network slices.
A program that functions as.