JP7209985B1 - Computer system and computer-implemented method for displaying multidimensional data - Google Patents

Abstract

Kind Code: A1 A disclosed system is a computer system configured to execute a display process for displaying multidimensional data, the display process comprising: receiving a user's operation of determining a dimension to be a seek bar dimension and a dimension to be a non-seek bar dimension, which is a dimension other than the seek bar dimension, among the multi dimensions in the multi-dimensional data; displaying the seek bar for indicating a display location to be displayed as the display data in the seek bar dimension determined based on the above; generating and displaying the display data based on the operation; generating and displaying non-seekbar dimension data correspondence information based on the manipulation. [Selection drawing] Fig. 1

Description

The present disclosure relates to computer systems and computer-implemented methods for displaying multidimensional data.

A seek bar is sometimes used to indicate the playback point of time-varying data such as moving images. A conventional seek bar includes, for example, a slider movably provided on a bar that indicates from the beginning to the end of time-varying data such as moving images. The position of the slider indicates the playback time point of the data.

Patent Literature 1 discloses displaying a moving image together with a seek bar.

JP 2019-97137 A

A conventional seek bar as shown in Patent Document 1 only indicates the display position of data in the time dimension. That is, in the conventional seek bar, the seek bar dimension, which is the dimension indicated by the seek bar, is the time dimension.

Here, since the multidimensional data displayed using the seek bar is multidimensional, there may be restrictions on display on a display or the like. Display constraints can make data difficult to observe or analyze.

The present inventors came up with the idea of facilitating observation or analysis of multidimensional data by displaying multidimensional data in various modes. By displaying multidimensional data in various modes, it is expected that such difficulties will be alleviated.

Therefore, it is desirable to be able to display multidimensional data in various forms.

One aspect of this disclosure is a computer system. The disclosed system is a computer system configured to execute display processing for displaying multidimensional data with seek bar dimensions whose dimensions are indicated by the seek bar and non-seek bar dimensions which are dimensions displayed by display data. Then, the display processing is performed by a user who determines a dimension to be the seek bar dimension and a dimension to be the non-seek bar dimension, which is a dimension other than the seek bar dimension, among the multi-dimensions in the multi-dimensional data. receiving an operation; displaying the seek bar for indicating a display location to be displayed as the display data in the seek bar dimension determined based on the operation; and generating the display data based on the operation. generating and displaying non-seek bar dimensional data correspondence information based on the operation, and generating the data for display includes, of the multidimensional data, the operation wherein the non-seek bar dimension data corresponding to the non-seek bar dimension data determined based on the It is information generated from each of the non-seek bar dimension data at each location of the seek bar dimension determined based on the operation, and is displayed corresponding to each location of the seek bar dimension on the seek bar.

Another aspect of the disclosure is a computer-implemented method. The disclosed method is a computer-implemented method performed by a computer for displaying multi-dimensional data, with seekbar dimensions where the dimensions are indicated by the seekbar and non-seekbar dimensions where the dimensions are displayed by the data for display. receiving a user's operation of determining a dimension to be the seek bar dimension and a dimension to be the non-seek bar dimension, which is a dimension other than the seek bar dimension, among the multi-dimensions in the multi-dimensional data; displaying the seek bar for indicating a display location to be displayed as the display data in the seek bar dimension determined based on the operation; generating and displaying the display data based on the operation; generating and displaying non-seek bar dimensional data correspondence information based on the operation; Seek bar dimension data, including generating the data at the display location indicated by the seek bar as the display data, wherein the non-seek bar dimension data correspondence information is determined based on the operation It is information generated from each of the non-seek bar dimension data at each location of the seek bar dimension, and is displayed corresponding to each location of the seek bar dimension on the seek bar.

Further details are described as embodiments below.

FIG. 1 is a configuration diagram of a computer system. FIG. 2 is a flow chart of a process performed by a computer system. FIG. 3 is a schematic diagram of input data, output data, and a classification model. FIG. 4 is a flow chart of data display. FIG. 5 is a schematic diagram (selected dimension=T) of the area for data display. FIG. 6 is a schematic diagram of the area for data display (selected dimension=T). FIG. 7 is a schematic diagram of the area for data display (selected dimension=T). FIG. 8 is a schematic diagram of the area for data display (selected dimension=T). FIG. 9 is a schematic diagram (selected dimension=Z) of the area for data display. FIG. 10 is a schematic diagram showing variations in representation of four-dimensional data. FIG. 11 is a conceptual diagram of generation of display data. FIG. 12 is a schematic diagram (selected dimension=Z) of the area for data display. FIG. 13 is a conceptual diagram of generation of display data. FIG. 14 is a schematic diagram (selected dimension=Z) of the area for data display. FIG. 15 is a schematic diagram (selected dimension=Z) of the area for data display. FIG. 16 is a schematic diagram of the area for data display (selected dimension=Y). FIG. 17 is a schematic diagram of the area for data display (selected dimension=X). FIG. 18 is a schematic diagram showing a modification of the seek bar.

<1. Computer system and computer-implemented method for displaying multidimensional data>

(1) A system according to an embodiment is configured to execute display processing for displaying multidimensional data in a seek bar dimension whose dimension is indicated by a seek bar and a non-seek bar dimension that is a dimension displayed by display data. wherein the display processing includes: among the multidimensional data in the multidimensional data, a dimension serving as the seek bar dimension; a dimension serving as the non-seek bar dimension, which is a dimension other than the seek bar dimension; displaying the seek bar for indicating a display location to be displayed as the display data in the seek bar dimension determined based on the operation; based on the operation, the generating and displaying data for display; generating and displaying non-seek bar dimensional data correspondence information based on the operation; generating the data for display includes generating and displaying the multidimensional data; wherein the non-seek bar dimension data determined based on the operation, wherein the data at the display location indicated by the seek bar is generated as the display data, the non-seek bar dimension data The corresponding information is information generated from each of the non-seek bar dimension data at each location of the seek bar dimension determined based on the operation, and is displayed corresponding to each location of the seek bar dimension on the seek bar. .

In this case, the seek dimension and the non-seek bar dimension can be arbitrarily determined by the user's operation, so the dimension indicated by the seek bar (seek bar dimension) can be changed. In addition, since the dimension displayed by the display data (non-seek bar dimension; display dimension) is also changed with the change of the seek bar dimension, the display mode of the display data can also be changed.

Furthermore, in the present embodiment, non-seek bar dimension data correspondence information is generated and displayed corresponding to each location of the seek bar dimension. The user can be presented with information that serves as a reference for the user to specify. Moreover, since the non-seek bar dimension data correspondence information is changed in accordance with the change of the non-seek bar dimension, appropriate non-seek bar dimension data correspondence information is displayed.

Note that the multidimensional data may be four-dimensional spatio-temporal data having a three-dimensional spatial dimension and a temporal dimension. The multidimensional data may be three-dimensional data, or may be five-dimensional data or more. The number of dimensions of the displayed multidimensional data is the sum of the number of dimensions of the seek bar dimensions indicated by the seek bar and the number of non-seek bar dimensions (display dimensions) that are the dimensions displayed by the display data. be. The number of seek bar dimensions may be 1 or 2 or more. The number of non-seek bar dimensions (display dimensions) is, for example, 2 or more, preferably 3 or more. The display data may be, for example, two-dimensional image data or three-dimensional image data. The display data may be composed of, for example, a combination of two-dimensional data or a combination of three-dimensional data.

(2) Preferably, the seek bar is configured to simultaneously display the non-seek bar dimension data correspondence information at each location of the seek bar dimension.

(3) It is preferable that the non-seek bar dimension data correspondence information is displayed on the seek bar corresponding to the location specified by the user, and non-specified locations are not displayed.

(4) Preferably, the multidimensional data includes a multidimensional heat map. Preferably, the non-seek bar dimension data correspondence information is generated from the heat map of the non-seek bar dimension at each location of the seek bar dimension determined based on the operation.

(5) the display data is displayed in a display area for displaying the display data, and when the non-seek bar dimension is changed based on the operation, the display data displayed in the display area is It is preferable that the non-seek bar dimension display data before change is switched to the non-seek bar dimension display data after change.

(6) When the non-seek bar dimension is changed based on the operation, in the seek bar, the non-seek bar dimension data correspondence information displayed corresponding to each location of the seek bar dimension is the non-seek bar dimension before the change. It is preferable that the non-seek bar dimension data correspondence information is switched to the changed non-seek bar dimension data correspondence information.

(7) A method according to an embodiment is a computer-implemented method for displaying multi-dimensional data, with seekbar dimensions whose dimensions are indicated by the seekbar, and non-seekbar dimensions, which are dimensions displayed by data for display. In the implementation method, a user's operation of determining a dimension to be the seek bar dimension and a dimension to be the non-seek bar dimension that is a dimension other than the seek bar dimension among the multi-dimensions in the multi-dimensional data displaying the seek bar for indicating a display location to be displayed as the display data in the seek bar dimension determined based on the operation; generating the display data based on the operation; , displaying, generating and displaying non-seek bar dimensional data correspondence information based on the operation, and generating the display data includes, among the multidimensional data, based on the operation generating the determined non-seek bar dimension data at the display location indicated by the seek bar as the display data, wherein the non-seek bar dimension data correspondence information is generated based on the operation It is information generated from each of the non-seek bar dimension data at each location of the seek bar dimension determined by the above, and is displayed corresponding to each location of the seek bar dimension on the seek bar.

<2. Examples of computer systems and computer-implemented methods for displaying multidimensional data>

FIG. 1 shows a computer system 10 according to an embodiment. The system 10 executes processing such as display processing of multidimensional data. System 10 may be configured by one or more computers. In FIG. 1, the system 10 is configured by one computer as an example. A computer that configures the system 10 is, for example, a server connected to the network 15 . The server can execute processing for displaying multidimensional data and other necessary processing on terminals 30 (client terminals) connected via a network. Network 15 is, for example, the Internet or a local area network. Note that the system 10 may be of a stand-alone type.

Terminal 30 is a computer connectable to network 15 . A computer that configures the terminal 30 is, for example, a personal computer, a smart phone, or a tablet. Terminal 30 may comprise a display 31 . Terminal 30 may be connected to system 10 via network 15 . Display 31 of terminal 30 may display multi-dimensional data. As an example, the system 10 according to the embodiment executes processing and the like for displaying multidimensional data on the display 31 of the terminal 30 .

System 10 may be configured by a computer comprising processor 10A and storage device 10B. The storage device 10B is connected to the processor 10A. The storage device 10B includes, for example, a primary storage device and a secondary storage device. A primary storage device is, for example, a RAM. The secondary storage device is, for example, a hard disk drive (HDD) or solid state drive (SSD). Storage device 10B comprises a computer program 10C executed by processor 10A. The processor 10A reads and executes a computer program 10C stored in the storage device 10B.

The computer program 10C has program code indicative of instructions for displaying multi-dimensional data and other necessary instructions.

Storage device 10B may store various data or information. The storage device 10B can store various data used for processing for displaying multidimensional data. The storage device 10B can store, for example, input data 100, output data 150, and a trained model 180, which will be described later.

FIG. 2 shows an example of processing that the system 10 according to the embodiment performs for data display. The process of FIG. 2 includes obtaining input data 100, obtaining output data 150 from input data 100, and displaying output data 150. FIG. Below, output data 150 is displayed by system 10 . The output data 150 is, for example, data obtained by adding data indicating a region of interest in the input data 100 to the input data 100 .

Input data 100 and output data 150 are multidimensional data. The input data 100 and the output data 150 are, for example, four-dimensional data. The number of dimensions of the input data and the output data 150 may be 3 or 5 or more. The input data 100 and the output data 150 shown in FIG. 3 are, for example, four-dimensional spatio-temporal data having three-dimensional spatial dimensions (X, Y, Z) and one-dimensional temporal dimension T. FIG. However, the type of each dimension in multidimensional data is not particularly limited. Multidimensional data need not have a time dimension. The spatial dimensions may be two dimensions (eg, X and Y dimensions). Multidimensional data need not have a spatial dimension. Dimensions in multidimensional data need not be physical dimensions with physical meaning, such as spatial and temporal dimensions. The dimensions in multidimensional data can be freely defined by the designer or user of system 10 .

The three-dimensional spatial dimensions shown in FIG. 3 have a dimension X, a dimension Y orthogonal to dimension X, and a dimension Z orthogonal to dimensions X,Y. Input data 100, which is four-dimensional data having a time dimension T, is configured by arranging three-dimensional spatial data 101, 102, 103, 104, and 105 at a plurality of different time points along the time dimension T, for example. . Similarly, the output data 150 is also configured by arranging three-dimensional spatial data 101, 102, 103, 104, 105 at a plurality of different times along the time dimension T. FIG. The input data 100 and the output data 150, which are four-dimensional data, change over time as three-dimensional spatial data.

Four-dimensional data having a three-dimensional spatial dimension and a temporal dimension is, for example, a three-dimensional moving image. A 3D moving image is data in which a 3D image changes over time. A three-dimensional image is, for example, voxel data or polygon data. Voxel data is data representing a three-dimensional image with elements called voxels. Polygon data is data representing a three-dimensional image with elements called polygons.

A three-dimensional image is, for example, a medical image. A three-dimensional medical image is, for example, an image of the interior of a human or animal that is stereoscopically photographed. A medical image shows the structure or function of an internal organ or the like. The medical image is, for example, a Computed Tomography (CT) image or a Magnetic Resonance (Magnetic Resonance; MR) image.

A medical image as four-dimensional data is, for example, a combination of three-dimensional medical images taken at different times. A medical image as four-dimensional data has three spatial dimensions (X, Y, Z) and a one-dimensional temporal dimension T. A medical image as four-dimensional data visualizes movements inside the body in a three-dimensional space. A Magnetic Resonance Imaging (MRI) apparatus capable of outputting four-dimensional data is also called a four-dimensional MRI apparatus.

Returning to FIG. 2, the system 10 acquires the input data 100 (step S21). The system 10 acquires, as input data 100, a four-dimensional medical image output from a medical imaging apparatus such as a four-dimensional MRI apparatus, for example. A medical imaging device is connected to the system 10 via a network 15, for example. System 10 obtains input data 100 , for example, via network 15 . The system 10 stores the acquired input data 100 in the storage device 10B.

System 10 provides input data 100 to classification model 180 to obtain classification results (step S22). Classification model 180 is, for example, a trained model machine-learned to classify input data 100 . Model 180 may be referred to as a classifier. Machine learning is, for example, deep learning. Classification model 180 is, for example, a model having a neural network. The neural network is, for example, a convolutional neural network (CNN).

When the input data 100 are medical images, the classification model 180 classifies medical images taken inside a patient's body, for example, to diagnose or assist in diagnosis of the patient. That is, model 180 outputs the predicted classes. Classification of medical images can detect, for example, patient diseases.

The model 180 shown in FIG. 3 is, as an example, a four-dimensional Convolutional Neural Network (4D-CNN). A model 180 shown in FIG. 3 includes, as an example, a four-dimensional convolutional layer 181 and a fully connected layer 183 . The four-dimensional convolutional layer 181 receives four-dimensional input data 100 and outputs features 185 of the input data 100 . Feature 185 is, for example, a feature map. The feature 185 here is a 4-dimensional tensor, corresponding to the input data 100 being 4-dimensional data. Features 185 are subject to processing such as pooling as necessary.

Feature 185 is provided in all-bonded layer 183 . Fully connected layer 183 outputs a classification result 187 . Classification results 187 are, for example, class scores.

The system 10 includes, for example, a class activation map (CAM) generator 190 . The CAM indicates which regions in the four-dimensional input data 100 contributed to the classes predicted by the model 180 . Regions that contributed to the predicted class are called regions of interest. CAM provides a visual explanation of predictions made by neural networks. CAM is represented as a heat map, for example. In the CAM represented by a heat map, the degree of contribution to the prediction class is represented by shades of color. Corresponding to the input data 100 being four-dimensional data, the CAM here is four-dimensional data. As shown in FIG. 3, when the input data 100 is data that changes with time, attention area data 151, 152, 153, 154, and 155 such as CAM are also data that change with time. Attention area data such as CAM has the same number of dimensions as the input data.

The generator 190 generates a CAM by, for example, the Gradient-weighted Class Activation Mapping (Grad-CAM) method (step S23 in FIG. 2). The Grad-CAM method uses the generated feature map 185 to calculate how much influence each region of the input data 100 has on the predicted class. Influencing gradients and weighting factors are utilized in this calculation.

The CAM generation method may be a gradient method (steepest descent method), Guided Backpropagation, or Guided Grad-CAM.

Further, the generation unit 190 may generate a map (region-of-interest map) by another method for visualizing the region-of-interest of the input data 100 . A CAM is also called a saliency map. The generating unit 190 may generate the map using a technique known for generating saliency maps. Generating a saliency map is also called saliency detection.

As shown in FIG. 3, as an example, the system 10 superimposes the map (attention area data 151, 152, 153, 154, 155) generated by the generation unit 190 on the input data 100, thereby generating the output data 150. Generate (step S24 in FIG. 2). The system 10 executes processing for displaying the output data 150 and the like (step S25). When the attention area data 151, 152, 153, 154, 155 are represented as a heat map, the heat map can be superimposed on the input data 100 with a certain degree of transparency. The transparency of the heat map results in output data 150 that allows observation of both the input data 100 and the heat map.

The system 10 transmits data for displaying the output data 150 and the like on the display 31 of the terminal 30 to the terminal 30 .

Note that the system 10 does not have to have the function of generating the output data 150 from the input data 100 . The system 10 may acquire data corresponding to the output data 150 from the outside and perform processing for displaying the acquired data. Data corresponding to the output data 150 may be stored in the storage device 10B of the system 10 in advance. Also, the data displayed by the system 10 may be data corresponding to the input data 100 . Below, the output data 150 may be referred to as "four-dimensional data 150" or simply "data 150".

FIG. 4 shows a processing procedure for displaying data in step S25 of FIG. As an example, the system 10 generates the display data 300T-1 for displaying the output data 150 by the display method set in the initial settings. This display data 300T-1 is also called initial display data. The system 10 displays the initial display data 300T-1 on the display 31 (step S41). The initially set display method is, for example, to display the three-dimensional image at the start of the moving image data 150 on the display 31 as a planar image projected in a predetermined direction.

In step S42 of FIG. 4, an operation is performed to select a dimension indicated by the seek bar (seek bar dimension) among the four dimensions (X, Y, Z, T) of the four-dimensional data 150. FIG. The selection operation in step S42 is performed in area 210, which will be described later. By selecting the dimension, the display mode in the area 250 described later is switched. Here, when the seek bar dimension is determined by the operation, the remaining dimensions other than the seek bar dimension among the multi-dimensions (four dimensions) are determined as non-seek bar dimensions (display dimensions).

In step S43 of FIG. 4, display data representing the data 150 in non-selected three dimensions (non-seek bar dimensions; display dimensions) not selected in step S42 among the four dimensions (X, Y, Z, T) is generated and displayed. Display data is displayed in an area 250, which will be described later. In step S44 of FIG. 4, an operation on the seek bar or display data is accepted, and the display data displayed in the area 250 is changed according to the operation. By changing the display data, the data 150 can be observed from various viewpoints. Details of these steps S42 to S44 will be described later.

FIG. 5 shows an example of the area 200 where the display data 300T-1 and the like are displayed. The display content of the area 200 shown in FIG. 5 can correspond to the display content in step S41 of FIG. The area 200 is displayed on the display 31 of the terminal 30, for example. The system 10 transmits to the terminal 30, for example, the data required to display the area 200 on the display 31 in order to display the display data and the like. The area 200 is configured as, for example, a viewport of a WEB browser installed on the terminal 30 or a viewport of other applications installed on the terminal 30 . A WEB browser or other application installed on the terminal 30 displays data provided by the system 10 . The area 200 can be configured as a graphical user interface that receives user operations such as operations using the mouse pointer P or tap operations using the user's finger.

The area 200 can include an area 250 (display area 25) for displaying the display data 300T-1. The area 250 is, for example, a viewport in which display data, which is three-dimensional data (three-dimensional image), is drawn as two-dimensional data (planar image). A region 250 in FIG. 5 can display a three-dimensional still image at an arbitrary point in the four-dimensional data 150, which is a three-dimensional image moving image, as display data 300T-1. Also, the area 250 in FIG. 5 can display the 3D moving image, which is the 4D data 150, as the display data 300T-1. Area 250 can be configured as a graphical user interface that receives a user operation (operation of step S44) for changing the display method of display data 300T-1. This point will be described later.

Area 200 may include seek bar 230 . The seek bar 230 shown in FIG. 5 indicates the display time point td1 (playback time point td1) of the image displayed in the area 250. FIG. A display time point td1 in FIG. 5 is a display position of the time dimension T in the four-dimensional data 150 . The display time td1 can be indicated by a slider 231 movably arranged along the seek bar 230, for example. Seek bar 230 is configured as a graphical user interface that allows user operation (operation in step S44) to specify display time point td1. Although one seek bar 230 is shown in FIG. 5, there may be two or more. By increasing the number of seek bars 230, it is possible to suppress an increase in the number of dimensions of the display data even if the number of dimensions of the multidimensional data 150 increases.

The seek bar 230 shown in FIG. 5 can be set to an arbitrary position between the start time Tmin of the moving image displayed in the area 250 and the end time Tmax of the moving image displayed in the area 250 by a slider 231 indicating the longitudinal position (horizontal direction position in FIG. 5) of the seek bar 230 . can indicate the location of That is, the seek bar 230 shown in FIG. 5 indicates where in the time dimension T of the four-dimensional data 150 the display data 300T-1 displayed in the area 250 is an image. Note that the display indicating the display time td1 in the seek bar 230 is not limited to the slider 231 .

In FIG. 5, a planar image obtained by projecting a three-dimensional image at time td1 in a predetermined line-of-sight direction is displayed in area 250 as display data 300T-1. The display data 300T-1 in FIG. 5 also has a heat map 350T-1 included in the three-dimensional image at time td1. Therefore, a user observing the area 250 can observe both the display data 300T-1 and the heat map 350T-1.

The slider 231 can be moved in the longitudinal direction of the seek bar 230 by user operation using the mouse pointer P or the like. As shown in FIG. 5, when the display time td1 is specified in the seek bar 230, in the area 250, the three-dimensional data of the display point td1 in the time dimension T in the four-dimensional data 150 is displayed as display data 300T-1. Is displayed. That is, when the four-dimensional data 150 is expressed as (X, Y, Z, T) using the dimensions X, Y, Z, T, the display location td1 (dimension T = td1) is specified in the seek bar 230 , the display data 300T-1 of (X, Y, Z, td1) is displayed in the area 250. FIG.

When the user performs a user operation to move the slider 231 of the seek bar 230 from the position td1 shown in FIG. 5 to the position td2 shown in FIG. Display data 300T-2 is displayed. FIG. 6 shows display data 300T-2 at display time td2. (X, Y, Z, td2) is a three-dimensional image at time td2 in (X, Y, Z, T), which is the four-dimensional data 150 . The display data 300T-2 also has a heat map 350T-2 at display time td2.

In this way, the four-dimensional data 150 here is three-dimensional data that changes over time. Therefore, when the display position in the time dimension T is changed, the display data displayed in the area 250 also changes. can. Also, when the display location in the time dimension T is changed, the heat map 350T-1 displayed in the area 250 can also be changed.

The seek bar 230 shown in FIG. 5 is configured to indicate the strength of the heat map at each point in time (at each point) in the time dimension T, which is the seek bar dimension, as non-seek bar dimension data correspondence information. The non-seek bar dimension data correspondence information is information for visualizing the non-seek bar dimension data that is not displayed as display data and facilitating designation of the display location on the seek bar 230 . Four-dimensional data 150, which is multidimensional data, includes a heat map as described above. The heat map strength (non-seek bar dimension data correspondence information) in the seek bar dimension (time dimension) is obtained from each non-seek bar dimension (spatial dimension) heat map at each point (each point in time) in the seek bar dimension (time dimension). Generated by

The seek bar 230 shown in FIG. 5 indicates the strength of the heat map by gradation in the seek bar 230. FIG. That is, this gradation indicates the non-seek bar dimension data correspondence information at each location in the time dimension T, which is the seek bar dimension. This gradation has shading corresponding to the intensity of the heat map at each point in the seek bar dimension (time dimension T). For example, a dark portion (black portion) in the seek bar 230 indicates that the value indicated by the heat map is strong, and a light portion (white portion) indicates that the value indicated by the heat map is weak. The strength of the heat map here is, for example, the sum, average value, median value, or maximum value. The strength of the heat map indicated by the seek bar 230 may be other values obtained from the values indicated by the heat map included in the three-dimensional data (X, Y, Z). The intensity of the heat map represented by the seek bar 230 does not need to be obtained from the entire area of the three-dimensional data (X, Y, Z), but any part of the three-dimensional data (X, Y, Z). It may be the sum total, average value, median value, maximum value, or the like of values indicated by a heat map in a limited area such as a dimensional area.

Since the seek bar 230 indicates the temporal change of the heat map, the seek bar 230 can visually provide the user with the temporal change of the heat map. Therefore, with reference to the gradation displayed on the seek bar 230, the user operates the seek bar 230 to display the three-dimensional image (X, Y, Z) at the time when the heat map is strong or weak in the area 250 for observation. can do.

Here, the seek bar 230 will be explained from a different point of view. The seek bar 230 shown in FIG. I can say. In FIG. 5, the values for region 350T-1 that contribute to the predicted class are indicated by gradations within seek bar 230. FIG. The value for region 350T-1 that contributes to the predicted class is the sum, average, median, maximum, or other value of the degree of contribution to the predicted class at each point in time.

The change over time of the heat map or the change over time of the values relating to the region 350T-1 contributing to the predicted class may be displayed outside the seek bar 230. FIG. That is, the change over time of the heat map or the change over time of the values for the region 350T-1 contributing to the predicted class may be displayed separately from the seek bar 230. FIG.

As shown in FIG. 5, the area 200 includes a video playback button 235 . The play button 235 is a graphical user interface that can be selected by user operation. When the play button 235 is selected, the moving image is reproduced from the start time Tmin of the moving image or the position indicated by the slider 231 . The reproduction here is to change the display data 300T-1 displayed in the area 250 with time. That is, the reproduction here is to change the three-dimensional space data 300T-1 forming the four-dimensional data 150 along the time dimension T. FIG. When the four-dimensional data 150 is reproduced as a moving image, in the area 250 the three-dimensional image 300T-1 changes with time. In this case, the slider 231 automatically moves toward the end point (to the right in FIG. 5) as time progresses during playback.

During playback, display of the playback button 235 switches to display of a button for performing a stop operation. A button for the stop operation is a stop button or a pause button. When the button for the stop operation is operated, the movement of the slider 231 stops, and a still image at the point of stop is displayed in the area 250 .

As described above, the area 250 can be configured as a graphical user interface that receives a user operation (operation of step S44) for changing the display of the display data 300T-1. The change in display in the area 250 is, for example, at least one of a change in the visual field direction of the display data 300T-1 and a change in the display surface of the display data 300T-1. Changing the display surface of the three-dimensional data may include changing the display surface showing the inside of the three-dimensional data.

FIG. 7 shows an example of how the viewing direction of the display data 300T-1 displayed in the area 250 is changed. Changing the viewing direction is to change the viewing direction of the object displayed in the area 250 (the object is, for example, a cube appearing as the display data 300T-1 in FIGS. 5 and 7). The viewing direction is changed, for example, by moving the mouse pointer P in the left, right, up, or down direction on the display data 300T-1 or in the vicinity of the display data 300T-1. FIG. 7 shows that the cube appearing as the display data 300T-1 is displayed tilted compared to FIG. Thus, in FIG. 7, the viewing direction is changed compared to FIG.

FIG. 8 shows an example of how to change the display surface showing the inside of the display data 300T-1 displayed in the area 250. As shown in FIG. Area 250 may include cross-sectional indications CS-X, CS-Y, CS-Z as a graphical user interface for changing the display surface showing the interior. The cross-section indication displays CS-X, CS-Y, and CS-Z are movable by user's operation. The display data 300T-1 is configured to display the cross section designated by the cross section designation displays CS-X, CS-Y, and CS-Z in the three-dimensional image (X, Y, Z).

The cross-section indication display CS-X is used to specify the position in the X-dimensional direction of the cross-section parallel to the YZ plane of the three-dimensional data (X, Y, Z) by user operation. For example, the cross-section designation display CS-X is selected by the mouse pointer P1 and moved in the horizontal direction (X-dimensional direction) in FIG. By changing the position of the cross-section indication display CS-X, the YZ cross-section displayed as the display data 300T-1 in the three-dimensional image changes.

The cross-section indication display CS-Y is used to specify the position in the Y-dimensional direction of the cross-section parallel to the ZX plane of the three-dimensional data (X, Y, Z) by user operation. For example, the cross section indication display CS-Y is selected by the mouse pointer P2 and moved in the diagonal direction (Y dimension direction) in FIG. By changing the position of the cross-section indication display CS-Y, the ZX cross-section displayed as the display data 300T-1 in the three-dimensional image changes.

The cross-section indication display CS-Z is used to specify the position in the Z-dimensional direction of the cross-section parallel to the XY plane of the three-dimensional data (X, Y, Z) by user operation. For example, the section indication display CS-Y is selected by the mouse pointer P2 and moved in the vertical direction (Z-dimensional direction) in FIG. By changing the position of the cross-section indication display CS-Z, the XY cross-section displayed as the display data 300T-1 in the three-dimensional image changes.

Being able to change the displayed cross section facilitates observation of the interior of the 3D image and the 3D heat map of the interior of the 3D image.

Note that changing the display of the display data 300T-1 in the area 250 may be performed when the area 250 displays a still image, or may be performed when the area 250 displays a moving image. good too. Further, the technique for changing the display of the display data 300T-1 in the area 250 is not limited to the technique shown in FIGS. can be adopted.

Returning to FIG. 5, the area 200 includes a display 220 showing the classification result of the input data 100 (see step S22 in FIG. 2). The display 220 indicating the classification result indicates, for example, the patient's disease name corresponding to the medical image that is the input data 100 . The disease name in this case is the disease name inferred by the system 10 .

For example, a doctor who is a user of system 10 can refer to the disease name shown in display 220 showing classification results. The doctor diagnoses the patient by referring to the disease name guessed by the system 10 . The doctor, for example, refers to the region 250, examines the validity of the prediction by the system 10 from the heat map 350T-1 indicating the region (region of interest) that contributed to the prediction of the disease name by the system 10, or examines the validity of the heat map 350T. A more detailed diagnosis can be made by carefully observing the region of interest indicated by -1.

In the case of a disease for which the way the body changes over time rather than the static state of the body is important for diagnosis, displaying the four-dimensional data 150 showing the change over time in the body assists the doctor's diagnosis. It is advantageous over

However, the displays shown in FIGS. 5 to 7 have limitations and are not necessarily sufficient for observation or analysis of data by a user such as a doctor. One of the limitations of the display modes shown in FIGS. 5 to 7 is that the dimension indicated by the seek bar 230 is limited to the time dimension T. FIG. Due to this constraint, in the display mode of the area 250 shown in FIGS. 5 to 7, for example, data at different locations in the time dimension T cannot be displayed at the same time.

To relax this constraint, the area 200 shown in FIG. 5 can include an area 210 for selecting the dimension indicated by the seek bar 230 (seek bar dimension). A region 210 shown in FIG. 5 is a graphical user interface for selecting one dimension indicated by the seek bar 230 among the four dimensions X, Y, Z, and T of the four-dimensional data 150 (operation of step S42). is configured as Hereinafter, among the four dimensions X, Y, Z, T, the dimension (seek bar dimension) selected by the area 210 will be referred to as the "selected dimension". By determining the selected dimension, the remaining dimensions other than the selected dimension among the four-dimensional data X, Y, Z, and T are naturally determined. Among the four-dimensional data X, Y, Z, and T, the remaining dimensions other than the selected dimension are called "non-selected dimensions". Non-selected dimensions may be referred to as "non-seek bar dimensions" or "display dimensions." Display data representing the four-dimensional data 150 in non-selected dimensions can be displayed in the area 250 . Note that the operation of step S42 may be configured as an operation of selecting a non-seek bar dimension (display dimension), and the seek bar dimension may be automatically determined by determining the non-seek bar dimension based on the operation.

The number of selected dimensions may be two or more as long as it is less than the number of dimensions of the data 150 . For example, two dimensions, dimension T and dimension Z, may be selected dimensions. In this case, the area 200 may comprise a seek bar indicating dimension T and a seek bar indicating dimension Z. FIG. Also, in this case, in the area 250, display data representing the four-dimensional data in two dimensions, that is, the dimension X and the dimension Y, which are non-selected dimensions, can be displayed.

The selection dimension (seek bar dimension) may initially be the time dimension T, as an example. That is, when the area 200 is displayed in the initial setting (step S41 in FIG. 4), as shown in FIG. Three-dimensional data of certain non-selected dimensions (X, Y, Z) can be displayed as display data 300T-1.

The user of the system 10 can switch the dimension indicated by the seek bar 230 and the dimension of the display data in the area 250 by operating the area 210 for selecting the dimension (switching in step S43). Further, by operating the area 210, the gradation displayed on the seek bar 230 (information corresponding to non-seek bar dimension data) is also switched. For these switching, as shown in FIG. 9, the area 210 includes dimension switching buttons 210X, 210Y, 210Z, and 210T. Button 210X is a button for selecting dimension X as a selected dimension. Button 210Y is a button for selecting dimension Y as a selected dimension. Button 210Z is a button for selecting dimension Z as a selected dimension. Button 210T is a button for selecting dimension T as a selected dimension. Among the dimensions X, Y, Z, and T of the data 150, the dimensions not selected as the selected dimension (seek bar dimension) become non-selected dimensions (non-seek bar dimension).

For example, as shown in FIG. 9, the user can change the dimension indicated by the seek bar 230 from the dimension T shown in FIG. 5 to the dimension Z by operating the mouse pointer P and selecting the button 210Z. can.

As shown in FIG. 9, the area 200 has a display 233 that indicates the type of dimension indicated by the seek bar 230 . That is, display 233 shows the selected dimension. The display 233 of FIG. 9 includes the letter "Z" indicating that the dimension indicated by the seek bar 230 is the Z dimension. On the other hand, the display 233 of FIG. 5 includes the letter "T" indicating that the dimension indicated by the seek bar 230 is the time dimension.

When the selected dimension (seek bar dimension) is the time dimension T, the four-dimensional data 150 is configured as a moving image in which the three-dimensional spatial data (X, Y, Z) changes over time. Similarly, if one of the dimensions X, Y, and Z other than the time dimension T is the selected dimension (seek bar dimension), the four-dimensional data 150 is the remaining three-dimensional spatial data that is the non-selected dimension (non-seek bar dimension). can be configured as a moving image that changes over time. That is, when the selected dimension is the time dimension T, the four-dimensional data 150 is grasped as data in which the three-dimensional data (X, Y, Z) changes along the dimension T, like the data 150T in FIG. be. Data 150T can be displayed as a moving image in which three-dimensional data (X, Y, Z) changes over time.

Also, when the selected dimension is the dimension Z, the four-dimensional data 150 can be grasped as data in which the three-dimensional data (X, Y, T) changes along the dimension Z, like the data 150Z in FIG. . The data 150Z can be displayed as a moving image in which the three-dimensional data (X, Y, T) changes over time, with the dimension Z being regarded as the time dimension.

Also, when the selected dimension is the dimension Y, the four-dimensional data 150 can be grasped as data in which the three-dimensional data (X, Z, T) changes along the dimension Y, like the data 150Y in FIG. . The data 150Y can be displayed as a moving image in which the three-dimensional data (X, Z, T) changes over time, with the dimension Y as the time dimension.

Also, when the selected dimension is dimension X, the four-dimensional data 150 can be grasped as data in which three-dimensional data (Y, Z, T) changes along dimension X, like data 150X in FIG. . The data 150X can be displayed as a moving image in which the three-dimensional data (Y, Z, T) changes over time, with the dimension X as the time dimension.

Returning to FIG. 9, when the dimension Z is selected as the selected dimension (seek bar dimension) and the selected dimension is changed, the display data displayed in the area 250 is the non-seek bar dimension display data before the change (for example, (X, Y, Z) three-dimensional data) is switched to three-dimensional data (X, Y, T), which is non-seek bar dimension display data after change. Also, the seek bar 230 is switched to indicate where the image shown in the area 250 is the three-dimensional image in the dimension Z (seek bar dimension) of the four-dimensional data. Furthermore, when the non-seek bar dimension is changed with the change of the selection dimension (seek bar dimension), the gradation of the seek bar 230 (non-seek bar dimension data correspondence information) is changed to the non-seek bar dimension data correspondence information of the non-seek bar dimension before the change. , to the non-seek bar dimension data correspondence information of the non-seek bar dimension after change.

A seek bar 230 shown in FIG. 9 indicates an arbitrary point between the minimum value Zmin and the maximum value Zmax of the dimension Z in the four-dimensional data 150 . The minimum value Zmin of the dimension Z is the start point of the animation 150Z displayed in the area 250, and the maximum value Zmax is the end point of the animation displayed in the area 250. FIG.

A seek bar 230 shown in FIG. 9 indicates a display portion za1 (playback portion ta1) of the image displayed in the area 250. FIG. The system 10 generates the three-dimensional data (X, Y, za1, T) of the display location za1 in the dimension Z from the four-dimensional data 150 as the display data 300Z-1, and generates the generated display data 300Z- 1 is displayed in area 250 .

FIG. 11 shows a conceptual diagram of a method of generating display data 300Z-1, which is three-dimensional data (X, Y, za1, T) of the display location za1 in the dimension Z from the four-dimensional data 150. FIG.

As shown in FIG. 11, the four-dimensional data 150 is three-dimensional data (X, Y, X, Tmin), (X, Y, Z, t1), . . . , (X, Y, Z, Tmax). In this case, the system 10 generates three-dimensional data (X, Y, Z, Tmin), (X, Y, Z, t1), . From (X, Y, Z, Tmax), X-dimensional and Y-dimensional data (X, Y, za1, Tmin), (X, Y, za1, t1), ..., ( X, Y, za1, Tmax). Each extracted data may include a region represented by a heat map.

The system 10 processes the extracted data (X, Y, za1, Tmin), (X, Y, za1, t1), . By arranging them, display data 300Z-1, which is three-dimensional data (X, Y, za1, T) of the display location za1, is generated. The generated display data 300Z-1 is displayed in the area 250 as shown in FIG.

In FIG. 9, a plane image obtained by projecting a three-dimensional image at the display point za1 of the dimension Z in a predetermined line-of-sight direction is displayed in the area 250 as the display data 300Z-1. The display data 300T-1 in FIG. 9 also has a heat map 350T-1 included in the three-dimensional image at the point za1. Therefore, a user observing the area 250 can observe both the display data 300T-1 and the heat map 350T-1. By observing the area 250 in FIG. 9, the user can grasp the change in the time dimension T of the XY plane at the point za1 of the four-dimensional data 150 from one three-dimensional image 300Z-1.

When the user performs a user operation to move the slider 231 of the seek bar 230 from the location za1 shown in FIG. 9 to the location za2 shown in FIG. Data 300T-2 is displayed. (X, Y, za2, T) is a three-dimensional image at location za2 in dimension Z in (X, Y, Z, T), which is four-dimensional data 150 . The display data 300Z-2 also has a heat map 350Z-2 at the display location za2.

FIG. 13 shows a conceptual diagram of a method of generating display data 300Z-2, which is three-dimensional data (X, Y, za2, T) of the display location za2 in the dimension Z from the four-dimensional data 150. FIG. The method shown in FIG. 13 is the same as the method shown in FIG. 11 except that the display locations are different.

That is, as shown in FIG. 13, the system 10 provides three-dimensional data (X, Y, Z, Tmin), (X, Y, Z, t1) at points Tmin, t1, . ), . , . . . , (X, Y, za2, Tmax) are extracted. Each extracted data may include a region represented by a heat map.

The system 10 processes the extracted data (X, Y, za2, Tmin), (X, Y, za2, t1), . By arranging them, display data 300Z-2, which is three-dimensional data (X, Y, za2, T) of the display location za2, is generated. This display data 300Z-2 is displayed in the area 250 as shown in FIG.

Thus, the four-dimensional data 150 is represented as display data according to the position in the Z dimension of the three-dimensional data (X, Y, T). Therefore, when the display position in the dimension Z is changed by the seek bar 230, the display data displayed in the area 250 can also be changed. Also, changing the display location in dimension Z may change the heat map displayed in area 250 .

The seek bar 230 shown in FIGS. 9 and 12 is configured to indicate, as non-seek bar dimension data correspondence information, the strength of the heat map at each point in the dimension Z, which is the seek bar dimension. The seek bar 230 shown in FIG. 5 indicates the strength of the heat map by gradation in the seek bar 230. FIG. This gradation indicates the non-seek bar dimension correspondence information for each location in the dimension Z, which is the non-seek bar dimension. This gradation has a shade corresponding to the intensity of the heatmap at each point in dimension Z, which is the seek bar dimension. For example, a dark portion (black portion) in the seek bar 230 indicates that the value indicated by the heat map is strong, and a light portion (white portion) indicates that the value indicated by the heat map is weak. The strength of the heat map here is, for example, the sum, average, median, or maximum value of the values indicated by the heat map included in the three-dimensional data (X, Y, T) at each location in the dimension Z. or The strength of the heat map indicated by the seek bar 230 may be other values obtained from the values indicated by the heat map included in the three-dimensional data (X, Y, T). The intensity of the heat map represented by the seek bar 230 does not need to be obtained from the entire area of the three-dimensional data (X, Y, T), but any part of the three-dimensional data (X, Y, T). It may be the sum total, average value, median value, maximum value, or the like of values indicated by a heat map in a limited area such as a dimensional area.

Since the seek bar 230 shows the change in the Z dimension of the heatmap, the seek bar 230 can visually provide the user with the change in the Z dimension of the heatmap. Therefore, the user operates the seek bar 230 with reference to the gradation displayed on the seek bar 230 to display the three-dimensional image (X, Y, T) at the time when the heat map is strong or weak in the area 250 for observation. can do.

Here, the seek bar 230 will be explained from a different point of view. The seek bar 230 shown in FIGS. It can be said that there is 9 and 12, the values for region 350Z-1 that contribute to the predicted class are indicated by gradation within seek bar 230. FIG. The value for the predicted class contribution region 350Z-1 is the sum, average, median, maximum, or other value of the degree of contribution to the predicted class at each location in the Z dimension.

The change over time of the heat map or the change over time of the values relating to the region 350Z-1 contributing to the predicted class may be displayed outside the seek bar 230. FIG. That is, the change over time of the heat map or the change over time of the values for the region 350Z-1 contributing to the predicted class may be displayed separately from the seek bar 230. FIG.

In FIG. 9 or 12 , when the play button 235 is selected and operated, the moving image is played in the area 250 from the position indicated by the minimum Z-dimensional value Zmin or the slider 231 . The animation here shows the change in the dimension Z of the three-dimensional data (X, Y, T) as a change over time.

A region 250 shown in FIG. 9 can also receive a user operation for changing the display method of the display data 300Z-1 in the same manner as described with reference to FIGS.

For example, as shown in FIG. 14, it is possible to change the viewing direction of the display data 300Z-1. The viewing direction is changed, for example, by moving the mouse pointer P in the left, right, up, or down direction on the display data 300Z-1 or in the vicinity of the display data 300Z-1. FIG. 14 shows that the cube appearing as the display data 300Z-1 is tilted compared to FIG. Thus, in FIG. 14, the viewing direction is changed compared to FIG.

Further, as shown in FIG. 15, the display surface showing the inside of the display data 300Z-1 displayed in the area 250 can be changed. Area 250 may include cross-sectional indications CS-X, CS-Y, CS-T as a graphical user interface for changing the display surface showing the interior. The cross-section indication displays CS-X, CS-Y, and CS-T are movable by user's operation. The display data 300T-1 is configured to display the cross section designated by the cross section designation displays CS-X, CS-Y, CS-T in the three-dimensional image (X, Y, T).

Thus, when dimension Z is selected as the selected dimension, the seek bar dimension indicated by seek bar 230 is dimension Z, and the three-dimensional data for the non-seek bar dimensions displayed in area 250 is (X, Y, T). Other than that, the display is the same as when the time dimension T is the selected dimension, and the same operations are possible.

FIG. 16 shows the case where dimension Y is selected as the selection dimension (seek bar dimension). In this case as well, the dimension indicated by the seek bar 230 is the dimension Y, and the dimension of the three-dimensional data displayed in the area 250 (non-seek bar dimensions) is (X, Z, T). The same display as in the case of (seek bar dimension) is made, and the same operation is possible. By observing the area 250 in FIG. 16, the user can grasp the change in the time dimension T of the XZ plane at the location yb of the four-dimensional data 150 from one three-dimensional image 300Y (display data 300Y).

Also, the seek bar 230 in FIG. 16 shows changes in the Y dimension of the heat map. That is, the seek bar 230 of FIG. 16 can visually provide the user with changes in the Y dimension of the heat map by means of the gradation, which is the non-seek bar dimension data correspondence information. Therefore, the user operates the seek bar 230 with reference to the gradation displayed on the seek bar 230 to display the three-dimensional image (X, Z, T) at the time when the heat map is strong or weak in the area 250 for observation. can do.

FIG. 17 shows the case where dimension X is selected as the selected dimension (seek bar dimension). Again, the dimension indicated by seek bar 230 is dimension X, and the time dimension T is the selected dimension, except that the three-dimensional data displayed in area 250 has dimensions (non-seek bar dimensions) of (Y, Z, T). The same display as in the case of , and the same operation is possible. By observing the area 250 in FIG. 17, the user can grasp the change in the time dimension T of the YZ plane at the point xc of the four-dimensional data 150 from one three-dimensional image 300X (display data 300X).

Also, the seek bar 230 in FIG. 17 shows changes in the X dimension of the heatmap. That is, the seek bar 230 of FIG. 17 can visually provide the user with changes in the X dimension of the heat map by means of the gradation, which is the non-seek bar dimension data correspondence information. Therefore, with reference to the gradation displayed on the seek bar 230, the user operates the seek bar 230 to display the three-dimensional image (Y, Z, T) at the time when the heat map is strong or weak in the area 250 for observation. can do.

As described above, according to this embodiment, the display mode of the data 150 in the area 250 can be switched by switching the selected dimension. Therefore, the user can observe the data 150 from various viewpoints. When the data 150 is a medical image, the present system 10, which facilitates observation from various viewpoints, is useful for assisting a doctor's diagnosis.

FIG. 18 shows seek bars 230A and 230B according to modifications of the seek bar 230. FIG. For reference, FIG. 18 also shows an outline of the seek bar 230 shown in FIG. The seek bar 23 shown in FIG. 5 is configured to simultaneously display the non-seek bar dimension data correspondence information at each location of the seek bar dimension. That is, since the gradation, which is the non-seek bar dimension data correspondence information, is displayed entirely within the seek bar 230, the user can simultaneously visually recognize the non-seek bar dimension data correspondence information for each of a plurality of locations in the seek bar dimension. can. On the other hand, the seek bar 230A shown in FIG. 18 has a bar 237 on which a slider 296 is movably provided. Seek bar 230A does not show heatmap variation in the selected dimension like seekbar 230 shown in FIG. 5, but instead slider 238 shows the intensity of the heatmap where slider 238 resides. there is For example, as shown in FIG. 18, when the selected dimension is the time dimension T and the slider 238 is positioned at time t3 by the mouse pointer P, the color within the slider 238 in the seek bar 230A at time t3 is The strength of the heat map of three-dimensional data (X, Y, Z) is shown. Also, when the slider 238A is placed at the position of time t2 by the mouse pointer PA, the color inside the slider 238A indicates the strength of the heat map of the three-dimensional data (X, Y, Z) at time t2. , mouse pointer PB to position slider 239 at time t1, the color within slider 238B indicates the intensity of the heat map of the three-dimensional data (X, Y, Z) at time t1. In this way, the non-seek bar dimensional data correspondence information (heat map strength) must be displayed on the seek bar 230A at locations specified by the user, and not hidden at unspecified locations. good too.

Also, the display 239 indicating the strength of the heat map does not need to be indicated by a slider, and may be indicated by a display box 239 separate from the slider 231 like the seek bar 230B shown in FIG. For example, as with the seek bar 230B shown in FIG. The strength of the heatmap is indicated by display box 239 . When the mouse pointer PA is placed at the position of time t2 on the seek bar 230B, the strength of the heat map of the three-dimensional data (X, Y, Z) at time t2 corresponding to the mouse pointer PA is displayed. Indicated by box 239A. When the mouse pointer PB is placed at the position of the time t1 on the seek bar 230B, the strength of the heat map of the three-dimensional data (X, Y, Z) at the time t1 corresponding to the mouse pointer PB is displayed. Indicated by box 239B.

In this way, the seek bars need not be configured to indicate the values (for example, the intensity of the heat map) of the data 150 at each location in the selected dimension as shown in FIG. It may be configured to show the values (eg, the intensity of the heatmap) for the data 150 at the locations specified above. In this way, the non-seek bar dimension data correspondence information (heat map strength) only needs to be displayed corresponding to each location of the seek bar dimension in the seek bar, and does not need to be displayed on the seek bar. Outside the seek bar, It may be displayed in association with each location of the seek bar.

The present invention is not limited to the above embodiments, and various modifications are possible. For example, the data displayed by the system 10 is not limited to 4-dimensional spatio-temporal data having 3 spatial dimensions and temporal dimensions, but is 3-dimensional spatio-temporal data having 2 spatial dimensions and temporal dimensions. good too. In this case, one selected dimension of the 3D spatiotemporal data can be indicated by seek bar 230 and the 3D spatiotemporal data can be displayed in area 250 in the remaining two dimensions.

Also, in the above description, the selected dimension is selected from among all the dimensions (X, Y, Z, T) of the multidimensional data 150. It doesn't have to be selectable. That is, the selected dimension may be selected from a plurality of dimensions (for example, dimension Z and dimension T) among all dimensions of the multidimensional data 150 .

10: Computer system 10A: Processor 10B: Storage device 10C: Computer program 15: Network 30: Terminal 31: Display 100: Input data 101: Three-dimensional space data 102: Three-dimensional space data 103: Three-dimensional space data 104: Three dimensions Spatial data 105: three-dimensional spatial data 150: output data 150T: data 150X: data 150Y: data 150Z: data 151: attention area data 152: attention area data 153: attention area data 154: attention area data 155: attention area data 180 : Classification model (learned model)
181 : Four-dimensional convolution layer 183 : Fully connected layer 185 : Feature map 187 : Classification result 190 : Generation unit 200 : Region 210 : Region 210T : Dimension switching button 210X : Dimension switching button 210Y : Dimension switching button 210Z : Dimension switching button 220 : Display 230 : Seek bar 231 : Slider 233 : Display 235 : Playback button 250 : Area (display area)
300T-1: display data 300T-2: display data 300X: display data 300Y: display data 300Z-1: display data 300Z-2: display data 350T-1: heat map 350T-2: heat map 350Z-1 : Heat map 350Z-2 : Heat map CS-X : Cross-section indication display CS-Y : Cross-section indication display CS-Z : Cross-section indication display CS-T : Cross-section indication display P : Mouse pointer

Claims (7)

A computer system configured to execute display processing for displaying multidimensional data with seek bar dimensions whose dimensions are indicated by the seek bar and non-seek bar dimensions which are dimensions displayed by display data,
The display process includes
Receiving a user's operation to determine a dimension to be the seek bar dimension and a dimension to be the non-seek bar dimension, which is a dimension other than the seek bar dimension, among the multi-dimensions in the multi-dimensional data;
displaying the seek bar for indicating a display location to be displayed as the display data in the seek bar dimension determined based on the operation;
generating and displaying the display data based on the operation;
generating and displaying non-seek bar dimension data correspondence information based on the operation;
including
Generating the display data includes the non-seek bar dimension data determined based on the operation among the multidimensional data, and displaying the data at the display location indicated by the seek bar. including generating as data for
The non-seek bar dimension data correspondence information is information generated from each of the non-seek bar dimension data at each location of the seek bar dimension determined based on the operation among the multidimensional data, and the A computer system that is displayed corresponding to each point in the seekbar dimension. 2. The computer system according to claim 1, wherein the seek bar is configured to simultaneously display the non-seek bar dimension data correspondence information at each location of the seek bar dimension. 2. The computer system according to claim 1, wherein the non-seek bar dimension data correspondence information is displayed on the seek bar in areas specified by the user and not displayed in areas not specified. The multi-dimensional data includes a multi-dimensional heat map, and the non-seek bar dimension data correspondence information is generated from the heat map of the non-seek bar dimension at each location of the seek bar dimension determined based on the operation. Item 1. The computer system according to item 1. the display data is displayed in a display area for displaying the display data;
When the non-seek bar dimension is changed based on the operation, the display data displayed in the display area is switched from the non-seek bar dimension display data before the change to the non-seek bar dimension display data after the change. 2. The computer system of claim 1, wherein: When the non-seek bar dimension is changed based on the operation, in the seek bar, the non-seek bar dimension data correspondence information displayed corresponding to each location of the seek bar dimension is the non-seek bar dimension of the non-seek bar dimension before the change. 2. The computer system according to claim 1, wherein the data correspondence information is switched to non-seek bar dimension data correspondence information of non-seek bar dimension after change. A computer-implemented method, implemented by a computer, for displaying multidimensional data with seekbar dimensions, the dimensions of which are indicated by the seekbar, and non-seekbar dimensions, the dimensions of which are displayed by the data for display, comprising:
Receiving a user's operation to determine a dimension to be the seek bar dimension and a dimension to be the non-seek bar dimension, which is a dimension other than the seek bar dimension, among the multi-dimensions in the multi-dimensional data;
displaying the seek bar for indicating a display location to be displayed as the display data in the seek bar dimension determined based on the operation;
generating and displaying the display data based on the operation;
generating and displaying non-seek bar dimension data correspondence information based on the operation;
including
Generating the display data includes the non-seek bar dimension data determined based on the operation among the multidimensional data, and displaying the data at the display location indicated by the seek bar. including generating as data for
The non-seek bar dimension data correspondence information is information generated from each of the non-seek bar dimension data at each location of the seek bar dimension determined based on the operation, and corresponds to each location of the seek bar dimension in the seek bar. Computer-implemented method displayed as

Images