Creating an ASM Disk on Exadata FlashDisk

Recently, we noticed performance drops at certain times on an Exadata server operating as a data warehouse (DWH). Upon analyzing AWR reports, the following wait event stood out:

direct path read/write temp waits

We knew this event was caused by temporary (temp) data being written to or read from disk during query execution. Although we had applied various SQL optimizations, they did not significantly reduce the wait times.

Root Cause: Heavy TEMP Tablespace Usage

Large SQL queries in data warehouses—especially those involving operations like hash joins, sorts, and group by—require extensive use of TEMP space. In our case, the TEMP tablespace was located on high-latency disk groups by default, which increased IO wait times.

Solution: Move TEMP Tablespace to FlashDisk on Exadata

To directly improve performance, we decided to move the TEMP tablespace to the FlashDisk layer on Exadata. This layer offers much higher IO throughput compared to traditional disks, making it particularly advantageous for handling temporary data operations.

P.S. Since we will drop and recreate the existing flash disks during this operation, there may be minor negative impacts during the process and until the flash cache warms up again. Therefore, this work should be performed during a planned and quiet maintenance window.

dcli –g cell_group –l root cellcli -e "alter flashcache all flush"
dcli -g cell_group -l root cellcli -e "LIST CELLDISK ATTRIBUTES name, flushstatus, flusherror" | grep FD
dcli -g cell_group -l root cellcli -e drop flashcache
dcli -g cell_group -l root cellcli -e create flashcache all size=20.2875976562500T;
dcli -g cell_group -l root cellcli -e CREATE GRIDDISK ALL FLASHDISK PREFIX='FLASHTMP';

sqlplus / as sysasm

alter system set asm_diskstring='o/*/DATA_*','o/*/RECO_*','o/*/FLASHTMP*';

CREATE diskgroup TEMPDG normal redundancy disk 'o/*/FLASHTMP*' attribute 'compatible.rdbms'='19.0.0.0.0', 'compatible.asm'='19.0.0.0.0', 'cell.smart_scan_capable'='TRUE', 'au_size'='4M';

sqlplus & as sysdba

CREATE TEMPORARY TABLESPACE TEMP_FLASH TEMPFILE '+ TEMPDG' SIZE 32G EXTENT MANAGEMENT LOCAL UNIFORM SIZE 1M;

ALTER DATABASE DEFAULT TEMPORARY TABLESPACE TEMP_FLASH;

DROP TABLESPACE TEMP INCLUDING CONTENTS AND DATAFILES;