Exadata Flash Cache WriteBack
Posted by Vishal Gupta on Oct 2, 2012
With the announcement Exadata X3, Oracle has introduced a new feature called “FlashCache Writeback” to allow writes to cell Flash Cache (aka Exadata Smart FlashCache) in WriteBack mode. Earlier with WriteThrough mode, writes were not written to FlashCache, instead they were written directly to cell disks. Exadata software used to decide whether to cache these writes back into FlashCache or not. In WriteBack mode, writes are written to cell FlashCache and acknowledgement is given back to calling process as soon as data is written to flashcache. Exadata Server software de-stages the dirty writes in flashcache to spinning disks in the background. Writes to flashcache are persistent, so there will be no data loss in case of cell node failures.
Advantages of FlashCache WriteBack
- It can significantly improve database write speeds, as write to PCI FlashCards is faster than spinning disks by a order of magnitude.
- It will improve the speed of database checkpoints.
- Control files writes will be automatically done to flashcache, which can improve performance of logswitches, checkpoints.
- It can significantly improve direct path writes i.e. when sort/merge/hash join operation spill over from PGA memory to temporary tablespaces, they can be written to FlashCache instead of being written to slower spinning disks. This will speed large parallel query operations.
- Now there is no need to create separate ASM DiskGroup with disks carved out of flash disks to hosts tablespace (temporary or permanent) which require faster read/write response times. FlashCache WriteBack feature eliminates this need.
- Exadata V2/X2 full rack has 5.3TB of FlashCache and X3 has 22.4 TB of flashcache, which can be used for caching writes. This same FlashCache will be shared for both reads and writes, pretty much like what most of the SAN arrays do to use same RAM cache for “write pending cache” and “read cache”.
- Online read logs were already being written to FlashCache via FlashLogs, where by every write to online redo logs is done to both flashcache and disks. As soon as write completes on either of media, control is returned back to LGWR immediately. In some outlier cases, writes to flashcache can be slower in which case if write to disk finishes first, control is returned back to LGWR. This reduced waits on “logfile parallel write” event. FlashLog feature has been available since Exadata Server Software version 11.2.2.4.0 and above.
Good news is FlashCache WriteBack feature has been back ported to Exadata V2, X2, X2-8 models as well via Exadata Server Software version 11.2.3.2.0. You can improve the performance of your existing Exadata V2, X2-2 and X2-8 models by 10 times for FREE by simply upgrading the software. No need to swap out older Exadata models with new ones to get the performance boost. This is the part I like most.
Bad news is FlashCache WriteBack feature will only work if Grid Infrastructure and Database software version are at 11.2.0.3 BP9 (11.2.0.3.9) and above. This is because 11.2.0.3.9 or later include fixes for following bugs which are the marker bugs to add Flash Cache Write-Back support to ASM and RDBMS software.
Bug 14132953 Enhanacement to add Write-back flash cache resilvering support
Bug 14143451 Enhancement for ASM write-back flash cache resilvering support
Oh well, before older versions of GI and database software go out of support, one needs to upgrade them to latest patchset in any case, so bad news is not so bad after all.
Vishal Gupta's Blog 
Sumukh Keni said
Hi Vishal,
Thanks for this gr8 post. Do know of anyone who would have upgraded their existing Exadata infrastructure and infiniband software for leveraging write back FlashCache features and are doing well?
Thanks,
Sumkh
Vishal Gupta said
This patch came out just few days ago, so i doubt anyone might have updated their Exadata’s yet. But people might have tried it in their labs. Enkitec have exadata for their testing, they might have tried it. Andy Colvin may be able to tell you.
YD Kim said
In case of Flash failure, How All data which are written into flash are protected? Can you explain the procedure in detail?
Tim Hopkins said
Hi Vishal (been a while!),
Did you read that they’d implemented flash write back for direct path temp writes or assumed they had? It’d be nice but I haven’t actually seen that explicitly mentioned in what I’ve read so far.
Cheers,
Tim
Vishal Gupta said
Tim,
Direct path temp writes are written to temporary tablespace, since FlashCache writeback is enabled for all write operations for all tablespaces (in addition to already implemeted online redo logs using FlashLogs), then even the direct path temp write would start using Write-Back FlashCache feature. They would also automatically benefit from faster writes to FlashCache.
What i am not clear about it how is the dirty writes to FlashCache (not de-staged to spinning disks) are protected from FlashCard failures? I am not sure if they is any redundancy (mirroring) implemented for FlashCache writes as well, just like it is implemented for ASM disks. Also considering that 4 FDOMs are provisioned from the same FlashCard, if there is indeed any redundancy, is it implemented across to different cards (to protect against card failure) or different cells (to protect against cell failure)?
Vishal
Vishal Gupta said
Tim,
By the way, we are having our own set of fun ever since we have upgraded to Exadata Storage Server 11.2.3.2.0.
– Initially our V2 nodes were rebooting after every few hours due to loss of infiniband connectivity to few cells at random. This is fixed either by disabling irqbalance service with UEK kernel or by reverting to non-UEK kernel which comes along with 11.2.3.2.0 image on all compute nodes. Cell nodes can remain on the UEK kernel without any changes.
– Now we are battling with drop in the redo apply rate on standby database sitting on ZFS appliance using NFS over infiniband. For no reason, ever since we upgrade to 11.2.3.2.0, speed has dropped.
Umesh said
Very clearly written difference between write back and write through.
Rameez said
Nice post ..redundancy is covered for flashcache as per oracle
Click to access exadata-smart-flash-cache-366203.pdf