Vishal Gupta's Blog

Exadata Storage/Compute Node remote re-imaging

Posted by Vishal Gupta on Oct 23, 2011

When you have a Exadata machine in the lab and you are testing lot of different things or giving hand-ons training to production DBA on lab Exadata to familiarize them with Exadata patching, one has to frequently start from scratch i.e . a particular storage/compute node image. Wouldn’t it be nice if you could write a script to re-image the servers? But alas!!! that is not possible with Exadata. One has to do all the reimaging manually. Even with the manual process, one has to insert the external USB into each storage/compute node and then remote it before reboot to complete the process. With ILOM (Integrated Lights Out Management) it is possible to mount a remote cdrom image (*.iso) file as a virtual cdrom. One can create an storage/compute node image iso using computeImageMaker_<imageversion>.x86_64.tar or cellImageMaker_<imageversion>.x86_64.tar file.

For example for compute node, please note we navigate into dl380 for compute node

# tar -pxvf computeImageMaker_11.2.1.2.3_LINUX.X64_100305-1.x86_64.tar
# cd dl380
# makeImageMedia.sh computeNodeImage.iso

Or for cell node, please note we navigate into dl180 directory for cell.

# tar -pxvf cellImageMaker_11.2.1.2.3_LINUX.X64_100305-1.x86_64.tar
# cd dl180
# makeImageMedia.sh cellImage.iso

Would it not have been nice, if one could just mount this iso file via ILOM as virtual cdrom, change the boot order in BIOS by booting into bios, which can also be force either via ILOM gui or via /usr/bin/biosconfig -set_boot_override <xmlfile> command, and choose virtual cdrom as first boot device. But problem with this approach is cell/compute node imaging process resets the ILOM, which means that even our virtual cdrom iso image is also removed during this process and this results in imaging process not completing properly.

One could try leaving a external USB permanently attached on lab exadata, and then via ILOM try to reimage, so that during ILOM reset boot image device is not detached and once can simply reconnect to ILOM to continue answering the on-screen messages. But one problem with this approach is, as part of imaging one has to remove the external USB stick otherwise automated configuration scripts dont funtion properly. I tried it on the lab exadata and could get the cell to reimage properly with external USB still attached. Even after manually changing the boot order to internal USB and then harddrive, which is what cell checks for during each reboot validation, cell just sat on a blinking prompt without going forward. I left the cell overnight as well, thinking that properly some processing is going on, but with no luck. Once i removed the external USB stick, automated configuration scripts were able to properly complete the imaging process and came to automated run of ipconf script, which set the various setting on the cell at first boot after reimaging process.

Would it have been nice, it all this could be done remotely via ILOM once ilom has been connected to the network, as ilom network configuration is not reset during re-imaging process. But that would be wishful thinking !!!

I was trying this with 11.2.1.2.3 image, as that was my starting image on compute/cell nodes in prod/dr/test, so wanted to replicated the same history. I have not tried with later image versions yet to see if process has been improved in this regard, that trial would be for some time later.

Regards,

Vishal



Advertisement
			

			
		
Posted in Exadata Installation, Exadata Patching, Oracle Exadata |    4 Comments »

		


	
		

			
Exadata Storage Server 11.2.2.4.0 Patching

			
Posted by Vishal Gupta on Oct 22, 2011

			

				





Non-interactive shell issue for Database Host minimal pack 


Recently i set about patching Exadata Storage Server software to from 11.2.2.x.x to 11.2.2.4.0, which is the latest patch from Oracle Corporation. I was testing and documenting the process for one of my client and wanted to automate this as much as possible, as in past people actually executing the commands had missed running few commands on certain nodes. As with any Exadata storage server software patch, there is cell node component of patch which is patched using patchmgr either in rolling or non-rolling fashion. And there is database host component, called database minimal pack. Release note of 11.2.2.4.0 asks the install the patch (after running some prerequisites) using ./install.sh -force option. I was taking the approach of install patch on one cell node and if successful, then apply on rest of the cell nodes in parallel. Similarly apply the database minimal pack patch on one compute node, then if successful, apply it on rest of the compute nodes in parallel. And what could be more convenient to run command in parallel on many nodes than dcli command. So i programatically created an dbs_group_withoutfirstnode file with all the compute nodes apart from first compute node. Then installed the patch on first compute node, which was successful. After that using dcli i transferred the patch to other nodes, extracted its contents in parallel. Then using dcli command ran the (cd <patch_directory>; ./install.sh -force) command on rest of the compute nodes. But guess what, compute node patch does not like the running via dcli. DCLI simply runs the command on a remote host using “ssh command” method in simple terms. Though its slightly more complex. Effect of running command via dcli is that, all command are run in non-interactive session i.e. without tty terminal or standard output/error. It means that if your script is not redirecting all standard output and standard error messages to a file, then it will exit with a non-zero (i.e unsuccessful) exit code. install.sh script gives a call to dopatch.sh, which in turn calls a series of functions listed. As part of one of the function, it tries to set update the image version and adds it to image history. In this function, it tries to output the error messages explicitly to /dev/stderr device. As a result of this, if compute node patch is run via some automated script, it exits at this step and fails to run any further steps which include firmware update to ILOM and BIOS upgrade etc.


Now after this has happened, imageinfo command will show the new version, but there will be empty status and activation date. imagehistory will also not show the new image version. If you try to rollback the patch using ./install.sh -rollback-ib command, it will complain that version is not valid, as it is not set with success status. So if you try run /opt/oracle.cellos/imagestatus -set success , then it will complain. But you can force it by using /opt/oracle.cellos/imagestatus -set success -force db_patch. After this you will be able to use the rollback. And then you can install the patch again using an interactive shell.


grub.conf Symoblic link Issue


I also noticed that symbolic link /etc/grub.conf which points to /boot/grub/grub.conf is missing on OEL5.5 compute/cell nodes. OEL5.5 is installed starting with 11.2.1.3.1 cell image.


Suggestions for Oracle Exadata Development


Exadata development team could write their upgrade/patching so that they are compatible with dcli, it allows to automated the patch and save lot of hassle.


Summary


– Don’t use non-interactive shell or dcli to run compute node patching commands.

– Check your /etc/grub.conf symbolic link exists which needs to point to /boot/grub/grub.conf.


Hopefully this will save some hassle to someone out there patching production Exadata’s.


[Update, 05-Nov-2011]


One can redirect all the standard output and standard error to a file, then it will be possibile to run install.sh to install compute minimal patch via dcli.


cd /opt/oracle.Support/onecommand/
dcli -l root -g dbs_group "mkdir -p /opt/oracle.Support/onecommand/patches/patch_11.2.2.4.0.110929"

# Transfer the compute node minimal patch file
dcli -l root -g dbs_group -d /opt/oracle.Support/onecommand/patches/patch_11.2.2.4.0.110929/ -f /opt/oracle.Support/onecommand/patches/patch_11.2.2.4.0.110929/db_patch_11.2.2.4.0.110929.zip

# Unzip the compute node patch file
dcli -l root -g dbs_group  "(cd /opt/oracle.Support/onecommand/patches/patch_11.2.2.4.0.110929/; unzip -o db_patch_11.2.2.4.0.110929.zip)"

# Run the compute node patch
dcli -l root -g dbs_group "(cd /opt/oracle.Support/onecommand/patches/patch_11.2.2.4.0.110929/db_patch_11.2.2.4.0.110929 ; ./install.sh >> install.sh.log 2>&1)"




Cheers,

Vishal Gupta

			

			
		
Posted in Exadata Patching, Oracle, Oracle Exadata |    Leave a Comment »

		


	
		

			
Oracle Exadata Documentation

			
Posted by Vishal Gupta on Oct 15, 2011

			

				
As  most of you are aware, Oracle does not publish the Exadata documentation. Apparently this is protect their software from competitors. But in my opinion, Oracle could increase their Exadata sales by making their documentation freely available for everyone to read, just like their other product’s documentation.  Wouldn’t it be nice, when someone asks you regarding a feature of Exadata, if you had the ability to direct them to a link of Exadata documentation for that feature and they could be read at their leisure in detail.


I feel if more and more people know about Exadata, they would be able to advocate about Exadata and try to convince their company’s decision makers into buying Exadata machines. So making Exadata documentation publicly available should help oracle rather than their competitors. Also a knowledgable dba would be able to manage Exadata better than un-knowledgable dba.


Many Oracle DBA managing Exadata are not aware that Exadata documentation is available on every cell under /usr/share/doc/oracle/Exadata/doc folder. But if you dont have access to a Exadata cell, then how do you view the Exadata documentation? Well you have few options for that.


Ways to obtain Exadata Documentation


1.  Ask your friendly DBA/SystemAdmin who has access to Exadata cell to give your entire (including sub-directories) contents of /usr/share/doc/oracle/Exadata/doc folder. Then using your favourite internet browser open the index.html file to view documentation.


2.  Extract the exadata documentation from cell images. You don’t have to install the Exadata Storage server software to obtain its documentation.


    

  1. Create an Oracle Web Account at Oracle Technology Network aka OTN . Account creation direct link.
    2. 

  2. Wait for account validation email to come, click on link to verify your Oracle single sign on account.
    3. 

  3. Go to Oracle eDelivery site, now know as Oracle Software Delivery Cloud at http://eDelivery.oracle.com and login using your Oracle Web account.
    4. 

  4. Under Media pack search, select product pack as “Oracle Database” and platform as “Linux x86-64” and hit Go.
    5. 

  5. It will list the Oracle Exadata Storage Server (CELL) Software Media pack links, download the storage server software. 11.2.2 media pack contains all 11.2.2.x.x cell and compute images.
    6. 

  6.  Extract the files.
    7. 

  7. For older cell images (upto 11.2.2.3.3), you will find the cellImageMaker_<imageversion>.x86-64.tar.zip.
    8. 

  8. Extract the tar file from zipfile. In older cell images (upto 11.2.2.2.2), it will prompt you for a password which can be obtained from Oracle support. This password is common on all cellimage zipfiles. In newer cellimage from 11.2.2.3.2 onwards, zipfile is no longer password protected.
    9. 

  9. On any linux/unix machine extract the Exadata documentation from tar file using “tar -pxvf cellImageMaker_<imageversion>.x86-64.tar dl180/boot/cellbits/doclib.zip” command. This extracts the doclib.zip file containing Exadata documentation for that cell image version.
    10. 



Enjoy learning Oracle Exadata from documentation.


[Update] – Exadata Documentation is now available via Patch 10386736


[Update, 28-Feb-15] Oracle Exadataion documentation is now available online at following link – http://docs.oracle.com/cd/E50790_01/welcome.html


Disclaimer – Please ensure that you or your company has the Exadata software license before obtaining the Exadata documentation, as it is Oracle’s copyright protected material and only meant to exadata license holders. I will not be liable for any license or copyright breaches due to you or your company not having a valid license before obtaining the Exadata documentation.

			

			
		
Posted in Oracle Exadata |    16 Comments »

		


	
		

			
Oracle 11gR2 on RHEL6

			
Posted by Vishal Gupta on Sep 19, 2011

			

				
Just a quick note to say that RedHat submitted certification test result of Oracle 11gR2 on RHEL6 to Oracle Corporation on 09-Aug-2011, so we should expect the formal certification around last week of Sep-2011.


More news at – http://www.redhat.com/about/news/blog/Red-Hat-Submits-Oracle-11gR2-on-Red-Hat-Enterprise-Linux-6-Certification-Test-Results-to-Oracle


[Update: 17-Apr-2012]


Oracle has finally certified 11gR2 on OEL6 and RHEL6. See the announcement here

			

			
		
Posted in 11gR2, Linux, Oracle |    17 Comments »

		


	
		

			
Direct Path Reads – 11g Changed Behaviour

			
Posted by Vishal Gupta on Aug 19, 2011

			

				
In 10g, serial full table scans for “large” tables used to always go through cache (by default). If table was small it was placed at most recently used (MRU) end of the buffer cache. If table is large it is placed at least recently used (LRU) end of the buffer cache.


In 11g, full table scan do not always go through the buffer cache. Decision to read via direct path or through cache is based on the size of the table, buffer cache and various other stats. Table is considered to be small or large based value of _small_table_threshold internal parameter.  Default value of this parameter is 2% of buffer cache size and is specified in blocks. This means any object (table) smaller than 2% of the buffer cache will be read via the buffer cache and not using direct path read. And tables larger than 2% of the buffer cache are read via direct path read and not via buffer cache. With AMM (Automatic Memory Management) or ASMM (Automatic Shared Memory Management), buffer cache could drop to a lower value if memory is falling short for shared pool. In such a case after restart of instance, _small_table_threshold parameter would become even lower due to decreased buffer cache.


By enabling event 10949 at either session or system/instance level one can disable the autotuning of direct path reads for full table scans. It means full table scan on every table whether small or large, will go via buffer cache. This could flush out the already cached cached objects. I would strongly advise against setting this for all the databases in our standard build.


[/source]SQL> alter system set events '10949 trace name context forever' scope=spfile;

$ oerr ora 10949 10949, 00000, "Disable autotune direct path read for full table scan"
 // *Cause:
 // *Action: Disable autotune direct path read for serial full table scan.




If you are doing full table scans on the same LARGE table too often, then I would suggest to tune the query or create an index on the table. You could also set the minimum value for db_cache_size, so that it does not fall below the minimum (normal workload) levels and small_table_threshold parameter value does not fall below desired threshold.


Related Oracle MOS Notes


Doc ID 793845.1 – High ‘direct path read’ waits in 11g


Doc ID 787373.1 – How does Oracle load data into the buffer cache for table scans ?

			

			
		
Posted in Oracle |    9 Comments »

		


			

			
« Previous Entries

			
Next Entries »

		


	





 











	

		

	

	

	

	

	



		
		

		
		



	
	


	

	
				




	
	

		

			%d bloggers like this: