Oracle RMAN command prompt hang???
Posted by Vishal Gupta on Jun 22, 2011
This afternoon on a Oracle 11.2.0.2 installation, i was trying to run the Oracle RMAN utility. Rman cli was just hanging and not going into RMAN.
[oracle@linux2 ~]$ . oraenv ORACLE_SID = [oracle] ? 11202 The Oracle base has been set to /opt/oracle/product/rhel4/database/11.2.0.2 [oracle@linux2 ~]$ rman
After entering above rman command, nothing happened. It just hung on the command prompt. I did not receive any error at all. Hmm… whats happening? I issued an CTRL-C to cancel it and tried again. Still same result. Okay lets trace it using Linux strace utility.
oracle@linux2 ~]$ strace rman execve("/usr/X11R6/bin/rman", ["rman"], [/* 31 vars */]) = 0 uname({sys="Linux", node="linux2.vishalgupta.com", ...}) = 0 brk(0) = 0x5a3000 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a95556000 access("/etc/ld.so.preload", R_OK) = 0 open("/etc/ld.so.preload", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0444, st_size=23, ...}) = 0 mmap(NULL, 23, PROT_READ|PROT_WRITE, MAP_PRIVATE, 3, 0) = 0x2a95557000 close(3) = 0 readlink("/proc/self/exe", "/usr/X11R6/bin/rman", 4096) = 19 open("/stub/lib64/libcwait.so", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\3>\1\20\6"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=4176, ...}) = 0 mmap(NULL, 1051008, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2a95558000 mprotect(0x2a95559000, 1046912, PROT_NONE) = 0 mmap(0x2a95658000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0) = 0x2a95658000 close(3) = 0 munmap(0x2a95557000, 23) = 0 open("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/tls/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/tls/x86_64", 0x7fbfffec30) = -1 ENOENT (No such file or directory) open("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/tls/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/tls", 0x7fbfffec30) = -1 ENOENT (No such file or directory) open("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/x86_64/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/x86_64", 0x7fbfffec30) = -1 ENOENT (No such file or directory) open("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib/libc.so.6", O_RDONLY) = -1 ENOENT (No such file or directory) stat("/opt/oracle/product/rhel4/database/11.2.0.2/home/lib", {st_mode=S_IFDIR|0755, st_size=12288, ...}) = 0 open("/etc/ld.so.cache", O_RDONLY) = 3 fstat(3, {st_mode=S_IFREG|0644, st_size=146311, ...}) = 0 mmap(NULL, 146311, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2a95659000 close(3) = 0 open("/lib64/tls/libc.so.6", O_RDONLY) = 3 read(3, "\177ELF\2\1\1\3>\1p\305\1"..., 832) = 832 fstat(3, {st_mode=S_IFREG|0755, st_size=1636072, ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a9567d000 mmap(NULL, 2330696, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x2a9567e000 mprotect(0x2a957ae000, 1085512, PROT_NONE) = 0 mmap(0x2a958ad000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x12f000) = 0x2a958ad000 mmap(0x2a958b3000, 16456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x2a958b3000 close(3) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a958b8000 mprotect(0x2a958ad000, 16384, PROT_READ) = 0 mprotect(0x3e48a14000, 4096, PROT_READ) = 0 arch_prctl(ARCH_SET_FS, 0x2a958b86e0) = 0 munmap(0x2a95659000, 146311) = 0 brk(0) = 0x5a3000 brk(0x5c4000) = 0x5c4000 fstat(0, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a95659000 read(0, <unfinished ...> [oracle@linux2 ~]$
Had to again issue a CTRL-c to come out of trace, as it was just hanging again. I tried connecting to database using sqlplus, that worked fine.
oracle@linux2 ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.2.0 Production on Mon Jun 20 06:08:58 2011 Copyright (c) 1982, 2010, Oracle. All rights reserved. Connected to: Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL>
Hmm… Lets recompile the Oracle binaries, it case something funny was going on with libraries. And then we will re-run our RMAN cli.
[oracle@linux2 tns_admin]$ relink writing relink log to: /opt/oracle/product/rhel4/database/11.2.0.2/home/install/relink.log [oracle@linux2 tns_admin]$ rman
Hmm… still the same.
It turns out the there is Linux RMAN command as well, which is installed by xorg-x11-devel rpm package.
[oracle@linux2 ~]$ which rman /usr/X11R6/bin/rman [oracle@linux2 bin]$ man rman PolyglotMan(1) PolyglotMan(1) NAME PolyglotMan, rman - reverse compile man pages from formatted form to a number of source formats
So lets run the actual Oracle RMAN command.
[oracle@linux2 bin]$ $ORACLE_HOME/bin/rman Recovery Manager: Release 11.2.0.2.0 - Production on Mon Jun 20 07:14:58 2011 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. RMAN>
So, moral of the story is set your PATH correctly. I hope someone will find it useful, in case their rman command is hung.
Jeevan said
Nice one Vishal.
Gokhan Atil said
Nice article. Thanks for sharing!
goryszewskig said
Hi,
actually You nailed it in trace , check first line:
oracle@linux2 ~]$ strace rman
execve(“/usr/X11R6/bin/rman”, [“rman”], [/* 31 vars */]) = 0
Regards
GG
Vishal Gupta said
GG,
you are right, problem was solved from strace itself, but since the output was too long, i was focussing at the last section of strace for clue why it was hanging, rather than paying attention that it was calling the right RMAN command. I never expected a linux (non-oracle) rman command to be there. So i wrongly assumed that it was calling the right rman. Now i think back, strace always gices you the full path of command getting executed in the very first line of it.
Thanks for pointing this out to me.
Gaius said
A mutual associate of ours has the letters “rman” in his surname – so whenever he was logged in,
wc -l `ps -ef|grep [r]man`
would be non-zero regardless of whether RMAN was running. That took me a while to figure out…Vishal Gupta said
Nice. There was person called Norman with us. And a colleague of mine always use to call him NO RMAN 🙂
rmanbackup said
Thanks for sharing , Great information
Abhinav Gupta said
Which version of Linux it was?
Regardless, if we set the PATH with “$ORACLE_HOME/bin” preceding to other ones in profile, we will avoid it.
Vishal Gupta said
Abhinav,
Version of RMAN does not matter here. Problem here is how you setup your PATH env variable. If there is any rman command (in this case reverse compile man page command) in any of the directories appearing in your PATH before $ORACLE_HOME/bin, then it would create the problem.
Vishal
gmesquitax said
Thank you for sharing. I was having the same issue and could solve using your post.
Carl Trimble said
I have this same issue and am having trouble resolving it…
When I type “man rman”, I get “No manual entry for rman”
I am an oracle noob, but would love it if anyone could point me in the right direction.
My strace says…
“/usr/X11R6/bin/rman”
khaleel said
Thanks buddy….
WillieCnpaoq said
I see your site needs some unique & fresh content.
Writing manually is time consuming, but there is solution for this hard
task. Just search for – Miftolo’s tools rewriter
모바일바둑이 said
모바일바둑이
Oracle RMAN command prompt hang??? « Vishal Gupta's Blog