Archive for the 'Linux / Unix' Category

Oracle Clusterware 11.2: ASM crashes at startup

These days, a customer’s Oracle Clusterware (2 nodes) crashed one ASM instance at every startup.

More Facts:

  • It was not possible to start it manually, too.
  • The CSSD was running.
  • For obvious reasons, CRSD did not start.
  • The other ASM instance in the cluster recognized CLUSTER RECONFIGURATION for a short period of time.

The ASM Alert Log file looked like:

Sun Nov 13 13:44:08 2011
 MMNL started with pid=21, OS id=7783
 lmon registered with NM - instance number 2 (internal mem no 1)
 Sun Nov 13 13:46:05 2011
 System state dump requested by (instance=2, osid=7684 (PMON)),
         summary=[abnormal instance termination].
 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_7706.trc
 Sun Nov 13 13:46:05 2011
 PMON (ospid: 7684): terminating the instance due to error 481
 Dumping diagnostic data in directory=[cdmp_20111113134605], requested by (instance=2, osid=7684 (PMON)),
         summary=[abnormal instance termination].
 Instance terminated by PMON, pid = 7684

Strange problem. Looking up device permissions, read write tests, rebooting the cluster in a downtime window – nothing.

To make a long story short: The NTP daemon did not get his time synchronisation, but was running. Thus, CTSS was in observer mode, and server time started drifting apart. Fixing NTP, fixed the cluster.

Nota bene

Linux Network bonding – setup guide

After looking up Linux bonding stuff for the thrid time, I planned to write an article aubout it. But there are lots of good blog posts on this, so just click here at unixfoo:

Linux Network bonding – setup guide

(strange link, I know, but it works)

Hope it helps for you next high avaliability project, like Oracle RAC, Oracle Grid Infrastructure or Oracle DataGuard.

Take care

Native ZFS for Linux

After long-aking FUSE implemented ZFS on Linux, Lawrence Livermore Labs have released a reimplemented ZFS for native use with a Linux Kernel.

See details here:

Still have to test it, but I’d daresay, it was time.
Good pooling

AIX: Avoiding “ORA-27126: unable to lock shared memory segment in core”

Did you ever change lock_sga = true for an Oracle Instance on AIX 5.3, and did see

ORA-27126: unable to lock shared memory segment in core
IBM AIX RISC System/6000 Error: 1: Not owner

in Return?

Do the following:

# vmo -o v_pinshm
v_pinshm = 0
# vmo -o v_pinshm=1
Setting v_pinshm to 1
# lsuser -a capabilities oracle
# chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle
# lsuser -a capabilities oracle

It should do now.


Oracle 11gR2 ASM / ACFS: A first benchmark (poorly)

Hi folks,

since Oracle 11g Release 2 is out now, I had to test one of the most-missed ASM features: the ASM cluster file system ACFS.

My Setup:

  • Two VMware nodes with 2 CPUs and 1,5GB of RAM each
  • Oracle Enterprise Linux 5.3 x86_64
  • Four virtual cluster disks from the ESX server, 10GB in size each
  • Building disk group DATA from them, with redundancy NORMAL
  • containing four failgroups with each one of the disks within
  • In DATA, one ACFS volume of 1 GB in size, mounted to /acfs1

Read more…

Oracle 11g JDBC driver hangs blocked by /dev/random – entropy pool empty

On a headless (=without console) network server, the 11g JDBC driver used for (java) application connect may cause trouble. In my case, it refused to connect to the DB without any error, trace or log entry. It simply hung. After several hours, it connected one time, and freezed again. Remote debugging done by the development clarified that it locks after calling SeedGenerator() and SecureRandom().
Read more…

Setting ulimit -m in AIX

AIX has its own rules, as I have to discover these days. Coming across ulimit, I found out that smitty configures value A here and value B completely elsewhere. Hardest one so far has been “ulimit -m” (“Specifies the size of physical memory, in number of K bytes.” a user can assign), since I could not find where to specify it in smitty. At some point, I stopped investigating smitty, and went directly to


but it did not become easier, nobody could tell me what’s the parameter for -m in limits’ terms. Some suspicion did go in direction “rss”, but oddly, lsuser delivered “rss=65536″ and ulimit -a delivered “32768”. So at first glance, they did not seem to be connected despite a similar function description in documentation.

But finally, I came over Unix Essentials Blog and was told:

“ulimit -m” is “rss” in limits, so i specified

rss = -1
rss_hard = -1

and was happy. The lsuser command shows “rss=-1″ and ulimit -m shows “unlimited” now. By the way: ulimit -m is specified in KB, rss in blocks …

Thanks to morsing in freenode’s #aix as well!


Single Sign On for Apache 2.2 and Active Directory 2003 R2 with SuSE 10.2, mod_auth_kerb and MIT Kerberos

this is a small howto out of my needs, suggestions are always welcome!

  1. Assumptions for this paper
    1. You know and basically follow, but there are some facts missing/wrong.
    2. 2003R2 KDC’s are and (DNS: and
    3. Kerberos Realm for ADS is ADS.EXAMPLE.COM (upper case required!)
    4. Web server (hostname.domain) is
    5. DNS and reverse DNS on and for the machine in FQDN works, verified with “dig” and “dig -x”

    Read more…

Switch workspace on command line

Sometimes, it’s nice to change the current windowmanager workspace on command line, for example if you desire to start a application exactly HERE or THERE.

For KDE you may want to do it with “kstart”. But a tool like kstart might not be available for your windowmanager, so have a look at “wmctrl” as well:

Have fun

Oracle 10g Data warehouse ORION benchmark, size 20TB, 1200MB per second

During the last days, I have had the opportunity to test and benchmark a data warehousing hardware, that’s really fast for its money. It’s not suitable for real/available production, since it depends on disk striping over a bunch of components, but I considered it as a good way to push the limits a bit.

Result: A throughput of about 1200MB/s and nearly 1200 iops. Period. You may quickly want to see detailed results on the bottom of the page, but for fully understanding of the results look up the setup as well.
Read more…

You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.