Archive for the 'Linux / Unix' Category

Oracle Clusterware fails: Can’t install ohasd service: Inappropriate ioctl for device line 5427

Setting up Oracle Clusterware on SuSE Linux Enterprise Server (SLES) SP2 should work flawlessly, but sometimes it does not. :) It turned out that this would become a pair of blog entries. Second one is about “USM driver install actions failed (oracleoks.ko)“. But step by step. On Saturday morning, failed with the following error:

Failed to install ohasd startup script, error: Can’t install ohasd service: Inappropriate IOCTL (I/O-Control) for device

Can’t install ohasd service: Inappropriate IOCTL (I/O-Control) for device at /u01/app/grid/11.2.0/crs/install/ line 5427.

/u01/app/grid/11.2.0/perl/bin/perl -I/u01/app/grid/11.2.0/perl/lib -I/u01/app/grid/11.2.0/crs/install /u01/app/grid/11.2.0/crs/install/ execution failed

There are several “My-Oracle-Support” (MOS) entries (bug notes and documents) for failing in, but not for line 5427 – and the line really matters! This script does a lot, and usually different things in different lines. :)

Whenever dealing with malfunctions, the rootcrs logfile ($GRID_HOME/cfgtoollogs/crsconfig/rootcrs_<hostname>.log) is your best friend. It appears in a not-too-verbose style, and if invokes OS- or third party commands, it quotes those outputs in a useful way – Bravo Zulu for the Oracle scripters here.

In my particular case, the problem was related to Linux’ insserv command, thats used to integrate ohasd into the SYS V startup script structure. My IBM Storage Manager Agent (service SMagent) and Oracle’s Trace File Analyzer (service init.tfa) had a dependency loop (dumbass SMagent depends on $all, /*NO COMMENT*/). In my case, I happily removed the $all dependency, and off it went.

Good luck with your GI

Oracle on AIX: How to find out the process memory usage

Calculating memory on Unix is tricky business. Especially when a complex software like Oracle Database has shared memory segments like SGA and Code Area.

One might be convinced to use the following construction to calculate the overall memory footprint of Oracle processes running on this machine:

ps -elf |egrep " oracle* | ora_.*_* " | grep -v egrep \\
| awk '{sum += $10} END {print sum/1024/1024}'

But that’s bad, since the sum is based on the SZ column of the “ps -elf” command. Unfortunately, SZ displays the full core image, but most of it is shared (remember the Oracle Code Area from the architecture diagram). So we greatly overestimate the memory footprint this way.


When you use “ps v” for a given PID, you get it more detailled: SIZE is the non shared data rump, TSIZE the shared text component of the image. In sum, they roughly add up to SZ.
(Units are all in KB)

I tried to find a solution. This is the original, overestimated version:

# ps -elf |egrep " oracle* | ora_.*_* " | grep -v egrep \\
| awk '{sum += $10} END {print sum/1024/1024}'

This one extracts the PID from “ps -ef”, executes “ps v” for each and adds them up. The greps might be a bit ugly, but it works for Oracle. :)

# for X in $(ps -ef | egrep " oracle* | ora_.*_*  " | grep -v egrep | awk '{print $2}'); \\
do ps v $X | grep ora | awk '{print $6}'; done \\
| awk '{sizesum += $1} END {print sizesum/1024/1024}'

I ran both commands on the same prod database system within the same second, so the difference should be realistic.

Stay safe

Thanks to Maxym’s old blog entry for great impressions!

Additional reading:

Speaking at COLLABORATE 14: “YOUR machine and MY database – a performing relationship!?”

I’m excited to announce that IOUG accepted my talk

“YOUR machine and MY database – a performing relationship!?”

for COLLABORATE 14 in Las Vegas.


I’d love to see you there – for tech talk, gossip and meeting old and new friends!


Databases affect machines, machines affect databases. Optimizing one is pointless without knowing the other. System administrators and database administrators will not necessarily have the same opinion – often because they know little about the opposite’s needs. This lecture was made to promote understanding – showing how the database can stress the server, and how the server can limit the database. And why two admins sometimes don’t speak the same language, not even with a developer as an interpreter.

  • Recall the different needs of different technical layers underneath a database system.
  • Understand the technical collaboration of hardware, operating system and database.
  • Plot ways how to avoid collisions, competition and concurrency.
  • Promote collaboration!

Date, time and location:

Thu, Apr 10, 2014
01:00 p.m. – 02:00 p.m.

Level 3, Lido 3003

The Venetian and Sands Expo Center
201 Sands Ave
Las Vegas, NV 89169

Presentation and papers


Martin Klier now on twitter

After ignoring the little bird telling things for quite a while, I decided to join the tweeters. Twitter might bring more color into my daily reading. :)

If you feel like, just follow me – @MartinKlierDBA

Oracle: RHEL6 and Oracle Enterprise Linux 6 certified for Database

Hi Linux-DBAs,
Red Hat Enterprise Linux 6 and Oracle Enterprise Linux 6 are now certified for Oracle Database

Please see this link:

Best regards

DOAG Conference 2011 – Impressions and Look-at’s

Once again this year, the German Oracle Users Group has its annual conference and exhibition in Nuremberg. (DOAG Konferenz und Ausstellung 2011, Nürnberg). Being there is nearly a must for Oracle guys in German speaking countries.

As usual, here comes my unordered, incomplete and ad-hoc list of things I wrote down to have a closer look at in the next year, coming up during or from random talks I attended.

Day 1 (Tue 15.11.2011)

  • AVG_ROW_LENGTH of a table vs. Blocksize
  • analyze table X validate stucture cascade
  • Linux: Transcendent Memory
  • Linux: CleanCache and zcache
  • Linux: Cgroups
  • Linux: Transparent Huge Pages (wow!)
  • Linux: DTrace
  • Linux proprietary: Ksplice
  • View: registry$history for the REAL version number
  • Rolling Upgradable patches means minimal downtime on on Single Instance DBs, in combination with Out-of-Place-Upgrade)
  • Bug 10187168 in PSU (_cursor_features_enables=1026
  • Typeset conversions: CSscan and DMU utilities
  • AIX patch following note 1246995 (Memory Footprint)

Read more…

Oracle Clusterware 11.2: ASM crashes at startup

These days, a customer’s Oracle Clusterware (2 nodes) crashed one ASM instance at every startup.

More Facts:

  • It was not possible to start it manually, too.
  • The CSSD was running.
  • For obvious reasons, CRSD did not start.
  • The other ASM instance in the cluster recognized CLUSTER RECONFIGURATION for a short period of time.

The ASM Alert Log file looked like:

Sun Nov 13 13:44:08 2011
 MMNL started with pid=21, OS id=7783
 lmon registered with NM - instance number 2 (internal mem no 1)
 Sun Nov 13 13:46:05 2011
 System state dump requested by (instance=2, osid=7684 (PMON)),
         summary=[abnormal instance termination].
 System State dumped to trace file /u01/app/oracle/diag/asm/+asm/+ASM2/trace/+ASM2_diag_7706.trc
 Sun Nov 13 13:46:05 2011
 PMON (ospid: 7684): terminating the instance due to error 481
 Dumping diagnostic data in directory=[cdmp_20111113134605], requested by (instance=2, osid=7684 (PMON)),
         summary=[abnormal instance termination].
 Instance terminated by PMON, pid = 7684

Strange problem. Looking up device permissions, read write tests, rebooting the cluster in a downtime window – nothing.

To make a long story short: The NTP daemon did not get his time synchronisation, but was running. Thus, CTSS was in observer mode, and server time started drifting apart. Fixing NTP, fixed the cluster.

Nota bene

Linux Network bonding – setup guide

After looking up Linux bonding stuff for the thrid time, I planned to write an article aubout it. But there are lots of good blog posts on this, so just click here at unixfoo:

Linux Network bonding – setup guide

(strange link, I know, but it works)

Hope it helps for you next high avaliability project, like Oracle RAC, Oracle Grid Infrastructure or Oracle DataGuard.

Take care

Native ZFS for Linux

After long-aking FUSE implemented ZFS on Linux, Lawrence Livermore Labs have released a reimplemented ZFS for native use with a Linux Kernel.

See details here:

Still have to test it, but I’d daresay, it was time.
Good pooling

AIX: Avoiding “ORA-27126: unable to lock shared memory segment in core”

Did you ever change lock_sga = true for an Oracle Instance on AIX 5.3, and did see

ORA-27126: unable to lock shared memory segment in core
IBM AIX RISC System/6000 Error: 1: Not owner

in Return?

Do the following:

# vmo -o v_pinshm
v_pinshm = 0
# vmo -o v_pinshm=1
Setting v_pinshm to 1
# lsuser -a capabilities oracle
# chuser capabilities=CAP_BYPASS_RAC_VMM,CAP_PROPAGATE oracle
# lsuser -a capabilities oracle

It should do now.


You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.