Oracle RAC 12.2 on SLES12 – How to disable Hardware Lock Elision Support

The Problem

In the last week, I had to set up an Oracle RAC (and thus, the Clusterware) version 12.2.0.1 on SuSE Linux Enterprise Server 12 SP2 (SLES12 Sp2). The gridSetup.sh failed latest in root.sh of the first node with a kinda non-intuitive error:

CRS-5804: Communication error with agent process
CRS-4000: Command Start failed, or completed with errors.
2017/07/10 10:18:49 CLSRSC-119: Start of the exclusive mode cluster failed
Died at /u01/app/12.2.0/grid_1/crs/install/crsinstall.pm line 2053.
The command ‘/u01/app/12.2.0/grid_1/perl/bin/perl -I/u01/app/12.2.0/grid_1/perl/lib -I/u01/app/12.2.0/grid_1/crs/install /u01/app/12.2.0/grid_1/crs/install/rootcrs.pl ‘ execution failed

Thank you – for nothing.

The rootcrs.log in /tmp/GridAction<date> directory was a bit more enlightening, but not much:

CRS-5804: Communication error with agent process
CRS-4000: Command Start failed, or completed with errors.
The exlusive mode cluster start failed, see Clusterware alert log for more information
Executing cmd: /u01/app/12.2.0/grid_1/bin/clsecho -p has -f clsrsc -m 119
Command output:
> CLSRSC-119: Start of the exclusive mode cluster failed 
>End Command output
CLSRSC-119: Start of the exclusive mode cluster failed
###### Begin DIE Stack Trace ######
Package File Line Calling 
--------------- -------------------- ---- ----------
1: main rootcrs.pl 287 crsutils::dietrap
2: crsinstall crsinstall.pm 2053 main::__ANON__
3: crsinstall crsinstall.pm 1963 crsinstall::perform_initial_config
4: crsinstall crsinstall.pm 653 crsinstall::perform_init_config
5: crsinstall crsinstall.pm 813 crsinstall::init_config
6: crsinstall crsinstall.pm 380 crsinstall::CRSInstall
7: main rootcrs.pl 446 crsinstall::new
####### End DIE Stack Trace #######

Eh… yes.

The error in the cluster alert log finally was better, and made me curious:

[ORAROOTAGENT(20948)]
CRS-8503: Oracle Clusterware process ORAROOTAGENT with operating system process ID 20948 experienced fatal signal or exception code 11.
Errors in file /u01/app/oracle/diag/crs/myserver08/crs/trace/ohasd_orarootagent_root.trc (incident=1):
CRS-8503 [__lll_unlock_elision()+48] [Signal/Exception: 11] [Instruction Addr: 0x7f5df148a4a0] [Memory Addr: (nil)] [] [] [] [] [] [] [] []

Ah. Of course. :)

The System

CPU:
Intel(R) Xeon(R) CPU E5-2643 v4 @ 3.40GHz

Kernel:
# uname -a
Linux myhost08 4.4.59-92.24-default #1 SMP Thu Jun 22 14:29:09 UTC 2017 (d11a83a) x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/SuSE-release
SUSE Linux Enterprise Server 12 (x86_64)
VERSION = 12
PATCHLEVEL = 2

First Conclusions

This unlock_elision thing was unusual enough to raise suspicions. First, a little research about the term: Seems that Lock Elision is a part of Intel® Transactional Synchronization Extensions (Intel® TSX) and is described here. Long story short: A CPU extension to work with mutexes more efficiently, dealing with transactional serialisation on the chip. Cool stuff.

And for one or other reason, Oracle does not like that on this SLES12 SP2 system with this E5-26xx chip. Oracle came back with a short and somewhat cryptic hint: “Would it be possible to disable HLE and retry again?” Guess HLE means “Hardware Lock Elision”.

If we can’t have it, how can we get rid of it, to free our way?

SLES12 – How to disable Hardware Lock Elision Support

Lock Elision on Linux is used by the libpthread libraries, and thankfully, the distributor has prepared a plan B for people who are not blessed with this performance booster: in /lib64 is a directory “noelision” containing a symlink to an appropriately pre-configured libpthread-x.xx.so. All we have to do now is to ensure that dynamically linked binaries will use this instead of the “normal” one. I decided not to use ld.so.conf directly, but to use a “distributor-proof” approach and generate a noelision.conf in /etc/ld.so.conf.d, just containing:

/lib64/noelision

And, of course run “ldconfig -v” afterwards. Check its output for the new libpthread symlink, should be in the upper part.

There’s no point in checking the linker search path now (with gcc -v or ld itself), they will not change. Since we talk about the effect of dynamically linked libaries on started programs, all you can do is to look into the output of ldconfig -v as described above to check before re-testing gridSetup.sh.

Yes, that did the trick, and I am grateful.

Be careful how you choose your hardware, folks… Best regards
Martin

Thanks and Acknowledgements

 




You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

One Response to “Oracle RAC 12.2 on SLES12 – How to disable Hardware Lock Elision Support”

  1. Martin Klier Says:

    Two links on the technical background, thanks to Tanel Poder:

    “There was a bug with TSX in some earlier CPU versions:”

    http://www.pcworld.com/article/2464880/intel-finds-specialized-tsx-enterprise-bug-on-haswell-broadwell-cpus.html

    https://bugzilla.redhat.com/show_bug.cgi?id=1406791

Leave a Reply