Oracle Dataguard: ORA-00600 [krsu_upi_atc.7] – crash when primary ships the first time

After setting up a new Oracle Dataguard system (primary plus one standby DB), everything looked promising.

But after activating the log shipping from primary, and after archiving a redo log for the first time, the primary instance crashed with ORA-00600 [krsu_upi_atc.7]. Without the standby system available (DB idle or listener off), no error occurred.

******************************************************************
 LGWR: Setting 'active' archival for destination LOG_ARCHIVE_DEST_2
 ******************************************************************
 LGWR: Archival destination is a Primary RAC instance: 'ITWMT2'
 Errors in file /u01/app/oracle/diag/rdbms/itwmt/ITWMT/trace/ITWMT_lgwr_22151.trc  (incident=18089):
 ORA-00600: internal error code, arguments: [krsu_upi_atc.7], [], [], [], [], [], [], [], [], [], [], []
 Incident details in: /u01/app/oracle/diag/rdbms/itwmt/ITWMT/incident/incdir_18089/ITWMT_lgwr_22151_i18089.trc
 Use ADRCI or Support Workbench to package the incident.
 See Note 411.1 at My Oracle Support for error and packaging details.
 Errors in file /u01/app/oracle/diag/rdbms/itwmt/ITWMT/trace/ITWMT_lgwr_22151.trc:
 ORA-00600: internal error code, arguments: [krsu_upi_atc.7], [], [], [], [], [], [], [], [], [], [], []
 LGWR (ospid: 22151): terminating the instance due to error 470
 Wed Mar 19 15:27:07 2014
 ORA-1092 : opitsk aborting process
 Wed Mar 19 15:27:07 2014
 System state dump requested by (instance=1, osid=22151 (LGWR)), summary=[abnormal instance termination].
 System State dumped to trace file /u01/app/oracle/diag/rdbms/itwmt/ITWMT/trace/ITWMT_diag_22141_20140319152707.trc
 Dumping diagnostic data in directory=[cdmp_20140319152707], requested by (instance=1, osid=22151 (LGWR)), summary=[abnormal instance termination].
 Instance terminated by LGWR, pid = 22151

Instance terminated by LGWR did not look promising. Plus no search-engine-of-choice hits, no MOS search result. But re-reading the configuration unveiled a very basic mistake: The DB_UNIQUE_NAME of the two databases (primary and standby) was the SAME – not exactly the purpose of a UNIQUE name…. Changing it on standby side, and off it went.

Let me tell you, read carefully.
Martin Klier



DOAG Database Conference Düsseldorf: “YOUR machine and MY database” was accepted

DOAG just informed me that my talk, “YOUR machine and MY database – a performing relationship!?” was accpeted for DOAG Database Conference Düsseldorf. Last year, the first Database Conference at Düsseldorf was a great success, I enjoyed it very much. I’m glad to be part of the speaker’s corps this year.

Hope to see you there!

June 3rd, 2014
Van der Valk Airport Hotel
Düsseldorf

Event: http://www.doag.org/de/events/konferenzen/doag-2014-datenbank.html

 

 

2014-DB-Banner-600x160



Klug GmbH integrierte Systeme wins Oracle Excellence Award Germany 2013 category “ISV”

Klug wins Oracle Excellence Award

I’m proud to announce that my employer, Klug GmbH integrierte Systeme, Teunz (Germany), won the Oracle Excellence Award Germany 2013 in the category of Independent Software Vendors (ISVs).

Read more…



Oracle Clusterware root.sh issue: USM driver install actions failed (oracleoks.ko)

As I already said in my last post about “Can’t install ohasd service“, setting up Oracle Clusterware 11.2.0.4 on SuSE Linux Enterprise Server (SLES) SP2 should work flawlessly, but sometimes it does not. :) This time, it was about the USM drivers.

USM driver install actions failed
/u01/app/grid/11.2.0/perl/bin/perl -I/u01/app/grid/11.2.0/perl/lib 
-I/u01/app/grid/11.2.0/crs/install
/u01/app/grid/11.2.0/crs/install/rootcrs.pl execution failed

USM drivers are components (Kernel object files, extension .ko) enabling ACFS – I don’t use it on this system, but root.sh (in fact, rootcrs.pl) needs a decent directory structure related to the Linux Kernel version: Again, the log file “$GRID_HOME/cfgtoollogs/crsconfig/rootcrs_<hostname>.log” was my friend: It unveiled, that the problem was somewhat related to loading oracleoks.ko. And this file is in directory “$GRID_HOME/install/usm/Novell/SLES11/x86_64/<your-kernel-version>/default/bin”. Trouble is, that good old SLES 11 SP2 has a Kernel that was not foressen by the Oracle folks implementing this piece of software.

Read more…



Oracle Clusterware root.sh fails: Can’t install ohasd service: Inappropriate ioctl for device crsconfig_lib.pm line 5427

Setting up Oracle Clusterware 11.2.0.4 on SuSE Linux Enterprise Server (SLES) SP2 should work flawlessly, but sometimes it does not. :) It turned out that this would become a pair of blog entries. Second one is about “USM driver install actions failed (oracleoks.ko)“. But step by step. On Saturday morning, root.sh failed with the following error:

Failed to install ohasd startup script, error: Can’t install ohasd service: Inappropriate IOCTL (I/O-Control) for device

Can’t install ohasd service: Inappropriate IOCTL (I/O-Control) for device at /u01/app/grid/11.2.0/crs/install/crsconfig_lib.pm line 5427.

/u01/app/grid/11.2.0/perl/bin/perl -I/u01/app/grid/11.2.0/perl/lib -I/u01/app/grid/11.2.0/crs/install /u01/app/grid/11.2.0/crs/install/rootcrs.pl execution failed

There are several “My-Oracle-Support” (MOS) entries (bug notes and documents) for root.sh failing in crsconfig_lib.pm, but not for line 5427 – and the line really matters! This script does a lot, and usually different things in different lines. :)

Whenever dealing with root.sh malfunctions, the rootcrs logfile ($GRID_HOME/cfgtoollogs/crsconfig/rootcrs_<hostname>.log) is your best friend. It appears in a not-too-verbose style, and if rootcrs.pl invokes OS- or third party commands, it quotes those outputs in a useful way – Bravo Zulu for the Oracle scripters here.

In my particular case, the problem was related to Linux’ insserv command, thats used to integrate ohasd into the SYS V startup script structure. My IBM Storage Manager Agent (service SMagent) and Oracle’s Trace File Analyzer (service init.tfa) had a dependency loop (dumbass SMagent depends on $all, /*NO COMMENT*/). In my case, I happily removed the $all dependency, and off it went.

Good luck with your GI
Martin



Oracle on AIX: How to find out the process memory usage

Calculating memory on Unix is tricky business. Especially when a complex software like Oracle Database has shared memory segments like SGA and Code Area.

One might be convinced to use the following construction to calculate the overall memory footprint of Oracle processes running on this machine:

ps -elf |egrep " oracle* | ora_.*_* " | grep -v egrep \\
| awk '{sum += $10} END {print sum/1024/1024}'

But that’s bad, since the sum is based on the SZ column of the “ps -elf” command. Unfortunately, SZ displays the full core image, but most of it is shared (remember the Oracle Code Area from the architecture diagram). So we greatly overestimate the memory footprint this way.

aix-memory-calculation

When you use “ps v” for a given PID, you get it more detailled: SIZE is the non shared data rump, TSIZE the shared text component of the image. In sum, they roughly add up to SZ.
(Units are all in KB)

I tried to find a solution. This is the original, overestimated version:

# ps -elf |egrep " oracle* | ora_.*_* " | grep -v egrep \\
| awk '{sum += $10} END {print sum/1024/1024}'
19.0745
(GB)

This one extracts the PID from “ps -ef”, executes “ps v” for each and adds them up. The greps might be a bit ugly, but it works for Oracle. :)

# for X in $(ps -ef | egrep " oracle* | ora_.*_*  " | grep -v egrep | awk '{print $2}'); \\
do ps v $X | grep ora | awk '{print $6}'; done \\
| awk '{sizesum += $1} END {print sizesum/1024/1024}'
1.57206
(GB)

I ran both commands on the same prod database system within the same second, so the difference should be realistic.

Stay safe
Martin

Thanks to Maxym’s old blog entry for great impressions!

Additional reading:
https://www.ibm.com/developerworks/community/blogs/aixpert/entry/aix_memory_usage_or_who_is_using_the_memory_and_how20?lang=en



Speaking at COLLABORATE 14: “YOUR machine and MY database – a performing relationship!?”

I’m excited to announce that IOUG accepted my talk

“YOUR machine and MY database – a performing relationship!?”

for COLLABORATE 14 in Las Vegas.

collaborate14-logo

I’d love to see you there – for tech talk, gossip and meeting old and new friends!

Abstract:

Databases affect machines, machines affect databases. Optimizing one is pointless without knowing the other. System administrators and database administrators will not necessarily have the same opinion – often because they know little about the opposite’s needs. This lecture was made to promote understanding – showing how the database can stress the server, and how the server can limit the database. And why two admins sometimes don’t speak the same language, not even with a developer as an interpreter.

  • Recall the different needs of different technical layers underneath a database system.
  • Understand the technical collaboration of hardware, operating system and database.
  • Plot ways how to avoid collisions, competition and concurrency.
  • Promote collaboration!

Date, time and location:

Thu, Apr 10, 2014
01:00 p.m. – 02:00 p.m.

Level 3, Lido 3003

The Venetian and Sands Expo Center
201 Sands Ave
Las Vegas, NV 89169
USA

Presentation and papers

2014_141_Klier_odp_v1
2014_141_Klier_v1_doc



Oracle Datenbank Architektur – nicht nur für Einsteiger (DOAG Konferenz 2013)

Talk at DOAG 2013

Thanks everybody for attending my talk “Oracle Datenbank Architektur – nicht nur für Einsteiger” at DOAG Conference 2013. It was a great feeling to have a packed room there. As promised, here comes my presentation and whitepaper (both in German).

OracleArchitektur-DOAG
OracleArchitekturNichtNurFürEinsteiger_Klier

Feedback is always appreciated!



DOAG Presentation: Oracle Standard Edition RAC

Wednesday last week, I had a presentation for my regional Oracle Users Group (DOAG Regio Nürnberg). The month before I was asked to display the difference between Enterprise Edition and Standard Edition RACs.

Here comes the presentation (German), for questions and suggestions just let me know.

Stay highly available
Martin Klier



Martin Klier now on twitter

After ignoring the little bird telling things for quite a while, I decided to join the tweeters. Twitter might bring more color into my daily reading. :)

If you feel like, just follow me – @MartinKlierDBA