Exadata troubleshooting easy with GetExaWatcherResults.sh

If you have for example a performance problem or a crash on your Exadata Machine and you need an overview about the machine then use the very powerful tool „GetExaWatcherResults.sh“ which based on the data of Exawatcher.

GetExaWatcherResults.sh give you a details about:

  • CPU utilization and details
  • IO Summary
  • and more

GetExaWatcherResults.sh will generate very good graphics and that is what I will show now.

How is the way to get these graphs?

Login as "root"

cd /opt/oracle.ExaWatcher

GetExaWatcherResults.sh --from 02/23/2018_08:00:00 --to 02/23/2018_13:00:00 --resultdir /tmp/exawatcher_230218

Next step

cd /tmp/exawatcher_230218

bunzip2 the archive and un tar it

tar xvf ExaWatcher_exa31_2018-02-23_08_00_00_5h00m00s.tar


Change to the newly created directory and you find the following files



Start a Browser on the Server or copy the files to your notebook and start analyzing

open the "index.html" files

here the details



It is easy to answer questions like

  • What is the CPU usage on Friday between 09:00 an 14:00 o’clock?
  • Where do we have a CPU peek during the business hours?

No problem with „GetExaWatcherResults.sh“ and if needed go deeper with other tools for example „awr – report“ on the database itself or an „ash-report“

Try it. It’s very helpful :-)





Veröffentlicht unter Engineered Systems, Exadata | Verschlagwortet mit | Kommentar hinterlassen

Secure Erase of an Exadata System

What has to be done when the Lifecycle of an Exadata System comes to the end.

You need to do an secure erase of DB and Storage nodes.

By the way you can also secure erase the Switches and PDU, etc. but this is not described in this article.



and  My Oracle Support Doc ID 2180963.1

Steps to do

I use the method via bootable USB Stick

Download Boot Image via Patch 25470974

You need a complete list of all DB and Storage node names including all IP addresses of each ILOM Server

Tip: Before you start reset the password on each Server for example „welcome1“

Prepare USB Stick

I use my Mac book and do a „dd“ to copy the image to the USB Stick

dd if=image_diagnostics_12. of=/dev/disk2

Start a ILOM Web Console Login for the first Server

In parallel start a ILOM terminal console session to do a restart of the first Server

While the reset is running the Web  Console came up with the BIOS Splash screen. Please  enter


It takes a while and you see the boot menu

Select the USB Stick and press enter to start

the Boot Screen appears

Now it is time to check the output on the ILOM terminal console. You will see after a while a „login:“ prompt

Login as "root" password "sos1exadata" or "sos1Exadata"

If the password doesn’t work contact Oracle Support for help


Veröffentlicht unter Engineered Systems, Exadata, General, Oracle Storage Server | 1 Kommentar

Oracle Essential Support Tools im Exadata Umfeld

Mein DOAG Vortrag zum Thema

„Oracle Support wie gehe ich vor und welche Tools setze ich im Exadata Umfeld ein“



In der Zwischenzeit hat Oracle eine neue Version den Trace File Analyzer bereitgestellt die man dann auch einsetzen sollte. Einfach über die Doc-ID suchen.

TFA Collector – TFA with Database Support Tools Bundle (Doc ID 1513912.1)

Noch eine Anmerkung für alle die gerade PSU’s etc. eingespielt haben. Leider stellt Oracle Support nicht die oben genannte TFA Version mit dem letzten PSU zur Verfügung.  Die Installation des Trace File Analyzer muss separat durchgeführt werden.



Veröffentlicht unter Exadata, General, Oracle 12.1 | Verschlagwortet mit , | Kommentar hinterlassen

Exadata Flash Cache enabled for Write Back

During Tests for an Migration of a major customer application we saw in our AWR reports that most of the jobs are very write intensive. This was the point where we would like to test what happens when we change the Flash Cache Mode from Write Through to Write Back.

What are the main benefits of Write Back mode:

  • it improves the write intensive operations while writing to flash cache is faster than writing to normal Hard disks
  • on Exadata X3  and newer machines write performance can be improved up to 20X IOPS
  • The Write Back Flash Cache accelerates reads and writes for all workloads

First of all I take a look in Metalink and found a Doc ID 1500257.1 with more details.

What are the requirements?

since April 2017 it is default if the following conditions are full filled:

  • Grid and RDBMS Home
    • or higher
    • or higher
    • or higher


  • DATA diskgroup has HIGH redundancy

When should I use Write Back?

  • It makes sense if your application is write intensive
  • You find significant waits for „free buffer waits“
  • High IO times to check for write bottlenecks in AWR reports

What are the steps to enable Write Back?

We have the possibility to do it „offline“ so we stop the whole Grid & Rdbms Stack, but you can change it also in a Rolling manner.

  • Actual Flash Cache Mode

dcli -g cell_group -l root "cellcli -e list cell attributes flashcachemode"
cel04: WriteThrough
cel05: WriteThrough
cel06: WriteThrough
cel07: WriteThrough

  • Stop the whole Cluster

crsctl stop cluster -all -f

  • Check State of Flash Cache

name: cel04_FLASHCACHE
status: normal

name: cel05_FLASHCACHE
status: normal

name: cel06_FLASHCACHE
status: normal

name: cel07_FLASHCACHE
status: normal

  • Steps for the Change

These steps has to be done on every Cell Server here as an example

Drop the flash cache on that cell

CellCLI> drop flashcache;
Flash cache cel04_FLASHCACHE successfully dropped.

Shut down Cell service

CellCLI> alter cell shutdown services cellsrv;
Stopping CELLSRV services... The SHUTDOWN of CELLSRV services was successful.

Change Cell Flash Cache mode to Write Back

CellCLI> alter cell flashCacheMode=writeback;
Cell cel04 successfully altered 

Restart the Cell Service

CellCLI> alter cell startup services cellsrv;
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.

Recreate the Flash Cache

CellCLI> create flashcache all;
Flash cache cel04_FLASHCACHE successfully created

Finally check the State on all Cell Server dcli -g cell_group -l root "cellcli -e list cell attributes flashcachemode"
cel04: WriteBack
cel05: WriteBack
cel06: WriteBack
cel07: WriteBack

So the first step was done and now the tests can go on.

In a few weeks I will give a feedback what are the real improvements so stay tuned.





Veröffentlicht unter Exadata, General, Oracle Storage Server | Verschlagwortet mit , | Kommentar hinterlassen

Important Change for Grid 12.2 Upgrade and MGMTDB

I just did an Grid Infrastructure Update to Oracle 12.2 with the latest PSU Aug 2017.

I did my preparation with the help of the Doc ID 2111010.1

„12.2 Grid Infrastructure and Database Upgrade steps for Exadata Database Machine running and later on Oracle Linux“

In the Post-upgrade Steps was written that the Management DB should be deconfigured

  • Post-upgrade Steps

      • Deconfigure MGMTDB


The deconfigure is obsolete and the MGMTDB is now part of the Update. 

Here the original text from the Doc ID.

September 28,2017
  • MGMTDB will now be part of upgrade, the flags to reove and deconfigure are removed.



For me it looks like that the revision of the document is not yet finished

So if you plan a Grid Upgrade in the near future read the document very carefully and if needed open an SR

Veröffentlicht unter Exadata, General, Grid Infrastructure, Oracle 12.2 | 1 Kommentar

Exadata 18 available and installed

Yes, that great news for all Exadata guys.

The documentation is also updated. So please check first the Maintenance Guide.

Oracle Exadata Database Machine Maintenance Guide.

Have fun :-)

In the meantime I did an Upgrade to Exadata 18.1 Software. It was very smooth and runs without any problems




Veröffentlicht unter Exadata, General | Verschlagwortet mit | Kommentar hinterlassen

Flashback in a Data Guard Environment but what are the steps

The Setup of an Data Guard environment with Physical Standby via Data Guard Broker was done including an Switchover. Everything fine and works as expected. (Oracle Release is and PSU Jan 2017).

The Dev Team asked for a Flashback Database if possible a few days ago. Okay no problem. But what do I have to do exactly.

What are the steps to do? ( The requested timestamp for Flashback on the Primary DB is „07.09.2017 17:15“ )

Stop Apply on the Standby

DGMGRL> connect sys@gtdb10_11_DG
Connected as SYSDBA.
DGMGRL> edit database gtdb10_11 set state='APPLY-OFF';

To get the Primary easily after the Flashback in Sync with the Standby you need to flashback the Standby a short time before you flashback the Primary.

Primary =   „07.09.2017 17:15“

Standby =  „07.09.2017 17:00“

sqlplus / as sysdba

flashback standby DATABASE TO TIMESTAMP(to_date('07.09.2017 17:00','dd.mm.yyyy hh24:mi'))

Now to the Primary Database and I describe the easiest way

srvctl stop db -d gtdb10_12
sqlplus / as sysdba
SQL*Plus: Release Production on Mon Sep 11 13:12:32 2017
Copyright (c) 1982, 2014, Oracle. All rights reserved.
Connected to an idle instance.
SQL> startup mount

SQL> flashback DATABASE TO TIMESTAMP(to_date( '07.09.2017 17:15','dd.mm.yyyy hh24:mi'))

After Flashback the Primary open the DB with resetlogs

SQL> alter database open resetlogs;

# Stop the mounted database
SQL> shutdown immediate

# Restart the Primary
srvctl start db -d gtdb10_12

Flashback done

# Now I restarted the Standby is normally not needed
srvctl stop db -d gtdb10_11
srvctl start db -d gtdb10_11

# Start the Apply on the Standby Database
DGMGRL> connect sys@gtdb10_11_DG
Connected as SYSDBA.
DGMGRL> edit database gtdb10_11 set state='APPLY-ON';

The Dev Team can go on :-)


Important is that you have to start with Flashback on the Standby Site not on the Primary Site. Otherwise you need to recreate the whole Standby Database and that could be very time intensiv by having the risk that you work without a Physical Database during the Setup. Also the Switchover has to be redone because whole Standby Database was newly created.

Restore Points

If you work with „restore points“ than keep also in mind that you create the first restore point on the Standby Database and then the Restore Point on the Primary Site.








Veröffentlicht unter General, Oracle 12.1 | Verschlagwortet mit | 1 Kommentar

Problems with Adaptive Features in What’s to do?

The Adaptive Features is divided in two parts

  • Adaptive Plans
    • the possibility to change plans during execution
  • Adaptive Statistics
    • the statistics allow plans to be built from the results of previous executions

Version 12.1 

In 12.1 both components  will be managed by the parameter „optimizer_features_enable“ and is per default on „TRUE“.

Version 12.2

In 12.2 the parameter „optimizer_features_enable“ obsolete. The adaptive feature is now controlled by two parameters

  • optimizer_adaptive_statistics default „FALSE“
  • optimizer_adaptive_plans default „TRUE“

What are the recommendation for if you do a upgrade. You should directly adopt the 12.2 default  by applying the following Patches:

  • Patch 22652097 splits the parameter optimizer_adaptive_features into two, as above, and disables adaptive statistics.
  • Patch 21171382 disables the automatic creation of extended statistics unless the optimizer preference AUTO_STAT_EXTENSIONS is set to ON.


If you Upgrade to 12.1..2 set directly the Default for Oracle 12.2.

Yes, the next Upgrade to Release 12.2 came sooner or later.




Veröffentlicht unter General, Oracle 12.1, Oracle 12.2, Oracle Database | Verschlagwortet mit , | Kommentar hinterlassen

Oracle RU is available

Today Oracle released a very large „monster“ Upgrade. This July 2017 Update includes the first time the new RU „Release Upgrade“ and RUR „Release Update Revision“ Patches.

To be up to date read the following My Oracle Support Doc-ID’s

Actually I can’t find a FAQ but I think this will come in a few weeks.

Stay tuned :-)

Veröffentlicht unter General, Oracle 12.2, Oracle Database | Verschlagwortet mit , , | 1 Kommentar

Oracle ACE

Vor zwei Jahren als Oracle ACE Associate gestartet, wurde ich nun zum Oracle ACE „befördert“.  Vielen Dank

Two years ago I started as Oracle ACE Associate and now I was „promoted“ to the Oracle ACE status. Many Thanks







Veröffentlicht unter General | Verschlagwortet mit , | Kommentar hinterlassen