We patch our cluster with the a PSU but we did not recognize that the cluster was afterwards in rolling patch state. A few days later we had a defect disk in one cell server. The disk was changed and we got an error message.
ASM alert.log SQL> /* Exadata Auto Mgmt: DROP ASM Disk */ alter diskgroup DBFS_DG drop; disk DBFS_DG_CD_08_S1R8CD5; force rebalance nowait ORA-15032: not all alterations performed ORA-15137: The ASM cluster is in rolling patch state.
So what can be done to get the disk group rebalanced. First we stop the rolling patch mode on the ASM Instance which could be done with the following command
ALTER SYSTEM STOP ROLLING PATCH;
Looking to the Cell Server we saw that the new disk was now known by the Systems but the Grid disks are offline.
CellCLI> list griddisk attributes name,asmmodestatus,asmdeactivationoutcome DATA_CD_08_s1r8cd5 OFFLINE Yes DBFS_DG_CD_08_s1r8cd5 OFFLINE Yes RECO_CD_08_s1r8cd5 OFFLINE Yes So we saw that the Grid disks are offline CellCLI> alter griddisk DATA_CD_08_s1r8cd5 inactive; GridDisk DATA_DG_CD_08_s1r8cd5 successfully altered CellCLI> alter griddisk DBFS_DG_CD_08_s1r8cd5 inactive; GridDisk DBFS_DG_CD_08_s1r8cd5 successfully altered CellCLI> alter griddisk RECO_DG_CD_08_s1r8cd5 active; GridDisk RECO_DG_CD_08_s1r8cd5 successfully altered
While doing these steps the rebalance on the disk groups starts automatically.
ASM alert.log ... alter diskgroup DBFS_DG drop disk DBFS_DG_CD_08_S1R8CD5 force rebalance nowait; ... After a few hours the disk groups are in “sync”.
We fixed this problem on the ASM instance but on the Grid Infrastructure level we have a different patchlevel if we do an “opatch lsinv”. To fix this problem I write a second blog article. stay tuned