Sunday, November 05, 2006

To use DRM or not to use DRM

10g Real Application Clusters introduced a concept of resource remastering via Dynamic Resource Mastering (DRM). With DRM a resource can be re-mastered on another node in the cluster if it is found that the cache resource is accessed more frequently from that node.

In 10G R1 this was file based whereas the 10G R2 it is object based.
In 10G R1 due to a few bugs many related to high CPU usage during the DRM freeze window most customers disabled DRM by setting the following parameters
_gc_affinity_time=0
_gc_undo_affinity=FALSE

_gc_affinity_time defines the frequency in minutes to check if remastering
is needed.
_gc_affinity_limit defines the number of times a node must access an object
for it to be a DRM candidate
_gc_affinity_minimum defines the minimum number of times an object is accessed per minute before affinity kicks in

The performance problems may manifest themselves in terms of a DRM related wait event like 'gcs drm freeze in enter server mode'

In 10G R2 this feature appears to be more stable.

You can also manually remaster an object on a different node which
is different from the node on which the object is currrently mastered as shown below

SQL> select object_id,current_master, previous_master ,remaster_cnt from V$GCSPFMASTER_INFO where object_id = 144615

OBJECT_ID CURRENT_MASTER PREVIOUS_MASTER REMASTER_CNT
---------- -------------- --------------- ------------
144615 0 2 0


The object 144615 is currently mastered on node 0.
To remaster the object onto node 2 connect to node 2 as sysdba


Go to instance 2
NODE2> oradebug setmypid
Statement processed.
NODE2> oradebug lkdebug -m pkey 144615
Statement processed.

NODE2> select object_id,current_master, previous_master ,remaster_cnt from V$GCSPFMASTER_INFO where object_id = 144615

OBJECT_ID CURRENT_MASTER PREVIOUS_MASTER REMASTER_CNT
---------- -------------- --------------- ------------
144615 2 0 0

Note: In V$GCSPFMASTER_INFO you will also see resources with object ids in the 4Gb range (e.g. 4294950913)
These are for undo segments.

To dissolve remastering of this object on this instance
SQL> oradebug lkdebug -m dpkey 144615
Statement processed.

SQL> select object_id,current_master, previous_master ,remaster_cnt from V$GCSPFMASTER_INFO where object_id = 144615;
no rows selected


The remaster_cnt appears to be 0 for all objects. I have got Oracle
to log bug 5649377 on this issue.

SQL> select distinct remaster_cnt from V$GCSPFMASTER_INFO ;

REMASTER_CNT
------------
0


DRM statistics are available in X$KJDRMAFNSTATS

SQL> select * from X$KJDRMAFNSTATS
2 /

ADDR INDX INST_ID DRMS AVG_DRM_TIME OBJECTS_PER_DRM QUISCE_T FRZ_T CLEANUP_T REPLAY_T FIXWRITE_T SYNC_T
-------- ---------- ---------- ---------- ------------ --------------- ---------- ---------- ---------- ---------- ---------- ----------
RES_CLEANED REPLAY_S REPLAY_R MY_OBJECTS
----------- ---------- ---------- ----------
200089CC 0 1 32 214 1 3 14 0 0 0 99
0 2441 6952 30

The column MY_OBJECTS denotes the number of objects mastered on that node.
This should match with the following

SQL> select count(*) from V$GCSPFMASTER_INFO where current_master=0
2 /

COUNT(*)
----------
30

4 comments:

Alex Gorbachev said...

oradebug staff is quite cool. thanks.
I hope I'll find some time/project to play with it.

Anonymous said...

Hmmm ... having been bitten by this DRM issue, in 10.2.0.2, we had to apply patch 5071492 to effectively resolve the problem.

It caused so much pain.

PRASAD said...

How to disable DRM on 10R1.

alter system set _gc_affinity_time=0 scope=spfile
*
ERROR at line 1:
ORA-00911: invalid character

Thanks
-Prasad

Fairlie Rego said...

_gc_affinity_time needs to be put in double quotes

Thanks
Fairlie