RMAN Active Duplicate Advantages

Introduction

We are planning to start upgrading some of our enterprise Oracle databases from 10g to 11g and I am evaluating the new features of Oracle 11g that our organization can be benefited from. A 11g new feature that has caught my attention is the 'ACTIVE Database Duplicate' option in Oracle RMAN. This feature allows a database to be duplicated directly from its live source database instead of using its backup.

Advantages

Instant Database Clones possible (On Demand Cloning)

If one requires a database clone, all one has to do is to run the active database duplicate command after specifying the source and destination databases. I call this Cloning onDemand ! This helps organizations create test database environments in a rapid and simpler fashion.

EMC TimeFinder BCV sync/split is not necessary

Consider an organization relying on BCV sync/split technology to perform database clones. In this case one must take the help of a system administrator to perform the BCV sync/split operation. A database is put in hotbackup mode and the BCV split process is performed. Once the operation is completed, the database is taken out of backup mode. This is all additional work and the Active Database Duplication eliminates all these steps as it performs a direct copy.

Staging ASM Diskgroup is not required

Consider the case when ASM (Automatic Storage Management) is used as the underlying disk management technology for an Oracle database. An ASM-2-ASM database clone always requires a consistent BCV sync/split operation and this during process the ASM disk headers are copied to the destination host. This requires staging (intermediate) ASM Diskgroups on the destination host with the same name as that of the source database's ASM diskgroups. After the consistent BCV sync/split is completed, data is again copied from the staging ASM diskgroups to the clone database's diskgroups(DEV/TEST diskgroup names). Active Database Duplication eliminates the need for staging ASM diskgroups, thus saving a lot of diskspace (equal to the total size of the database being cloned)

No additional space required

In the traditional backup-based database duplication, one has to make the backups accessible to the destination host before the duplication is started. For tape backups, this equates to moving the tape to a drive attached to the destination host or using a network-accessible tape server. For disk backups, this equates to copying the backup to the destination host (additional space). Active Database Duplication does not require backups and hence the additional disk space requirement is no longer needed on the destination host.

Restrictions

Incomplete recovery NOT possible (no SET UNTIL)

Active Database Duplication cannot be used to duplicate a database until a point in time. One has to rely on the backup-based database duplication method if one wants to create a duplicate database until a specific point in time ( a specific time, log sequence or SCN). Furthermore, active duplicate copies data only until the last archived log file in the source database. The contents of the online redologs will not be copied to the duplicate database.

FRM-40734:Internal Error:PL/SQL error occurred

Introduction 

 

It was one of those days when things get delayed and you go past the deadline. Last week I faced a similar situation when an Oracle R12.1 (E-Business Suite 12.1) clone was not completed successfully.  

 

Problem 

 

All the cloning tasks were completed, however, during testing of the cloned instance, I discovered that Oracle Forms was having a problem.

1. Launch any form ( For e.g Login as SYSADMIN=> System Administrator => Concurrent => Requests )
2. The forms window opens, but it goes into an hour glass mode and after a few minutes it displays the following error message at the bottom of the form

FRM-40734:Internal Error:PL/SQL error occurred  

 

Troubleshooting 

 

I searched for the occurrence of the above error in all the logfiles in the $INST_TOP/logs/ora/10.1.3/opmn directory including the forms_default_group_1 subdirectory. But everything seemed normal in these logfiles with no reference to the above error code and message. I also performed a routine check of the $INST_TOP/logs/ora/10.1.3/j2ee/oacore directory, but no clue at all. This seemed to be one of those errors when the logfiles do not give much information for troubleshooting purposes. 

 

Solution 

 

Because of the several steps involved in the cloning process, I was under the impression that I could have done a mistake or missed a step or two. Hence I executed all the cloning steps again. But the issue refused to go away. I used the adadmin utility to run the 'Compile APPS schema', 'Generate form files', 'Compile menu information' and the 'Compile flexfields' programs. But the issue still refused to go away. Finally, we ran the 'Generate product JAR files' adadmin program which helped fix the problem.  

 

Conclusion 

 

So you now know the trick when you face the same symptoms that I faced above !!

Codeline B Patch readme containing Codeline A prereqs and Codeline C postreqs

Introduction 

 

Oracle has introduced the concept of "Codelevels" and "Codelines" in the new E-Business Suite Release 12. The first base release in R12 is 12.0 (also called Point Release). All the patches introduced on top of this point release are said to be on Codeline A. 12.0 is also called the Codelevel A. The next Codelevel in R12 is Codelevel 12.1 and its Codeline is B. Similary 12.2 and Codeline C. Compare this with the E-Business Suite 11i version when Codelevels were instead called as maintenance packs (11.5.8, 11.5.9, 11.5.10 etc.) Furthermore, Oracle says in one of its support notes, that it is "not possible to apply a Codeline A patch on an E-Business Suite system that is already on Codeline B (12.1 point release)". So that should mean that Codeline A patches are not compatible to apply on a Codeline B system.  

 

Problem 

 

Our E-Business Suite system is on 12.1 point release. The other day I was analyzing a patch xxyyzz.R12.AR.B to be applied for fixing an "Accounts Recievables" issue and I noticed a few discrepancies in the patch's README file. 

 

 Issue #1 (Prereqs) 

 

The README mentioned several prerequisite patches to be applied to be applied prior to applying the actual patch. One of the prereqs mentioned as a Codeline A patch aabbcc.R12.AR.A. I checked with the Oracle Support team and they mentioned that its a "typo" in the README and it was immediately taken care of. 

 

 Issue #2 (Postreqs) 

 

The README mentioned several postrequisite patches as well and one of them was a Codeline C patch ! ppqqrr.R12.AR.C. This time the answer from Oracle Support was not an expected one. As per Oracle Support, some of the products on 12.1 release have been using the C codelevel due to an intermediate release of some vertical products falling in the same product line. So in 12.1, one can see some products with the C label too.  

 

Conclusion 

 

Please seek a clarification from Oracle Support (in the form of a Service Request) if you find a patch that contains prereqs and postreqs from different Codelines. Most of the times it could be a typo in the patch README and sometimes it may be a valid case !!!

ASM Disk Controller numbers can get changed if the SAN Switch is replaced

Introduction

We have setup our Oracle E-Business Suite database to be highly available by using Oracle 10g RAC technology using the ASM feature. We use EMC Symmetrix for storage and a Brocade device for the SAN switch. Our SAN fabric was recently upgraded and among many changes, the legacy Brocade SAN switches were replaced by Cisco switches.

Problem

Due to this change, we rebooted both the Oracle RAC database nodes. But the database would not start because the ASM diskgroup could not be mounted. We have hardcoded the asm_diskstring parameter to a specific set of disk controller numbers. While mounting a diskgroup, the ASM instance will discover only those devices that match the asm_diskgroup parameter. Upon further investigation, we found out that the disk controller numbers got changed after the new switch installation.

Note the changed disk controller numbers in the below image



Solution

We use EMC Symmetrix as our SAN storage solution. During the initial ASM conversion project, we managed to save the syminq numbers of all our ASM devices. Because the syminq numbers do NOT change even upon a switch replacement, we could use the same syminq numbers to find out the new disk controller numbers. We then updated the asm_diskgroup initialization parameter with these new controller numbers and then the ASM diskgroup was successfully mounted and the database started.

How to avoid username conflict in OID during AD synchronization

Introduction

Our master source of truth for authenticating Single Sign On (SSO) users for access to the E-Business Suite is the Active Directory (AD). We periodically syncrhonize data from AD to the Oracle Internet Directory (OID) using the syncrhonization profiles.

Because Oracle SSO can work with only OID, we cannot afford any failures/mismatches between AD and OID and hence we do not let the syncrhonization to continue if an error is reported. The 'Continue after error' is set to FALSE.

One common problem in a directory is managing usernames for people with identical names. For example, an organization can have two employees with the same name viz. John Smith and John Smith respectively. The AD administrator creates these accounts as JSMITH and JSMITH1 respectively because the username has to be unique across the organization. During initial propagation to OID, these accounts will have two seperate entries in OID as the same JSMITH and JSMITH1 respectively. When JSMITH leaves the organization, the other John Smith can request for a change in his username from JSMITH1 to JSMITH (JSMITH looks good compared to JSMITH1).

Problem

The update of JSMITH1 to JSMITH generates a new change number in AD and the next syncrhonization cycle will attempt to make the corresponding change in OID too. We had such an update in AD yesterday and OID failed to process the change. The following is the error that is reported in the syncrhonization profile logfile.

Exception Doing ModRDN operation : javax.naming.NameAlreadyBoundException: [LDAP: error code 68 - Entry Already Exists]; remaining name 'cn=jsmith1,ou=us,dc=mycompany,dc=com'
Ignore modrdn.

Solution

At the time of processing the update, OID had the following two entries

'cn=jsmith1,ou=us,dc=mycompany,dc=com'
'cn=jsmith,ou=us,dc=mycompany,dc=com'

AD had only one entry JSMITH (after the username change by the administrator).

'cn=jsmith,ou=us,dc=mycompany,dc=com'

Because OID has an jsmith entry already, the update of JSMITH1 to JSMITH fails with the [LDAP: error code 68 - Entry Already Exists] error.

I solved the problem by deleting both the entries jsmith and jsmith1 in OID and making a fake update in AD (removed the telephone number of jsmith). The next syncrhonization job picked up the telephone number change in AD and created a new jsmith entry in OID. Once the OID processing has become successful, I corrected the telephone number in AD.

Unable to login to R12 after SSO integration

Introduction

I am now working on the E-Business Suite Release 12 upgrade project. The login page broke after integrating the R12 development system with Oracle Single Sign-On. We recieved 'HTTP 500 Internal Server Error'. Furthermore, the local login page (/OA_HTML/AppsLocalLogin.jsp) too was not getting rendered.

Problem

This certainly seemed to a java servlet initialization problem in the Oracle OC4J. I noticed the following errors in the $INST_TOP/logs/ora/10.1.3/j2ee/oacore/oacore_default_group_1/application.log

Error initializing servletjava.lang.NoClassDefFoundError: Could not initialize class oracle.apps.fnd.profiles.Profiles

html: chain failed javax.servlet.ServletException at com.evermind[Oracle Containers for J2EE 10g

I could not a direct answer after searching the Oracle Support website. I was able to better understand the problem after reviewing the $INST_TOP/logs/ora/10.1.3/opmn/oacore_default_group_1/oacorestd.err logfile.

Exception in static block of jtf.cache.CacheManager. Stack trace is: oracle.apps.jtf.base.resources.FrameworkException: IAS Cache initialization failed.

Solution

The problem has something to do with the Java Cache. I went ahead and disabled the cache mechanism by setting the parameter LONG_RUNNING_JVM to false in the $INST_TOP/ora/10.1.3/j2ee/oacore/config/oc4j.properties and this solved the problem.I will update this article once I know the root cause why the Cache failed to initialize after the SSO integration.

Re-Registering the Oracle E-Business Suite after an SSO database RAC conversion

Introduction

After converting the Oracle SSO database to RAC for High Availability, one has to re-register the provisioning applications such as the E-Business Suite to let them know about the new database configuration. In this case, to update the provisioning applications with the new RAC TNS string.  

Problem

 

One of our provisioning applications is the Oracle E-Business Suite. The provisioning application has to be deregistered first with the txkrun.pl -deregister option. We recieved the following error during the deregistration process.

txkrun.pl -script=SetSSOReg -deregister=Yes

txkSetSSOReg_Fri_Feb_27_15_03_19_2009.log FUNCTION: TXK::advconfig::SSO::validateParams [ Level 1 ]ERRORMSG: Invalid ORASSO database user credentials.Connecting to ORASSO schema using dbhost.mycompany.com:1521:OIDDB failed.  

Solution

The ORASSO password was correct. So the above error has nothing to do with the password. Upon further investigation, I found out the script is unable to establish the SSO database connection. Because the database has been converted to RAC, the current SID OIDDB is not applicable anymore. During the RAC conversion, a new SID has been generated for each of the two instances in the RAC viz. OIDDB1 and OIDDB2 respectively. The service name is still the same OIDDB, but Oracle is trying to establish a connection using OIDDB as a SID, resulting in the above error.

More often that not, Oracle utility scripts show the input arguments so we can choose the right argument for the database connection string and pass it to the script. However, txkrun.pl does not seem to be good at it. txkrun.pl -help too does not provide the input argument list. The only way to find the txkrun.pl input arguments is to check its failed logfile in $COMMON_TOP/rgf/$CONTEXT_NAME/sso/txkSetSSOReg*.log file ( infraconnstr )  

txkrun.pl -script=SetSSOReg -deregister=Yes -infraconnstr="(Description =(ADDRESS_LIST =(LOAD_BALANCE = yes)(ADDRESS = (PROTOCOL = TCP)(HOST = node1-vip)(PORT = 1521))(ADDRESS = (PROTOCOL = TCP)(HOST = node2-vip)(PORT = 1521)))(CONNECT_DATA =(SERVICE_NAME = OIDDB)))"  

Conclusion

I hope txkrun.pl provides a better way to find its input argument list in the future. This can be an enhancement request for Oracle. Moreover, the script should prompt for the service name instead of the database SID currently (Enter the Oracle iAS Infrastructure database SID ? ) if one does not intput the infraconnstr argument. Service name is a much better way of referencing a database instead of a SID, because the former remains constant across RAC conversions, not the latter.