Thursday, June 12, 2008

RMAN debug

RMAN automates Oracle database's backup and recovery processes. If you would like to know what happens inside an RMAN command,(RMAN Internals), then try the debug option. Also, when you have a case opened on any RMAN issue, the debug logfile can help a support analyst quickly locate the source of your RMAN problem.

Its very easy to turn on the debug feature.

/home/oracle/backup_testing> rman log=output.log trace=debug_info.log
RMAN> debug on
RMAN> RMAN Command 1;
RMAN> RMAN Command 2;
....
....
....
RMAN> exit

The debug_info.log contains all the internal details. BTW, you will not find information about this feature in the Oracle Documentation.

Tuesday, June 3, 2008

central tnsnames.ora

Introduction

For hassle-free management of enterprise wide databases connection information, the connect descriptors can be stored in a centralized repository. There are several solutions to implement a centralized repository ranging from the basic tnsnames.ora that is stored on a NFS to the advanced LDAP complaint directory services implementation through OID (Oracle Internet Directory).

Problem

Several of our Oracle developers and end users rely on 3rd party tools like TOAD, SQL*Navigator to communicate with Oracle databases. We have put the tnsnames.ora containing all the oracle databases connection information for these utilities on a shared network drive in windows. This solves the problem for clients using Windows. However our system administrators do not allow NFS access on Unix servers. Due to this restriction, it has become a hassle to maintain a separate tnsnames.ora in every Unix server that hosts oracle installations(client or server). If we create a new database or change an existing database's connection settings, we have to update the tnsnames.ora on all other Unix servers with this new connection information.

Solution
While discussing in Oracle forums, i stumbled upon a specialized tool tnsManager built by Andrew Barry that acts as a simple LDAP compliant server to serve just TNS information. It is the kind of solution that i was looking for solving the problem. It is quite easy to use and has a very small footprint. The 'HOW TO' guide is here.

Conclusion
We will be implementing Oracle Identity Management (that will contain OID) by the end of the year that provides a much more sophisticated mechanism for organizing and managing centralized directory services like applications login accounts, centrally managing database connection information etc. The tnsManager utility can be a good alternative which will satisfy our basic needs for storing just the database connection information in a central tnsnames.ora until the full blown Identify Management project is implemented. Best of all its just one executable that Andrew supplies and it takes 5 minutes to install and get it going ! As far as a DBA is concerned, all he or she has to do is edit the central tnsnames.ora that is read by the tnsManager utility whenever an existing database connection settings are changed or a new database is added in the enterprise. No extra configuration necessary.

Thursday, May 29, 2008

RMAN DUPLICATE - automatic temp files recreation

As part of our 10g upgrade project, we are testing our existing RMAN scripts, especially the DUPLICATE command that we use to clone databases. In one my previous posts, i have already outlined the procedure to clone a database using the DUPLICATE command.

It works fine in 10g too; however, an added benefit in this release is that it recreates the temp files automatically. The temp files for the destination database will be created in the directory specified by the DB_FILE_NAME_CONVERT parameter with file names and file sizes similar to the source database. This eliminates the manual recreation of the temp files after every database clone performed with the DUPLICATE command.

With this new feature in 10g (specifically it is an Oracle 10g R2 new feature) the DUPLICATE command eliminates one more manual step performed during the database cloning process.

Monday, May 19, 2008

How to become an Oracle DBA

I was on a vacation for three weeks in April and hence could not blog since more than a month. We are also currently busy upgrading our E-Business Suite 11.5.10.2 on 9.2.0.6 database veresion to 10g R2 version and also completing Linux Migrations of the rest of the Oracle systems that we have here. The upgrade is still in the development phase and we are planning to golive in July this year. We are all very excited about the upgrade especially because of all the goodies and new features that come with 10g release. So far the upgrade looks smooth that but i will post my experiences with issues (if any) once we are ready to GOLIVE.

Yesterday i got an email from a regular visitor of my blog who happens to be a fresh computer science engineering major and he wanted to know how can be become a good Oracle DBA with no prior experience.

One has to start with a Professional training course in Database Administration. I recommend DBA University courses. Learning by oneself is also a good idea especially if one is a computer science major already because one picks up RDBMS concepts in college usually. I recommend getting trained with Oracle Real Applications Clusters (RAC) directly. A useful book that i came across is the 'Personal Real Application Clusters' . This book is from Rampant Press (well known for their specialized Oracle concentration). Though i did not go through the book, the topics and chapter look quite interesting.

It will be a very good experience setting up your own personal RAC system. The industry is converting more and more Oracle single instances into multi node clusters (Real Application Clusters) for high availability and scalability. The favorite platform for RAC is Linux and of course it will be become the platform of the future. The book seems to be a good handbook for building your own Oracle RAC on Linux Platform easily and in a cost effective manner. Increased confidence and a sense of accomplishment will be the result of this exercise.

Wednesday, April 2, 2008

Oracle DBA interview questions

Introduction
Last night i got a call from my friend who wants to "job-hop" to get a 200% pay raise and he wanted answers to some popular interview questions.

There is no one line objective answer to these questions. The answers get better as you gain more experience in the art called database administration. If i were asked these questions below in an interview, my answers will be like the ones in this post. For simplicity, i will assume the database in question is a 9i or a 10g database and also an OLTP one. Some of the answers may not be applicable or may have got changed in other versions.
1) Suppose if i am having RAM of 30GB ,What would be the size for SGA ? what is the ideal calculation for configuring sga?
A) I will go with the assumption that you already have a database available of a particular size and now want to use newly provided 30GB RAM to configure SGA for this existing database. I will go with 1% of the DB size for SGA. So if the database size is 200 GB, the SGA i will recommend is 2GB. Similarly if the database size is 1000GB, i will go with 10 GB SGA. I will follow this formula up to 1000GB (1 Terabyte database). It is not directly proportional afterwards. For very large databases (more than 1 Terabyte), SGA sizing has to be done based on the demand and requirement after getting valuable inputs from performance reports like Statspack reports etc. If i had a 10 TB database i would stop at 10GB SGA and check performance reports to take a suitable decision whether to increase the SGA any further. This formula is purely based on my experience. There is no hard and fast rule. You are always free to add more SGA if you have the physical memory. But read the next question below to know why it may not be of good benefit if the existing SGA is already well configured.

2) If i increase/decrease the sga size what would be the effect to the database.
A) For majority of the cases there may not be any effect. Here is why. You should take statspack or other performance reports and find if the hit ratios are above 98% (I like 99%+ hit ratios). If the LIBRARY CACHE HIT RATIO and BUFFER CACHE HIT RATIO are below 95%, its better to increase the SGA. However if there are full table scans on big tables in your queries, any amount of increase of SGA (DB_CACHE_SIZE) will be futile because the LRU algorithm puts data accessed from table scan in the "Least Recently Used" category thereby causing them to get flushed out of the DB CACHE more frequently than others like those accessed through an index path. Similarly for a query which executes frequently and using literals instead of bind variables. They fill up the SHARED POOL immediately. I have seen databases getting ORA-01403 (Shared Pool out of memory) with a 5 GB SHARED_POOL_SIZE and i have also seen databases running fine with just 80MB SHARED_POOL_SIZE ! Statspack reports are going to be your bible in a 9i database, though you may rely on other components like the ADDM in 10g. In short, i would not increase the SGA unless i go through the performance reports.

3) Say my select query is going on,but some how my db buffer cache got full,so now what will happen to the query, under this scenario from where it will read the data.
A) It will read data from disk. There will not be any failure. You can see slow response (usually only the first time) from the database because it will read data from the disk. But the buffer cache will get flushed soon, because of the LRU algorithm and soon your select query will have free space in the buffer cache (unless your select query is the culprit here doing a lot of full table scans) .
4) Under what condition should i need to increase the sga max size
A) SGA_MAX_SIZE is a "nice to have" init.ora parameter. If you want to experiment with different SGA sizes while having the database up and running, you should configure SGA_MAX_SIZE. This parameter allows you to change individual SGA component sizes like DB_CACHE_SIZE, SHARED_POOL_SIZE dynamically while your database is online. You can increase these SGA components during peak load for e.g when there is a quarter end or month end reporting job that accesses a lot of data.
5) Say a query is not performing well today,but that query was running well so far,some how today it is taking a lot much time ,as a DBA what basic strategy is needed to check the performance of the query ?

A) First get the current sql of the query. You can join v$session and v$sqltext dynamic performance views based on the sql address and get the current SQL that is running (if its active). Every sql statement gets a unique address in the SHARED_POOL_SIZE (think it like an unique identifier throughout its existence in the SHARED POOL) or else you can get it from GUI tools like OEM, SQL Navigator or Embacardo tools.Then it depends why all of a sudden your query is having a higher response time today. May be there are other jobs currently running that are taking lot of resources, or there could have been a code change yesterday night by a database developer or there could have been a sudden data load into the underlying tables since last night. There are several possibilities. I will try to check the num_rows in dba_tables and the actual number of rows using a count(*) and then decide if i want to analyze the tables. Taking a trace (level 12) is a good place to start to get to the bottom of the problem. There could also be a I/O problem in the system level. So i will also check with system administrators.
6) Say like i have executed delete from a table( it containing 5 records only),but it is not processing the delete. I check and see there are some locks in the database. How much time do i need to wait for my delete operation to succeed ? If Oracle does not release the locks, do i have to bounce the database to resolve the problem ?

A) You should find out what kind of lock is there. If its an enqueue lock you have to look at the other offending session that is blocking this session. Enqueue locks are usually a result of bad code. To let the delete statement pass through, I can call the user who is running the offending session and then remove the session from the database. However the root cause lies in the code design. In future the same delete statement may have the problem. Therefore fixing the code will be a long term fix for enqueue kind of locks.
7) When ever we run statspack,we see some top 5 wait events like "log file sync" and "db file scattered read",when these events occur and how to resolve them ?

A) I will not worry about familiar events like "db sequential read" or "sql *net message from client" etc. However i will be concerned if the events are "log file sync", "enqueue" or "file scattered read" etc. if there is huge transactional activity in the database, you should get a trend of your expected log switches per day and if the latest statspack report has this event in the top, then you should find out if any long running DML jobs were running that is causing too many waits on "log file sync". Some of the reasons could be a disk I/0 problem that is causing slow log file syncs or there may be too frequent commits in your long running jobs that is causing frequent log buffer flushes to the online redo logs. DB FILE SCATTERED READ is a scary wait event. If its in the top 5, then that means user sessions are waiting frequently on data accessed from disk. This also indicates that the database is undergoing a lot of full table scans or index fast full scans which will slow down the database.

Friday, March 14, 2008

Oracle Database is the skill with the biggest demand

Since the last few days i am only hearing discouraging news about the US economic slowdown and a possible recession in the coming days , but there is one news item that will bring a smile to all the Oracle DBAs !

According to The Dice Report, the latest snapshot on the technology job market published by dice.com for the month of March 2008, Oracle Database is the skill thats the most in demand. This is certainly heartening especially for the Oracle DBA community.

Click here for more details....

Oracle Business Intelligence (Discoverer 10g) Linux Migration

Introduction

As part of the ongoing Linux Migration project, we successfully migrated the Oracle Business Intelligence (Discoverer 10g) from HPUX to Linux last weekend. The Discoverer migration is straightforward for the most part and many steps can be done well in advance to avoid increased system downtime during the cutover. Most of the procedure involves a new linux installation and configuration and only a part of the procedure involves “actual migration” of files from the existing source HPUX platform to the target linux platform. Below are the high level steps that we followed.
Advance Steps
  1. Our existing version on HPUX is 10.1.2.0.2. We downloaded the same Linux x86 version from the OTN site. Oracle Discoverer 10g is also called Oracle Business Intelligence SE (Standard Edition).
  2. Install the Linux 10.1.2.0.2 software on the target linux host.
  3. Create $ORACLE_HOME/discoverer/secure directory
  4. To provide transparency and a much more friendly user URL to end users, we got a DNS alias for the linux host. After the installation, $ORACLE_HOME/chgip/scripts/chgiphost.sh -mid is run to update the discoverer installation's configuration files with the DNS alias (Compare it with adautocfg.sh in Oracle E-Business Suite 11i.
Cutover Steps (Downtime)
  1. Shutdown the discoverer processes on the existing HPUX platform.
  2. Migrate the discoverer preferences from the source HPUX to the target Linux platform. Discoverer preferences are settings that define the Discoverer environment and control the behavior of Discoverer Plus and Discoverer Viewer. Whenever an individual user changes preferences in discoverer, thet are stored in a hidden file $ORACLE_HOME/discoverer/.reg_key.dc. This file contains values in a platform specific format and therefore needs to be converted to the target platform format.
  • Backup the target linux platform’s $ORACLE_HOME/discoverer/.reg_key.dc
  • Copy the $ORACLE_HOME/discoverer/.reg_key.dc from source platform to Linux as hpux_reg_key.dc
  • On Linux , cd $ORACLE_HOME/discoverer/util and run the below conversion script
  • perl convertreg.pl hpux_reg_key.dc $ORACLE_HOME/discoverer/.reg_key.dc

3. Apply any one-off patches on Linux that have been applied in source platform

4. Migrate tnsnames.ora and dbc files from source to Linux.

5. Start the linux discoverer processes (opmnctl startall and emctl start iasconsole)

6. Oracle Discoverer is now LIVE on Linux !