This blog contains experience gained over the years of implementing (and de-implementing) large scale IT applications/software.

Oracle Documentation Is a Moving Target

I appreciate that internet content is better when it’s dynamic, but sometimes there’s a need for static content; or should I say more accurately, static links to content.

I’m right in the middle of compiling a new tutorial on installing Oracle 11gR2 on Oracle Enterprise Linux 6.1.
The Oracle docs in HTML format have been on-line for years now, but I’ve never really referenced them over a period of time with such specific requirements.
I’ve been using the software pre-requisites section of the Oracle 11gR2 docs and have bookmarked them in my browser of choice.  At least I thought I had.  Until today, when I click the link I get an Oracle web page stating “404 not found”.
What happened?  What caused the move of the actual URL?
Here’s what I was using before:
https://download.oracle.com/docs/cd/E11882_01/install.112/e16763/pre_install.htm#CIHIDJAH

Here’s the same content today:
https://download.oracle.com/docs/cd/E11882_01/install.112/e24321/pre_install.htm#CIHFICFD

Looks like the e16763 has been replaced with e24321 and the internal bookmark (anchor) in the page (CIHIDJAH) has been changed to (CIHFICFD).

All links internally on the page are relative links and not absolute, and when you try and navigate up the path, you get a 404.  So I guess that Oracle themselves don’t want you to bookmark the links.

Lesson learned, always capture what you need elsewhere and duplicate everything in case you can’t find it or it gets removed…

Disaster Recovery – Where To Start

Perhaps IT implementors think that Disaster Recovery is a dark art, or maybe it’s just another item to de-scope from the implementation project due to time restrictions or budget cuts.
But I find that it’s coming up more and more as part of the yearly financial system audits (the ones we love to hate!).
Not only is it an action on the main financial systems, but the auditors are training there sights on the ancillary systems too.

So where do I normally start?
Usually it’s not a case of where I would like to start, it’s where I *have* to start due to missing system documentation and configuration information.
Being a DBA, I’m responsible for ensuring the continuity of business with respect to the database system and the preservation of the data.
So no matter what previous work has been completed to provide a DR capability, or a HA capability (so often confused) it’s useless without any documentation or analysis to show what the tolerances are (how much data can you afford to lose – RPO, and how much time it will take to get a system back, RTO).

First things first, ensure that the business recognises that DR is not just an IT task.
You should point them at the host of web pages devoted to creating the perfect Business Continuity Plan (BCP).
It states that a BCP should include what *the business* intends to do in a disaster.  This more often than not involves a slight re-work to the standard business processes.  Whether this involves going to a paper based temporary system or just using a cut-down process, the business needs a plan.
It’s not all about Recovery Point Objective and Recovery Time Objective!
A nice “swim lane” diagram will help a lot when picturing the business process.
You should start at the beginning of your business process (normally someone wants to buy something), then work through to the end (your company gets paid).  Add all the steps in between and overlay the systems used.  Imagine if those systems were not present, how could you work around them.

Secondly, you need to evaluate the risks among the systems you support.  Identify points of failure.
To be able to do this effectively, you will need to know your systems inside out.
You should be aware of *all* dependencies (e.g. interfaces) especially with the more complex business systems like ERP, BPM, BI, middleware etc.
Start by creating a baseline of your existing configuration:
– What applications do you have.
– What versions are they.
– How are they linked (interfaces).
– Who uses them (business areas).
– What type of business data do they store (do they store data at all).

Third, you should make yourself aware of the hardware capabilities of important things like your storage sub-system (SAN, NAS etc).  Can you leverage these devices to make yourself better protected/prepared.
An overall architecture diagram speaks a thousand words.

Once you understand what you have got, only then can you start to formulate a plan for the worst.
Decide what “the worst” actually is!  I’ve worked for companies that are convinced an aeroplane would decimate the data centre at any moment.  All the prep-work in the world couldn’t prevent it from happening, but maybe you can save the data (if not yourself 😉 ).

Back once again (with the renegade master…)

Some hefty time off spent on the beach in the south of Italy (ah che bello) with the in-laws, has revealed a slight sun-tan and a new sense of purpose.
– Part III of my SAP Basic Performance Tuning Guide is nearly ready for posting.
– A new job opportunity has come my way.
– The VMWare test rig will be getting another hammering once I have got some post holiday chores out of the way (still can’t believe I got it working in 30 minutes!).

Bring on August!

Use of Oracle AWR / ASH leading to bad coding?

I had a brief email exchange with another Oracle guru the other day.
He suggested that the quality of Oracle coding in PL*SQL and Plain Jane (www.medicaltextbooksrevealed.com/blog/2010/02/plain-jane/) SQL had gone down hill.

This could be attributed to two factors:
1, The level of coding experience has dropped.  Older more experienced coders have filled into the new architect roles and the void is being filled quickly by newer in-experienced coders.
2, The rigour with which debugging, testing and tuning is performed has become somewhat lax (www.thefreedictionary.com/lax ) because there’s just no emphasis on the developers to tune their code when the DBA has such great tools to do it for them.

Is it possible that the use of the additionally licensed tools such as AWR (Automatic Workload Repository) and ASH (Active Session History) introduced in Oracle 10g, have provided an easy mechanism for DBAs to seek out better performance.
I don’t think these tools are just for DBAs, but the way they are marketed makes me feel they are pushed that way.

Are you sure I’m not a consultant

I’d like to mention the growing (in my mind anyway) disparity between an operational support person and a consultant.
Having spent some time in contact with various recruitment agencies, I can understand how single words specified on a requirements brief from the client must be followed as accurately as possible.
The client, after all, wants the perfect candidate don’t they?!

I’ve worked on both sides of the coin you see.
I started off as an apprentice in an IT department in the late nineties.
Customer satisfaction is key when you’re in an IT department working the helpdesk.  Even if you’re second line (which infers a military comparison, doesn’t it? https://www.google.co.uk/search?hl=en&biw=1385&bih=862&q=second+line+of+defence&aq=f&aqi=g7g-v2g-sv1&aql=f&oq=), you may be rated on your comms skills, how efficient you were and how accurate (first time fix).
So I can understand that supporting something that is already installed and working would be viewed as operational support and not consultancy.

I’ve also worked for a consultancy, quite a big one.  I wasn’t called a consultant, but if you work for a consultancy you must be? right?

But wait, doesn’t an internal IT department upgrade stuff?  performance tune?  generate ideas?
Are we saying that IT departments just do these things without talking to the business side.  I think not.  I remember many meetings where new ideas had been conceived in the IT department and were conveyed to the business.
Is this not consulting? Advising what would be best for the business and the technology.  I don’t know any IT department today that would contemplate changing anything without consulting with the business first.

So if we’re saying that IT departments do consult, the difference between a consultant and operational support must be in the *type* of consulting right?
You could argue that a consultant has more “soft” skills.  Um, like a salesman perhaps?
Or would you argue that a consultant may have broader knowledge.  Possibly.  More of a fish eye? (https://en.wikipedia.org/wiki/Jack_of_all_trades,_master_of_none).
Hmm, I can’t see how this is different either.  You are either very good at these things (IT skills) or you’re not.
The only difference is when you consider a company wanting change, but not having the skills internally.  Wouldn’t that force them to use an external consultancy?  Someone *else* to consult with.

I think a “consultant” today is someone who has managed to escape the IT department.  He’s a free radical (https://en.wikipedia.org/wiki/Radical_(chemistry)).  But without the other elements he’s not got anything to bond with.  He’s able to offer unbounded ideas and generally gets the biggest budget (and sometimes some training), but he’s not able to evolve into anything more complex.  He came, he saw, he implemented, he left.  There’s a bigger side to this implementation lark, and it happens to be operational support!
Hurrah to the helpdesk operators, the backup tape changers and the email mailbox size limiting people (
https://www.winserverkb.com/Uwe/Forum.aspx/exchange-admin/5313/Why-limit-people-s-mailbox-size).

Some of the best, most talented people I’ve ever worked with have been working in operational support.  Fantastic, creative ideas ingeniously devised in the smallest of rooms at the eleventh hour (https://en.wikipedia.org/wiki/The_Eleventh_Hour) to stop the world ending (or the system from crashing) on a system that was implemented 20+ years ago.  How can these people not be consultants?  Maybe they’re too honest 😉