Project Management and Data Warehouse An Introduction

INTRODUCTION
This series of blog forms part of my assessment for my Masters of Business Information Systems course at Monash University, Victoria. The subject matter is – “How to successfully project manage a data warehouse (DW) implementation and avoid potential pitfalls during such implementation”.

Topics that will be covered in the coming weeks are:
1. Data Warehousing and their stakeholders – who is driving the need for a DW, top down or bottom up;
2. DW Projects – failures and how we can avoid them, Technologies Compatibilities (including evolutionary development), Database maintenance (data quality), Implementation, Analysis, ETL, project methodology;
3. Summary – closing notes.

It is assumed that the reader will have some Project Management and DW background.

I was a Project Manager working for the largest vendor for JD Edwards globally. I project managed Enterprise Resource Planning (ERP) system implementations for industries such as Transportation, Specialised Manufacturing, Telecommunication and Small Goods Manufacturing.

Most of the ERP projects had requirements for better decision support systems for the ability to track usage (costs), revenue, show market trends and produce timely reports with drill-down capabilities, or a traffic light type of executive dashboard.

ERP projects are very much function-driven and very specific in terms of data entry requirements. Users of ERP system are typically operational users who enter data into the system or act on operational reports from the system for a specific function in a simple or complex business process. Success of an ERP system is measured and determined when the system is delivered on budget, on time and according to the requirements specification. The project can also be deemed successful when the end-users are successfully trained to use the system correctly.

DW users on the other hand, are usually at management or executive levels who are decision makers in the organisation (Shanks, O'Donnell et al. 1997). They need to view the data at an aggregate level and typically have to ensure that the business’s KPIs (Key Performance Indicators) are being met. The data is often not from one source system alone and the reports are not operational in nature. In fact, reporting from a data warehouse involves analysis of the information.

So, how is the success of a DW project measured? Is a DW project successful when data is correctly extracted, transformed and loaded from source systems to a DW? Or is it deemed to be successful only when reports are produced? Or is a DW only useful when it can collect and aggregate all possible data from all possible source systems of the particular organisation?

The above questions will gradually be answered as I explore DW stakeholder requirements, expectations and issues as well as examine case studies of other data warehousing implementation project pitfalls, in the next few blogs.

There are few interesting similarities between some of the critical factors leading to a perceived ‘failure’ of an ERP and a DW implementation:

1. Organisational Culture & Change Management

When management buys technology with the intention of wanting the system to force change on organisational culture, they will be disappointed with the results because an ERP system is only the ‘technology’ part of the business. People (organisational and change management) and process issues have to be taken into consideration in the project plan.

2. Scope Creep

No matter how well requirements are written, it is inevitable that requirements may change during the process as issues are uncovered or new business requirements crop up. A project that has good requirements sign-off methodology, change control procedures and stakeholder management will ensure happy customers even if the project takes longer to deliver and cost more than originally budgeted (due to an increase in scope). However, if the scope is not clearly defined at the onset of the project, the project’s timeline will be stretched with no clearly-defined reasons.

3. Poor Requirements

When a project fails, the blame generally falls on the system and its inability to meet the original requirements. This might be because there was a lack of mutual and agreed understanding between both Project Manager and the Steering Committee with regards to the requirements (Sauser, Reilly et al. 2009).

PROJECT MANAGEMENT
In this section of the blog, I want to explore whether good project management is vital to the success of a data warehousing implementation. Common sense tells me the answer is an obvious yes. The question should really be: “Can good project management PREVENT a data warehouse project from being considered a failure?”

Every IT project will need to follow some Software Development Life Cycle (SDLC) methodology. However in DW projects, there is a greater need to understand the data requirements behind the reports requested. The reporting and analysis requirements can come from top down or bottom up and it is important to include all relevant stakeholders during the requirement gathering process. The importance of this will be discussed more in my subsequent blog.

Other areas of project management will include and not limited to, project scoping (what are the deliverable, milestones), ensuring that the project is on time and within budget, testing all deliverable, getting sign-off’s and user training.
 
DATA WAREHOUSING
A DW is designed to be the single repository to aid in the decision making process of the engaging organisation usually defined as a "subject-oriented, integrated, time-variant, and non-volatile collection of data in support of management's decisions” (Inmon and Hackathorn 1994). The data sources for decision making are generally from legacy systems, external systems or special purpose data (Rachmat 2000).

Users can query the DW to extract information to understand trends that had occurred within the organisation down to the division or department level, if required. This information is useful for management to forecast or even predict an outcome for better decision making purposes, in short improving the organisation through historical data.

Although it could be argued that DW is also known as a Decision Support System (DSS) from a business perspective (Westerman 2001), DW is actually neither a DSS nor an Executive Information System (EIS). It is the central repository of data collected as discussed above from various sources so as to better provide the user with reports for decision making purposes. Therefore it can be said that the DSS and the EIS are derived from a DW.

DSS and EIS are time critical systems. When I was in the consulting field, one of the most commonly asked question with managers are: “When will I get my report and will it look the same after you have “played” with the system?”. Without a DW, it is harder if not close to impossible to have an efficient DSS or EIS system due to the size of data collected and the time it takes to extract all the information in a meaningful manner. By joining tables of a normalised functional database, it will tend to degrade performance and the report may take a long time to run. Therefore it is important to construct a DW so that data can be accessed by systems such as DSS and EIS. According to Bill Inmon, who is often called the "father" of data warehousing, a DW is "a collection of integrated, subject-oriented databases designed to support DSS function (Lewis 2001). Bill Inmon did not say that DW has another name called DSS.

Ralph Kimball, another data warehousing expert, explains that a data warehouse is “a copy of transaction data specifically structured for query and analysis" (Lewis 2001).

Some analytics vendors will try to persuade customers to commence their data warehouse project by buying a commercial DSS rather than building a DW from scratch. The main advantage, they argue, is that the user gets to see the results quicker. As revealed above, it is in fact the DW that facilitates the speed of this DSS or EIS system to access the data.

Let’s now combine project management with data warehousing. I talked about the elements of a good project plan and the need to understand the data requirements for a DW. The nature of DW requirements is such that the requirements are vague at the start and can only become clearer once the users have a taste of the sort of information they can get out of the data warehouse. This straight away means that keeping to a project schedule is challenging at best, when the requirements are not as clear or specific enough. (You don’t know what you don’t know).

REFERENCES
Inmon, W. a. and R. Hackathorn (1994). Using the Data Warehouse. New York, John Wiley and Sons.

Lewis , W. J. (2001). A crash (or collision) course: A history of data warehousing. Data Warehousing and E-Commerce Prentice Hall.

Rachmat, S. (2000). Australian Data Warehouse Practice Monash University. Masters Thesis.

Sauser, B. J., R. R. Reilly, &, et al. (2009). "Why projects fail? How contingency theory can provide new insights – A comparative analysis of NASA’s Mars Climate Orbiter loss." International Journal of Project Management Available online 14 February 2009: 15.

Shanks, G. G., P. A. O'Donnell, and , et al. (1997). Data Warehousing: A Field Study. Proceedings of the 8th Australasian Conference on Information Systems. Adelaide: pp350-365.

Westerman, P. (2001). Data Warehousing: using the Wal-Mart model, Academic Press.