http://insights.iolap.com/author/nlarsen/The total cost of operations (TCO) of Business Intelligence (BI) systems is often measured in three categories: time-to-completion of projects, on-budget completion of projects, and cost per user of BI applications. There is a key process in every project that impacts all three categories: Business Requirements Engineering.
An effective requirements methodology ensures that project scope is clearly understood and costs accurately estimated. At the same time, when we deliver what users want, usage and adoption of the solution increase the user base. Why then do so many programs not take a closer look and the effectiveness of their approach to this key part of the process?
Requirements gathering methodologies have been refined for decades, but we consistently see inadequacies and gaps with our end users. The “Business Requirements” deliverable has become a checkbox for most Project Managers and a nightmare for Business Analysts not familiar with BI.
The biggest issues with traditional methodologies are:
- They are “one size fits all.” The plans are used for large or small projects, BI applications, line of business (LOB) applications, or customer-facing applications.
- They are not specific enough for the BI application to be developed.
- They lock down. Changes based off data profiling or other factors are challenging to make.
Agile methodologies are becoming more and more popular in some types of projects. We are seeing higher user engagement in the entire process and of course, quicker deliveries (albeit with iterative functionality). But, how do we improve the requirements process for traditional BI methodologies? How do we deliver a world class product that attracts a large user base? How do we deliver functionality faster and at a lower cost?
I believe it starts with formulaic requirements tied to particular project types. Too many times, we hear Business Analysts say, “OK business units, what do you want?” It usually starts with a report mockup and they begin documenting all the fields and metrics. As page filler, we see things like:
- Ability to drill
- Support role based access
- Support ad-hoc report building
- Provide a single version of XYZ metric
- Consistent and accurate data
- Improve analytics capability
There is a place for these type of requirements, but they certainly lack the detail necessary to build a successful solution. Instead, we must consider the type of project we’re undertaking first.
This project could be the introduction of an enterprise data warehouse (EDW) and BI platform where none existed before, or replacing existing infrastructure. Most commonly, the project involves the creation of a new BI application, report suite, and/or the integration of new data sources into an existing solution. Other types of projects exist, but are generally IT-driven such as platform upgrades, application simplification, and so on.
New EDW/BI Platform
These projects are usually driven from the consolidation of disparate departmental solutions. Ralph Kimball and others have done a great job of providing the recipe for kicking off new programs:
- Prepare the Business Case. This starts defining business drivers and understanding goals, measures, and objectives. The case is solidified by presenting the core business improvement opportunities that will be realized from the program.
- Define the Program and Roadmap. Identify and rank subject areas. Solicit conceptual requirements. Prepare time and cost estimates.
The key activities in the requirements stage involve a deep partnership with executive sponsors and steering committees. User interviews are conducted but the requirements are kept at a much higher (conceptual) level than specific applications. The goal is to gather enough information to present both a business and technical roadmap for the program.
New Infrastructure
Both program and project level initiatives lead to new infrastructure. External forces such as procurement requirements, product support deprecation, or organizational changes often impact platform change decisions as well.
The requirements process is very different for this type of project. Consider organizing the specifics into big buckets, including:
- Security
- Data sizes – both storage and processing
- Network and overall hardware topology
- Data volatility
- Vendor support and service level agreement (SLA) needs
- User/query concurrency and application use cases
Most BI platform vendors have similar approaches when sizing your system and licensing needs. You can leverage this during your requirements sessions.
Data
We frequently see philosophical “chicken or the egg” debates with these projects. Consider a scenario where a new transactional system is being developed or a new third party extract is available for consumption. Do you proactively go after this source and integrate it into your environment at an atomic level following your existing best practices? Or do you wait for business funding and a specific BI application need to tie it back to? Said another way, do you build it and hope they will come knowing that your architecture will support most business requirements?
I’m a big fan of integrating as much data as possible as quickly as possible. Too many times source systems purge data limiting the types of analytical solutions that can leverage it. More data means better data profiling. This approach mitigates this risk and reduces integration costs. Don’t get me wrong, there will be enhancements and rework needed once the BI application requirements start flowing in, but hopefully you are 80% there.
More often than not, funding wins this battle and the integration must wait for official projects. Regardless of the inception method, the project will be most successful with hardened non-functional requirements centered around:
- Compliance and security
- Data latency and delivery SLAs
- Conformity and granularity
- Archiving and lineage
- Quality, recoverability, and restartability
Start by building frameworks and guidelines around these types of requirements. When the business project hits, you can easily work through variances.
Before starting the detailed data requirements, architects need some cycles with the data itself. This involves profiling, looking for relationships, and business scenarios. This puts them in a good position to take decision points to requirements sessions rather than rework requirements after design as started.
Functional requirements are more centered on the specifics of the source and desired state. Consider these areas in your framework or requirements solicitation:
- Calculations and transformations
- Dimension strategies (Type 1, 2, 3, and so on)
- Late arriving scenarios
- Bad data scenarios
BI Applications
These could be reports, dashboards, mobile applications, or any other presentation of the data requirements to the users. These have their own non-functional areas such as compliance, security, and performance, but most of the time they will inherit overall constraints driven from the platform.
Once a good understanding of the data source for the application is achieved, requirements should focus more around functionality. We classify them into:
- Presentation
- User interactions
- Elements and calculations
- Consumption/distribution
The major gap we see in BI application requirements is a lack of communication. These are living documents that must be allowed to evolve throughout the project and as the solution is showcased to the users. During the process, be careful to protect scope but be mindful to the wins you can achieve by deviating from the exact mockup to provide additional business value.
Summary
Enable your Business Analysts and Architects to be more effective during requirements engineering sessions by providing frameworks and guidelines targeted for specific project types. The requirements you gather are the key to delivery success, perceived or otherwise.