We’re always on the lookout for tools that help customers understand the value of cloud-based business analytics. If you’re considering bringing SaaS BI into your organization, we highly recommend you check out this post from Forrester Research analyst Boris Evelson on the Forrester Blog for Business Process & Applications Professionals. In “BI SaaS Vendors are Not Created Equal,” he provides a valuable framework for evaluating vendors before you invest.

We shared our point of view and wanted to continue the discussion around some of the key questions he raises in his framework:

VC backing: Evelson asks, “Is the firm backed by a VC with good track record in information management space?”

We’d frame this a little differently: Does the VC understand the SaaS revenue model? I would suggest that you look for SaaS pedigree in the VCs that are supporting a SaaS BI/BA vendor. Investing in a SaaS business is different payback model than the traditional enterprise software investment model. Experienced SaaS VCs understand it and know how to build a financial funding model to support it.

He also asks about profitability or loss run rate. “Is the business profitable or is the loss rate manageable, predictable and adequately financed till the planned breakeven / profitability goal?”

Every business needs to have a legitimate business plan and viable financial model with a management team that understands how to bring a company to profitability. It is not a viable business model to bet the farm on the fact that a large vendor will pay exorbitant premiums because of the numbers of subscribers doing analysis and reporting for “free” or on a trial basis. We believe the SaaS BI vendor HAS to deliver on the value of its service to both end users AND the IT organizations in a model that produces a fair return for the SaaS vendor.

Evelson mentions Salesforce.com recommendation. “Does Salesforce.com (or another major vendor on whose platform, data source, etc analytics are based) provide favorable or unfavorable recommendation?”

We agree that strong strategic partnerships are a good leading indicator. If the application is solely designed around better analysis of Salesforce data, I would suggest reviewing the customer ratings and testimonials on the AppExchange.

He suggests customers evaluate the management team, asking, “Does the management team have a good track record with successful startups and solid BI experience?”

We strongly agree with Evelson that the SaaS BI management team needs to be comprised of individuals that have relevant experience in both SaaS and BI.

Here are a few more areas Evelson mentions:

Architecture and technology -

Focusing on architecture and technology alone misses the point of the SaaS delivery model. Our customers do not care if we use relational, columnar, MP, MPP, “shared nothing”, ROLAP, MOLAP, HOLAP, Hadoop or Houdini. They all come to us with a set of requirements and expectations and price for a service they felt was appropriate. We happen to believe we have a better, more scalable solution and tend to rattle on for hours about it in a sales process, but once a customer has made a decision to use our service, they all focus on the richness of the functionality and how rapidly it can be deployed to a large user community in a low cost, low risk service model. How we get them there is no longer their concern.

Multitenant architecture: Is the architecture truly multitenant or is SaaS offering really an ASP/MSP under the covers? In other words, can a customer swipe a credit card and get provisioned instantly without a necessity for any manual intervention, setup, etc?

This, by the way, is the main reason why I often exclude mainstream BI vendors from BI SaaS category – but I welcome their challenge to this point.

Some of the largest SaaS application vendors in the world in Payroll (ADP, PayChex), Benefits (Ceridian) and Healthcare (eHealth) might question the credit card logic. I think the point is about self-service and low startup costs with minimal IT impact, not the actual form of payment.

Salesforce.com dependency; is the software offering 100% dependent on a single data source? Does the vendor have anything else to fall back on should that one single dependency not work out? Some other typical sources include NetSuite, Quickbooks online, and click stream data from Google, payroll data from PayChex, and others.

The true value of BI or BA is the end user’s ability to combine multiple data sources for improved visibility into the business. Customers should ask about connectors to SaaS vendors Evelson mentioned, but also should ask about SAP, Oracle Financials, Microsoft Dynamics applications, PeopleSoft, and other operational applications beyond Salesforce.

Metadata: Does the tool give you capability to import/export metadata so that business and technical metadata can be integrated and reused with other enterprise applications?

This is an excellent question – internally at PivotLink we call these design initiatives “being a good corporate citizen”. This drives our development around security and integrating with existing standards in an organization and it needs to go beyond just metadata. Customers should question whether the SaaS vendor’s security component is implemented as tightly as their own for the class of data that is included in the design. Will the service support single sign on and work with the authentication models already established in my organization? Will the service provide a means to get “my data” back if the customer decides to terminate the service.

APIs, Web Services. Can product functionality be modified, exposed and reused in other applications via APIs or Web Services?

Excellent point, because web services, mashup Gadgets and downloadable applets are defining the next generation of BI user experience.

Architectural secret sauce. Since any SaaS business is in danger of becoming commoditized does a vendor have a “secret sauce”, a protectable IP that truly differentiates them?

The secret sauce for any SaaS BI vendor is its ability to offer a high quality service, rapid time to value and continuous innovation. Come to think about it, isn’t that what the best companies in the world are also trying to do in their business?

Are customer references available? From customers in production (not POCs or prototypes)? What is the total number of verifiable customer logos in production? How diverse is this verifiable customer base by industry, enterprise size, function?

Agreed – references are KEY as they speak to quality of the service and the ability of the organization to deliver on its SLAs over time.

Work by firms like Forrester helps customers frame important questions to ask when evaluating vendors. Hats off to them for opening this up for discussion and inviting many points of view as this area evolves.