The Big Integration

I guess I was naive. I thought that Web analysts were the ones who had to live in an approximative world, because of the nature of our data. Boy, can data get dirty in the BI world! I attended the full-day course today on Data Conversion, Consolidation, and Cleansing – Practical Skills given by the passionate Arkady Maydanchik. Arkady can pack the most number of pieces of information in 10 seconds I have ever seen! He actually made what many would consider a rather dry topic, something exciting. I am not kidding; data cleansing is very complicated to execute well. I’m sure it can be very absorbing.

So, yes, data can get really really bad in that world, but at least, they have means to work on that and make it better. I realized today again how little control we have on our data, since we basically have to trust the vendors on that. What is a bad “record” in Web Analytics? Is getting back to analyzing IIS logs a solution? Well, I wouldn’t go that far; I am a true believer in tagging. But I wonder if we could not get rid of proprietary data format, and work as a community towards standardized structures in how the data is collected. This would mean that logs collected from tagging could be analyzed by whatever Web Analytics product you purchased (or didn’t). I guess this is similar to the meta data situation with BI application vendors. It was clear yesterday at the night course that none of them wanted to make their meta data readable by other vendors. I don’t know anything about that field, but it seems that meta data schemes are pretty much what they base in part their competitiveness on.

I am not sure that a call for standardized log structure (in tagging I mean; I know IIS or Apache are already vendor-neutral logs) would be such a terrible thing to vendors in our field. In the process of doing so, I guess we could also think about how we could re-structure that data to be easier to integrate with other systems in the business. Anyway, food for thoughts.

Tomorrow is the last day of the conference. I will attend another full-day class, this time on technology architecture for BI.

Today I was in new territories. I took both sessions on Data Requirements Analysis and Data Profiling for Reengineering, Knowledge Discovery, and Information Quality, both given by David Loshin. We spent the morning discussing (well, not me!) data discovery, data assessment, and data requirements. We went deep and in all details into David’s methodology. It felt like speaking a foreign language at an upper-intermediate level: I cold understand all we talked about, but missed several of the subtleties.

I was very impressed with the level of scrutiny data warehouse and BI people put in their assessment of the data quality. I think we should pay a little more attention about that in Web Analytics. Of course, we are talking here (BI) about a way more complex world. Still, I think we should do a better job at questioning our data. Looking at all that can and should be done with data assessment, I even wonder now how the ASP model is good for us, if we can’t have immediate and full access to the data/logs. I mean, when you stop thinking about it, what validates the accuracy of the results in Google Analytics? Yes, the brand. That’s it! No access to the data; got to trust Google that the numbers are good. Not acceptable to a professional web analyst I think. This means that the Omniture and WebTrends (OnDemand version) of this world should make it very easy to access the logs.

The afternoon session was basically focusing on what can go wrong with data. Wow! a lot, as you can imagine. I can’t go through the details of all that. It would be too long and I haven’t digest everything yet (if I ever will!). I was just extremely happy to be exposed to all that.

In early evening, I listen to a vendor panel featuring SAS, Business Objects, Oracle, MicroStrategy, IBI, and Cognos. Besides the very deep technical stuff their customers in the room asked them, they all agreed that text/verbatim analysis was very hot. Most of them are choosing to acquire vendors in that field instead of developing that capacity.

OK, that’s it for today. Tomorrow is going to be even deeper in the data. I wonder if I will emerge from this week in one piece…

OK, now I’m beginning to understand why people who were telling me that 6 days was exhausting were smiling. I’m starting to feel it in my bones. This was another day packed with information. The day started with Michael L. Gonzales again with a session called “HandsOn-Statistical Analysis for BI”. I have known that I needed to revisit statistics for a while (on Jim Novo’s advice), and I got confirmation again today. Actually, Michael was telling us that BI people as well tend to forget their importance. So, we went over exciting stuff such as descriptive statistics, linear regression, and control charts. I’m not being sarcastic here: it is really cool and exciting stuff. I wish I knew more, and will definitely hit the books. When I asked him about what books could be good, Michael suggested the two tomes in the For Dummies collection, even though he does not like them in general. However, these ones seem to be particularly good for a re-introduction to stats.

The afternoon was spent with Wayne Eckerson, whose book Performance Dashboards, has been an inspiration to me for some time. Although Wayne had way too much material to be covered in half a day, we went over very fundamental stuff in dashboarding: KPIs, business architecture, system architecture, design and layout, etc. I can tell you that dashboarding in BI is very heavy stuff. Projects can run in the hundred of thousands of dollars, given their complexity. This has nothing to do with what our Web Analytics vendors call dashboard. Aren’t we tired of those meaningless graphs we often scroll down over? I actually wish they were all under the tables!

OK, you can tell my the length of this post that I’m fried. Got to go to bed early tonight; tomorrow will be spent on data requirements analysis and profiling. Gulp!

I just spoke to Paul Katza, TDWI’s Director of Education, and when I told him that I was someone from Web Analytics here to spy on them BI people, he brought to my attention the San Diego event. One of the two keynote speakers will be Bill Schmarzo, VP Advertiser Analytics. His presentation is titled “BI Meets Web Analytics: Through the Looking Glass”.

Here’s what the brochure says:

“Like Alice through the looking glass, what happens when you transport an old-school business intelligence and data warehousing guy into the fast and furious world of Web Analytics? What is life like in the world of Web Analytics [hey! that’s why I’m here this week, wondering what life’s like in the BI world! JW] How does our knowledge of traditional business intelligence and data warehousing impact the Web analytics world? How do you leverage the best of both the BI and Web analytics worlds to deliver something new and compelling [my emphasis]?

The last question is what this blog/forum are all about. Too bad I do such a lousy job at maintaining them. But this is a very important question.

Anyway, since I’ll be in San Francisco then (the XChange event will just have finished), chances are that I’ll hop on a plane to go listen to Bill.

Today was HandsOn-Business Analytics with the very interesting Michael L. Gonzales. Michael walked us through some high level topics in today’s BI: the promise of BI’ the BI gap, filling the BI gap, dashboards and scorecards, advanced visualization, location BI, and data mining.

As in Web Analytics (I believe), BI is facing some frustration regarding made promises on which it still does not deliver well: “traditional warehouse technologies and techniques fall short of delivering the promise of BI” says Michael Gonzales. All those systems have been generating tons of reports, leaving the analyst will all the insight lifting to do. In short, too much reporting and not enough actionable information.

Gosh, does that ring a bell for us Web analyst!!

Michael stressed the importance of seamless integration of technologies, and simplified data delivery, as key components of the foundation of successful BI. He mentions that more and more the integration and data management defectiveness are embedded in today’s database products. ETL, for example, has just disappeared has a field in itself, and the reporting applications, such as Business Objects, Cognos, and Hyperion, belong now to the likes of Oracle, IBM, and SAP. That made me think that the latter, not the former ones, could be the potential buyers for high-end products in the Web Analytics field (no, not Google Analytics). When I look at all that is being done here, and the difference in size of the players, I can’t see what kind of long-term future there is for those vendors as stand-alone companies.

We discussed the difference between scorecards and dashboards, but I didn’t always agree with him. Michael Gonzales says that scorecards are the ones handling the KPIs, while dashboards are for operational use. To me, it seems to be more a question of vocabulary than true differences. Following W. Eckerson (whose workshop on dashboard I will attend tomorrow afternoon), I prefer to talk about strategic and tactic/operational dashboards. We did a little lab with Microsoft Scorecard Manager; quite neat (although a little poor on the design side).

Michael also introduced us to the new trend of visual analysis, which I was first exposed in June 2007 during Stephen Few’s first public seminar. It makes SO much sense to “look” at data to discover insights. We got to play with tools such as Advizor Analyst (ouch!), WebFOCUS Visual Discovery, ESRI Business Analyst (cool spatial analysis tool), and Tableau Software, for which I got a sweet spot.

We ended the day with data mining. M. Gonzales stressed the high quality of BI going on in that field, calling for the necessity to make it an important part of the BI portfolio.

This was a great day of learning, and I certainly look forward to tomorrow’s session, when I will attend, in addition to Eckerson’s class, another Michael Gonzales’s seminar on statistics.

I just got back from this first day very happy that I came here this week. We spent the whole day listening and interacting with Jill Dyché of Baseline Consulting. I knew Jill from two books I read a couple of years ago: e-Data: Turning Data Into Information With Data Warehousing, and The CRM Handbook: A Business Guide to Customer Relationship Management.

It was quite a fascinating day. Not only was it the very first session I have ever attended in BI, but I realized how much that world has in common with Web Analytics. The difficulties we often face (executive buy in, problems with planning, human resources, over-focus on tools, etc.) seem to be daily challenges in BI. Actually, I thought that field had left to prove than us, but I guess that, like any other practices that demand culture change, there is no easy path to success.

Jill’s session was called “BI from Both Sides: Aligning Business and IT”. You see? This could easily be the title of a Web Analytics presentation! She shared with us 11 techniques for aligning business and IT around BI. We could tell that all that information was coming from someone who has been at it for years. Besides having to deal with systems far more complex than Web Analytics, BI faces the same high-level challenge as WA. The human factor, i.e. champions, steering committee, executive sponsorship (which, to tell the truth, Web Analytics does not always enjoy), etc., is at the heart of the success of BI. One of the most important things to do, is to map the corporate strategy to the BI Portfolio (i.e. applications, systems, and data). Sounds familiar?

Another tip? Get quick wins. Gosh, don’t we know that!

So, you will excuse me if I don’t write more about today. I’m beat… and it’s only the first day!!

I must confess that I am very excited to attend TDWI’s conference on BI in Chicago next week. I am sure I’ll learn a lot.

Here are the workshop I am going to attend:

 – BI from Both Sides: Aligning Business and IT with Jill Dyché (I love her books)

 – HandsOn-Business Analytics with Michael Gonzales

 – Performance Dashboards: Measuring, Monitoring, and Managing Your Business with Wayne Eckerson (Wayne’s book on performance dashboards is just awesome!)

 – HandsOn-Business Intelligence Strategy with Michael Gonzales

 – Statistical Modeling for Non-Statisticians with Michael Barry

 – Data Conversion, Consolidation, and Cleansing—Practical Skills with Arkady Maydanchik

 – TDWI Technology Architecture for BI: Planning and Design of the Technical Infrastructure with Deanne Larson

If you are going to be there too, let me know!

I created a new Human Resources category in the Forum, with a first post coming from Judah Phillips’s latest blog post about what types of people are needed to build an awesome multi-channel team.

Judah kindly gave me permission to reproduce it in the Forum, and I thank him.

If you’re a Web analyst like me, and are convinced that there is an upcoming integration of the web data with the other enterprise data (after all, you read this blog!), you believe that there is an all new set of knowledge, and expertise that will be necessary to to understand, if not master.

I registered today to the full six days of the TDWI World Conference on BI. It’s full of full-day and half-day sessions which will allow me to be exposed to a lot of stuff that I have been curious about for a long time. I’ll be attending courses for six days, starting Sunday May 11th.

So, if you happen to attend, let me know; that would be cool to have a drink and talk about the future of Web analytics, and the coming Big Integration!

Hi all, it’s been a while.

I just wanted to let you know that Hurol Inan has made a good general introduction to the necessity of integrating web data with other systems in the enterprise. You can find the article from his newsletter here.

Good reading!