As a general guide, Marketers prefer Google Analytics, IT departments prefer Urchin…

This statement is reflected in the user-base for each product. Urchin is mainly used for measuring intranets (an area of the business that Marketing is not involved in), and used by web-hosting providers where deployment scalability for a large number of websites is important.

On the other hand, Google Analytics, apart from being a free service (Urchin requires a purchased license), is used by organizations that wish to have greater control of their individual web analytics implementation.

Those are not mutually exclusive rules or requirements, so lets look at some criteria.

When GA is the best fit

Select Google Analytics if you are measuring the success (or not) of your website, its ability to convert, and the effectiveness of online marketing. Google Analytics is much easier to implement, has stronger AdWords integration, and by comparison is maintenance free. Site overlay and internal site-search reports are only available in GA.

Google Analytics also has AdSense integration, which Urchin does not, and has the greatest potential for other Google product integrations. This is because Google Analytics is SaaS (Software as a Service) solution running on the Google network. Essentially, it is much, much easier for a vendor to integrate data within its own network, rather than an external stand-alone system.

These are key reasons why marketers love Google Analytics.

When Urchin is the best fit

Select Urchin if:

• You have an intranet site behind a firewall that blocks internet activity. Google Analytics is a hosted solution that needs access to the internet in order to work.
• You are unable to tag your pages*.
• You are a hosting provider wishing to offer visitor reports to thousands of customers. Urchin has a command-line interface that can be scripted to create and modify multiple website reports at once. That is, Urchin has greater flexibility when it comes to large-scale, multi-user deployments.
• You wish to track individual visitors (anonymously). Google Analytics does not track individuals as it is against Google’s privacy policy. Hence GA data is both anonymous and aggregate.

*Note: Some third-party hardware solutions can automatically insert Google Analytics page tags for you on the fly, that is, as the page is requested from your web server. They achieve this by using proxy servers that sit within your network (in front of your web server) and insert the code for you. See, for example, www.click-stream.com, www.coradiant.com.

When you need both

Select both if you need the flexibility of maintaining your own visitor data, for example, for third-party auditing purposes. Combining Google Analytics with Urchin Software gives you the best of both worlds—the advanced features of Google Analytics (free) and the flexibility of Urchin (data control). This article: Backup your Google Analytics data and use Urchin describes how you can configure your page tags to stream data both to Google Analytics and Urchin simultaneously.

Recommendations

My recommendation is to use Google Analytics wherever possible. It is easier to implement, has a slicker user interface (with best-in-class geomap overlay reports), and is primarily aimed at Digital Marketers. Urchin Software should be used where there is a specific technical need that Google Analytics cannot fulfill. Urchin currently lacks some marketing features that GA has, though it can track individual visitors anonymously. if you can, use both tools.

Are you considering the merits of Urchin v GA? Please add your thoughts with a comment.

© Brian Clifton for Measuring Success, 2010. | Permalink | 8 comments | Add to del.icio.us
Post tags: hybrid, log file, logfiles, Urchin, urchin 6, urchin software

Urchin Software Inc. is the company and technology that Google acquired in April 2005 and went on to become Google Analytics—a free web analytics service that uses the resources at Google (I explain more about its history in Google Analytics – Fours Years on). Urchin software is a downloadable web analytics program that runs on a local server (Unix or Windows).

Typically, this is the same machine as your web server. The Urchin Software creates reports by processing your web server logfiles (including hybrid logfiles) and is commonly referred to as server-side web analytics. Example screenshots of Urchin Software (version 6) are shown in Figures 1 and 2.

Figure 1: Urchin 6 initial configuration screen

Figure 2: Urchin 6 visitor report showing A) individual (anonymous) visitor information, B) visits by date, and C) path and resultant transaction information for a particular visit .

Urchin is essentially the same technology as Google Analytics—the difference when using Urchin is that your organization needs to provide the resources for log storage and data processing. Urchin can provide complementary reports that Google Analytics currently does not (or cannot because of its methodology). Let’s look at some examples:

• Visitor history report
  Tracking individual visitors enables you to view the path a visitor takes through your website as well as their referral information. For privacy reasons Google has deliberately taken the decision not to track individuals with Google Analytics. However, with the data collection and processing under your control, you have the freedom to do this with Urchin. Each visitor is tracked anonymously.

• Error page and status code reports
  More than just reporting completed page views (as is the case for Google Analytics), Urchin can report partial downloads and any error code. It is possible to configure your website to report error pages within Google Analytics. However, Urchin software reports on errors out of the box because your web server tracks these by default.

• Bandwidth reports
  Reporting on bandwidth allows you to view how “heavy” your pages are and how this impacts the visitor’s experience.

• Login reports
  If your website has a login area, you can report on this access by username. This supports standard Apache (.htaccess) or any authentication that logs usernames in the logfile.

Differences between Google Analytics and Urchin

With two analytics products from Google to choose from, how do you determine which one of these is right for your organization? As you may have guessed, Google Analytics is perfect for most organizations, for two very simple reasons:

• Google Analytics is a free service
  This is generally considered a major benefit for small and medium-size organizations where budgets for analysis are tight. Urchin software is a licensed product and therefore must be purchased (currently $2,995 per installation).

• Google Analytics handles a large part of the IT overhead
  That is, Google conducts the data collection, storage, program maintenance, and upgrades for you. This is generally considered a major benefit for large organizations where web analytics is a priority for the Marketing department and less so for the IT department. If your organization is using Urchin software, it is responsible for the IT overhead. Hence, good interdepartmental communication (IT and Marketing) is required.

The second point is not trivial. In fact, in my experience, the IT overhead of implementing tools was the main reason why web analytics remained a niche industry for such a large part of its existence. Maintaining your own logfiles has an overhead, mainly because web server logfiles get very large, very quickly. As a guide, every 1,000 visits produce approximately 4 MB of log info. Therefore, 10,000 visits per month are approximately 500 MB per year. If you have 100,000 visits per month, that’s 5 GB per year, and so on. Those are just estimates—for your own site, these could easily double. At the end of the day, managing large logfiles isn’t something your IT department gets excited about.

Urchin also requires disk space for its processed data (stored in a proprietary database). Though this will always be a smaller size than the raw collected numbers, storing and archiving all this information is an important task because if you run out of disk space, you risk file or database corruption from disk-write errors. This kind of file corruption is almost impossible to recover from.

As an aside, if you maintain your own visitor data logfiles, the security and privacy of collected information (your visitors) also become your responsibility.

Why, then, might you consider Urchin software at all?

Urchin software does have some real advantages over Google Analytics. For example, data is recorded and stored by your web server, rather than streamed to Google, which means the following:

• Data processing and reprocessing
  Urchin can process data as and when you wish, for example, on the hour, every hour. You can also reprocess data—to apply a filter retroactively or to correct a filter error. Google Analytics reports are three to four hours in arrears and cannot reprocess data retroactively (in my opinion, the benefit of reprocessing data is the strongest advantage of Urchin).

• Unlimited data storage
  Urchin can keep and view data for as long as you wish. Google Analytics currently commits to keeping data for a maximum of 25 months, though to date, Google has made no attempt to remove data older than this—see Figure 3.1.

• Third-party auditing
  Urchin allows your data to be audited by an independent third party. This is usually important for publishers who sell advertising space on their site, where auditing is required to verify visitor numbers and provide credibility for advertisers (trust in their rate card). Google Analytics does not pass data to third parties.

• Intranets and firewalls
  Urchin works behind the firewall; that is, it’s suitable for intranets. Google Analytics page tags cannot run behind a closed firewall.

• Database access
  Urchin stores data locally in a proprietary database and includes tools that can be used to access the raw data outside a web browser, allowing you to run ad hoc queries. Google Analytics stores data in remote locations within Google datacenters around the world in proprietary databases and does not provide direct access to the raw data for ad hoc queries. That said, the Google Analytics API does allow you to query your processed data.

Note: Urchin is sold and supported exclusively through a network of Urchin Software Authorized Consultants. For a full list of USACs, see www.google.com/urchin/usac.html.

When it comes to adoption numbers, Urchin is certainly the smaller sibling compared to GA. However, when appropriate I recommend using both side by side.

Many people first come into contact with Urchin from their ISP/hosting account. Are you an Urchin user? Are you on version 5 (2004) or version 6 (2008)? I would love to hear your feedback. I am planning the next post on Urchin to discuss the criteria for selecting GA v Urchin. If that interests you, let me know – it motivates me to write it…!

© Brian Clifton for Measuring Success, 2010. | Permalink | 14 comments | Add to del.icio.us
Post tags: Urchin, urchin 6, urchin software

A brief history of Urchin

Urchin analytics has been around for some time. In fact since 1997, Urchin software has been slowly and quietly building a strong reputation for its server-side web analytics software. I first came across it in 2003, where its lightening fast processing power, small resource footprint and good value for money caught my attention. Even on a moderately specified Linux box, Urchin’s number crunching performance far exceeded anything else on the market – and still does today. This has made it particularly attractive to ISPs and web hosting companies that remain its largest customer base.

A differentiator for the Urchin product is its hybrid approach – combining data collection from both web server logfiles and page tag beacons results in a greater accuracy of visitor tracking. This is still the most accurate way to track web visitors, that is, without having force your visitor to login into your website (view the Accuracy Whitepaper for a detailed description). However, by 2003 the market was beginning to move away from server-side web analytics software as the method for benchmarking performance.

Instead, Marketers wanted more control. Server-side analytics is in the realm of IT departments – a place were, historically, marketing performance and user-experience measurement are not a priority. Now marketers wanted to be in the driving seat – accessing the data in order to manage and optimise campaign performance. The industry alternative was “page tagging” – that is no web server logfiles (and hence no IT) to worry about. These simplified implementations gave more independence to the marketers for web measurement.

In 2004, the San Diego based company launched their page tag web measurement service, called Urchin On-Demand. IT teams were no longer required to install or manage software. Although IT/developer input is still needed for page tag deployment, it is a much ‘lighter’ touch and less resource required than before as the responsibility for data collection, storage, archiving and reporting goes direct to Urchin. Only a page tag beacon (a small snippet of JavaScript) on each web page is required to do this.

In addition, Urchin On-Demand gained improved marketing reports – particularly for AdWords. It was this simple, yet powerful combination of a web measurement solution aimed at digital marketers, that caught Google’s attention. By April 2005 Google’s acquisition of Urchin Software Inc. was announced.

The Google impact

Since 2005, there have been 3 key phases of Google Analytics development and I believe 2010 will mark the beginning of Phase 4:

Phase 1 – Free and almost scalable (Nov 2005)

On November 11th 2005 Google Analytics was launched. A major part of the announcement was that the product was now free (originally Urchin On-Demand started from $500/month). This was a tipping point in the industry. Overnight Google re-wrote the entire industry business model – giving away a deep dive web analytics tool for free while everyone else charged based on volume of traffic.

The impact of that decision was dramatic. An industry that once counted its customers in the tens of thousands, now exploded. In fact so dramatic was the uptake of the service that it had to close to new subscribers for 10 months while new machines were allocated to the number crunching tasks at Google’s data centres. However, once re-opened, the user base of Google Analytics rapidly expanded and went beyond a million in a matter of months.

Looking back, the launch was a bit of a  PR disaster – the company took a lot of flak for under estimating demand (though I have no idea how it could have been done better!). But we had an alternative viewpoint – the launch still remains one of Google’s most successful product launches ever. Those within the company that had not heard of the team until then, certainly had by Friday 15th November. We were the new kids on the Googleplex block. That was a great week.

Phase 2 – The new User Interface (April 2007)

To all who witnessed the launch of Google Analytics in 2005, the product simply appeared to a re-branded version of the Urchin On-Demand service – just given away for free. Of course a great deal of hard work went into scaling the product so it could handle very large data sizes – both from the huge user base it was accumulating, but also from clients such as MySpace, YouTube etc. that had truly massive pageview volumes.

However, the fact remained that the user-interface, the most important part of any reporting tool, didn’t stand out from any other reporting tool. The launch of the new UI in April 2007 changed that (and as I was in the middle of writing the book, required  re-write!).

It was worth the wait. The same UI design is still used today and has proved so successful in user experience studies, that the format is being adopted throughout Google – notice the rounded style graphs you now see in AdWords and geo map overlay of Google Insights for example. The team responsible for that came from Adaptive Path , a Google acquisition of 2006.

Phase 3 – Enterprise features that stand out (Oct 2008)

Numerous enterprise clients get real value from Google Analytics, whether its the BBC, Roche, Tetrapak or non-profits such as Unicef or the World Health Organisation, many choose Google Analytics because it answers their questions quickly and efficiently (see a fuller list of Google Analytics users ). However, naysayers have commented on a number of missing “enterprise” level features. This was addressed in October 2008 with a plethora of new announcements. These include (still in beta):

• Advanced segments On the fly visitor segmentation. For example, show me paid search visitor metrics alongside organic visitor metrics. Custom segments can also be define e.g. show only visitors that had visited 3+ times, show visits from the UK and US combined compared to all visits, show only visits that spend at least 30 seconds on site, show only mobile visits etc, etc.
• Report customisation Don’t like the layout of a particular report? Want to include a metric from one report e.g. revenue, in another report e.g. top content? No problem
• AdSense integration If you are a publisher you will be familiar with the AdSense model i.e. showing Google pay-per-click ads on your site for a share of the click revenue. Now you can get your AdSense account information imported into your Google Analytics reports in a similar way to the AdWords reports. For example, impression and click through data.
• Motion Charts Data in five dimensions! Essentially this is animated (Flash) statistics to aid with data visualisation. Its a cool feature that was the result of a Google acquisition for Trendalyzer software (a Swedish company) in 2007. Its one of the first charts I look at to gain a bigger picture overview prior to focusing down on a specific referrer/metrics/keyword etc.
• Data APIs Previously access to the report data within Google Analytics was a manual process – go to a report, click export and save the data for that report. This can now be automated and customised with the use of the Application Programming Interface. That’s a lot of possibilities…

If I had to choose one, I would say the advanced segments feature is probably the most powerful feature Google Analytics has ever had. It allows you to drill down and gain valuable insights as to how your web site is performing like never before (previously, filters and separate profiles were required).

Of course, there is still room for the competition to differentiate themselves. In fact, I was very disappointed to hear of Microsoft’s analytics demise recently. Deepmetrix (Microsoft’s  acquisition of 2007) was a respected tool and I evaluated it alongside Urchin in 2003. It was a very close call in terms of features and flexibility and my choice to go with Urchin essentially boiled down to the building of a remote relationship – a phone call with a lady called Megan Cash at Urchin Inc. made the difference. Lucky for me that it turned out to be the winning bet!

So what’s next for Google Analytics?

(...)
Read the rest of Google Analytics – Four years on (308 words)

© Brian Clifton for Measuring Success, 2009. | Permalink | 18 comments | Add to del.icio.us
Post tags: 4 years, Google Analytics, history, hydrid, integrate, integration, who uses google analytics

If you are looking for an outsourced backup service for Google Analytics, contact GA-Experts.com, who can do this all for you…

[This post updated: 04-Aug-2009]

Benefits of keeping a local copy of Google Analytics visitor data

What you can do with your local copy of your data:

• Greater control over your data e.g. for third-party audits
• Troubleshoot Google Analytics implementation issues
• Process historical data as far back as you wish – using Urchin
• Re-process data when you wish – using Urchin

Use the following 2-step guide to keep a local copy; 1) modify your Google Analytics Tracking Code – the GATC; 2) add a small transparent gif file to your web root.

1. Modifying your GATC

If you are using urchin.js in your Google Analytics tracking, then you should upgrade! However, if that is not possible just yet, add the following line immediately before urchinTracker(); as follows:

For urchin.js users:

var _userv=2;

For ga.js users, add the following line immediately before pageTracker._trackPageview(); as follows:

pageTracker._setLocalRemoteServerMode();

2. Add _utm.gif to your web server root

The consequence of step 1, is the request for a file named __utm.gif from your web server as your page is loaded. This is simply a 1×1 pixel transparent image that Urchin uses for processing. In fact, it is the only line in your logfile that Urchin uses (apart from error pages and robot activity. You can create the file for yourself or use this one (use right click, save file as) __utm.gif file. Upload this in your document root i.e. where your home page resides.

That’s it! With these two simple steps, Google Analytics visitor data is simultaneously streamed to your web server logfiles in addition to being sent to Google Analytics for processing. This is simple to achieve as all web servers log their activity by default, usually in plain text format.

Once implemented, open your logfiles to verify the presence of additional __utm.gif entries that correspond to the visit data as ‘seen’ by Google Analytics.

A typical Apache logfile line entry (line wrapped here) looks like this:

86.138.209.96 www.mysite.com - [01/Oct/2007:03:34:02 +0100] "GET /__utm.gif?utmwv=1&utmt=var&utmn=

2108116629 HTTP/1.1" 200 35 "http://www.mysite.com/pageX.htm" "Mozilla/4.0 (compatible; MSIE 6.0;

Windows NT 5.1; SV1; .NET CLR 1.1.4322)" "__utma=1.117971038.1175394730.1175394730.1175394730.1;

__utmb=1; __utmc=1; __utmz=1.1175394730.1.1.utmcid=23|utmgclid=CP-Bssq-oIsCFQMrlAodeUThgA|

utmccn=(not+set)|utmcmd=(not+set)|utmctr=looking+for+site; __utmv=1.Section One"

[GA-Experts.co.uk has a best practice guide on configuring an Apache logfile format ]

For Microsoft IIS, the format (line wrapped) can be as follows:

2007-10-01 01:56:56 68.222.73.77 - - GET /__utm.gif utmn=1395285084&utmsr=1280x1024&utmsa=1280x960

&utmsc=32-bit&utmbs=1280x809&utmul=en-us&utmje=1&utmce=1&utmtz=-0500&utmjv=1.3&utmcn=1&utmr

=http://www.yoursite.com/s/s.dll?spage=search%2Fresultshome1.htm&startdate=01%2F01%2F2010&

man=1&num=10&SearchType=web&string=looking+for+mysite.com&imageField.x=12&imageField.y=6&utmp

=/ 200 878 853 93 - - Mozilla/4.0+(compatible;+MSIE+6.0;+Windows+NT+5.1;+SV1;+.NET+CLR+1.0.3705;

+Media+Center+PC+3.1;+.NET+CLR+1.1.4322) - http://www.yoursite.com/

In both examples, the augmented information applied by the GATC is the addition of utmX name value pairs. This is known as HYBRID data collection – the benefits of which are discussed in the post Software v Page Tags v HYBRIDS .

The benefits explained in detail

1. Greater control over your data
Some organisations simply feel more comfortable having their data sitting physically within their premises and are prepared to invest in the IT resources to do so. Of course you can not simply run this data through an alternative web analytics vendor, as the GATC page tag information will be meaningless to anyone else. However, it does give you the option of passing your data to a 3rd party auditing service such as ABCE . Audit companies are used to verify web site visitor numbers – useful for content publishing sites that sell advertising and therefore wish to validate their rate cards.

NOTE : Be aware that when doing this, protecting end-user privacy (your visitors) is your responsibility and you should be transparent about this in your privacy policy.

(...)
Read the rest of Backup your Google Analytics data and use Urchin (545 words)

© Brian Clifton for Measuring Success, 2007. | Permalink | 12 comments | Add to del.icio.us
Post tags: back up, backup, local copy, software, Urchin, urchin 6, urchin software

Although not as feature rich as Google Analytics, Urchin is essentially the same technology that allows you to view historical data over any time period you have data for, as well as providing complimentary information not available in Google Analytics.

Data available in the latest Urchin beta release that Google Analytics does not have:
(click for screenshots – opens new window)

• Error page/Status code reports
• Bandwidth reports
• Login name reports – standard Apache .htaccess or any authentication that logs usernames in the logfile
• Visitor history report – tracking individual visitors (anonymously)
• Greater customisation – show as much or as little data as you want
• Data is stored locally on your server in a mysql database – allows for ad-hoc data queries

Reporting differences between Google Analytics and Urchin beta software:

• GA imports AdWords cost data with little fuss
  Just just two tick boxes to click and the data is collected daily. For Urchin, this is a manual process
• The GA dashboard is fully customisable
  With up to 12 different reports that can be changed and re-ordered on a per user basis. Urchin summary dashboards are fixed
• GA geographic overlay is best in class
  With zoom and continent/subcontinent breakdown, this GA report is one of the best out there. Urchin geographic overlay is a little more basic (same as GA v1.0), though still very good
• GA is available in 25 languages, Urchin is available in 12
• GA has internal site search reports, Urchin does not
• GA has event tracking (beta), Urchin does not

(...)
Read the rest of What is Urchin 5? (469 words)

© Brian Clifton for Measuring Success, 2007. | Permalink | 7 comments | Add to del.icio.us
Post tags: ABCE, audit, back up, backup, hosted, hydrid, log file, logfiles, software, Urchin, urchin software