Google Analytics actually uses cookies to do the brunt work. Doing a quick and simple check on the timestamp of the cookie vs current time tells them if they are a returning user, etc. If no cookie, they are a new user. After this, they can just increment a number in db for date range.

http://code.google.com/apis/analytics/docs/gaConceptsCookies.html
"Google Analytics uses cookies to define user sessions, as well as to provide a number of key features in the Google Analytics reports. Google Analytics sets or updates cookies only to collect data required for the reports. Additionally, Google Analytics uses only first-party cookies. This means that all cookies set by Google Analytics for your domain send data only to the servers for your domain. This effectively makes Google Analytics cookies the personal property of your website domain, and the data cannot be altered or retrieved by any service on another domain."

http://analytics.blogspot.com/2007/01/absolute-unique-visitors-versus-new.html
"This report classifies visits (sessions) by the visitor type (new or returning). So when somebody visits your site for the first time, the visit is categorized as “Visit from a new visitor.” If this user has browsed your website before, the visit is categorized as “Visit from a returning visitor.”"
Permalink | Report

Google Analytics uses a creative combination of cookie, session and environmental variables to determine Absolute Uniques.

Specifically, the unique ID within the cookie is made up of (1):
Name, which is merely a string identifier
Domain, which is the fully qualified domain of the cookie (e.g. www.example.com, sub.example.com, or sub1.sub.example.com)
Path, which is the root level by default

The cookie lasts a long time. 2 years to be specific:

"Google Analytics sets an expiration date of 2 years for unique visitor tracking. However, if your visitors delete their cookies and revisit your site, then Google Analytics will set new cookies (including new unique visitor cookies) for those visitors." (1)

So, they're not relying on their (albeit massive) databases to calculate Uniques. Rather, they simply tally from the cached that is retrieved from individual clients.

If your site was for sale and it was being professionally valued, the Absolute Unique visitor count could be used as a generality. If the real Absolute Visitor number was required, the Google Analytics data could be used in combination with the raw sever logs. Statistical analysis could result in a magrin of error with both the raw and tertiary count of Uniques.
Source(s):
(1) http://code.google.com/apis/analytics/docs/gaConceptsCookies.html

http://www.stonetemple.com/articles/interview-brett-crosby-011207.shtml

http://code.google.com/apis/analytics/docs/gaTrackingVisitors.html


Permalink | Report

A visitor is defined using a unique numeric identifier stored in the Google Analytics tracking cookies. This value is set when the visitor’s first visit is created.

Each visitor is counted only once in the Absolute Unique Visitor metric, regardless of how many times they return to the site during the reporting period.

While Google Analytics does share the same definition for a return visitor it does not does not count the number of returning unique people (visitors) that have visited the site during the reporting period. GA counts the number of VISITS generated by people coming .

GA identifies a return visitor as any visit generated by a person who’s unique identifier cookie was set prior to the reporting period.

Google Analytics will not track users who have disabled JavaScript and cookies

The number of Absolute Unique Visitors displayed in the Visitors Overview report previously did not match the number of unique visitors in the Absolute Unique Visitors report. This is because the Absolute Unique Visitor report shows data over time and was therefore summing daily unique visitors. The Absolute Unique Visitors report no longer sums the day by day totals but instead displays the absolute unique visitors metric that is displayed in the Visitors Overview.
Source(s):
http://www.epikone.com/blog/2008/09/21/google-analytics-compliance-with-waa...
http://www.google.com/support/googleanalytics/bin/answer.py?hl=en&answe...

Permalink | Report

My guess:

1. Batch calculate page-views/user per day.
2. Divide the total page-views for the requested period with the weighted average page-view/user per day, and return the result.
Permalink | Report

I don't have an answer for this question but agree with mckoss that none of the answers above are satisfactory.

GA reports daily unique visitors which can be easily counted in advanced and stored in the database for later retrieval and display.

To determine the unique visitors for a period of 4 days (for example) you can't just add the unique visitors for each day together as someone who visited on day 1 may have comeback on days 2, 3 or 4.

To do this properly you need to keep a record of every visitor for every day and then query the unique set of visitors across the 4 days of interest. The computational requirements to do this increase exponentially with the number of days and the number of visitors.

GA is able to generate a unique visitor count across any period almost instantaneously which suggests:
a) they are not counting the true unique visitors (maybe an approximation?)
b) they pre-count unique visitors for every combination of start and end date (even if done during pre-processing, this is even more computation than simply calculating the desired period when requested)
c) they have come up with a simply way of counting unique visitors

I would love to know how they do it as well!
Permalink | Report

Hi! Great blog mckoss! I agree, they key point seems to eludes everyone? Did u get any answers? mvh Nick
Permalink | Report

How certain are we that Unique Visitors is even accurate?
I have installed Google Analytics on my development server, which sits behind a NAT firewall. So far two people have hit it from inside the company (me and one other). Yet Analytics reports only 1 Unique Visitor. But two browsers, two operating systems, etc.
In addition to the tracking cookie, it seems Google is also relying on visitor IP address.
I know this doesn't answer the question, but perhaps the question is asking to prove something false? How do we know the world is flat?
Permalink | Report

With Google-sized clusters, this is actually not that difficult to do in real-time on-the-fly. Remember, this is the company that can serve up search results in a few 10s of milliseconds.

client_ids (hash of cookie/IP/User-agent/whatever) can simply be partitioned in a table across multiple BigTable nodes, such that a given client is always on the same, and only one, node. There would also need to be a column-family with a list of dates that the given visitor actually visited. This rows in this distributed table that have entries in the report date range can then be counted in parallel across all nodes. Indexes may also be leveraged to avoid having to scan the entire table each time.

I don't know this for a fact. Don't work for Google, but have solved the same problem myself for a large analytics application.

The other way to do it is to pre-compute all possible date ranges in a nightly batch job. For two year's history you have 365*2 possible ranges to calculate each night, for each site being tracked.

Good luck!
Permalink | Report

I believe I understand the question and while I can't offer 'the answer' I can provide a potential solution.

For unique visitors I believe its fairly easy as you can simply increment a site Id each time a visitor arrives that doesn't have the cookie yet (as inidcated above). If a visitor does have said cookie then an increment of the returned visitor count is made (probably).

Example: (sorry if this doesn't quite look right on you display)
where X=New, O=Returning

M T W Th
X O O
X O O
X
X O O
X

So for a report on Wednedsay (W)

U: 1, R: 2

A report for M-Th

U: 4, R:3 or 0?

I actually think the more interesting metric is the returning visitors as in the example above it could be 3 if you count unique visitors that returned on the next day or 0 if you dont. However you question was about Unique visitors which I think is fairly straightforward.

However would the metic be useful if you did omitt those returning. If you revisit a site outside of a 30min window that's generally considered a new session and thefore a new returning visitor.

As a side note the values in the overview of my site indicate (low traffic) that I have 848 Absolute Unique Visitors, yet the New vs returning metic tells me I have 826 new visitors. I can't see how an abosolute unique can be higher than any other unique count.
Permalink | Report

mkoss,

I am not positive I 100% understand what you are getting at but what about this:

If we can use the cookies (or whatever method you would like) to determine when a new user arrives. We simply create an entry in our new_visitor table with the current timestamp.

# id - auto increment
# arrived_at - unix timestamp
new_visitor(id,arrived_at)

With this table you can write very simple queries to know the number of unique visitors over any timespan you would like:

SELECT COUNT(*) as unique_visitors FROM new_visitor WHERE arrived_at > start_time AND arrived_at < end_time;

Where your timespan is (start_time - end_time)
Permalink | Report

One way to do this (and I've no idea if this is what Google uses or not) is:

Store the data like this:

Date-----VisitorID-------VisitedSite
1st Jan--VisitorA---------True
1st Jan--VisitorB---------True
10thJan-VisitorA----------True
10thJan-VisitorC---------True
1stFeb-VisitorA----------True

Then (assuming True = 1) you need a function that calculates:

Sum (max of VisitedSite for each unique VisitorID)

For whatever time period the query is.

e.g. if your query is for 1st Jan to 31st Jan

Max of VisitedSite for Visitor A = 1
Max of VisitedSite for Visitor B = 1
Max of VisitedSite for Visitor C = 1

Sum of (max of VisitedSite for each unique VisitorID) = 1+1+1 = 3

If your query is for all time then the answer is still 3 because the Max of VisitedSite for Visitor A still = 1.

That at least would give you a feasible way to calculate it on the fly for any time periods.
Permalink | Report