For a non-subscription service like Mahalo, it is much more difficult to predict churn because you are never 100% positive that a user is actually dead. Many analytics services use the measure of monthly active users, but this is dangerously incorrect. A user may be inactive for several months and then suddenly come back.

Without going into too much statistical details, the more appropriate way to do this is to model a user as having a natural rate of usage. Every time he uses Mahalo, he flips a coin with a probability of death. Every user has a different natural rate of usage and different probability of death. This model is called the Beta-geometric/Negative Binomial Distribution model. Paper is sourced. With this model, you can make statements like: given a user's past activity, what is the probability he is dead? What is his expected future behavior? If you are interested, I could take a stab at modeling this.

If you have a subscription service, modeling and forecasting churn becomes MUCH easier because you know whether someone is dead (they cancel the service).

As I understand them, churn and retention can be defined as the following: Churn is the percentage of users who stop using a service after a predefined amount of time. Conversely, retention is the percentage of users who continue using a service after a predefined amount of time.

If I was developing a churn and retention report for Mahalo, the goal of the report would be to quantify existing data, hypothesize why these results have occurred and finally to test my hypothesis with a controlled group of users.

Here is how I would do that:

1)
I would start by carefully defining the retention demarcation date. This is the date that someone either "churns" out, or "retains" their status in the community. This is the single most important variable when considering churn and retention. This date can also serve as a measure of success. The further the date moves away from churn and into retention, the better.

2)
Since Mahalo is a complex system with a variety of different groups of users, I would define one large retention demarcation date and several other minor retention dates to represent the different groups of users. In other online environments, we might call them classes, but here on Mahalo, these would be simple user groups with a list of unique attributes. There are a large number of unique groups of users on Mahalo. While members can exist in multiple groups, defining each group and applying a demarcation date to their involvement in that group would be valuable to understand both internal churn and off-site churn. After accurately defining the groups, attributes and current retention dates for those groups, I could start to develop the actual churn and retention report.

3)
Armed with raw statistical data and classifications, I could start to answer "why". Why does the current churn date exist? Why do users leave? Why do groups of users have the attributes they do and why do they exhibit common behaviors? I would likely combine statistical observation with samples of individual communication. After carefully observing and organizing the numbers, I would randomly select a sample of users who do not contribute any longer. I would survey these users with a standard quantifiable survey and regardless of submission, contact each person individually to conduct a personal survey. Online communities are complicated, personal, organic things and going to the effort of truly understand why people have left can provide a deep level of understanding beyond simple numbers.

4)
The fourth part of my report would be to assemble and quantify both the hard statistical data and the soft survey data that has been gathered. I would develop a model based on my findings and apply that model to a subset of new users. I would measure global churn and retention in contrast to my improved model. If my changes showed a significant improvement, I would implement those changes globally and create a new sample group with further refinements based on my observations and continued retention rate date calculation.

5)
The final component of my report would be a model that could be reproduced. I would define the variables, explain how to uncover and identify groups of users and their attributes, then I would show how to form conclusions based on these inputs to generate a conclusive report based on timeless information. This final step would allow me to continually run this report without a significant investment of time. My goal would be to build a process that would form an entire retention system.

Specific answers to the 4 questions asked:

Is there any open source software for this?
No. There isn't. I searched and there is no software that can accurately define churn or retention. I expected to find some sort of vBulletin (forum software) or Wordpress (blogging software) plugin or service but came up dry. Calculating a raw retention date would be simple programatically, simply define a rate of activity and average the number of users who contribute over x number of days.

Are there any SAAS solutions for this?
I've heard a company called Sysomos (http://www.sysomos.com/) talk a lot about this sort of analytic measurement of social media. Specifically, the CEO Nick Koudas is what I refer to as "crazy smart" and has built tools that can help quantify and measure this type of data.

Any examples of retention and churn reports?
There are a few examples for sale here: http://www.marketresearch.com
However, I think that churn and retention reports are such a business / site specific thing that I would doubt very much would be gained by purchasing one.

Any resources for this topic?
Jason has somewhat unparalleled access to people who have spent a lot of time thinking about this topic. Not only does he have access to the S.F. business crowd, but there are a group of users here on Mahalo who "live and die" in engagement. Talking both externally and very specifically internally about retention may be your largest resource. You would need to wade through the differences that others have experienced and the "good old days" conversations that would ensue, but I have to think that buried under these things are the most valuable resources you have access to.

Simple articles:
http://www.forumone.com/blogs/post/how-keep-users-coming-back-more
http://www.alistapart.com/articles/fromlittlethings/
http://carsonified.com/blog/business/how-to-grow-and-nurture-your-community/

Come check out kontagent analytics for some ideas. We were just on gigaom and TechCrunch yesterday.

Pentaho is an open source suite that does reports and dashboards, I have only used applications from Hyperion, Siebel and Oracle. You would be surprised at what you can create with Excel if you connect it to a database. I have come up with crazy solutions based on what client have lying around.

Robertdo is absolutely correct. I would only add that the problem with any method of calculating churn on the web - either through mining analytics data or enlisting some SAAS solution - is the way in which the data is collected. As the cliche goes - garbage in, garbage out.

One could collect the data necessary to make the model by monitoring activity of registered users (via the CMS) or culling information from an analytics package. Both of these rely on cookies to a varying degree, so the data would have to be adjusted for the probability that someone has deleted their cookies since the last session. Registered users also introduce a selection bias to the sample. This results in a ton of variability and a large margin of error. I learned this the hard way.

In order to collect the most clean data, one would have to track users' activity via a toolbar users install in their browsers (a la Google, Alexa, and Quantcast). But, here you have even more selection bias, not to mention some ethical questions.

When I went through this process, I spent a ton of time building a model that wasn't terribly accurate. So I reevaluated what led me to create the model in the first place. Why track churn rate? For example, if I determine that I have a retention rate of 40% (+/- 8%), is that good? I'm not sure. If I make some adjustments and my retention increases to 43% (+/- 8%), is that better? The data is not actionable.

I reevaluated. My goal is to maximize revenue (for the most part, it is an ad-supported CPC model). Through much trial and error, here are some performance metrics I found helpful (the community is forum-based, but the basics should apply to Mahalo).

There are two categories of posts on my site - helpful and useless.

The helpful posts have the following characteristics: They occur in a subset of the sections - 5 out of 43. They tend to contain a higher quality and quantity of keywords for our search engine optimization efforts. They have a relatively small number of responses (10 vs. 100). The visitors tend to be "new", come from search engines, and view more pages per visit than the new visitors to the other sections. The click through rates on the ads were also substantially higher. Basically, this information was much more useful to the visitor.

The useless posts have the following characteristics: They constitute 90% of the content created and pages viewed. They are created by 1% of the users who view 200+ pages a month (this is the limit of what Google tracks). They necessitate heavy moderating. The bounce rates for new visitors are much higher than that of other sections. The click through rates are near zero. (I think they only click on ads accidentally.) In essence, these are people chatting with their friends and making posts like, "ok".

So, in order to minimize the bad and encourage the good, I created incentives for users to make valid posts in a few chosen sections. I use two types of conversions to track success. First, when a new post is made in those sections, and second, when the 10th comment is made. All of these metrics directly measure the efficacy of incentives given to make good content. Good incentives = quality posts = better search rankings = more new visitors (engaged with content and ads) = higher likelihood to post. Volume is less important.

Additionally, I track: bounce rate for search entries to the sections (from those whose query did not match the URL) and the number of users who made their first post in these sections. In Mahalo's case this would be "questions". Based on the data, these are the users that provide the most value to the community.

This is all a long way of saying, I thought retention rate was the best way to measure the development of my community. But I found that other, more actionable, more easily measured factors encouraged growth… and the retention followed.

a side note, I applied "forced churn" by eliminating trolls that were responsible for a good deal of the useless posts. As a result, the amount of time spent moderating dropped, and users who had not posted in a long time began to post again.

Mahalo is unusual in that you own a social community rather than playing on another platform which many tools are designed for. With Digg and Yahoo I've noticed that in the past they've pushed polls on their user base on a semi-regular basis. I wish i had done screen caps of their surveys, but they're usually a mix of gathering quantitative (i.e. rate this feature) and qualitative (i.e. you tell us what's wrong) data. This is an intrusive method but it can give some insight without writing custom software. By the way I notice that on most of those surveys they always delve into how many hours you spent on the net vs. how much time you spend with their service.

Side note: If you run a Facebook page you'll notice that they give owners access to "Insights" which seems to be a glimpse into their own stats software. They do a great job of measuring community interactions — you might take a look at this as an inspiration if you're writing custom software. I've also noticed that YouTube and Flickr offer similar glimpses — but the Facebook insights data is much better as they're aimed at a b2b audience rather than consumer.