How would you make Mahalo's pages load faster?
http://bit.ly/mahalospeed
This page "feels" absurdly fast when I load it, but it's gotten slower over the year according to pingdom:
http://www.mahalo.com/stem-cell-research
... and let me know how you would make things faster. Best answer, M$100!
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$40 Answers
According to this, Mahalo has made a significant improvement. Congratulations!
Here are some things that I would do:
1)
If Mahalo doesn't already run on an SQL alternative, I would consider moving. Something like Cassandra may work particularly well to increase overall speed.
2)
Next, I would carefully evaluate Mahalo's CDN (content delivery network). Often, speed can be increased by tailoring a CDN specifically to the traffic that is causing bottlenecks. If you're looking to maximize your speed, I would consider building your own CDN and leasing unused time back to smaller companies. A CCIE on staff (or contracted) with full control over your CDN could set up some wicked awesome routing that can truly change the game as far as bandwidth is concerned. Also a quick review of the DNS setup with proper configs can speed up the initial page load which is what most people judge speed on.
3)
Next, I would look at exactly how pages are cached and served. I would carefully review the routing configs, low level (server) hardware and every piece of software that contributes to the distribution of a every page on Mahalo. If you didn't have it already, this would be part of a "topology" project where I would map each and every device and route so that I could visualize and identify problems before they occurred. Reviewing both upstream and downstream bandwidth routes and reviewing existing bw commitments would be an important part of this step.
4)
Then, I would turn to design. I would cut back (way back) on the number of external program calls. I would carefully optimize the CSS. I would even go so far as to ensure that every sprite map and every image was optimized to the "point of no return" so that the content itself didn't cause a problem. It looks like M3 has already implemented a plan to streamline SERP's, but I think that deeper cuts and self hosted alternatives would provide an even more significant gain.
5)
Last, I would develop a clear and regular maintenance schedule, implement a real time alarm system (with limits for speed alarms), and ensure that every "staff" wheel was greased and completely understood every nut and bolt so that my ship would run like a Ferrari on the track. This is why I name my servers after race cars :)
So, that's a general overview of what I would do. This is a very difficult question to answer specifically because every (every single) setup is different. However, if these 5 pieces of the web world puzzle all work well and are managed well speed problems should be seen well in advance. In Mahalo's short history, it has grown so quickly that these types of fundamental actives would have been difficult to justify and budget. I think companies live and die in this sort of work.
I've been employed to do this sort of thing.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
B) Set up a second server to serve all static content (js, images, css, etc) and serve it cookieless (request headers add a little bit to each request). The example being that there is no reason I should be sending my cookies to content.mahalo.com. You need to buy a completely separate domain to handle this. Maybe like mahalo-static.com or some such.
C) I assume that a site as large as yours must have caching in place. Maybe you can be a bit more aggressive in your caching strategy. Talk to your developers.
D) Work with your ad providers. Some of the slowest loading things on your page are your advertising content. Mahalo is a pretty big site now. You might be able to throw some weight behind a complaint to Google. Any performance increases you can get benefits everyone. :D
E) I would say optimize your SQL, but I don't think you can be where you're at without already having pretty awesome SQL.
It loads fairly quick for me as well, and it never feels slow. So keep up the good work.
I think combining resources (listed as A and B) is really your best bet.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
The largest source of latency comes from third party uncompressed and uncached images coming from Flickr and lacortedelraydelpop.com. I used AOL Page Tests' http://www.webpagetest.org and Keynote's MyKeynote to help measure this. See the graphs at the bottom.
The long story:
By way of introduction, I’m the co-author of Complete Web Monitoring, which I wrote with Alistair Croll (@acroll). We’ve spent a lot of time looking at these types of problems.
Let’s begin:
If you go around tweaking things to make your site faster without a strategy, then you may wind up spending a whole bunch of time & effort on stuff without it having a noticeable impact on your business’ bottom line. To avoid performance tuning without wasting your time, there are a few steps that you should follow:
Figure out what you can fix (performance audit)
Prioritize what you can fix
Measure the impact of the fix
Figure out what you can fix:
First, figure out what hurts the most. To understand where things get slow, we need to use a cascade diagram of performance. These tools track every single connection that happens when you load a site and tell you how long each one takes by allowing you to visualize it through a cascade diagram.
In this case, let’s use AOL Page Tests’ utility (http://www.webpagetest.org) to measure Mahalo.com
The results can be found here: http://www.webpagetest.org/result/091127_38T2/
This first attached graph tells me that 27 connections are made every time mahalo.com is loaded. Calls to flickr.com are adding a whole bunch of load time on mahalo.com’s main page. You may want to create a script that locally cache images instead of calling them from a third party site.
Furthermore, you should invest in time / technology to make sure your images are compressed, reducing overall page load time.
The problem is that AOL Page Test is looking at what’s slow right away, instead of “on average, what’s the most slow”. To do this on average, you’ll need to use a more sophisticated tool.
Vendors in the space include: Webmetrics, Keynote, Gomez & Alertsite.
Prioritize what to fix:
I’m not going to spend a lot of time here, but it’s important to mention that you need to consider the cost of the varying forms of optimization. In other words, it may take 6 months of development time to make sure that you have compression and spriting in order, which could end up being more expensive than purchasing an appliance or CDN that can do this for you.
Make sure you take the time to consider how much your optimization efforts will cost so that you don’t end up raiding windmills.
Measure what you’ve fixed:
Also, find out how fast is “fast enough”. You can make your site load faster than light, but on average, it may turn out that Mahalo users may not care about performance increases as long as pages load faster than 2 seconds for them.
You need to figure out “how does web performance correlate to our business goals”. A business goal can be things like “adding an answer”, “reading an answer”, “commenting on a post” and so on.
To do this, split test your traffic, and make the site faster for a small percentage of your users. Drop a cookie in their browsers saying “I’m optimized!”. Then, use Google Analytics to compare your conversions (ad clicks, pages surfed, comments posted, etc) for the unoptimize segments and the optimized ones.
We’ve seen businesses do this by using web acceleration devices like Strangeloop Networks, Akamai or Limelight on 50% of their traffic, then using Google Analytics to track the impact on conversions as a result.
You can read more about this here (http://bit.ly/7ikPAr) with associated PDF here (http://bit.ly/5mrqLl).
www.webpagetest.org, www.watchingwebsites.com
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
This may also sound crazy but just to add to that.
Speed might not necessarily be the absolute BEST thing ALL the time, fluidity however might be more of importance.
Getting an idea of what Mahalo's user agent share looks like would be great.
I could imagine however atleast 35-50% of all mahalo's most active users use some form of the Gecko engine based version of firefox or some other derivative.
However the "noobs" (sorry to the noobs BTW) , might still be using IE or IE7/8.
Also keep in mind not everyone has a ultrafast connection, some places up north here still use dial-up. My 85yr old neighbor refuses to switch and still uses AOHell. :(
On the client side there's plenty that can be done to improve load times and user experience. Some have already been covered by other user's answers but I'll address them anyway.
1. Reduce requests
Between CSS, Javascript, and images Mahalo has a high number of server requests that can be significantly decreased. I'm seeing 19 Javascript files, 3 CSS files, and a varying number of images. Most of the Javascript files can be combined since they are loaded on all of the pages. An alternative would be to use a loader like LABjs (http://labjs.com/) to use non-blocking JS loading. The CSS files can also be combined. The site is already using sprites for its standard images so there probably isn't a lot that can be done to reduce the number of image requests.
2. Optimize Javascript usage
There is Javascript scattered throughout the pages with a large number of files included in the head. Much of this can be shifted to the end of the document to decrease page rendering time. All of your tracking JS can be moved to the end (minus the date at the very top of ).
3. Minify JS and CSS
It looks like most of your JS is already minified but your CSS is not. Minifying it will reduce the bandwidth required.
4. Reduce DOM objects and strip out HTML comments
Mahalo's HTML is pretty heavy. The more DOM objects the longer it will take for the page to render. Have your developers look through the DOM and remove unnecessary divs and other elements.
5. Replace eTags with Expires headers
This will show a BIG improvement for full cache page loads. None of Mahalo's static content uses Expires headers. By adding far-future Expires headers a page load with full cache will require a lot fewer requests and will load much faster.
6. Use multiple domains for static content
Use multiple domains (eg static1.mahalo.com, static2.mahalo.com) for static content. This will allow some web browsers to increase the number of parallel requests and download static content faster.
7. Get rid of unnecessary meta elements
All pages have Bing and Google verification meta elements. These can be removed and replaced with their verification files. It will reduce page weight.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$There are 18 JavaScript files served from content.mahalo.com. They should be combined into as few files as possible.
* /content/scripts/jquery-1.3.2.min.js
* /content/scripts/jquery-ui-1.7.1.custom.min.js
* mahalo3.js?v20091124
* /content/scripts/jquery.simplemodal-1.2.2.pack.js
* /content/scripts/jquery.autogrow.update.js
* /content/scripts/jquery.treeview.pack.js
* /content/scripts/jquery.rating.pack.js
* /content/scripts/jquery.hoverintent.min.js
* /content/scripts/jquery.cycle.all.min.js
* /content/scripts/jquery.bookmark.js
* jquery.ajaxupload.js
* /content/skins/mahalo3/scripts/jquery.htca.js
* /content/skins/mahalo3/scripts/jquery.metadata.js
* jquery.checkbox.min.js
* mahalo3-serp-common.js?v20091124
* mahalo3-serp-sheriff.js?v20091124
* /content/scripts/sheriff-0.1.js?v20091124
* /content/scripts/chartbeat/chartbeat.js?v20091124
via FireBug
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
I have just a couple more things to add:
a) before dumping EdgeCast, try setting up a custom domain (not sub-domain). Several Mahalo cookies (with a domain of .mahalo.com) are being sent to content.mahalo.com on each request, which aren't needed.
b) try using Google's Ajax Libraries API for the first piece of JavaScript (jquery-1.3.2.min.js). If the person has already visited a site that uses Google's Libraries API, then it will already have been cached in their browser.
I wouldn't just ditch EC right away, I would go to them and say...
Hey look
... this is whats going on.. if you want to keep our business.
..............This is what needs to change.
And then show them this question =)
Also you might want to dump EdgeCast and go with Akamai as your CDN. "Akamai is a Hawaiian word meaning smart or intelligent." So if you put Akamai and Mahalo together you have a website that is smart & polite at the same time. How do you like dem apples? ;)
The person doing the best research in this area right now is Google's Steve Souders (http://stevesouders.com). Here are the slides to a talk he gave about the topic at SXSW last year: http://www.slideshare.net/souders/sxsw-even-faster-web-sites
1.) Part of the build process should include collapsing JS files into one file, minifying them and so exposing only one, much smaller file to the browser. This can be done per group of pages that share the same code (search result page, front page, answers page whatever) ... For frameworks like Django, there are packages that do this for you (http://code.google.com/p/django-compress/) ... for custom codebases, it will require some work but is worth it.
2. ) The same should be done for css files.
3. ) Once JS and CSS files are auto generated per build, meaning a differently name file is generated per build, they should be sent by the server using "Far Futures Headers" which will ensure browsers will cache the files forever essentially helping repeat users not have to re-download JS/CSS.
4. ) If collapsing JS and CSS files is just too much work to implement onto your codebase then initially, at the absolute minimum, you should be minifying every JS and CSS file during a build. Server side tools exist to do this and should be incorporated into your build process (http://www.crockford.com/javascript/jsmin.html for JS, http://csstidy.sourceforge.net/ for CSS)
5.) It appears Mahalo is not compressing JS and CSS files right now. All JS and CSS should be GZIPed by the web server before being sent to the client. (This is on top of minifying). An exception should be put in for IE6 (at the server level based on User-Agent string) as it does not decompress JS file and execute them correctly for versions before XP SP2.
6.) It looks like Mahalo appends a lot of bits of JS at the end of the HTML for various services like Quantcast, ChartBeat, etc... In a JS driven site where certain actions may be triggered on page load, these scripts are not "free"! If they must be there, they must be included in such a way as to be loaded asynchronously so they don't block page load. There is alot more info in Steve Souders slides about how to do this, but, for example, here is one blog post about how to do this for Google Analytics (http://lyncd.com/2009/03/better-google-analytics-javascript/) ... The same basic method can be adapted to Quantcast, ChartBeat, and other services that require JS to be included in page.
...
Looking at the backend, not knowing any of the existing infrastructure, I would say, in general:
1.) Cache, cache and cache some more. Every where, in every layer of your service.
2.) Move away from relational databases where they don't make sense. If you're not doing join, than a "NoSQL" database can be orders of magnitude faster and easier to scale. Not knowing any of your infrastructure, it would be hard to make a recommendation as to exactly which one to use.
...
So that's the pretty basic stuff that should be part of bast practices for any large web service. Beyond that you're going to have to get pretty fancy.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$the images on the home page are your biggest problem - make the page load, then pull in the images behind the captions via a quick jquery ajax request instead of loading on document load - this will increase initial load time and not slow you down so much due to outside sites hosting images.
also - the fact that you jumped from low to way higher very shortly after launching 2.0 could point to another issue - it's possible pingdom changed something, or their algorithm figured out that your page is loading more content as it cycles through the results in the top section of the page. because of this, the point in time in the load where they stop measuring and declare the page "loaded" could have changed even though it's way past a true "loaded" state.
use the debugging tools in webkit browsers (safari, chrome) to look at load times for each individual file, as well as sizes of your external assets (js, css, img) to find easy "get this crap outta here" ways to reduce load time. it's likely that your backend architecture would result in much lower increases in performance, so i'd stick with these client side files first since they're often the most overlooked these days when developers are spending most of their time in the backend and designers to much of the front end dev.
also, it seems that many people are recommending other plugins or things to insert into the page to make it load faster - i'm not a fan, simply because it's another thing that can 1) break and 2) you have to keep updating and might not use in the future. web standards are standards for a reason, and usually going the simple yet creative route will yield the best possible page!
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
Just thinking out loud here:
Instead of loading Facebook Connect for every page request, why not only load it on the login page? Instead of displaying the FB connect button up in the top-right of every page, why not just include a link (including mention of Facebook Connect... maybe the FB connect image, but just linking) to the login page where the user can login with FB connect if they wish?
1) Defer all non-essential javascript loading until the very bottom of the page. By non-essential, I mean anything that won't cause an error if a user clicks a button etc before it's loaded.
Loading javascript source files blocks the browser from downloading anything else simultaneously. If you have a stack of them in the middle or start of the page, it's preventing other things from loading.
2) Minify all CSS and javascript files using something like this: http://developer.yahoo.com/yui/compressor/
3) Use CSS sprites (I think you're already doing this judging by most of your buttons).
4) Use a separate content server with far-future expires headers, no cookies, and gzip compression.
5) Optimise images
6) Follow this guide: http://developer.yahoo.com/performance/rules.html
If you're testing using firefox, install the Yslow and google's pagespeed plugins which will give you plenty of other suggestions.
From a UX perspective: One solution would be to split large content pages into separate sections where the user has to click a "read more" section to download the additional content via ajax or a similar mechanism.
http://developer.yahoo.com/performance/rules.html
firefox firebug, yslow, pagespeed plugins
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$I guess you are worried because many are complaining on slower computers mahalo is very tough to load. I don't have much difficulty on my end but im on a cable connection with a pretty quick computer.
Lets get to what is needed to lower this load time.
1st. While it is great to start buying new servers and such to lower the page load time, it is most probably all dependent upon how the coding is setup, possibly duplicate query's completing the same tasks.
2. I would take all your major pages being used like index.php and shrink them using html shrinker light. It is a free program and can reduce KB of your site dramatically.
3. As XDS stated start using sprites, it is personally used on my site shoptimesaver.com and I can say from experience our page load time has decreased by 50% using sprites.
4. Consider removing pages from the Data Base that are pointless, things like the sex pages in which take up much of the DB, most probably slowing your search down dramatically.
I have 2 expert programmers and 1 expert designer if needed that can easily incorporate sprites into the site and optimize the DB level if needed.
Take my advice, don't up on your servers or change anything on the server level, its not necessary to throw hardware at bad coding. Just get a great coder and they should be able to optimize this site so it runs 3 x as fast if not more.
If you need more info or would like to speak with my guys let me know.
Thanks, Mike !
Personal Experience on my site shoptimesaver.com
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
I also wanted to note as I was seeing some other comments, i want to second the motion on removing the quantcast code as it lags the site some when you get to the end of the loading. Quantcast is sort of unnecessary I believe, im not even sure why you would want your information out there like that, I know some of the big names are doing it but it just slows down the site so it's not worth it.
Using YSlow on the homepage yields the following simple server to browser fixes.
- reduce the number of requests
- There are 13 Javascript includes. Combine them into one file
- There are 3 stylesheets. Combine those into one file.
- There are 55 background images. Combine them into one file and use CSS sprites
- Use a CDN like Akamai or CacheFly.
- There are no expires headers. This is HUGE! Simply by adding a expiry years out into the future and then making your static files have date and time stamp arguments like ?20091127 you can have them cached in the browser forever speeding up subsequent requests
- A few components aren't being compressed (gzip)
- Not all JS and CSS have been minified
Those should help but as I mentioned you'd need to specify the backend technology.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$First and foremost.
& I see you are already doing this for parts of the main page but...
Sprites Sprites Sprites...
#1 >>>Make the entire page just one giant sprite.<<<<
Here's an example of a page sprite for those of you who don't know what i'm talking about.
Have all the pages load the same combined image, then pre-cache it.
Your developers should be able to find a way to have the users machine load the sprite continuously for each newly clicked page.
#2 Standardized thumbnails.
Find what the most common size is for graphics and use those instead of your 90x90's which you currently have there.
#3 Load JS and Styles LAST! (css before JS however)
For the life of me I can't understand why mahalo pages always insist on loading the java script with the pages as they are loading ?
Once a user already has a page cached in their temporary internet files that pretty much takes the place of what they necessarily "see" from here on out.
The first page might be like giving birth for the first time but everything after is like riding a bike.
#4 Ditch the meta tags/elements.
With a big site like this the major search engines don't really need to pay close attention to that stuff and it will only cause stopage.
MOST IMPORTANT
#5 Increase your ceiling.
I've noticed that in the early hours PDT and Mid day EST then right after the kids come home from school and when everyone first jumps on their system to check their Mail / Mahalo / Mahalo DQ's and everyone starts pounding the site comes to a screeching halt.
The site must be run on some type of scaling server. This isn't necessarily the best fit for a startup, you have issues with usage clumping.
Ask the devs to give mahalo's resources in some places a little more leeway or...
**************************************************************************************
Provide us a premium version of mahalo that we can use our mahalo dollars to buy every month that is ad free on our side as well as faster and more robust.
***************************************************************************************
Also if I may be so bold as to refer you to a "Squid CDN"
These guys ...... CacheFly.com setup delivery servers all over the world for hosting content if mahalo could partner with them it would make you guys unstoppable!
_
_
If you run your own in house servers/services some other things I would suggest......
#1 Get on a bigger badder backbone.
Get in touch with whoever controls the "pipes" coming in and out of the Mahalo NOC and ask them if they have any developing technologies that they can work with you on getting implemented their.
When the NECC (The non-profit I work for) moved into the North East cox was doing our net and services, they had many different options, but I insisted to meet with the regional product manager.
He sat down with me and had a long meeting, we discussed our options and he filled me in on everything cox would be beta testing that year.
Our north east offices where the first to get DS3.
And then later transitioned to fiber. If your serving over wireless get in touch with the guys over at RuckusWireless or the guys over at Xirus.
Either of which can help you widen your pipes for the escalating user base.
#3 Switch to linux from Windows IS/Windows Server
If you haven't already, my god...... WHY
Not only is linux open source
(so you can run anything your heart desires on it)
But its free, its far more secure if you know what you are doing, and can be allot more pleasant to use all around.
#4 Solid state media
Checkout solid state media and run front end graphics such as sprites and landing pages on it.
I would take a look into what crucial and OCZ are putting out!, the numbers are just insane.
...... PCI-E 2.0 drives with un heard of 1000/750 read/write speeds.
Checkout this video on the fusion io. VERY INFORMATIVE !
http://www.youtube.com/watch?v=VS9Zxnx-r0E
Sorry to say it but If you haven't found the solution by now then you simply need to start throwing some money at it. $$$
In other words yeah ....Upgrade the boxes.
Kind Regards,
@XDS
CacheFly - http://www.CacheFly.com
Mozilla Labs Sprite http://www.Mozilla.org
And whats left of my brain this month from all the monotone.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
http://code.google.com/speed/
You can also use either YSlow or Cuzillion.
YSlow
http://developer.yahoo.com/yslow/
Cuzillion
http://www.stevesouders.com/blog/2008/04/25/cuzillion/
Be sure to watch the videos below:
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
I'm guessing most of the items on the page are gen'd dynamically. Perhaps take a look at the widgets that are used and pre-cache them on an interval as static HTML. Very un-datadriven-web20 but, extra calls to amazon, youtube etc could be mitigated.
As an example: You know the stem-cell page is a high traffic, high latency page. Snatch the amazon stuff on n interval, write it to disk, rinse and repeat. Then pull the content locally. Perhaps, set expiration on it, and let the code dynamically re-cache it with the first request and update the file, It would manage itself almost, and save some bandwidth and hedge against the latency. In this case, you pull the amazon book list dynamically once, then serve it up repeatedly in low overhead html to the next 10,000 views, then update it at your set expiration.
Also, look internally at other data results that is low volatility and do the same (like the reference data etc...)
You get NEAR real-time results and the content quality is preserved along with saving a few bits on the wire.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
My friend surfs the net on dial up and I know the mahalo pages really are slow for her. I am on cable so Its speedy for me most of the time.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
-quote-
"The solution turned out the be simple, combine all the different files into a single large file and compress that file using gzip.
Instead of going for the easy – but hard to maintain – solution I decided to automate the process and thanks to a small PHP script and some clever URL rewriting I now have an easy to maintain method to speed up the loading of pages that use many or large css and javascript files.
The idea is that you have one directory for css files and one directory for javascript files on your server. If you rewrite the URLs that point to these directories to a small script that intercepts the requests for those files. The script loads the file from disk and compresses it using gzip.
You can concatenate different files by simply adding the names of the other files to the URL of the first file.
Take for example the following URLs:
http://www.creatype.nl/javascript/prototype.js
http://www.creatype.nl/javascript/builder.js
http://www.creatype.nl/javascript/effects.js
http://www.creatype.nl/javascript/dragdrop.js
http://www.creatype.nl/javascript/slider.js
You can combine all these files to a single file by simply changing the URL to:
http://www.creatype.nl/javascript/prototype.js,builder.js,effects.js,dragdrop.js,slider.js
Compressing those files takes some time and on a busy server that time will become large enough to negate a significant portion of the improvements you made earlier. But this problem can also be solved by simply adding a cache that stores an already combined and compressed version of the files. The cached version is automatically created the first time that particular combination of files is used and used every time – as long as the files are not changed. The result is that once the cache is created there is almost no overhead and the compressed file is delivered almost instantly.
First you need to download and configure the combine.php script. By default this script look in the javascript and css directory in the root of your website, but if you are currently using different directories you can change these values at the top of the combine.php script. Upload the combine.php script to the root of your website. Secondly you need to create a cache directory that is writable by the web server. Again, by default this script will look for the cache directory in the root of the website, but you can change this in the combine.php script. Finally you need to create or modify your .htaccess file. If you do not have a .htaccess file you can create it in the root of your website and add the following lines. If you already have an preexisting .htaccess file you can simply add the following lines to the file:
RewriteEngine On
RewriteBase /
RewriteRule ^css/(.*\.css) /combine.php?type=css&files=$1
RewriteRule ^javascript/(.*\.js) /combine.php?type=javascript&files=$1"
-end of quote-
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
1. Getting a high-speed server to host your site is a must. I think this is the first thing that big sites must do. In order for them to cater services to thousands or even millions of viewers, first, your host must be able to handle this many accessing users at the same time.
2. High Quality graphics and flash animations makes the site loads slower. If it's possible, avoid too big images or too heavy images. Flash sites isn't also advisable for big sites.
3. In CSS, it's more ideal to use external style sheets than the ones embedded on the same page. External CSS doesn't have to load everything on the page, it just takes what characteristics the classes must have for that page. Compared to the internal CSS where everything is loaded even the things you don't need for that page.
4. Creat efficient SQL statements. Ugly SQL makes accessing database slower. If your statements are more direct, results for a particular query will show faster.
5. Having indexed tables will also make faster access on the db. Instead of querying a table with thousand records, it will just search on the index where recently queried items are stored.
6. Database clean up... Maybe there are some unnecessary data on your tables that you don't need at all which sometimes are still being loaded on the page. To make the results on a certain page appears faster (without these unimportant data), deleting these items will help. Umm... I think the foolish questions around Mahalo lately must be removed right away. It would not only help on the community, but it will also lighten the load of the db.
I think that's all I know. I hope it helps somehow. :)
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$Assuming the part you care about is the pingdom graph data, then this is what I would do:
I assume Mahalo may have a lot of these in place already
- @mshubin said "For non logged in users a full page is served from memcache through nginx" - there is something wrong there if the page is still taking 2 seconds to load. One thing I would do is put a small, completely static page in the monitoring, which should load instantly on the server side and see how long pingdom takes to fetch that, since there may be a network issue or probably there is still some SQL in there.
- Put a timer on every SQL query on the page.
- Run the slowest queries again with EXPLAIN EXTENDED prefixed to them, look at the indexes being used and check that the query is using the index it was designed for (all queries should be designed with a particular index in mind)
- Setup a mysql slow query log and look through it for slow queries and again run EXPLAIN EXTENDED on it
- Increase the use of memcache so that it covers more queries if possible.
- look at using SSD disks in your database servers, a Ycombinator startup, ReThinkDB is doing some interesting work with that. http://www.rethinkdb.com/
- put more memory in your database servers, and tune to make use of it
- put more memory in the memcache cluster
- Talk to Sun Microsystems who give free help to startups under their startup essentials program
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$What feels "Absurdly" fast to you, in theory is NOT absurdly fast as well, even cached the page serves in just under 12 seconds. You are loading almost 1.2MB of data, I suggest again, you have somebody comb over your site and give suggestions on reduction, optimization and tips on how to fix this issue.
The easiest tip would be to implement local caching with smart Keep-Alive, and image size reduction through caching, AKA cache the image locally for an hour, resize it using PHP and serve it using a webserver that works well with fragmented file systems.
I do this for a living.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
Perhaps a small "Low Resolution" link somewhere at the top of each page that just toggles all image and video content into links. (then clicking "High Resolution" returns everything back to normal). Hopefully, the user's selection (low or high) would also carry over when viewing other pages.
This would help with users who are still stuck in dial up land as well as give a "Ridiculous Speed" option to those of us with fast connections.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
Speaking of speed, I just found out that Google is about to inject a new ranking factor into the algorithm – "site speed." check link below.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$http://www.aptimize.com
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
There are several things you can do that would give you immediate results..
The first one would be to compress your pages. The page in question for me takes around 500ms on your server then 1.2seconds for me to download, compression should improve the download times by around 70%.
Including all page components this page is around 880KB of which this includes many css, javascript and images. I notice that the jQuery javascript library you are using is minified, but you have not minified the other javascript files.
You have 28 components on this page.. try and merge your javascript and css into less number of files. try and move your images into sprites.
Add expire headers to your images js and css. ie google puts a 12 month or more expires on their images.
Regards.
Jason
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
Congratulations, no more login problems and pages load up instantly.
My one complaint with Mahalo 3.0 is unrelated to speed. I'm not sure what is accomplished by pushing featured photos and/or videos down close to the bottom of the page. Some pages benefit from a picture being at the top of the page. We live in a visual world!
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$2. If possible, reduce latency by having the closest server to the IP serve the content.
3. Create better organized database tables, distribute them into smaller relevant tables to reduce query time.
If you want more of an explanation, let me know.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
APC (advanced PHP Cache)
MCache
Which CDN do you use?
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
0) Contact Web Performance, Inc for a trial to run internally for baseline
1) Contact Keynote and get a trial to run externally
2) Contact Gomze and get a trial to run externally
3) Contact Webmetrics and get a trial to run externally
Ignore the sales agenda for Keynote, Gomez, and Webmetrics but consider keeping a license for Load Tester w/ASA as part of ongoing unit testing internally and for any issue or complaint pages.
Determine which of the these (1-3) your team actually likes as an external service and feels comfortable with as a published team metric tied to their yearly bonus calculation.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
For people who aren't logged in, you can just serve a full page cache most of the time (so practically zero execution time). For people who are logged in, it gets harder, but you can still cache most parts of the page, like for example on the homepage: top users, most popular tags, etc.
And anything which isn't easily cacheable: optimise, optimise, optimise. If Mahalo uses a framework (Ruby on Rails, Django, ect.), then you'll most likely find tons of helper method SQL queries (which are inefficient), which you can rewrite in raw SQL and get some quick improvement. You could also combine the many Javascript files Mahalo uses into one file, and let Google host the primary jQuery library, which will slightly help speed things up in certain browsers.
And finally, some extra or more powerful hardware can never hurt. :)
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
2) Use a content delivery network, such as Akamai for the static assets on the site. Though, as is, it appears you're hotlinking directly from google the lead-in image on the first paragraph. You're kind of at their mercy for how fast they want to serve that image.
3) I can't tell if pingdom is measuring Time To Render or just Time To Receive the whole page. Either way, one of the things slowing down rendering is your citations code. That javascript looks pretty chunky as it's searching the source of the HTML doc for cites and then building a list of what to show and hide. Calculate that from the backend instead and just have jscript show/hide those citation elements, instead of recalcing clientside *every* time when it's shown or hidden.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$Move JS to just before , where possible. By having scripts in the tag as you do now, slow-loading scripts can block the page rendering. In absolute terms the page should load at the same speed, but the browser will render it much faster so it will "feel" faster.
Compress your JS libraries into one file, or at least as few as possible. I see jQuery Core and three or four libraries in the header. Put them into a single file - preferably minified (not packed - you get smaller files, but the browser has to do a lot of extra work. Also, often times, minified+server gzip is smaller than packed as well). You'll cut out several HTTP requests which should help quite a bit.
Note - same applies for stylesheets, although it seems like it's not a problem here. Ideally, there would be a single external CSS and JS file, though that's not really practical. But one each main CSS, IE fixes CSS, JS libraries, and JS scripts isn't impossible or even too difficult. For page-specific scripts, having them inline at the bottom of the page is fine.
If you haven't already, try to use CSS sprites for images wherever it makes sense - gradients, icons, etc. This also helps remove unnecessary HTTP requests, among other things. On a similar note, check file size for gif/jpg/png - see which provides the smallest size for your requirements. There are also third-party compressor tools which can often knock another 20% or more off of file size regardless of file type and without loss of image quality. Seach around as I don't remember any URLs off the top of my head.
Analyze the page with Yahoo's YSlow or Google's Page Speed firefox extension. It gives lots of tips about server-side stuff with regards to caching, CDNs, and plenty of other areas. See: http://code.google.com/speed/articles/identifying-page-speed-problems.html
In short - minimize the number of HTTP requests, reduce file size, and make browsers cache whatever they can. That doesn't cover everything, but should follow the 80/20 rule nicely.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$Grade F on Make fewer HTTP requests
This page has 28 external Javascript scripts. Try combining them into one.
This page has 4 external stylesheets. Try combining them into one.
This page has 43 external background images. Try combining them with CSS sprites.
Decreasing the number of components on a page reduces the number of HTTP requests required to render the page, resulting in faster page loads. Some ways to reduce the number of components include: combine files, combine multiple scripts into one script, combine multiple CSS files into one style sheet, and use CSS Sprites and image maps.
ALSO: (not copy-paste)
Get your DBA to show you the average SQL execution plan for a page load - see what the cost is from the db side (I'm assuming chunks are served from your back-end database here obviously) - they should be able to explain themselves if there are any table-scans for example, indicating indexing optimisation is required. - also if a high percentage of your content (the sample page on stem-cell research for example) is not really dynamic (the article itself, and the references etc) , this should perhaps be cached rather then built in real-time from the DB (guessing at your site architecture)
without understanding your site architecture its difficult to help much further, but essentially your server admins should be able to prove you dont have a bottleneck server in the chain somewhere currently before you go making major changes.
HTH
Mart.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$From looking at the returned source of some of the mahalo pages, I can say that using more than one content server (eg: content1.mahalo.com, content2.mahalo.com, etc) and placing your scripts at the bottom (you currently have 16 external scripts before the page's content) should increase perceived and actual load times.
Without looking more deeply into redirects and caching, these would be some great starting points. In addition, you could add a "?^last-updated-timestamp^" to the end of css href's and js src's. When combined with far-field expiration dates on these assets, you can essentially tell most browsers to never re-download your assets unless they've been updated.
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
Also, to automatically figure out CSS sprites for any page, try http://spriteme.org/
That stuff is magic.
1. You've got an unmatched at line 121, and an unmatched at line 123. the mainpage document load time is 780ms for me.
2. under all the content.mahalo.com/content/ items, stray away from all the .PNG's, those have a pretty hefty load time, not to mention relatively not needed anymore with HTML5 canvas and CSS3.
3. Why are images being hosted on flicker? That's a huge page load time burden...
4. redo most of the legacy html in HTML5, drop most of the div id fields. for this type of splash page, you shouldn't need 2411 lines of code...
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$This question noticeably slowed down Mahalo today with everyone rubbernecking it...
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
-.- good observation, but it won't get you a helpful vote :P
This question should probably go to Digg as a 'believe it or not' "Man gets $10,000 in consultant time for $100!"
Make it so, Number 2!
http://digg.com/tech_news/Mahalo_CEO_gets_10_000_of_consultant_time_for_100
Now you are just being ridiculous :o) ;o)
Breaking news: Google has a website to help speed up your website: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=158541
Step 2 - Clear "History" in your browser. Frequently.
Step 3 - Restart computer.
Step 4 - Buy more RAM
I have yet to buy more RAM or get a bigger and better Mac from Santa, but I use the top 3 techniques almost daily, whenever the Internet gets sluggish and presto! Zippy! It's not just Mahalo, it's any media-heavy URL.
www.mahalo.com
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
Also, strip out as many SQL joins as you can. Joins are expensive.
The StackOverflow team explains this well in their podcasts.
http://highscalability.com/stack-overflow-architecture
You can leave an optional "tip" with Mahalo's virtual currency, Mahalo Dollars. If you are asking a difficult question that might require some research, or if you'd like a wide variety of feedback, a higher tip often leads to more answers to your question.
M$
@mshubin THANK YOU, For answering and adding to all of this.
First, can you give me your humble opinion about cachefly?
CS has been using them as their Squid CDN for a number of years now and are still partnered with them. They have been using them even as far back as 97.
I humbly believe as Mahalo is growing so is its market and target audiences !
This includes unfortunately some that might not have english as their primary language. I think this also is why we are receiving the amount of untimely and uncharacteristic anonymous questions. As well as the new user usage jump.
I'm going to finish reading what you wrote and give you a broader perspective of my opinions on the matter.
PS: VERY IMPORTANT!
Mahalo's greatest asset is its staff!
We should use that to our advantage somehow.
Hi Xds! In my answer, I mentioned that, "this is a very difficult question to answer specifically because every (every single) setup is different." By now, Mahalo must have accumulated a considerable of about of virtual and physical assets.
It could take a considerable amount of time first to untangle and understand that existing infrastructure and then lay out a specific, actionable plan to get that done.
Jason did not provide a lot of details. A lot of the details required to compose a step-by-step answer would require knowledge of the private and confidential inner workings of Mahalo. I don't have this knowledge. I believe what Jason has done here is post a thought provoking question and (I hope) is looking for an answer that outlines the generalities of what should be done to speed up Mahalo.
It's kind of unclear if mahalo does or doesn't run it's own NOC or not, although I would understand why you wouldn't want to say.
I do understand they use EdgeCast but for exactly what purposes, is this just as their CDN ? or click search provider ? or what ?
I've heard references allot of times to servers there AT mahalo. This is awfly confuzzling.
Also it would be great if we could get some stat data off the servers themselves no matter what the capacity. It's kinda hard to tell what the sticking points are and what could be in need of changing.
@mshubin , I'm so happy you replied FINALLY! , I guess you had a nice weekend :P
Happy to know mahalo still is planning for long term.
The good thing about cachefly is that they will work with you hand in hand with what your goals are and come to a equal mutual benefiting plan.
They are far less contract based and far more candid when it comes to seeing startup succeed. I implore you guys give them a call.
Note i don't have any affiliation with them whatsoever.
CS has used them since birth, I've watched them grow into the apple apps store delivering service they are today, with over 100 Million users in over 25 countries.
Also what is to become of this question, im just curious Jason has removed it from the front page :-/
I thought it should have hung out for a little longer than it did, specially for 100M$.
regarding #2, perhaps Mahalo could lease some of that unused time to sites we're "helping" like Conundrum?
@mshubin
If you still want to leave your options open for the future, for in house stuff (for now) I would recommend Hadoop , it has mapreduce, a high availability database, and a high availability file system visa vi bigtable .
All the big boys use it , google, yahoo, and even big blue (IBM).
( 1 tb of data sorted in 209 seconds ) =P
http://hadoop.apache.org/
For extremely small off site work check out nexenta http://www.nexenta.org/
This might sound selfish but I just hope the growth doesn't come at a cost to the rest of us :-/
@xds Basically we don't do a lot of big iron processing that requires a distribute model like a Beowolf cluster or a COTS solution from a company like SUN.
What we do and really need a better system for doing is storing large amounts of data and making said data highly scalable and highly available. Mahalo is "read-heavy" so having lots of machines available to accommodate these reads and store this data in a structured way is really what we need.
Leasing unused network resources back to smaller companies is a great way to earn a bit of extra money and keep an eye on the bottom line. If planned carefully, leasing time and especially support (if you have a rock star admin or network guru) can significantly offset startup infrastructure costs. Some smaller hosting companies and bandwidth providers jump all over this sort of setup so it can be pretty easy.
1) Right on. We currently run a couple of different NOSQL systems, but none of them are actually fully distributed like cassandra. We are currently looking at Cassandra to take over for these and to move other parts of the site to.
2) We use a CDN for our static content, but since mahalo is very dynamic using a CDN can help only so much.
3) This is all optimized. For non logged in users a full page is served from memcache through nginx (I have a solution in my head to use something like a CDN to serve these).
4) This is where we're going to get the most improvements, but there are other people at Mahalo who know more about this and why we do the things we do better than I do.
5) We are always looking at better ways to alert the staff when things go wrong. I will refrain form going too deep into this because of security reasons.
I really like you solutions. Honestly the other answers I've seen don't even address the issues you see on a highly dynamic, high traffic site. We've looked at the most obvious optimizations and now we need to look at the less obvious one.
@mshubin
Hope you don't think i'm being rash with these suggestions.. It's just the way I think.
A good place to start would be something like ROCKS or Kerrighed. To get a feel for DC.
....But yeah, putting this into a XUL extension would be the ideal idea.
The best part about it is, I think 99% of the active users on here trust the staff enough that they would gladly install and use something like this.
More importantly they trust the sites creator.
ps: Sorry if this sounded like a total noob comment rob theres just allot thats kinda unclear here.
@mshubin
First, Like i said above, thank you for taking the time out of your busy day especially on a friday night to come talk to all of us and fill us in on the current platform.
Why did I mention cachefly ?
I mention cachefly because of the shear amount of new traffic coming in from outside of North America. I believe as mahalo is growing it's user base will eventually widen. Mahalo will either need to tighten its belt (Like Hulu is doing) , or Loosen its grip. And start catering to other parts of the world.
( Like how Hispasec systems took VT ) ( Or Google took GoogleSearch )
Now this might sound a little insane...
Lets take a look at the hardware in the office.
Judging by what you have brought me you have told me more than enough, casandra or no casandra Mahalo's greatest asset is its user base & I think eventually what you guys need to push for is a system based on distributed computing.
Whether this is done through a Gecko 3.5 based XUL extension (firefox extention) or whatever.
What mahalo needs right now is for its users to also start pulling some of the weight. Especially if they are serious about their stay.
At first it wouldn't have to be much.
(I have heard threw the grape vine some of the staff want a Mahalo "bar" or "search bar" anyways)
Have it do some basic number crunching at first to get a idea of what you guys could be capable of.
And like i said, mahalo's greatest asset is its staff and user base.
USE THAT.
@xds I am familiar with distributed computing, in particular beowolf and ROCKS. In our particular case it does not apply and is not a replacement for a distributed, bigtable like, datastore with Map-reduce functionality like Cassandra.
In the case of CDNs (like cachefly) we'll go with the most cost efficient solution.
I do agree with you that our future traffic is going to be coming from outside North America, but we need to make sure out coverage in North America is sound before we concentrate on international Markets.
Mshubin: Thank-you for you're reply! I'm very happy to hear from one of the Mahalo "developers" although judging by your profile, being a storage guy, you live more on the server side of stuff.
Storage must be a pretty big deal to Mahalo. From distributed storage, to caching, to backup and recovery, you must have a pretty interesting job.
"We need to look at the less obvious one."
To me, this level of tuning is where proper planing and documentation start to pay off.
One of my favorite exercises is to have the Visio topology and functionality overviews printed off one a large format printer.. the kind that you can find at Kinkos, the ones Architects use. Toss these pirntouts down on the board room table. On a large whiteboard, write down the specific problems that need solved. Give everyone a different colored pen and say "have at 'er - fix these problems".
At the end of it, you'll (hopefully) be left with a series of actionable items and a couple of large maps with the steps that need to be taken to implement those refinements.
I have no idea at all if this will work for you guys and I wouldn't presume to make any real suggestions because from what I know, you guys are smarter than I am. It's just something that I've done in the past and has worked out really well.