Diggit Drupalextras

Aggressive crawlers that hit your web site a lot can cause performance problems.
There are many ways to identify aggressive crawlers, including writing custom scripts that analyze your web server logs.
One tool that we found to be useful in analyzing which crawlers hit the site the most today or yesterday is Go Access.
Getting Go Access
Go Access is available for Ubuntu Natty Narwahl (11.04) only, but not earlier LTS releases.

read more

When tuning sites for clients, we often see the usual symptoms and causes of why a site is slow, and occasionally we find an unusual reason.
For one client, after we diagnosed the main reason for a site being slow (bottleneck between database server and web server), they continued to experience slowdowns.
This was getting annoying as it happened several times a day, and their customers were complaining about the slowdown on a daily basis.

read more

If your Drupal site suffers occasional slow downs or outages, check if crawlers are hitting your site too hard.
We've seen several clients complain, and upon investigation we found that the culprit is Google's own crawler.
The tell tale sign is that you will see lots of queries executing with the LIMIT clause having high numbers.

Quick Tabs is a widely used Drupal module. Site builders like it because it improves usability in some cases by reducing clutter.
Incidentally, the way this module works has cause us to run across performance issues caused by certain uses. See previous article about Quick Tabs can sure use more caching and a case study involving Quick Tabs.

read more

I have just published an article about some new SEO strategies which including optimizing 2.0 websites and backlinking using your own personal Link Wheels designed to organically increase your keyword niche target traffic.

http://www.associatedcontent.com/article/2985966/backlink_strategies.htm...

Last week was a very frustrating time for me. For whatever reason, a unusually number of botnets decided to zero in on my Drupal site and created what I call an unintentional  Denial of Service attack (DOS). The attack was actually from spambots looking looking for script vulnerabilties found mainly in older versions of e107 and WordPress.

Hello,
We are looking for freelancers who can design a crawler for us and customise ( you can build on opensource).
http://www.manageability.org/blog/stuff/open-source-web-crawlers-java

If you are enthusiastic about it, then please write to me.

freelancer.openings@gmail.com

The payments will be good.

When we come together in Copenhagen in a few weeks for Drupalcon, we're doing an amazing thing. The Drupal community is gathering to share knowledge, experience and ideas to build our skills and advance the Drupal project. On our local level this is something we need to do every day.

A client site was facing intermittent outages, specially at peak hours.
We investigated the issue over a few days, and narrowed down the cause to certain slow queries, described in more details below.
They had tagadelic on every page, displaying a tag cloud, and from that, crawlers hit every term to paths like taxonomy/term/1581.
Slow queries causing server outage

read more