Meta: February 2004 Archives

Today I received the following message in my mailbox:

an improper scan has caused a ban on your site

date: Tue Feb 24 18:30:20 2004
ip: 66.77.73.32
host: shop-gw.sac.overture.com
agent: Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler

I regulary receive this kind of messages, usually created because bad robots or script kiddies access my spam trap. But Yahoo and Overture are well respected companies, and I would assume that they would have respected my robots.txt file, in which I explicitly deny access to the /private folder:

User-agent: * Disallow: /cgi-bin
Disallow: /dummy/dummy.html
Disallow: /errors
Disallow: /fimcap
Disallow: /js
Disallow: /mailtemplates
Disallow: /mt-static
Disallow: /private
Disallow: /spam

So I looked in my access log and found that they indeed violated my robots file!!

66.77.73.32 - - [24/Feb/2004:17:20:24 -0500] "GET /robots.txt HTTP/1.0" 200 758 "-" "Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler"
66.77.73.32 - - [24/Feb/2004:17:57:22 -0500] "GET /private/ HTTP/1.0" 200 4815 "-" "Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler"
66.77.73.32 - - [24/Feb/2004:18:30:20 -0500] "GET /private/welcome.html HTTP/1.0" 200 351 "-" "Yahoo-VerticalCrawler-FormerWebCrawler/3.9 crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler"

Notice that the page mentioned in the User Agent string states that Yahoo-Overture does support the robots exclusion protocol!

It has been very silent on this site lately.

This was not because I did not have time to write; it only takes about five minutes to write some lines.

This was not because I did not have anything to write about; I could have written about using Zempt for posting on this site, about the political situation in Spain, about my parents coming to visit me this week, about the fimcap.org server migration...

The only reason for my silence is that I did not feel like writing. But now I'm back again! I promise

Sometimes people ask me why this site is called 'Brain Tags'. Well, the origin comes from a nice song of Bettie Serveert called Brain Tag. Of course the word Tag links nicely to my interest in web development, while people knowing me only a little bit realise that the word Brain has no link to me at all...

How does this rhyme, cause every time I see you
well, I could swear, must've met you somewhere

Down under lock and key,
there's a brain-tag to every secret
If I could keep my mind shut,
there's a chance that I might even keep it
You feel familiar and I wanna feel some more
Have I ever laid my hands on you before

This must sound weird, it's like you peered inside me
cause every time, you recognise my state of mind

Down under lock and key,
we put a brain-tag to every secret
Now comes the time to figure out
whether we should keep it
You feel familiar and I wanna feel some more
Have I ever laid my hands on you before

Total freedom beneath the waves
Floating through the arcane waves
Synchronised in space and time
Leaving all desire behind