Google has updated its open source robots.txt parser code on GitHub the other day. Gary Illyes from Google pushed the update yesterday morning to the repository there.

Honestly, we’ve often thought about writing something that parses C code and stuffs it into a SQL database. This tool leverages the CLang parser and lets you write queries like: ...

Google announced yesterday as part of its efforts to standardizing the robots exclusion protocol that it is open sourcing its robots.txt parser. That means how GoogleBot reads and listens to ...

That shiny new module is XML::Parser, a bit of glue originally written by Wall and later adopted by Clark Cooper that provides a Perl interface to James Clark’s C-based expat parser.