The idea behind running a blog on GitHub Pages, was that if I’m committing changes to projects everyday, I might be more likely to commit blog posts more frequently. Also, I like using Git and Markdown to write/manage my Blog posts, instead of WordPresses PHP interface.

So if you have this Blog in your RSS Feed Reader, subscribe to the new Atom Feed:

http://postmodern.github.com/atom.xml

Besides the usual formatting mistakes that can easily crash SimpleMarkup, I and other users began noticing odd error messages when installing YARD documentation with SimpleMarkup:

Unhandled special: Special: type=65, name=CROSSREF,_SPECIAL_, text="Spidr"

Even after ditching RDoc for YARD, SimpleMarkup (the default markup used by YARD) was able to reach forth from it’s icy RubyGem and introduce bugs into my code. This was the last straw, SimpleMarkup is dead, time to move on.

Luckily, YARD supports using alternate markup engines, such as MarkDown or Textile. Having used MarkDown regularly in the past, and that it matched SimpleMarkup’s syntax in some ways, I decided to convert all of my Ruby project’s to MarkDown formatted YARD documentation.

First, I had to specify the --markup markdown option in the yard_options setting (provided by the hoe-yard plugin) in the Rakefile.

Second, I renamed the README.*, History.* files to README.md, History.md, respectively.

Finally, I began the tedious process of converting SimpleMarkup formatted text to MarkDown:

1. Replace the =, ==, === headings with #, ##, ###.
2. Indent all source-code examples not following a YARD @example tag by 4 spaces.
3. Convert any raw URLs to MarkDown links.
4. Replace any +code+ or <tt>code</tt> formatting with `code` or <code>code</code>. (Note: `code` can handle spaces, numbers and symbols within the ` ` characters, where as +code+ cannot.)

In conclusion, that is how I ridded myself of the last buggy vestiges of RDoc, and increased the reliability of my documentation using YARD.

When writing a library which bundles static-content in a directory named data/ or static/, you usually need to find the directory using something like File.expand_path(File.join(File.dirname(__FILE__),'..','..','..','..','static)).

But then you want to allow plugins to extend the functionality of your library, and even add their own static-content. Now you have to manage more than one directory path.

Enter StaticPaths, a Ruby library for managing and searching through directories containing static-content. StaticPaths helps manage directories across multiple libraries in much the same way that RubyGems manages lib/ directories using $LOAD_PATH. Except that StaticPaths does not use global variables.

Example Usage

require 'static_paths'

module MyLibrary
  include StaticPaths

  # define the static dir(s)
  register_static_dir File.join(File.dirname(__FILE__),'..','..','static')
end

register_static_dir will expand the path, make sure the path points to a directory, then adds the path to StaticPaths.paths and the local MyLibrary.static_paths.

One can also unregister directories using unregister_static_dir!.

MyLibrary.unregister_static_dir! File.join(File.dirname(__FILE__),'..','..','static')

Or, one can use unregister_static_dirs! to unregister all directories registered within a module/class.

module MyLibrary
  def MyLibrary.cleanup!
    unregister_static_dirs!
  end
end

To search through and access the content within registered directories, simply use the methods within the StaticPaths::Finders module.

module MyLibrary
  class UsesContent

    include StaticPaths::Finders

    def index
      find_static_file('index.html')
    end

    def file_dirs
      all_static_dirs('extra')
    end

    def copy(src,dest)
      each_static_dir(src) do |dir|
        FileUtils.cp_r(dir,dest)
      end
    end

  end
end

Installation:

$ sudo gem install static_paths

This rigidness of RDoc really shows up when you need to document ORM backed projects, such as one containing DataMapper models. DataMapper allows one to define the properties and relations between models using handy class-methods:

class MyModel

  include DataMapper::Resource

  # The primary key of the model
  property :id, Serial

  # The name of the model
  property :name, String

  # The many authors contributing to the model
  has n, :authors

  # The user that owns the model
  belongs_to :user

end

RDoc will not recognize property, has or belongs_to. Nor will RDoc know that property adds a class-method and instance reader/writer methods with the given name to the model. Documentation fail.

Enter YARD

YARD is a documentation generation tool for the Ruby programming language.
 It enables the user to generate consistent, usable documentation that can be
 exported to a number of formats very easily, and also supports extending for
 custom Ruby constructs such as custom class level definitions.

http://github.com/lsegal/yard

YARD organizes most of it's parsing logic into multiple handlers; essentially classes that inherit YARD::Handlers::Ruby::Base and define a process method. YARD also supports a plugin system, by loading any RubyGems installed on the system that are prefixed with yard- or yard_. Using these two features of YARD, one can easily create a YARD plugin gem containing custom handlers, which YARD can automatically load and use.

yard-dm

yard-dm is a YARD plugin for parsing DataMapper model syntax. The plugin can handle the following statements:

property :name, Type

has n, :things

has 1, :thing

has 0..n, :things

has 1..n, :things

has 2..5, :things

belongs_to :stuff

Install It

$ sudo gem install yard-dm

Use It

$ cd dm-project/
$ yardoc

It's that easy.

Next time you need complete documentation on your DataMapper backed project, just install YARD and yard-dm.

Since I use Hoe for all my gem publishing needs, I got tired of having to specify all of this information explicitly in my Rakefiles; after all Hoe is suppose to make one’s Rakefiles simple and elegant. So I took advantage of Hoe’s plugin system and created a Hoe plugin to properly configure my projects for YARD.

Introducing hoe-yard, a Hoe plugin that will:

• Automatically find the README and History files, irregardless of file extension.
• Sets has_rdoc to yard.
• Sets rdoc_options.
• Sets extra_rdoc_files.
• Adds YARD and hoe-yard as development dependencies.
• Adds the yard and docs Rake tasks.

Install

$ sudo gem install hoe-yard

Usage

require 'rubygems'
require 'hoe'

Hoe.plugin :yard

Hoe.spec('my_project') do
  # ...

  # optional YARD settings
  self.yard_title = 'My Project (0.1.0)'
  self.yard_markup = :markdown
  self.yard_opts = ['--protected'] # any additional YARD options

  # ...
end

Not only does hoe-yard make setting up YARD with a project extremely easy, it also ensures the documentation that is automatically generated when a gem is installed will match the documentation published for that gem.

Web Spider Obstacle Course (WSOC)

Spidr 0.2.2 now requires and makes use of the new Web Spider Obstacle Course (WSOC) for testing. Before one runs the RSpec test-suite for Spidr, the WSOC server must first be started:

$ wsoc_server

Then simply run the specs as usual:

$ rake spec

Cookie support

As of 0.2.2, Spidr now comes with a CookieJar, thanks to the work of @zapnap. Now when the Spidr::Agent visits a page, any new cookie values will be merged into the CookieJar, and sent back with any future requests.

Additionally, one can now access the Cookie values from a Spidr::Page object.

page.cookie
# => "COUNTRY=USA%2C97.100.45.38; expires=Mon, 18-Jan-2010 06:19:24 GMT; path=/; domain=.php.net"

page.cookies
# => ["COUNTRY=USA%2C97.100.45.38; expires=Mon, 18-Jan-2010 06:19:24 GMT; path=/; domain=.php.net"]

HTTP Basic Auth support

Spidr 0.2.2 now comes with a brand new AuthStore, for organizing HTTP Authentication credentials; also thanks to the work of @zapnap.

Provided you have the credentials for the various HTTP Basic Auth protected areas that are to be spidered, Spidr can automatically respond to Basic Auth challenges.

Simply specify the credentials to the Spidr::Agent and the agent will do the rest:

Spidr.host('corporation.com') do |agent|
  agent.authorized.add('http://corporation.com/private/', 'user1233', 'motivate synergize')

  agent.every_page do |page|
    if page.url.path =~ /private/
      # ...
    end
  end
end

URL Sanitization

A small yet important module was added in Spidr 0.2.2, and that is Spidr::Sanitizers. The Sanitizers module adds configuration settings to Spidr::Agent for how incoming URLs are to be sanitized.

For instance, URL fragments are removed by default, but this can be changed:

agent.strip_fragments
# => true
agent.strip_fragments = true

Additionally, perhaps one might wish to strip the query strings from all URLs:

agent.strip_query = true

Note: If YARD documentation generation fails when installing Spidr 0.2.2, this is due to a bug in RDoc/SimpleMarkup generation.

Now the Web Spider Obstacle Course (WSOC) has been rewritten as a Ruby Sinatra app that can be ran locally. Since most Ruby developers are familiar with Sinatra, it should not be too difficult to extend the Obstacle Course.

Currently WSOC tests a Web Spider’s ability to navigate:

• Empty links.
• Circular links.
• Links with relative-paths.
• Links with absolute-paths.
• Remote links.
• javascript: links.
• Links within frameset and iframe tags.
• Cookie protected pages.
• HTTP 300, 301, 302, 303 and 307 Redirects.
• HTTP Baisc Auth protected pages.

3… 2… 1… Go

Install:

$ sudo gem install wsoc

Run:

wsoc_server -p 8080

Visit:

firefox http://localhost:8080/

The front page of WSOC will explain the rules of the Obstacle Course and providing the Start URL for any would-be Web Spiders. The WSOC server still contains the Obstacle Course Specs, available at http://localhost:8080/specs. The Obstace Course Specs are essentially a list of URLs, the expected behavior of the spider for the URL (visit, ignore or fail) and a message about the given URL. These specs can be access by Web Spider test suites in JSON (http://localhost:8080/specs.json) or YAML (http://localhost:8080/specs.yaml) formats.

Use It

If you actively maintain a Web Spider, or are thinking of writing a Web Spider/Crawler/Scanner, I highly recommend you use the Web Spider Obstacle Course as part of your testing solution. The Obstacle Course will test more than a Web Spiders ability to resolve relative links or only visit a link once.

The internet is a wild place, and will not always obey XHTML 1.1 Strict or HTTP 1.0 standards. Test your Web Spiders fully before setting them loose.

ToorCamp: Ronin – A Platform for Publish and Mayhem

Original slides can be found here, in XHTML form.

Thanks go out to flatline and evoltech for going with me to ToorCamp, providing transportation and helping hitch-hike back when our van broke down. Also extra thanks to evoltech for letting me use his SUV sized laptop to run the slides on.

Evoltech correctly pointed out that the Wordlist::Builders::Website example was incorrect. The Website.build method must take a path argument and then any additional options.

For a while now people have asked/told me that I should write a Wordlist library for Ruby. People have also brought up the need to build wordlists from existing corpora; such as the text from a website. After seeing varying attempts at this task, I finally got serious and released Wordlist 0.1.0.

Currently, the design of the Wordlist Ruby library is quite simple. Wordlist::Builder is the base class for all wordlist builders. It handles parsing text, filtering out duplicate words, and writing the unique words to a file. The Builder class stores the CRC32 hash of each word in a Hash, mapping the word length to the Set of CRC32 hashes. Using a bucket system for the CRC32 hashes, keeps both word lookup time and memory usage to a minimum.

Wordlist::Builder.build('list.txt') do |builder|
  builder.parse(some_text)
  builder.parse_file('some/file.txt')
end

Wordlist also provides Wordlist::Builders::Website, which uses Spidr to completely spider a host, and build a wordlist from the inner-text recovered from every visited page.

require 'wordlist/builders/website'

Wordlist::Builders::Website.build('list.txt',:host => 'www.example.com')

Wordlist doesn’t just build wordlists, it can also enumerate them. Wordlist defines Wordlist::List as the base class for all wordlist readers. Per convenience, Wordlist also defines Wordlist::FlatFile which handles the enumeration of flat-file wordlists.

list = Wordlist::FlatFile.new('list.txt')
list.each_word do |word|
  puts word
end

The Wordlist::List also provides methods for defining text-mutation rules, which are applied in series to each enumerated word.

list.mutate 'o', '0'
list.mutate '@', 0x41
list.mutate(/[hax]/i) { |match| match.swapcase }

list.each_mutation do |word|
  puts word
end

A mutation rule takes a String or Regexp (which is used to match a sub-string within a word) and another String, Integer or block which is used as a replacement for the matched sub-string. A mutation rule will perform every possible combination of sub-string replacement, passing each mutated word to the next mutation rule, and so on.

One can install Wordlist using rubygems, by simply running the following command:

$ sudo gem install wordlist

The git repository for Wordlist is located on GitHub.

As of 0.1.0, Wordlist is fairly simple but it gets the job done. The library was fun to design and test, as it involved borrowing some techniques typically used in blind-fuzzing. Expect more features and updates to Wordlist as time goes on.

DataMapper 0.10.0

Ronin was probably one of the first big Ruby apps to upgrade to DataMapper 0.10.0. As a result of this, we had to postpone the release of Ronin 0.3.0, until DataMapper 0.10.0 had finished it’s rigorous two month Q/A cycle.

Once DataMapper 0.10.0 had been released, the two months of Q/A and almost a complete rewrite of the code-base really showed itself. I think @dkubb, @dbussink, @snusnu, @xshay, @myabc, @rsim, @namelessjon, @knowtheory and everyone else who helped with 0.10.0 deserves applause for their hard work and dedication.

YARD

As of 0.3.0, Ronin has successfully moved to YARD based documentation. The new Ronin API docs can be found in the usual location. YARD really helped improve the detail of the Ronin API docs, allowing developers to specify method arguments, accepted object-types, yield arguments, possible exceptions and return-types.

Additionally, YARD handlers were added to Ronin that parsed and regenerated documentation for DataMapper properties/relations. These additional YARD handlers helped improve documentation coverage for Ronin models, such as Ronin::Arch or Ronin::Exploits::Exploit.

Ronin::UI::CommandLine

Ronin::UI::CommandLine was almost entirely refactored in 0.3.0. Ronin::UI::CommandLine::Command was rewritten to use Thor. Switching to Thor greatly simplified the ronin commands, making it easier to write new commands. A good example of this, is the source-code for the install command from before and after 0.3.0.

Another benefit to using Thor, albeit cosmetic, was being able to use Thor’s ANSI color-output. Normal output will now appear in green, warnings in yellow and error messages in red.

New convenience methods were also added to Ronin::UI::CommandLine::Command to help format and print various data: indent, print_title, print_array and print_hash.

Ronin::UI::Output

As of 0.3.0, Ronin::UI::Diagnostics and Ronin::UI::Verbose were replaced with the brand new Ronin::UI::Output module. Ronin::UI::Output is designed to delegate how output is handled. By default all output is sent to Ronin::UI::Output::Handler, which prints the output to the terminal. One can override the default output handler by using Ronin::UI::Output.handler=, to specify the new handler module; which must define print_info, print_debug, print_warning and print_error methods.

Ronin::UI::Output also provides the Helpers module, which can be included into any class and adds the print_info, print_debug, print_warning and print_error methods.

New Convenience Methods

New TCP and UDP networking methods were added to 0.3.0. Now one can easily create TCP/UDP servers using, Net.tcp_server, Net.tcp_server_session, Net.tcp_single_server, Net.udp_server and Net.udp_server_session.

Ronin::Network::HTTP::Proxy was also added to 0.3.0. The Ronin::Network::HTTP::Proxy helps parse and represent HTTP proxy addresses, and is now used by Ronin::Network::HTTP.proxy and all :proxy options.

proxy = Network::HTTP::Proxy.parse('148.233.239.24:80')
# => #<Ronin::Network::HTTP::Proxy: 148.233.239.24:80>

The Proxy class can also be used to test a proxy.

proxy.valid?
# => true
proxy.lag
# => 0.003881

Ronin::Platform

The overlay.xsl file got a make-over in 0.3.0. overlay.xsl is a XML StyleSheet, which helps render ronin.xml files from Overlays into proper XHTML when they are viewed in a web-browser. Now ronin.xml files should appear more like a standard "About" web-page.

Reloading Overlays/Extensions while ronin is running became possible with the addition of Ronin::Platform.reload!. Now if an Extension is modified, one can simply call Platform.reload! and get the updated version.

Ronin Extensions gained convenience methods for defining reader/writer methods to instance variables: attr_reader, attr_writer and attr_accessor. They can be used in Ronin Extensions, just as one would use them in a class.

ronin_extension do

  attr_accessor :var
  attr_reader :result

  setup do
    @var = 5
    @result = nil
  end
  ...
end

Ronin Extensions also gained their own on-demand temp directory. When Ronin::Platform::Extension#tmp_dir is called, a temp directory specifically for the extension will be created within ~/.ronin/tmp/, and the path of the new temp directory will be returned.

Upgrading

As always, the rubygem files for Ronin 0.3.0 are available on rubyforge.org. One can update Ronin using rubygems by simplying running:

$ gem update ronin

Likewise, one can also install Ronin using rubygems:

$ gem install ronin

Also, as of Ronin 0.3.0 all releases of Ronin will also be made available on gemcutter.org.