Back to Repositories

Testing Security Warning Verification Generation in Brakeman

This test utility script automates the generation of test cases from Brakeman security warning reports. It provides a streamlined way to create structured test files based on detected vulnerabilities and security concerns.

Test Coverage Overview

The test generator provides comprehensive coverage of security warnings across controllers, models, and templates.

Key functionality includes:

Automated test case generation from warning reports
Coverage of multiple warning types and confidence levels
Fingerprint-based test verification
Line-specific code testing

Implementation Analysis

The implementation follows a template-based approach for generating Test::Unit test cases. It utilizes Brakeman’s internal reporting structure to create standardized test methods that verify specific security warnings.

The code employs Ruby metaprogramming patterns to dynamically generate test methods and leverages Brakeman’s warning tracking system.

Technical Details

Testing tools and components:

Brakeman security scanner integration
RubyParser for code analysis
Test::Unit framework
Custom report generation classes
Warning verification modules (BrakemanTester::FindWarning, BrakemanTester::CheckExpected)

Best Practices Demonstrated

The test generator implements several testing best practices including isolated test cases, consistent naming conventions, and comprehensive assertion checking.

Notable practices:

Structured test organization by warning type
Detailed warning attribute verification
Automated test scaffolding
Maintainable test generation patterns

presidentbeef/brakeman

test/to_test.rb

            
#This is a utility script for generating tests from reported warnings.
#
#It is not heavily tested. It is mostly for the convenience of coders. Sometimes
#it generates broken code which will need to be fixed manually.
#
#Usage:
#
#  ruby to_test.rb apps/some_app > tests/test_some_app.rb`

# Set paths
$LOAD_PATH.unshift "#{File.expand_path(File.dirname(__FILE__))}/../lib"

require 'brakeman'
require 'ruby_parser'
require 'ruby_parser/bm_sexp'
require 'brakeman/options'
require 'brakeman/report/report_base'

class Brakeman::Report::Tests < Brakeman::Report::Base
  def generate_report
    counter = 0

    name = camelize File.basename(tracker.app_path)

    output = <<-RUBY
abort "Please run using test/test.rb" unless defined? BrakemanTester

#{name} = BrakemanTester.run_scan "#{File.basename tracker.app_path}", "#{name}"

class #{name}Tests < Test::Unit::TestCase
  include BrakemanTester::FindWarning
  include BrakemanTester::CheckExpected

  def expected
    @expected ||= {
      :controller => #{@checks.controller_warnings.length},
      :model => #{@checks.model_warnings.length},
      :template => #{@checks.template_warnings.length},
      :warning => #{@checks.warnings.length} }
  end

  def report
    #{name}
  end

    RUBY

    output << @checks.all_warnings.map do |w|
      counter += 1

      <<-RUBY
  def test_#{w.warning_type.to_s.downcase.tr(" -", "__")}_#{counter}
    assert_warning check_name: #{w.check_name.inspect},
      type: #{w.warning_set.inspect},
      warning_code: #{w.warning_code},
      fingerprint: #{w.fingerprint.inspect},
      warning_type: #{w.warning_type.inspect},
      line: #{w.line.inspect},
      message: /^#{Regexp.escape w.message.to_s[0,40]}/,
      confidence: #{w.confidence},
      relative_path: #{w.file.relative.inspect},
      code: #{w.code.inspect},
      user_input: #{w.user_input.inspect}
  end
      RUBY
    end.join("\n")

    output << "\nend"
  end
end

options, _ = Brakeman::Options.parse!(ARGV)

unless options[:app_path]
  if ARGV[-1].nil?
    options[:app_path] = "."
  else
    options[:app_path] = ARGV[-1]
  end
end

tracker = Brakeman.run options

puts Brakeman::Report::Tests.new(tracker).generate_report