[DOCUMENTATION] It's not 100% clear how to contribute new benchmarks

There's this CONTRIBUTING file with some details https://github.com/fastruby/fast-ruby/blob/master/CONTRIBUTING.md

But I still have questions that I think it's worth documenting:

The README says All results listed in README.md are running with Ruby 2.2.0p0 on OS X 10.10.1. Machine information: MacBook Pro (Retina, 15-inch, Mid 2014), 2.5 GHz Intel Core i7, 16 GB 1600 MHz DDR3. Your results may vary, but you get the idea. : ) so it's not clear how and who should update the readme, maybe that should be removed from there and added for each benchmark so people adding benchmarks can add their specs?
It's not clear what to do with the Goal described in the Contributing file (for example, it says At least 12% improvement, does it mean that a benchmark showing less than that improvement is not accepted / useful?)
I see code that has methods named fast and slow (and even slow2, slow3 https://github.com/fastruby/fast-ruby/blob/master/code/proc-and-block/proc-call-vs-yield.rb), I don't think that's the expectation (it makes things harder to read when multiple methods are called slowX), I think it could be described better in the CONTRIBUTING file if that's actually the case

@arielj

@arielj Thanks for opening this issue, and I really agree on this hole in the process. I'm thinking that it will be better to not accept contributors' results in the README, only accept the code they want to benchmark, and the reports will be taken from Dockerized containers from our CI configuration, and that is going to help us reduce the big disruption. "In my machine this is the fastest."

fast-ruby/.github/workflows/benchmarks.yml

Lines 50 to 68 in 86836b0

    
           ruby: [ 
        
             'ruby_head', 'ruby_4.0', 'ruby_3.4', 'ruby_3.3', 'ruby_3.2', 'ruby_3.1', 
        
             'ruby_3.0', 'ruby_2.7', 'ruby_2.6', 'ruby_2.5', 'ruby_2.4', 'ruby_2.3', 
        
             'ruby_2.2', 'ruby_2.1', 
        
             'jruby_head', 'jruby_9.1', 
        
             'truffleruby_head', 'truffleruby_22' 
        
           ] 
        
           # '' is the plain Ruby. Each include below overwrites it, so GitHub 
        
           # adds it as a separate job instead of merging it into the plain one. 
        
           variant: [''] 
        
           include: 
        
             - { ruby: ruby_3.1, variant: yjit, flags: --yjit } 
        
             - { ruby: ruby_3.2, variant: yjit, flags: --yjit } 
        
             - { ruby: ruby_3.3, variant: yjit, flags: --yjit } 
        
             - { ruby: ruby_3.4, variant: yjit, flags: --yjit } 
        
             - { ruby: ruby_4.0, variant: yjit, flags: --yjit } 
        
             - { ruby: ruby_head, variant: yjit, flags: --yjit } 
        
             - { ruby: ruby_4.0, variant: zjit, flags: --zjit } 
        
             - { ruby: ruby_head, variant: zjit, flags: --zjit }

And then we are going to gather all the artifacts from those runs from Ruby 2.1 to the current latest Ruby version, with the variants of YJIT as well, and create a static site with the results, so we have them all in a single place, running with the "same possible" conditions in the machine, in order to return a better/accurate result.

Happy to hear from you what you think about this, cc @etagwerker

@arielj

Update: the results now come from CI, not from contributors' machines. Every benchmark runs in Docker on Ruby 2.1 through head, JRuby and TruffleRuby, with YJIT and ZJIT, and the results are live at https://fastruby.github.io/fast-ruby/ (#250).

The next step is to move the numbers out of the README so it keeps only the advice.

@arielj is this good enough for what you had in mind, or what would you add?

Navigation Menu

Repository navigation

[DOCUMENTATION] It's not 100% clear how to contribute new benchmarks #205

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	ruby: [
	'ruby_head', 'ruby_4.0', 'ruby_3.4', 'ruby_3.3', 'ruby_3.2', 'ruby_3.1',
	'ruby_3.0', 'ruby_2.7', 'ruby_2.6', 'ruby_2.5', 'ruby_2.4', 'ruby_2.3',
	'ruby_2.2', 'ruby_2.1',
	'jruby_head', 'jruby_9.1',
	'truffleruby_head', 'truffleruby_22'
	]
	# '' is the plain Ruby. Each include below overwrites it, so GitHub
	# adds it as a separate job instead of merging it into the plain one.
	variant: ['']
	include:
	- { ruby: ruby_3.1, variant: yjit, flags: --yjit }
	- { ruby: ruby_3.2, variant: yjit, flags: --yjit }
	- { ruby: ruby_3.3, variant: yjit, flags: --yjit }
	- { ruby: ruby_3.4, variant: yjit, flags: --yjit }
	- { ruby: ruby_4.0, variant: yjit, flags: --yjit }
	- { ruby: ruby_head, variant: yjit, flags: --yjit }
	- { ruby: ruby_4.0, variant: zjit, flags: --zjit }
	- { ruby: ruby_head, variant: zjit, flags: --zjit }

[DOCUMENTATION] It's not 100% clear how to contribute new benchmarks #205

Description

Activity

JuanVqz commented on Sep 29, 2026

JuanVqz commented on Sep 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions