The object of comparison is the system’s capacity to produce subsequent improvements. A study needs to identify what the system changed, what experience informed that change, and how the revised system performs on further research under controlled evaluation conditions. Empira brings the task contracts and revision records together for this analysis.

Task records preserve the upstream candidate interface, metric, protocol and requirements where declared, with pinned source revisions. The protocol design separates development feedback from final evaluation and accounts for related task families when defining splits. Recorded reference values retain their source context.

The comparison design matches task and evaluator versions and accounts for both system development and subsequent use. Current experiment views expose the models, calls and costs recorded by the native framework. Shared execution and accounting are part of the integration work.

Search policies, memory, tools, code and learning algorithms can be objects of revision. Each framework owns its state and update rules. An integration should make the initial state, retained experience and permitted modifications explicit, so their effects can be studied across revisions.

We’re happy to share MCP access for research collaborations and work on framework integrations. Tell us which tasks, systems and experiment interfaces you want to connect. We can agree on the access and adapter contract while your framework retains its own search and learning logic.

Contributions enter through reviewed pull requests with links to the original research and versioned source material. Continuous discovery of tasks and RSI systems is in development; reviewed snapshots currently populate the catalog. For a study, cite the original paper together with the task, evaluator and system revisions you used.