I created an AI skill for using Alloy

Tooling
1

I have been using Alloy 6 with AI models and I had some success but often I need to revise a few times and the AI model will also self heal on different failure modes like syntax. This skill aims to solve those issues before it happens.

1 Like
2

Thanks for sharing! Do you have any results from early evaluations?

When we built and tuned our Alloy skills, we found that we needed to focus more on pitfalls and patterns/style of how we like to build models (and not so much on Alloy in general). We ended up making two separate skills (alloy for small contexts and simpler problems, and alloy-more for larger context work). We continually update the skills internally, run evals, and re-tune in a AlphaEvolve-style loop, but you can see a version of these skills here: devbox/.opencode/skills at main · ohpauleez/devbox · GitHub – perhaps it’s useful in your own skill development.

1 Like
3

Thank you, your repo is very useful to improving my skill.

As for the eval, the honest answer is: not yet, at least not results I’d call a proper agent evaluation.

So far, we have validation evidence: the skill passes packaging checks, installs across Codex, Claude Code, Cursor, and Gemini CLI, and its bundled models execute with Alloy 6.2.0. We’ve drafted task and trigger evals, but haven’t yet run a controlled with-skill versus baseline benchmark or an AlphaEvolve-style tuning loop. I don’t want to misrepresent validation as evaluation.

Your point about emphasizing pitfalls and modeling style resonates. Our current approach is one small skill that loads focused references as needed, rather than an alloy/alloy-more split. I’m reviewing your patterns around disj, event depiction, trace scenarios, Markdown models, and cross-state reasoning; several look worth adapting and testing.

If you can share how you score outputs and decide when to use alloy versus alloy-more, I’d love to compare notes. Proper evaluation is the next gap we need to close.

4

I also added some eval cases:

5

Regarding alloy and alloy-more: we almost exclusively use alloy-more. We still run evaluations with weaker models (and local models) where context is limited and instruction bias (eg: models biasing the top and bottom 10% of a file, prompt, context) need to be considered. At this point, even local models have a context large enough to handle alloy-more comfortably.