Thanks Melvin, very valid and relevant points to which I don't have an easy answer :) In most leaderboards, as you know, we are usually benchmarking the same set of prompts across multiple LLMs.

In my experience of building prompt stores, we are usually working with a Business Unit (department) within a company who wants to build a prompt store for their team and a select group of internal stakeholders.

So building a prompt store that works across departments who have different needs and perceptions remains an open problem. Further, the theoretical answer to your question of customizing a prompt template to a specific LLM, is to maintain prompt specific variables that can be instantiated while querying that specific LLM. However, I am sure that there will be many challenges in implementing this in practice.