Logo LLI red

Generative AI Scorecard

Why did we act?

Progress in the development of so-called generative AI with applications such as ChatGPT has reached a point where there no longer is any doubt that artificial intelligence (AI) will become an integral part of legal work. AI will be used as an assistance system, or as an autonomous solution for answering simple questions, in law firms and legal departments as well as in courts, for administrative authorities, and probably also in the context of lawmaking.

For lawyers working with AI, it will be essential to know the strengths and weaknesses of the systems to be able to assess the opportunities and risks associated with its use correctly. An objective and standardized assessment requires a uniform catalog of criteria and a uniform scale of values applicable to the different systems. The results generated by the system will be assessed, and the goal of the assessment is to cover different abilities of the system (including descriptive, design, systematic, and analytical abilities).

What do we propose?

The Liquid Legal Institute e.V. develops a scorecard that allows for an objective and standardized assessment of the strengths and weaknesses as well as the opportunities and risks of ChatGPT and other forms of generative AI for use in the field of law.

The scorecard should perform the following functions:

  1. objective assessment of existing AI systems;
  2. creation of awareness regarding the capabilities of the AI systems used;
  3. benchmark in the further development and training of AI systems for the field of law.

How do we do it?

The scorecard will be developed within a LLI project, participation in which is open to all members. In a second phase, it will then be tested as part of a LLI lab.


Project lead(s): Dr. Jens Wagner

Contact: via LLI’s Microsoft Teams Platform (membership required)

Photo by possessedphotography on Unsplash