Reviewing: building rules
Last updated
Last updated
Before you can ask ClauseBuddy to automatically review documents, you must build one or more sets of rules. These rules can be managed through the Manage reviewing rules module on the home page of ClauseBuddy.
Your administrator must have enabled the right "Manage document reviews", otherwise you will not see this module on the welcome screen. Also note that the entire reviewing may have been disabled by your administrator for your entire firm/company.
For end-users, the admin must have enabled the right "Apply document reviews", and at least one set of rules must be available, otherwise end-users won't see the "Review document" module on the welcome screen.
When you first load this module, you will be invited to create a new review "category". As further explained below, you can create multiple categories — e.g., per legal domain, per type of contract or even per product/service that is being sold by your company.
Each review category can be assigned a title (in multiple languages if necessary), and can also be made subject to access rights, e.g. to prevent that department X would be able to access the review rules of department Y.
Once a category is created, you can update its properties (title and access-bundle) through the "..." button dropdown menu in the upper-right corner.
Throughout the reviewing module, textual elements can be specified in multiple languages, if you are using an account on a ClauseBuddy server a in multi-lingual jurisdiction.
These translations are mostly intended for your colleagues, and are not strictly necessary to complete. LLMs such as GPT can understand multiple languages, and will understand a Requirement written in French, and be able to apply it to an English contract.
Even so, it is often advisable to create translations. The reason is that the translation of some legal concepts can be dubious. In addition, when you give explicit instructions to the LLM on what to include in (or how to redraft) a clause, you may want to avoid the uncertainty associated with expecting the LLM to perform the translation for you.
You should be aware that one of the main limitations of LLMs consists of the length and complexity of the input you feed — i.e., both the content of your document and the rules you specify. While this is one of the primary domains of research, you should be aware that LLMs are very sensitive to the length, e.g. because doubling the input size causes the processing time to be four times as long.
GPT 4.5 (the LLM used by ClauseBuddy) seems to get into problems as from about 75 pages, literally "losing attention" and skipping items. Take into account that both your rules and the document's content count towards that 75 pages.
Accordingly, you should try to find a balance between writing your instructions in a compact way, yet at the same time being sufficiently detailed for the LLM to understand the instructions. In this regard, you may want to treat the LLM as your average junior team member: if you first ask it to read 50 pages of detailed legal instructions, and then ask it to review a 4-page NDA, it may deliver a worse job than when you had written a few key elements to look out for.
You can specify complex background details of any length through the "Context" box of a requirement (e.g., references to internal notes, practical tips for dealing with a counterparty during a discussion on a certain topic, etc.). That Context will not be submitted to the LLM, so feel free to provide all the details you want.
Taking into account the length constraints, you may want to skip content that you can assume the LLM already knows. The LLMs are trained on information available on the public internet, and it is often surprising what they know and do not know.
LLMs know a lot of facts, e.g. the capital of a country, what certain abbreviations (such as "GDPR" or "FDA") stand for, the functionality of certain product, etc. However, you should be aware that LLMs were trained up to a certain point (e.g., GPT-4.5 up to early 2023) and will not be aware of facts that happened afterwards. If those facts would be important for the review, then you need to tell the LLM explicitly.
LLMs also have a reasonably good information about well-known legal documents for which many examples exist, e.g. NDAs and license agreements, particularly in large jurisdictions such as the United States and the United Kingdom. Conversely, LLMs will have only limited information available about contracts in small jurisdictions and niche areas of law.
When you have create a category, you can add rules to them by clicking on the green "+" icon. Those rules can be one of the following elements:
A Group element allows you to add hierarchical structure to the other elements. You can organise your elements in groups, with sub-groups, sub-sub-groups, etc. any level deep.
A Requirement is the most important element. It specifies a rule, and can optionally include conditions and actions.
A Condition element specifies that a certain condition must be met for a Requirement to become applicable. You can attach questions to a condition, which must be answered by the end-user before the review process can take place (e.g., "What is the deal size?" or "How many products of type X are being ordered?"). The end-user's answers to those questions can then be used by the LLM to analyse whether the condition is met, i.e. to decide whether or not the Requirement applies or not.
An Action element specifies that a certain action must be taken when a Requirement is not met — e.g., that a clause must be deleted, adapted or swapped for some other clause.
An Insight element provides background information for the LLM, e.g. to explain what your company's product "AlphaDogs" is all about, or how your support department will deal with requests in the weekend.
Conditions and Actions do not necessarily have to be created as separate elements in a reviewing category. Instead, you can also specify them "ad-hoc", when creating a specific requirement. The advantage of creating a Condition or Action independently, is that you can then reuse it across different Requirements.
For example, a condition relating to the deal size may be specified in many different requirements. Instead of copy/pasting the same Condition, you can create an independent Condition and then simply refer to it within multiple Requirements.
Requirements specify a rule, i.e. something that must be met, either by being present, or instead not being presented, or having certain content. When the Requirement is not met, the associated Action will be presented on the screen.
When you edit a Requirement, you will see the following screen:
In the Title box, you can give a title to your Requirement, i.e. a short explanation of your requirement in a few words (e.g., "Our liability must be capped" or "Confidentiality obligations must not exceed 5 years").
In the Contents box, you can give more details about the requirement, e.g. a description on what you expect to see in a contract or clause.
It is not strictly necessary to complete the "Contents" box. If you do not specify any content here, ClauseBuddy will provide the Title to the LLM when submitting the review request. So if your Contents mainly repeats the Title, feel free to omit it.
In the Context box you can specify information for your human colleagues — e.g. internal hyperlinks with more information, internal contact information, "war stories" from the past on how to deal with counterparties on this specific topic, etc. This information will not be submitted to the LLM at all (it will only be presented towards the human end-user), so do not feel constrained in length.
Through the Conditions you can specify Conditions for your particular Requirement to become applicable (see below for more information). When multiple Conditions are specified, you can also specify whether all those Conditions must be met simultaneously, or whether instead it is OK for only one Condition to be met, or whether instead none of the Condition may be fulfilled.
Through the Actions button you can specify adhoc Actions for your particular Requirement. More on that below.
A Condition allows to specify towards the LLM when a certain Requirement must be completely ignored.
In principle, you can also specify condition-like elements in the Contents box of a Requirement. However, it is not advisable to this extensively, due to the advantages of using a separate Condition:
ClauseBuddy asks the LLM to first check all the separately listed Conditions of all the Requirements, and to assess whether those Conditions are met. The LLMs's opinion on whether or not the Condition(s) is/are met, can afterwards be consulted by the end-user.
Conditions can have Questions associated with them, allowing the end-user to convey deal-specific information towards the LLM, as further explained below.
Similar to the other reviewing elements, a Condition has a Title box in which you can provide a brief description of your Condition (e.g., "Contract value more than 5000 EUR").
If the Condition requires more detail, you can also use the Contents box. (If it contains no contents, ClauseBuddy will instead only submit the Title to the LLM).
You can associate one or more Questions with each Condition. Those Questions will be presented to the end-user, and the answer(s) given by the end-user will subsequently be submitted to the LLM.
Questions are ideal for modulating your review, by including deal-specific information. For example, when a Condition would specify that a certain Requirement is only relevant when the deal value is more than 5000 EUR, but this deal value is not explicitly listed in the contract itself, the LLM obviously would have no way of knowing whether this Requirement is relevant.
You can create any number of questions, and ask different types of question (true/false, text, number, date, duration, currency).
An Action will be presented towards end-users when a Requirement is not met. For example, when a Requirement is "The applicable court should be French" and the associated Condition is "Deals relating to Product X", then you may want to suggest the end-user to:
Insert a clause from your Quality Library that stipulates exactly that.
Ask the LLM to rewrite the clause that specifies something different. You can then specify the writing instruction.
Delete the clause in the document that specifies a different court.
Highlight the clause in the document
Add a comment in the document with predefined contents.
Note that you can create multiple Actions for a single Requirement. In fact, it may be helpful towards the end-user to offer multiple Actions for a single requirement, so the end-user can – for example – choose whether to delete an offending clause, or instead highlight it.
Insights allow you to provide background information to the LLM.
They are ideal for communicating information that is necessary for the LLM for assessing different Requirements. Instead of explaining certain internal concepts per Requirement, you are advised to create a single Insight that centrally lists that information.
An Insight only contains a Title and Contents box. As is the case with the other components of a Requirement, you can omit the Contents if it would not specify anything beyond the Title.
Once you have created several rules in a category, you can optionally create Review Sets by clicking on the "Combine" button.
The underlying idea is that not all your requirements will necessarily apply to every deal. Through Review Sets, you can specify which rules apply, and additionally pre-fill certain questions.
The end-user will then be able to specify the relevant Review Set when submitting an actual review request. For example, when licensing software, you may have 50 different rules that can theoretically apply, depending on the type of software being purchased, its mission critical nature, value, supplier, etc. Many of the more onerous requirements can probably be skipped when purchasing a low-value product, as opposed to purchasing an enterprise-wide multi-million dollar system.
If conditions have associated questions, then those can be optionally answered and stored within a Review Set. It is not necessary to answer all questions: questions that are left unanswered, will be asked from the end-user.
Only requirements that are selected will be submitted to the LLM.
It is not mandatory to create a Review Set. If you do not create any review set for a particular Review Category, and the end-user selects this Review Category, then all of the elements within that Review Category will be submitted to the LLM (preceded by asking the end-user all questions, since none will be answered).
