We tested three AI models live on one of the Norwegian Government Standard Terms and Conditions. See how AI can save you time and improve the quality of your contract work.
42 min read
25 May 2025 at 19:00
until 25 May 2025 at 20:06
AI in practical contract work – Who will win the battle over the Norwegian Government's Standard Agreement? Do you spend an unnecessary amount of time filling out contracts and tender documents? In this video, we test how artificial intelligence can make contract work faster, easier, and more accurate – using the Norwegian Government's Standard Agreement, Lille Sky, as our starting point.
In this video, you will see how:
* We put three of the most popular AI models to the test in a live competition:
Google Gemini 2.5, Microsoft Copilot, and Claude Sonnet.
Which one delivers the best quality? Which one understands the task? And which one could become your new sales assistant?
* Gain insight into how AI can be used in practical sales processes and see what sets the different models apart – both their strengths and weaknesses.
* You will also be introduced to how you can use Canvas to improve your workflow, and we share our experiences with challenges such as token limitations and the AI models' understanding of context.
CHAPTERS
00:00 Introduction: AI and contr
Do you spend an unnecessary amount of time filling out contracts and proposal documents? In this video, we show how artificial intelligence can simplify one of the most time-consuming parts of the sales process—namely, completing the Norwegian Government's Standard Agreement, Lille Sky.
Here, we put three leading AI models (Google Gemini 2.5, Microsoft Copilot and Claude Sonnet) to the test in a live competition: Which one performs best – and which one could become your new, highly efficient sales assistant? Watch the video here:
Frode and Adrian wrap up with a summary of the test and give you their conclusion based on their overall impression after the live test.
You can watch all the videos on our YouTube channel. Subscribe to stay up to date
The video is from April 2025. Transcript of the broadcast:
2025-04-04 AI and contracts
Frode Stenstrøm: Hello and welcome. Do you and your team spend hours filling out standardized contracts and proposal documents? Imagine being able to drastically reduce this time—time you could instead spend on customer engagement and value creation. Today, we are going to explore exactly that. We will look at how artificial intelligence, or AI, available here and now, can revolutionize one of the most time-consuming tasks in the sales process: completing the Norwegian government's standard agreements.
Frode Stenstrøm: We have put three of today’s leading AI models—Google Gemini 2.5, Microsoft Copilot, and Anthropic’s Claude Sonnet model—to a practical test: helping us complete the Norwegian Government’s Standard Cloud Agreement (Lille Skyavtalen). Together with my colleague Adrian Sjøholt, we will run a live competition to see which model actually works best today.
We’re not just looking at which one is the fastest, but also at the quality of what the AI models produce. The aim of today’s broadcast is to inspire you and show you what is possible with today’s technology [00:01:00] And how you might be able to work smarter, not harder. So let’s see how AI can become the new, highly efficient sales assistant, and stay with us until the very end of the broadcast, because that’s when we’ll reveal which model actually works best.
And with that, welcome back to the studio, Adrian. This is your second time here. For those who haven’t seen Adrian’s previous broadcast, we had Adrian in the studio to talk a little about approaches to AI strategy and, not least, to outline the current landscape and the AI models available today.
Adrian Sjøholt: Thank you. And that's partly why I'm here today.
Adrian Sjøholt: To take a closer look, there are so many different options out there. What actually works?
Frode Stenstrøm: And we presented a number of use cases in the previous episode, providing what I would call a very clear overview of use cases—in other words, what can you use artificial intelligence for right now?
Not what is coming, so to speak, but what is available now. And we promised to demonstrate the first use case, which is what we are going to do today.
Adrian Sjøholt: That's what we'll show today. Sales and contract generation.
Frode Stenstrøm: Contract generation, yes. And Adrian, just to briefly introduce you to those who haven't seen you before.
Adrian Sjøholt: Adrian Sjøholt, I work as a solution architect and enterprise architect here at ProsessPilotene.
I'm in my seventh year here now, so I've worked on a variety of projects.
Frode Stenstrøm: I think you've seen most of it. What you haven't seen may not be worth seeing.
Adrian Sjøholt: There has been a wide variety of things, and lately there has been a very strong focus on AI.
Frode Stenstrøm: Right
And my name is Frode Stenstrøm, and today on your show, while Adrian represents one AI model, I’ll compete against him with the other one. Then we’ll see which of these models is actually the best, and whether we switch.
Adrian Sjøholt: Maybe we'll switch.
Frode Stenstrøm: Maybe we'll switch. Just a little about the broadcast. When you're watching this broadcast and have questions, feel free to post them in the YouTube comments, and we'll answer them. [00:03:00] And please like the video, share it with anyone you think might be interested, and subscribe to our channel so you can see the other use cases we'll be demonstrating.
Adrian Sjøholt: Yes, this is just the first of several.
Frode Stenstrøm: It is, and now for today's agenda, Adrian.
Adrian Sjøholt: Yes, we're going to look at the challenge we've just presented. Yes. And then there's the question of which competitors are out there. We've already presented the three we're going to talk about. And then there's the competition itself—what actually needs to be done?
We'll come back to that. Then we'll analyze the results—not just the speed, but also which one we think works best. Which one produces the best text? And then we'll draw a conclusion, so that should be good.
Frode Stenstrøm: So I thought we'd start by setting the stage for today's competition, because now you and I are actually going to try to do the same job in parallel.
And when we start, I thought we could set a little timer so you can see how long it takes us and that we're not cheating. It's a live competition, so we may not manage it, but I thought we'd give ourselves no more than an hour to finish.
Adrian Sjøholt: How many hours would you have spent if you had filled it out without AI?
Frode Stenstrøm: Yes, because the background here is that one hypothetical scenario is that a supplier—meaning us in the process pilots, for example—is going to offer a cloud service to our customers. And in that case, we are very keen on using the Norwegian government's standard agreements. They have a number of templates, and it is not unusual to spend at least 40 hours, perhaps as many as 60, 70, or 80 hours, completing such a contract.
It can be anywhere from 50 to 150 pages of text. And doing the grunt work, Adrian, well, it gets to you every time.
Adrian Sjøholt: It’s time-consuming, and that means we task. The same goes for filling it out again and again and again.
Frode Stenstrøm: Time and time again. So what we’re looking for is whether we can use artificial intelligence to help us with the initial draft of the agreement, which we can then take to the customer and use to flesh out the [00:05:00] contract text in greater detail.
And we’re going to show some of the things we believe can save us time. And you’re right, as you say, this happens many times. And so, Adrian, our starting point is precisely this: we begin very simply by downloading the government’s standard agreement, the Lille Sky agreement, from the government website.
There are two documents: the main contract and the appendices, and it is the appendices that are central and need to be completed.
Adrian Sjøholt: Yes.
Frode Stenstrøm: Once we have done that, we will answer questions posed to us by the AI model, which is essentially what we need to complete the contract. We are going to ask users to use something called Canvas, which may be new to you. It allows you to see the result as you go, but you can also choose not to use it. We will go into that in a little more detail.
And our goal is to end up with a completed document, which, as mentioned, would normally have taken at least a week [00:06:00] to complete, and which can then be used to complete the contract. So that is our basis for making the competition fair. Adrian and I have prepared a little, which means that the questions we submit to the AI model are the same, as is the initial basis, but there may still be some differences in how they respond.
Adrian Sjøholt: It depends entirely on what questions they come back with, so, as you say, we have prepared what we are going to ask initially, as well as which customer details and solutions we are going to enter.
Frode Stenstrøm: So we'll see how it goes. Could you tell us a little about it? We're going to have three competitors, Adrian, and the first is Google's model, where we're using the very latest Google Gemini 2.5 model, and
it's free, so that's interesting.
Adrian Sjøholt: And you mentioned that the latest one also has the largest context window. It has a million tokens, which is
Frode Stenstrøm: So it should easily be able to handle that 60-page contract document [00:07:00]?
Adrian Sjøholt: He should be able to handle that just fine.
Frode Stenstrøm: So I’m pretty confident about that one. I’m rooting for it a little, and I think it’s going to win, but it may be able to convince me.
Adrian Sjøholt: It still has a few teething problems. It is quite new, after all.
Frode Stenstrøm: Yes, you might get to see it. And before the broadcast, we did this with Anthropic's Sonnet, and we've also prepared it for demonstration now.
And I know exactly how long it took me from start to finish to turn this into a finished product. And then you're going to test Microsoft.
Adrian Sjøholt: Yes, I do have a bit of a soft spot for Microsoft. I hope they can prove that they’re capable of beating both Google and Anthropic, so we’ll see. But maybe you’ll win me over here?
Frode Stenstrøm: Yes, we'll see. It's very, very exciting, and perhaps the conclusion of this episode will be that I, too, become an avid Copilot user. We'll see, we'll see.
Frode Stenstrøm: So the first thing we're going to start with in today's competition, and the first question we're going to ask the AI model, is this.
"I want to complete the appendices to the Norwegian government's standard agreement for cloud services (Lille Sky), version 2021, so that I can sell the following services." This is followed by some text describing which services are to be included in the delivery. The appendices should be created in such a way that I receive a customized contract template that I can use in my sales.
I add the agreement text and the appendices, and then comes the part that is quite important in our prompt to the AI model: "I now want you to ask me questions that will enable you to complete the appendices for me." So instead of simply completing the appendices right away, we ask all three AI models to ask enough questions for us to actually get a result.
And with that, Adrian, we're simply going to get the competition underway. I'm going to paste in that text, and you've pasted in the text.
Adrian Sjøholt: What is it they usually say? We've cheated a little.
Frode Stenstrøm: Yes, we have the glue, we've cheated a little, so now I'm simply going to paste that text in here. And then you've done the same thing in Copilot.
Adrian Sjøholt: Yes, and I'm also using [00:09:00] the free version.
Frode Stenstrøm: You’re using the free version, yes. That’s very important.
Just to get a good comparison
Adrian Sjøholt: here, so we don't end up using paid versions for testing.
Frode Stenstrøm: So you use the free version of Copilot, I use the free version of Google Gemini, and I have also used the Claude Sonnet model.
Frode Stenstrøm: And before we get started, I'll start the timer, and I just want to give you a little context. In the Google Gemini world, you'll see that we have the Flash model, as well as the Experimental and Deep Research models. In short, Flash is the most commonly used model. You can use it much like you would use ChatGPT.
Flash Thinking makes the model reason through your question before it actually... answers. Deep Research is really useful, because it will typically search the internet for more information and can also use the information it finds in the contract work.
Adrian Sjøholt: It’s also important to point out that Google’s model searches in real time and retrieves up-to-date information.
Frode Stenstrøm: It does.
Adrian Sjøholt: If you try to do the same thing in Copilot, you’ll often get a response saying that it can’t be done.
Frode Stenstrøm: Because he doesn't know that.
Adrian Sjøholt: It simply doesn't have that data available.
Frode Stenstrøm: As mentioned at the outset, there is a canvas feature that you can toggle on and off here. When you turn on the canvas feature, instead of revising the entire text again, you can, just like in Copilot, highlight a paragraph and revise only that paragraph.
It's kind of all right, because it means not having to rethink things over and over again. So I've pasted the documents in here in Gemini; I could paste the documents in here. In AI Studio, I have to grab the PDFs, Adrian, but there's not much difference, so I grab them and drag them in.
Adrian Sjøholt: Exactly, you just make this into a PDF, using the same files.
Frode Stenstrøm: Yes, these documents that we are actually going to try to fill out, so that those at home can see them, are this [00:11:00] standard agreement based on the government's standard agreement. The template itself is 34 pages long, and it is not unusual for it to quickly grow to more than 60 pages once it has been completed with all these appendices and chapters.
So our goal is to fill all of this out clearly and thoroughly. So, Adrian, we can start the stopwatch now, and then I'll click Run.
Adrian Sjøholt: Run. Then it starts generating.
Frode Stenstrøm: So now I’m excited to see what happens. We’re not going to cheat on the timing; we’ll just let it run.
Adrian Sjøholt: Here he has started processing!
Frode Stenstrøm: Are you in the lead?
Adrian Sjøholt: My leader so far.
Frode Stenstrøm: What you'll see with Gemini is that it's a reasoning model. That means it looks at what I've asked for, it looks at the source material and the text, and then it starts analyzing what I'm actually asking.
And that’s quite interesting, Adrian. Here you can actually see the entire reasoning process and how it works. And now [00:12:00] it generates—well, I don’t need to see the reasoning; I can just close it—but now it starts talking to me. And then there’s a canvas view here showing the answer to the question. And what I can now do is actually answer the question here in the canvas window.
But I don't want to do that, because then I might as well have filled out the document.
What does it look like on your end, Adrian?
Adrian Sjøholt: Yes, I’ve received a number of questions that he wants answered. And then he can fill out the supporting documents according to what I need.
Frode Stenstrøm: Correct.
Adrian Sjøholt: That's what I've been told, at least, so it'll be exciting to see.
Frode Stenstrøm: But the questions you've been given now—how are they structured, Adrian? Are you comparing them? Do you have some general questions?
Adrian Sjøholt: Yes, I have general questions about the services, followed by specific questions about the appendices.
Frode Stenstrøm: Correct.
Adrian Sjøholt: That’s how I approach the requirements specification and that part of it. And those were really the two main categories I came up with.
Frode Stenstrøm: And if we compare the questions you've received—and we can simply show both screens—we'll first look at the Copilot questions, where you can see that you have the general questions. If you switch over to my screen, we can see that we have similar general questions, and I've also received questions about the appendices.
Adrian Sjøholt: I have that down here too—appendices 1, 2, 3, and 4. So here, it has been adjusted to one question per appendix.
Frode Stenstrøm: Yes, and I have actually seen that too.
Adrian Sjøholt: So, so far, the initial overview looks pretty much the same.
Frode Stenstrøm: Now I’ll also show you that I asked the Claude model the same question. It says exactly the same thing. What is interesting to note here is that even though I provided this document in Norwegian and asked the question in Norwegian, the response is in English.
Adrian Sjøholt: Mine were in Norwegian.
Frode Stenstrøm: Okay, interesting. Here are the same questions, you see, but I think there were perhaps roughly the same number of questions per attachment. I see there are two questions per attachment. How about you?
Adrian Sjøholt: I have received two questions per attachment.
Frode Stenstrøm: Yes, and if we look here, we can see that these are roughly, perhaps slightly more specific questions from Gemini.
I see that it asks quite a lot about the different things. And if I then try the AI study that actually goes with the model, just to see whether Google is consistent, I can see that I get many of the same questions here too. Okay, but what we've done, Adrian, is put together similar answers to what we're going to ask about.
Adrian Sjøholt: It is, after all, to provide the best possible basis for comparing the models.
Frode Stenstrøm: Yes, exactly. So I think we should use the answers we have. I’m not saying they address all the appendices precisely, but at least they provide some more information. And this is typically information you get from the customer during the sales process.
When you’ve spoken with the customer, you learn things. And that’s what we’ve gathered here, so I’ll copy it now, and then at [00:15:00] I’ll go into Gemini, and then we’ll click paste.
Adrian Sjøholt: Here’s a quick tip for those of you who need to copy and paste several things at once. Instead of using Ctrl-V, use the Windows key plus V, and you’ll see a small list of the different things you’ve copied.
Frode Stenstrøm: So, now I've pasted the two things in here as well, and while these models are working in Gemna, I thought I'd just show you what this looked like in Claude Sonnet
Frode Stenstrøm: No, the competition continues. And what I did in Google Gemini was that I just took an answer here, and then I started cutting and pasting a question and answering underneath it. And I actually think this is a bit messy, because when you do this in Gemini—no, sorry, in Claude Sonnet—and don't have the Canvas model, you have to do everything in the chat.
Whereas in Google Gemini, I could now have listed the questions below in Canvas. And what’s kind of cool about that, from a process perspective, is that I could have compiled these questions, brought them into a customer meeting, gotten answers from the customer, and then simply filled them in afterward.
Adrian Sjøholt: That's really great. It also provides a starting point for the meeting.
Frode Stenstrøm: Yes, and then you can also have a completely different kind of communication with the customer, who will then feel a greater sense of ownership of the contract. And what’s rather interesting now, Adrian, is that thanks to the summary, there is enough detail to start filling in the appendices. So I would actually have started now.
Adrian Sjøholt: Has he started, then?
Frode Stenstrøm: Yes.
Adrian Sjøholt: Yes, I have received a summary of the information I provided. Thank you for the information. Here is a summary of how this will populate the appendices. Then, if there is anything more specific you would like to include or change in the appendices, I can go through it if anything was unclear or if it was...
misinterpreted, then I can go in and change it.
Frode Stenstrøm: Yes.
I found this exciting, because here you can actually see that I have two canvases like this. I can open the questions here that formed the basis, and then I can open the final result here, and the entire agreement text is actually shown below. But I thought it was a little thin, so now, Adrian, I’m going to see how the other model did.
It's actually the same Gemini, but from AI Studio instead of the ...
Adrian Sjøholt: I haven't gotten the results yet, but I do need to confirm that what I've entered looks correct. And what I think is great about Copilot now is that you can get suggestions for questions you could elaborate on and ask the model to fill in more information for you.
But shall we say we're happy with the starting point and ask it to finish generating?
Frode Stenstrøm: No, I think we should—now we have to—I now have to quality-check what it has produced, because I'm not so sure this is 100%, because I can only send
it off. No, so while you're quality-assuring it, look at your questions and see if there are things that should have been filled in but aren't there.
I'm going to do the same thing here. What I did when I did this in Claude Sonnet was, as I mentioned, to go through these questions, ask them iteratively one after another, and fill out each individual section. By then, I had answered all the questions. I completed the appendix one round at a time, taking a slightly more iterative approach, and that's how it will work in practice when you do this.
So once I had gone through everything, you can see that it had created complete documents and SLAs and so on, and then it said, “Now I have everything I need.” It then produced a finished result, and the first result I got from it gave me a very nice presentation of the contract. It included the appendix text, as you can see.
It has managed to check off yes and no and indicate which appendices are included in the agreement, which actually impresses me quite a lot.
Adrian Sjøholt: He has a pretty good understanding of what it [00:19:00] does on its own.
Frode Stenstrøm: I also think the Claude Sonnet model presents it in a very readable way. And then come the appendices to the agreement.
And this here, I thought it was—let's see—it starts there. And here it basically states what the customer's requirements are, what it concerns, what the deliverable is, and it has defined the deliverable. And what I did when I was working on this, Adrian, with the Sonnet model, was that I noticed that some of these points had been placed incorrectly.
For example, I saw that some things had been categorized under operations when they should have been under management. Some things were under services when they should have been under, in a sense, custom solutions. And this type of contract is a little unusual in that some things are cloud services, while others are add-ons to the cloud services.
So I had to spend some time talking to him, and this is where I spent most of the remaining time in the hour I worked on this in practice, providing feedback and moving things around. And one of the really good things I noticed about Adrian, which is quite interesting, is that if I moved an item from the customer's requirements in Appendix 1 and noted that it was, for example, part of management rather than operations, he updated Appendix 1, he updated Appendix 2, which is the supplier's response, and he updated the pricing appendix.
Adrian Sjøholt: And those are things you can easily almost forget to do yourself.
Frode Stenstrøm: A huge advantage. And then we went through some of the WCAG accessibility standards, so when I did this, we asked quite a few questions further down, and what the other models have received is a summary of these questions.
But I can already reveal that from the time I started this until I was finished, I spent about four hours in total on absolutely everything, and do you know what took the longest? Getting it into Word.
Adrian Sjøholt: But you didn’t get the finished Word document out?
Frode Stenstrøm: Well, yes, but that's for Word and formatting, which we've never had any trouble with, so I'm putting that aside for now.
Frode Stenstrøm: So I think we'll just take a look at the draft now, Adrian, read through it a bit, and see if there's anything we should have included. [00:21:00] Because I thought the wording here was a little different. When you look at the knowledge specification in the appendices now, what does it look like on your end?
Adrian Sjøholt: I haven't received the text yet.
Frode Stenstrøm: So does he create the Word document for you, then? The interesting thing about that, Adrian, is that if you have Copilot in Word and you get the groundwork done, you can use Copilot in Word to continue working on it.
Adrian Sjøholt: Then you can switch over to Word and simply continue working there.
So now mine has started generating. We'll see what the result is. In the meantime, we can take a closer look at yours.
Frode Stenstrøm: Yes, because I’m not very happy with the wording here. I think this is a typical requirement. This is somewhat imprecise contract language. So I’m going to say: use language that is precise, not [00:22:00] promotional, but neutral. This is supposed to be a contract, after all.
Adrian Sjøholt: It's the .docx file that's in the payload here
Frode Stenstrøm: Done?
Adrian Sjøholt: No, there's another issue with that file because it isn't a .docx file
Frode Stenstrøm: Interesting. Now I'm going to ... I'm going to give that feedback to both of these models of mine. And this is live work, so there's no guarantee that this will turn out right ...
Now I'm not so sure it'll actually get finished here
Adrian Sjøholt: Yes, I've managed to open my document.
Frode Stenstrøm: Okay, now I’m curious. Could you show us—just go ahead and see what you got out of it? Because this looks very elegant.
Adrian Sjøholt: Got the table of contents with all the items. Yes. Including appendices and everything that goes with it.
Frode Stenstrøm: Impressive.
Adrian Sjøholt: Comment.
Frode Stenstrøm: So he has kept the instructions that were in the template, then?
Adrian Sjøholt: Yes, he has. He is not an exciting you. The customer must file off the customer. The orderer's performance.
Frode Stenstrøm: But this was just the template. It was just the template, Adrian.
Adrian Sjøholt: Yes, now I’m a little curious to see what he’s done here.
Frode Stenstrøm: It certainly doesn't look like he's done a thing.
Adrian Sjøholt: No, this doesn't look good. He's only given me the template.
Frode Stenstrøm: But have you downloaded the template? Is it user error?
Adrian Sjøholt: It’s probably Copilot that hasn’t done what it was supposed to do.
Frode Stenstrøm: It hasn't really filled anything in. If we compare it with Claude Sonnet now, you can see that it has—here I had actually gotten—I'm really very pleased with the language here, because it says things like the customer needs a blah blah blah, very short sentences. It doesn't use that “should” wording, which is risky in the contract; it says “shall,” “shall.” There's clear text all the way down here. Tables and such, and all the appendices here—the sections have really been completed down to the last detail.
Adrian Sjøholt: The first draft here wasn't very good; here he has just put everything at the end
Frode Stenstrøm: Oh, he put it all together at the end. Yes [00:25:00] Okay, yes
then I think you should go back to Copilot and try talking to Copilot a little and see if you can redeem Copilot's honor and redeem Copilot's honor
Adrian Sjøholt: Yes, we hope we can
Frode Stenstrøm: I can see that when you look at Google AI Studio, where I've been working, I now have some text.
And I have to say that I understand that precision and neutrality are crucial for contract language. I adjust the wording, as instructed. We remove promotional language and focus exclusively on the actual obligations. Right after this, I think Google has provided a very good answer here. It also says something else.
References to clauses in the main agreement are retained. This provides necessary context, and I am somewhat impressed by that. Because what is a little concerning when dealing with these agreements is that you can write all sorts of things in the appendices, but the main agreement takes precedence. So he has retained that. And then he has written it out, and now I am beginning to see that this appendix is taking shape according to the template where ... the customer's requirements are ... And performance..
Adrian Sjøholt: No
Frode Stenstrøm: The plan was okay, but not very clear. The service level is okay. So far, I have a slight preference for ... the phase breakdown. I think it is written much more precisely and clearly. What happens when you switch on Copilot, Adrian? What does it look like there? At least it looks better now. Have you explained how to create a new version by filling out the appendices?
Adrian Sjøholt: Yes, we'll take a look at that now if we jump down to compare the two.
Let's see which one we should start with.
But they’ve just used the standard text again. That’s the question.
Frode Stenstrøm: But he really has done a good job here, hasn't he? Because now you're in...
Adrian Sjøholt: Now we're attachments. But he puts everything in at the end here too, you know.
Frode Stenstrøm: But could you retrieve the data text from Copilot in that chat, and does it put everything directly into the Word document?
Adrian Sjøholt: It puts everything directly into the Word document.
Frode Stenstrøm: Because I see a certain significant difference between these three models: you’re working on it in Word now. I haven’t gotten as far as creating documents, but I talk and use the canvas to present the result. And I see that in Claude, you get this document view on the right-hand side, which I find very neat and very easy to read.
Adrian Sjøholt: It’s easy to make changes. In my case, I now have to go back to Copilot, give it the new instruction, and then reload this file.
Frode Stenstrøm: And when I did this, Adrian, you can see that this is version 1.
There are 52 versions of this document. And that was because I kept correcting it as I worked on it. And what that really means is that if I had been sitting with a client, asking questions and answering them, I would have fed it all into the model, and then I ended up with 52 revisions, which is not common when working on a contract.
And that’s why I said that when I did this, I spent four hours, because I went back and forth between myself and the customer until we had a solid basis for the contract, and it was only once I had done that that I exported it as a PDF or Markdown, depending on what was needed, or simply copied the entire text and pasted it into Word.
And that way, I’m able to retain some of the formatting. That’s what I was hoping Copilot would do in the Microsoft ecosystem now.
Adrian Sjøholt: The formatting is there, but I wouldn't say the solution is good.
Frode Stenstrøm: The solution is not good.
Adrian Sjøholt: It just pasted everything at the end. Done.
Frode Stenstrøm: No, but that was interesting. Let's take a look at what this looks like in Google's world. In Google Gemini, I've enabled thinking.
Then I'll close that part, and take a look at the opening of the contract document.
Here it is ... What I really liked about Adrian's Gemini model, which Claude Sonnet didn't do, was the references.
It looks at the customer's appendices, then writes a text, but refers to the clauses in the main agreement to which it relates. And that is slightly different from the Sonnet model. Because if you look at this model, you have to do this when reviewing the appendices.
Appendix 1, for example, essentially follows the template. Claude Sonnet does not make that reference, but it retains the structure as it appears in the agreement template text. And [00:31:00] I think that is really how people are used to reading it. I can see that I am not particularly fond of typical.
Adrian Sjøholt: No, you need to be more specific.
Frode Stenstrøm: But the fun thing about it is that when you use Canvas, I can ask Gemini, like—Be specific here, set requirements.
And then I can, instead of revising the entire contract text.
Adrian Sjøholt: Then you'll get a new revision.
Frode Stenstrøm: Yes, you get a new revision that then just takes—look here.
Adrian Sjøholt: Yes, the needs ...
Frode Stenstrøm: There. And then it gets updated live, right?
Adrian Sjøholt: Is he thinking about the other one too now, or is he only thinking about that one?
Frode Stenstrøm: No, it’s only what I selected in it just now, and that’s what I think is quite sensible, and I think this is better than the Sonnet model’s default, because it just rewrites everything over and over again.
I also know that Sonnet is introducing similar Canvas features, and I think you get something similar in Copilot as well, but this is a really useful feature. And what I really like about it is that it typically says “other needs.” So I now have the option to enter other needs as well.
Frode Stenstrøm: But what I am going to try to test, which I find interesting, is that when you look at application operations, I can say to Adrian: Follow the standards set by IT Service Management. And specify the requirements that normally apply to application operations and application management. Divide these two into separate main requirements.
Adrian Sjøholt: It’s very easy to edit and provide feedback along the way.
Frode Stenstrøm: I think so, because in a way you've been chatting back and forth with Mento and rewriting everything over and over again. Now I'm asking you to do those things, and for this to work, you actually need to check online what the requirements are in context.
Frode Stenstrøm: That's fine, I understand. I'll split it into separate sections in Canvas and formulate it.
Here, he has linked it to familiar concepts and processes that the customer will often be familiar with. Wow. I'm actually a little impressed now.
Adrian Sjøholt: Yes, and then there’s the fact that he uses requirements rather than needs.
Frode Stenstrøm: Now it was worded using the correct contract language. Now I was ... Look here. Yes, he uses it. Okay, but I thought that was good.
But then I'd like to say something, because now I'm going to try to challenge him a little.
Update Appendix 2 to match the customer's requirements.
There. Now that we’ve made a change to the customer’s requirements, we also need to revise the requirements in Appendix 2, which would normally take a lot of time. While it’s working, how are things going for you, Adrian? Have you made any more progress with Copilot, or is it still producing the same output?
Adrian Sjøholt: No, I wouldn't say it's been that good so far.
Frode Stenstrøm: But what is the chat experience like in this?
Adrian Sjøholt: It generates a new document every time, and I have to write what I want it to do. In a way, I can't use the interface in the same way to go in and edit specific text, because it's a Word file.
Frode Stenstrøm: So if you had Copilot in Word, you could have done that?
Adrian Sjøholt: I probably could have done that.
Frode Stenstrøm: But it is a commercial product, and now we're essentially comparing free with free.
Frode Stenstrøm: What is impressive now, if we look back at the Gemini model here, Adrian, is that we changed quite a few of the requirements in Appendix 1, and now Appendix 1 is starting to become quite clear, I think, in terms of the customer's requirements.
Absolutely. And if we compare the requirements that have now been included in Appendix 1 below with, for example, the requirements in Anthropix, I still have to say, Adrian, that I prefer this wording. It is simpler: The customer needs a modern, user-friendly digital collaboration solution, then we communicate with it, and so on.
Adrian Sjøholt: It is easy and clear to read.
Frode Stenstrøm: Yes, you manage to convey a great deal very concisely about what matters most for the customer's requirements. The requirements for cloud services are very accurately worded, because in contractual terms, cloud services have a very specific [00:36:00] meaning. You also have the option to improve the text directly. That wasn't available when I tested this a few weeks ago.
And you can see what the deliverable is and what it includes, so I liked this one. I have a slight preference for Sonnet. But I have to say that this is... And if you look at the response to Appendix 2, you can see that he has now actually managed to describe the whole thing.
Adrian Sjøholt: It has 4,000 tokens in the free version.
Frode Stenstrøm: Look at the count now—we've built up 40,000 tokens.
Adrian Sjøholt: Yes, so that's why we're not getting an answer out there.
Frode Stenstrøm: But if Copilot were paid, would that increase it to something more, or?
Adrian Sjøholt: We can ask Copilot about that.
Frode Stenstrøm: Yes, that should be enough, because there was very little, I think. Because I can see now that the only thing I have left to do, Adrian, is to export this to documents. I just press a button.
And I’ve just clicked that one button, and I’m very excited to see what happens here now. There we have the contract. The whole thing—admittedly in Google Docs, but it’s easy enough to just copy and paste it and turn it into a Word document. And this document here, Adrian, it ended up not being that long, actually—13 pages—but it is.
Adrian Sjøholt: How many tokens did you use for this one?
Frode Stenstrøm: For one of them, I’m at forty thousand and something, but for the other one I’ve probably passed fifty.
Adrian Sjøholt: So either way, Microsoft will actually be out of reach. The paid version of Copilot Studio costs 32,000.
Frode Stenstrøm: That isn't enough either
Adrian Sjøholt: So here it is very important to choose tools based on your needs
Frode Stenstrøm: Because when I was working with Claude, as I said, I saw that it was able to do the job, and it did a good job. But as I mentioned, I noticed that it started to struggle when I got to a certain point here because of the token limit, but I still managed to use Claude Sonnet to get it done. And the final document I created looks like this; I transferred it to Word. The interesting thing here is that this was the contract text, but if you look at the appendices themselves...
This is my final result, and I think this document is ... How many pages did it end up being? I think it came to...
Adrian Sjøholt: 60 pages, yes,
Frode Stenstrøm: 60 pages, yes! So, what I did—and I’m going to show you a little trick—is that one of the things I really, really, really like about this contract work, and I’ve tested this with both Gemini and Claude, when you have that many tokens, is that the vast majority of deliverables have a responsibility matrix.
And I’m going to show you that responsibility matrix, because writing a precise responsibility matrix that aligns with the rest of the contract is a real nightmare. And I’m going to show you the responsibility matrix I wrote for this contract. I’ll get to it very soon.
(scrolling)
There it is. So what I did here—this responsibility matrix, Adrian—the AI wrote it for me. It uses standard responsibility matrices as a basis. It identifies the parties involved. And then it lists all the tasks, the customer, the supplier, the cloud provider, and a reference to where it appears in the agreement.
Adrian Sjøholt: Very much so, yes.
Frode Stenstrøm: So one of the things I do that really saves me a lot of time when working on contracts, and that people generally don't do well enough, is to describe the responsibilities of the parties. And the way I did this was actually—now I almost made some changes to this document of mine, and I don't want that.
So the way I did it was that I opened Google's Gemini model. As you can see here, I have a responsibility matrix, and I used Google AI Studio here, but it doesn't really matter—you can do it with Gemini as well. The point is that I took my completed document, with all the appendices I had finished
Adrian Sjøholt: So you generated it first as well?
Frode Stenstrøm: Yes, so I had sort of gone in and talked to people and customers about what it should be like. Then I uploaded the document, which was 15,700 tokens, and said: use this agreement and create the responsibility matrix in Norwegian, using the usual standard. And then it actually produces the entire completed responsibility matrix. And because I said I was going to use IT, or use those processes and things like that, it includes [00:57:00] general matters, service delivery, SLA and support, security and privacy, commercial terms, and termination.
And then you get the agreement, along with a very clear and useful responsibility matrix. Filling this out manually, which I’ve done a few times, also takes quite a few hours of work. So I think we’re getting to the point where we can stop the timer. We’ve managed to create a draft of the agreement.
Adrian Sjøholt: Absolutely.
Frode Stenstrøm: Then the clock will show how long we’ve spent on the whole thing.
So, we’ve had the robot answer questions, Adrian, we’ve had a dialogue with the robot, and we’ve tried to get it into Word. I thought we could talk a little about the role of AI agents, because we’ve now done a number of practical things—we’ve tried it, you’ve tried it, and I’ve tried it. But what you saw examples of along the way is that one example of an AI agent could be performing a Google search.
And now Google Gemini could search the internet and retrieve factual information, such as [00:58:00] data processing agreements, procedures, and guidelines that apply to your industry. This ensures that what it writes in the contract text is accurate and based on what is actually the case, rather than what it assumes.
Adrian Sjøholt: Yes, and an up-to-date person, not least
Frode Stenstrøm: And what’s quite interesting—and we’ll talk a little more about this—is that we mentioned MCP agents or an MCP server last time, but if you have an agent that can read your ERP system, e-commerce system, product catalog, or inventory system, the contract can also retrieve data from any other systems you may have and update the contract document with the information in your authoritative records. That’s where it really starts to feel like magic. Contracts very often need to refer to certain rules, laws, or something similar, so creating an agent that retrieves that information enables Gemini and the other models to draft even more precise contracts.
Adrian Sjøholt: So if there is a new revision of any legislation or regulations, you'll hear the updated version
Frode Stenstrøm: No, exactly, and then it’s kind of like: Shall we try to summarize?
Adrian Sjøholt: We pretty much have to, since we promised we would.
Frode Stenstrøm: Did Copilot get the job done, then?
Adrian Sjøholt: This one gets few tokens.
Frode Stenstrøm: This one gets few tokens, so there’s no tool to do it?
Adrian Sjøholt: Not the right tool for this job. It’s very good for small tasks. As we saw, the free version supports 4,000 tokens and the paid version up to 32,000, which is still extremely low compared to Claude’s and Gemini’s models.
So it will be able to handle its use cases. With 4,000 tokens in the free version, you might get answers to your questions or be able to generate simple things, but that is more or less where it stops.
So that rules it out for document generation.
Frode Stenstrøm: That rules it out. And I note that it is precisely the issue of document size, because we have now created a contract document.
But the advantage of having such a large context window, capable of processing so many characters, is that if, in the next round, we want to create a project plan based on the one set out in the agreement, it will have the entire agreement in its memory and can do exactly that, whereas Copilot obviously cannot.
Adrian Sjøholt: It wouldn't have stood a chance. We saw that with a completed file we generated, which came to 15,000. At that point, you've almost used up your tokens on the paid version of Copilot.
Frode Stenstrøm: And unless you're drafting a two-page agreement, I would say that for most agreements that are somewhat complex, where the point is to save 40 hours of work, we've now managed to produce a draft agreement in—well, the timer here shows how long it actually took us to finish it.
Even though we chatted a bit among ourselves, that’s how long it took.
Adrian Sjøholt: And comparing a little, you would have managed even faster if you were on.
Frode Stenstrøm: Exactly.
Frode Stenstrøm: And I have to say that we now have two winners. They are Google's latest model, Gemini 2.5, and Claude Sonnet 3.7. Initially, [01:01:00] I actually preferred the presentation and visual experience in Claude Sonnet to Gemini, but that is also partly a matter of getting used to it. As I see it, only one of them can handle the full context, but Anthropic, the company behind the Sonnet model, has said that during April 2025, they will increase the context window to 500,000 tokens.
Adrian Sjøholt: Yes.
Frode Stenstrøm: And then those models will essentially be on an equal footing, and it becomes more a matter of preference and which one you think is best to use.
Adrian Sjøholt: That way, you avoid having to extract and split up the document.
Frode Stenstrøm: Switching and splitting and things like that, yes. And I have to say that one thing Google does well is exporting it to Google Docs. It was just one click, and there it was. Then you're up and running, and your document is ready.
Adrian Sjøholt: So, are there many people who use Google Docs regularly as well?
Yes, you can hear that clearly.
Frode Stenstrøm: Yes, so even though we sell a lot of workshop technology, we still have to say what is... What was best today, then. And that raises the question: who actually won? What do you think?
Adrian Sjøholt: If I were starting now, I would simply choose Google's Gemini model.
Frode Stenstrøm: And I would have done that too, because it’s free.
I pay $20 a month for the Claude Sonnet model. So I would have chosen Google, quite simply, for that reason alone. Google also has a search engine feature that lets you connect it to search the web. So if I were to do this again, I would use Google's. And imagine a customer meeting where you are discussing contract details with the customer: you can have that canvas open, sit down with the customer, and specify points directly in the text, which means you can move forward very quickly.
And that essentially means that if you do the things we’ve just demonstrated, you can easily produce a 50–60-page basis for a contract in an hour or two.
Or however long we just spent—we'll have to check the timer; I don't know.
Adrian Sjøholt: We cheated a little with the questions. You would have spent a bit more time finding [01:03:00] answers and things like that, so we have to add that time, but you're still saving a fair amount compared to a week's work.
Frode Stenstrøm: And when I did this last week using the Sonnet model, Adrian, as I said, I timed myself—it took four hours from when I started until I was finished, and most of that time was spent struggling to get it into Word.
Adrian Sjøholt: So that may be a more realistic time estimate.
Two to three hours may be realistic for producing a draft that you can then take to the client, discuss the details, and revise—that's the usual communication process—but it means that a process that could easily have taken 40–60 hours can now be completed in a single working day from start to finish.
You can write two a day.
That is really what we wanted to show you today as inspiration. To summarize, we looked at the challenge of saving time on contract work, we looked at the models, we competed live, and we will see how much time we spent, what we analyzed, and the conclusion. And with those [01:04:00] words, I would like to thank Adrian for his contribution to today's broadcast.
Ask a question or share what helped you.
Share a question or reflection.
Be the first to contribute.
