Creating Test Cases

A good test suite is a handful of inputs that represent what the Canvas will see in production, each with a result you have checked. The fastest way to build one is not to write Test Cases from scratch. It is to save the runs you already judged: a preview you liked in Craft, a task your team corrected in the Workstation. Creating by hand and dropping files are for the inputs you still need to cover. For the invoice Canvas, the suite is: a clean PDF invoice, a scanned invoice, an invoice with two currencies, and a document that is not an invoice. The first two came out of Craft, the third out of the Workstation, and the last one was added by hand.

From a run in Craft

You have just clicked Run Preview on the clean invoice and the four attributes look right. Save the run before you touch the prompt again.

Review the result in place

Hover an attribute in the result and click the thumbs up or thumbs down. The marks you make here are saved with the Test Case as its first review, so the Test Case is born already reviewed.

Open Add to Test Cases

Under the result, the Add to Test Cases card shows a suggested name under Set a name, and how many attributes are positive, negative and pending. Click Save as a Test Case.

Name it and describe it

In New Test Case from Preview, give it a title under Set a title (for example Clean invoice, single currency) and, under Add a description, say what scenario it covers. The description is stored as a note on the run. Click Create.

Craft of the invoice Canvas after Run Preview, with the Add to Test Cases card under the result

New Test Case from Preview dialog with a title and a description

The … menu of the Inputs panel also has Create Test Case, which opens the same dialog for the current inputs and result. The Test Case keeps the variables, the files and the chat messages of the preview, and shows Manually Added as its source.

The Inputs panel has a Tests / New toggle. Tests lists your Test Cases, so you can run the prompt in Craft with the inputs of an existing Test Case instead of uploading the invoice again. Details opens the files of that Test Case.

In Workflows

After a Workflow run completes in Craft, the same Add to Test Cases card appears in the result pane. It saves the Inputs of the run, including files, and the output of every step. See Running, testing and publishing.

In Agents

In the Test your agent panel, click Review after a run to save it as a test case and open it in the Test experience. See Testing an Agent.

From a reviewed Workstation task

A task your team reviewed in a Workstation App is the best Test Case you can get: a real document, and an output a person checked and corrected. Suppose a reviewer opened the invoice with two currencies, found total_amount in the wrong currency and fixed it before approving.

Right after Approve & Continue, Tela suggests creating a Test Case from the task. Tick Snooze Test Case suggestions for today. to stop the suggestion for a day.
Later, open the approved task and choose Create Test Case in the … menu. The option only appears once the task is approved.

The Create a Test Case from Task N dialog asks for a Test Case Title and, when the task was corrected, to Explain why you made the corrections. Click Create.

Workstation review dialog of an approved task with the … menu open on Create Test Case

What Tela saves:

The task’s files and variables become the inputs of the Test Case.
Every attribute the reviewer left untouched is saved as an Expected Answer and marked Good.
For every attribute the reviewer edited, the original value is saved as a Wrong Answer and the corrected value as an Expected Answer, and the attribute is marked Bad on the original run.
Your explanation is stored as a note on the run.

The Test Case appears in the Test experience of the Canvas behind the App with the source Production. Failed tasks cannot be turned into Test Cases, and with Approval Required on in the App settings, tasks must be approved first. See Turning a task into a Test Case.

From files

To cover an input you have not run yet, drop the file on the Test experience, or click Drop files here / browse your computer.

When the Canvas has a single variable, each file becomes one Test Case named after the file, and it runs right away.
When the Canvas has several variables, or you selected several files for one Test Case, the Create Test Cases dialog opens. For each Test Case, pick a name and Assign file to each variable. Add another test case adds a row; Create N test cases confirms. Every Test Case must have all required variables assigned.

Up to 100 files per drop. Unsupported types are skipped with a warning; see Supported file formats. A Test Case created this way has no reviews yet: run it, then review it.

Untitled Test Cases get a title generated from the prompt and the input once their first run finishes. Rename it from the detail view with Edit Name if the suggestion is off.

By hand

Click New test case (Create manually). The dialog has a Name (prefilled as Untitled Test Case #N) and, under Content, one field per variable where you write text or attach a document. Required variables must be filled. Then:

Finish and Run creates the Test Case and runs it against the version you are viewing.
Finish and Create New creates it and opens an empty dialog for the next one.

This is how the Not an invoice case was added: a purchase order attached to invoice, to check that the Canvas does not invent a total_amount.

New test case dialog with the Name field and the invoice variable with an attached PDF

To change the inputs of an existing Test Case, open it and choose Edit Inputs from the … menu; Save and Run saves and runs it again. Duplicate, in the row menu of the table or in the rerun dialog, creates a copy named (Copy) with the same inputs and no runs, useful for a variant of an existing case.

Through the API

POST /test-case creates one Test Case, or several when the body is an array. The required fields are promptId (the Canvas ID from Connect via API), messages (an empty array for a Canvas without chat) and variablesRichContent. Text inputs go in variables; files go in files, each pointing at a file in your Vault and at the variable it fills.

curl -X POST https://api.tela.com/test-case \
    -H 'Content-Type: application/json' \
    -H "Authorization: Bearer $TELA_API_KEY" \
    -d '[
        {
            "promptId": "5f0c1a2e-8b7d-4c3a-9e1f-2d4b6a8c0e12",
            "title": "Clean invoice, single currency",
            "messages": [],
            "variables": {},
            "variablesRichContent": {},
            "files": [{
                "index": 0,
                "name": "acme-2024-0412.pdf",
                "mimeType": "application/pdf",
                "vaultUrl": "vault://invoices/acme-2024-0412.pdf",
                "variableName": "invoice",
                "url": "https://files.example.com/acme-2024-0412.pdf"
            }],
            "metadata": { "source": "test-case" }
        },
        {
            "promptId": "5f0c1a2e-8b7d-4c3a-9e1f-2d4b6a8c0e12",
            "title": "Not an invoice (purchase order)",
            "messages": [],
            "variables": { "invoice": "PURCHASE ORDER PO-2291 ... (text of the document)" },
            "variablesRichContent": { "invoice": "PURCHASE ORDER PO-2291 ... (text of the document)" },
            "metadata": { "source": "test-case" }
        }
    ]'

Creating a Test Case does not run it. Call Run a test case with the promptVersionId to run, or Run many test cases. The full schema, including expectedOutput, is in Create Test Cases; PATCH /test-case/{id} updates a Test Case and DELETE removes it.

The Tela SDKs cover the Completion API. Test Case endpoints are called over HTTP, in camelCase as shown above.

Organizing with labels

Labels group Test Cases so you can run and read a subset: scanned, multi-currency, regression. Open a Test Case and use the label selector next to its title to Assign label or Create label with a name and a color. In the table, the Labels column shows them and the Labels filter narrows the list; with a filter on, the run button becomes Run Filtered. Labels belong to the Canvas. Through the API, create them with POST /test-case-tags, assign them with POST /test-case/{id}/tags or in bulk with POST /test-case/bulk-add-tags, and pass tagIds in the filters of Run many test cases to run only those.

Test Case labels are not the Tags stored with runs for Usage. The two are separate.

Get started

Build

Test

Operate

Integrate

Creating Test Cases

From a run in Craft

In Workflows

In Agents

From a reviewed Workstation task

From files

By hand

Through the API

Organizing with labels

​From a run in Craft

​In Workflows

​In Agents

​From a reviewed Workstation task

​From files

​By hand

​Through the API

​Organizing with labels

From a run in Craft

In Workflows

In Agents

From a reviewed Workstation task

From files

By hand

Through the API

Organizing with labels