Fleet AI builds training environments that teach AI models how to reason through real-world work. As a task writer, you design challenges for these models by writing prompts and demonstrating how to solve them inside simulated applications. The environments you will work in are interactive replicas of real software, things like email clients, CRMs, project management tools, booking platforms, and more. Your job is to use the environment and the data provided to come up with realistic tasks a user of that software might need to accomplish, write a prompt describing the goal, and then show that the goal can be reached.
<aside> 👉
Before you watch! Please make sure you are able to complete Steps 1-4 in the “Your Assessment” section below. If you run into technical issues, reach out to participant support with the study title “Interested in Paid AI Training Work?”
</aside>
https://www.loom.com/share/981910df40794d03ac3b2e932cf2b306
For your given environment and provided tools, here is how your task (task prompts + API tool workflow) will be graded:
| Category | Explanation |
|---|---|
| Workflow requirements (CRUD operations) | - Your workflow must use at least 4 API tool calls. There is no upper limit! |
| - The final step must be a Create, Update, or Delete (CUD) operation. Do not end with a read (R) operation | |
| Key Criteria: Unambiguous (One correct answer) | Someone needs to read your task prompt and replicate EXACTLY the CUD operations you performed and parameters you used (not necessarily the same R operations). For example, if you say “book a hotel under $500” and there are multiple hotels under $500 to choose from, then that is an ambiguous task prompt ****(in this case, book the hotel under $500 with the highest rating would be unambiguous if there are no ties for “highest rating”) |
| Most Important Criteria: Creative “puzzle” to challenge AI models | We want creativity in logic and reasoning, not writing style (good bonus but not as significant). This is the most important criteria that will decide if you pass. |
A creative task is one that explores the available API tools and data, then connects that information in thoughtful ways to form a logical puzzle to challenge an AI model (e.g., conditional logic, sorts, filters, comparisons, alternate paths). See examples of “puzzles” | | Realistic | Please make sure task prompts would be something that someone would actually do in that application. | | Task prompt = conversational (”prompt”) tone | Please make sure the task prompt has a conversational tone that a real human would use to ask an LLM like ChatGPT (e.g., book the most expensive hotel under $150 instead of book the first hotel with max_price = $150, sorted by max_price in descending order) | | Spelling/Grammar | Tasks with spelling or grammar issues will be rejected. Use a grammar corrector if needed. | | Available Cities | Data within the environment is available for the following cities: Amsterdam, Austin, Bangkok, Barcelona, Berlin, Boston, Chicago, Denver, Dubai, London, Los Angeles, Miami, New York, Paris, Portland, Rome, San Francisco, Seattle, Singapore, and Tokyo | | Credit Card Numbers | If needed for your task, use one of the following card numbers (valid test numbers):
Please note these are snippets, not full prompts. Below are examples for how to increase complexity in your prompts.
| Poor | Good |
|---|---|
| [Email Environment] Send an email to [email protected] that he’s been giving a raise |
Why: This is ambiguous! 10 people would read this and write 10 different emails. | [Email Environment] Send an email to [email protected] with the subject: “Update on Salary” and body: “You’ve been given a raise. Congrats on the hard work!
Why: 10 people would read this and write the same email 10 times. However, see snippets below on how to add complexity while still remaining unambiguous | | [Shopping Environment] From Falmart, add one “Schweppes Ginger Ale” from the soda aisle and one “Kirkland Spring Water 1 Gal” from the bottled water aisle to my cart.
Why: Store name + products were named explicitly. The model can easily accomplish this task without exploring the environment data. | Select the store with the fastest delivery. My wife never lets me drink Schweppes so browse the soda aisle and put one of each in my cart. If the total before taxes/fees exceeds $15, take out the most expensive option and replace it with the cheapest Kirkland item in the bottled water aisle instead.
Why: Good use of conditional logic to add complexity. Store and product names were not named explicitly, which forces the model to explore the environment data to determine the correct answer. | | [HR Environment] Give Ethan Foster a $5,000 raise. Send an email to [email protected] with the subject “Raise” and body “I’ve processed your $5,000 raise.”
Why: This is a two-step task, where each step of the task can be completed independent of each other. You can delete the first sentence and still be able to complete the second part of the task. | Give Ethan Foster a $5,000 raise. Send him an email with the subject “Raise” and body “Your raise has been processed. Your new yearly salary is $XX,XXXX”
Why: Placeholders add complexity (this prompt is written in a way where the actual value would replace the $XX,XXX). This is called data-dependent chaining: the second step depends on the result of one or more prior steps. If you delete the first sentence, the second part of the task is no longer solvable. | | [Sales CRM Environment] Move “Dolphin Phonology Innovations Annual Subscription Deal 1” to the “Presentation Scheduled” stage. Send an email to [email protected], the Director of Technology, and cc [email protected] with the subject “Meeting next week” and body “It was great speaking with you today. Looking forward to our demo next week”
Why: Deal name is named explicitly. You can delete the first sentence and still be able to complete the second part of the task. | There are multiple deals associated with Dolphin Phonology Innovations, find the one with the highest deal size. We have scheduled a presentation for next week, please update the deal stage accordingly. Send an email to the Director of Technology; cc Ashley O'Brien if the deal size is higher than $100K. Use the subject "Meeting next week", and body "It was great speaking with you today. Looking forward to our demo next week"
Why: Avoids referencing objects explicitly while remaining unambiguous (one correct answer). Good use of conditional logic and data-dependent chaining. |