- Name–the user-defined name of the project.
- Routing Rule–the routing rule for the project. Routing rules are set up on the Routing Rules page.
- Status–the status of the project: Active or Inactive. Active projects receive routed traffic.
- Monthly Spend–the monthly spend allotted to the project, along with the amount spent this month. For example, $100 / $700 means $100 of the $700 monthly budget has been spent.
Add a Project to the LLM Gateway
To add a project, do the following:1
Click + Add Project. An Add Project dialog appears.
2
In the Details tab of the dialog (Step 1), enter a Name and Description for the project. Click Next.
3
In the Routing Rule tab of the dialog (Step 2), select an Existing Routing Rule or Create a New Routing Rule. See Routing Rules for more information on creating routing rules. Click Next.
4
In the Limits tab of the dialog (Step 4), add the following:Spend Limits section:
- Budget—enter the monthly budget for this project, or select Unlimited. If you select Unlimited, the remaining Spend Limits fields are hidden.
- Limit Type—toggle Hard Limit or Soft Limit. A hard limit stops requests after the budgeted amount. A soft limit sends a notification that the limit was reached, but allows requests to go through.
- Notification Threshold—set a threshold level, such as 75% of budget, for sending a notification (required for soft limit).
- Notification Frequency—select Once or Hourly for sending threshold notifications.
- Request Limit—enter the maximum number of requests and select the time period: minutes, hours, or days.
- Token Limit—enter the maximum number of tokens and select the time period: minutes, hours, or days.
5
Click Add to add the project. The new project’s summary appears on the project’s details page.
6
You can issue a virtual key for the project or return to it later. See Virtual Keys for details.
7
Make any other changes to the project and click Save Changes. The new project appears on the Projects tab with a default status of Active.
Edit a Project
To edit a project, do the following:1
On the Projects tab, click the edit icon for the project you want to edit. The project’s details page appears.
2
Edit the Name and Description as needed.
3
In the Routing Rule tab, change the routing rule or the prompt enrichment settings. See Routing Rules for details.
4
In the Spend Limits tab, edit the spend limits for the project. See Spend Limits for field descriptions.
5
In the Rate Limits tab, edit the rate limits for the project. See Rate Limits for field descriptions.
6
To manage virtual keys for the project, see Virtual Keys.
7
Click Save Changes.
Spend Limits
The Spend Limits tab controls the monthly budget and notification settings for a project.- Budget—enter the monthly budget for this project, or select Unlimited. If you select Unlimited, the remaining Spend Limits fields are hidden.
- Limit Type—toggle Hard Limit or Soft Limit. A hard limit stops requests after the budgeted amount. A soft limit sends a notification that the limit was reached, but allows requests to go through.
- Notification Threshold—set a threshold level, such as 75% of budget, for sending a notification (required for soft limit).
- Notification Frequency—select Once or Hourly for sending threshold notifications.
Rate Limits
The Rate Limits tab controls the maximum number of requests and tokens for a project. Select Unlimited to apply no rate limits. Otherwise, enter the following:- Request Limit—enter the maximum number of requests and select the time period: minutes, hours, or days.
- Token Limit—enter the maximum number of tokens and select the time period: minutes, hours, or days.