Improve error handling for HTTP 4xx errors, particularly 400 and 429

[KernelDeimos]

Problem\n\nThe current fetch tool implementation doesn't properly handle HTTP 4xx errors, particularly:\n\n1. 400 Bad Request errors: These should be retried with potential fixes to the request.\n2. 429 Rate Limit Exceeded errors: These should implement exponential backoff and potentially switch to a "slow mode" to avoid hitting rate limits.\n\n## Proposed Solution\n\n1. Implement retry mechanism for 400 errors with potential request fixes\n2. Implement exponential backoff for 429 errors\n3. Add a "slow mode" option that can be enabled when rate limits are encountered\n4. Improve error logging to provide more context about failed requests\n\n## Technical Details\n\nThe implementation will be in the fetch tool located at:\n\n\nThis will help make the API interactions more robust and prevent failures due to rate limiting or temporary issues.

[bhouston]

Issue Triage Report

Issue Classification

• Type: Enhancement
• Labels: enhancement, help wanted

Initial Assessment

This issue proposes improving error handling for HTTP 4xx errors in the fetch tool, particularly 400 Bad Request and 429 Rate Limit Exceeded errors. The issue description provides clear information about the problem and proposed solutions.

Current Implementation Analysis

After examining the fetch tool implementation in packages/agent/src/tools/fetch/fetch.ts, I found that:

1. The current implementation uses the native Node.js fetch API
2. It logs request details and response status
3. It processes responses based on content type
4. It returns an object with status, statusText, headers, and body

However, the implementation lacks:

1. Specific handling for HTTP error status codes (4xx, 5xx)
2. Retry mechanism for failed requests, particularly 400 errors
3. Rate limit handling for 429 responses
4. Exponential backoff strategies for rate-limited requests
5. "Slow mode" mechanism to throttle requests when rate limits are encountered

Technical Investigation

The implementation will need to:

1. Add configuration options for retries, backoff, and slow mode
2. Implement a retry mechanism with exponential backoff for failed requests
3. Add special handling for 429 responses, including respecting Retry-After headers
4. Track rate limit information from response headers
5. Implement a "slow mode" that throttles requests when rate limits are approached
6. Enhance error logging to provide more context about failed requests

The existing sleep utility in packages/agent/src/utils/sleep.ts can be used for implementing the delay between retries.

Complexity Estimation

• Difficulty: Medium
• Estimated Work: 2-3 hours for an experienced contributor

Next Steps

1. Create a design document outlining the retry and backoff strategy
2. Implement the changes in the fetch tool
3. Add tests to verify the new error handling behavior
4. Update documentation to reflect the new features

Would you like me to assist with any specific part of the implementation or provide further details on any aspect of this issue?