Can't work AI Gateway with bedrock

[watany-dev]

Existing documentation URL(s)

Probably a problem with the signing process, this sample doesn't work(gateway/bedrock), IAM hardcoded something strong so it's not an AWS IAM issue and I can confirm that I have sufficient permissions on the AI Gateway (/app).

https://developers.cloudflare.com/ai-gateway/providers/bedrock/

import { Hono } from "hono";
import { CloudflareWorkersAI } from "@langchain/cloudflare";
import { logger } from "hono/logger";
import { streamText } from 'hono/streaming'
import { AwsClient } from 'aws4fetch';

const app = new Hono();
app.use(logger());

app.get("/app/:msg", async (c) => {
    const msg = c.req.param('msg')
    const cloudflareAccountId = c.env.CLOUDFLARE_ACCOUNT_ID
    const cloudflareApiToken =  c.env.CLOUDFLARE_API_TOKEN

    const model = new CloudflareWorkersAI({
      model: "@hf/mistralai/mistral-7b-instruct-v0.2", // Default value
      cloudflareAccountId,
      cloudflareApiToken,
      baseUrl: `https://gateway.ai.cloudflare.com/v1/${cloudflareAccountId}/${c.env.CLOUDFLARE_AI_API_NAME}/workers-ai/`,
    });
    
    const responseStream = await model.stream(msg);
    console.log(responseStream)
    return streamText(c, async (stream) => {
        for await (const chunk of responseStream) {
            console.log(chunk)
            await stream.writeln(chunk)
            await stream.sleep(100)
        }
    })
});

app.get("/gateway/bedrock/:msg", async (c) => {
    const msg = c.req.param('msg');
    const cfAccountId = c.env.CLOUDFLARE_ACCOUNT_ID;
    const gatewayName = c.env.CLOUDFLARE_AI_API_NAME;
    const region = 'us-east-1';

    // AWS Bedrockの正式なAPIエンドポイントURLを使用して署名
    const modelUrl = `model/amazon.titan-embed-text-v1/invoke`;
    const awsUrl = `https://bedrock-runtime.us-east-1.amazonaws.com/${modelUrl}`;

    const awsClient = new AwsClient({
        accessKeyId: c.env.AWS_ACCESS_KEY_ID,
        secretAccessKey: c.env.AWS_SECRET_ACCESS_KEY,
        region: region,
        service: "bedrock"
    });

    const requestData = { inputText: msg };
    const headers = { 'Content-Type': 'application/json' };

    const presignedRequest = await awsClient.sign(awsUrl, {
        method: "POST",
        headers: headers
    });

    // Cloudflare AI Gatewayを経由するURLを設定
    const gatewayUrl = new URL(`https://gateway.ai.cloudflare.com/v1/${cfAccountId}/${gatewayName}/aws-bedrock/bedrock-runtime/${region}/${modelUrl}`);

    // AWS署名ヘッダーを追加
    const finalHeaders = new Headers(presignedRequest.headers);
    finalHeaders.set('Host', gatewayUrl.host);

    const response = await fetch(gatewayUrl.toString(), {
        method: 'POST',
        headers: finalHeaders,
        body: JSON.stringify(requestData)
    });

    if (!response.ok) {
        return new Response("Invalid response", { status: response.status });
    }
    const data = await response.json();
    return c.json(data);
})

export default app

What changes are you suggesting?

I would like a sample code that works well enough

Additional information

No response

[kodster28]

Howdy @watany-dev, the example you provided here has quite a bit more than the example in our docs (combining some of the Workers AI stuff, which is great overall but makes it harder to eval what exactly is wrong).

If you just run the example present in our docs, does that one work? Or does it also error out on you?

[github-actions]

This issue was closed automatically because there has been no response from the original author. As it stands currently, we don't have enough information to take action. If you believe this issue was closed in error, a) apologies and b) open a new issue and reference this one in the body.

[kittykatattack]

Hello!

I can confirm that the sample in your docs also does not work. This is the code that I am using:

import { AwsClient } from 'aws4fetch'

export default {
	async fetch(request, env, ctx) {

  
	  const requestData = {
		inputText: "What does ethereal mean?"
	  };
  
	  const headers = {
		'Content-Type': 'application/json'
	  };
  
	  // sign the original request
	  const stockUrl = new URL("https://bedrock-runtime.us-east-1.amazonaws.com/model/amazon.titan-embed-text-v1/invoke")
  
	  const awsClient = new AwsClient({
		accessKeyId: accessKey,
		secretAccessKey: secretKey,
		region: region,
		service: "bedrock"
	  });
  
	  const presignedRequest = await awsClient.sign(stockUrl.toString(), {
		method: "POST",
		headers: headers
	  });

	  console.log(presignedRequest);
  
	  // change the signed request's host to AI Gateway
	  const stockUrlSigned = new URL(presignedRequest.url);
	  stockUrlSigned.host = "gateway.ai.cloudflare.com"
	  stockUrlSigned.pathname = `/v1/${cfAccountId}/${gatewayName}/aws-bedrock/bedrock-runtime/${region}/model/amazon.titan-embed-text-v1/invoke`
  
	  // make request
	  const response = await fetch(stockUrlSigned, {
		method: 'POST',
		headers: presignedRequest.headers,
		body: JSON.stringify(requestData)
	  })
  
	  if (response.ok && response.headers.get('content-type')?.includes('application/json')) {
		const data = await response.json();
		return new Response(JSON.stringify(response));
	  } else {
		// Display the data in the console
		const data = await response.text();
		console.log(data);
		return new Response("Invalid response", { status: 500 });
	  }
	},
  };

The data response I get is always:

{"message":"The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details."}

I can also confirm that it is not a problem with my credentials, and I can see the request reaching Cloudflare's AI Gateway in my dashboard. It dies on hitting the amazon endpoint. I think it has something to do with the presignedRequest but I have no idea what. I've tried a few different models for which I have access and the results have been the same.

[kodster28]

Thanks for re-flagging @kittykatattack, re-opening and raising for our ENG team.

[mamtou]

Is there a solution?

[abir-taheer]

Still running into this issue, this is greatly making me rethink using Cloudflare's AI gateway on top of some other issues with token tracking with anthropic :/

[MingkaiChen]

We are also experiencing this issue. We hope this will be fixed very soon :(

[daisyfaithauma]

#21457 Fixed with this PR