Create your first worker

We will create a simple web scraping worker that listens for URLs to scrape, fetches the content, and logs the results.

Web scraping is a perfect use case for task queue workers, because of built in retries and ability to fetch websites in parallel.

Setting it up is straightforward and takes just a few minutes.

Requirements

Before starting, please read the Install Edge Worker guide.

1. Create your worker

   First, create a new Edge Function using the Supabase CLI. Then, replace its content with this code:

   import { EdgeWorker } from "jsr:@pgflow/edge-worker";
   
   EdgeWorker.start(async (payload: { url: string }) => {
     const response = await fetch(payload.url);
   
     console.log("Scraped website!", {
       url: payload.url,
       status: response.status,
     });
   });

2. Start Edge Runtime

   Start the Edge Runtime with the following command:

   npx supabase functions serve

   This makes Supabase listen for incoming HTTP requests, but does not start your worker yet.

   Restarting Edge Runtime

   You must stop and start the Edge Runtime every time you make changes to your workers because of the per_worker policy.

3. Start your worker

   Start the worker by sending an HTTP request to your new Edge Function (replace <function-name> with your function name):

   curl -X POST http://localhost:54321/functions/v1/<function-name>

   This will boot a new instance and start your worker:

   [Info] worker_id=<uuid> [WorkerLifecycle] Ensuring queue 'tasks' exists...

4. Process your first message

   Your worker is now polling for messages on the tasks queue (which was automatically created during startup).

   Send a test message:

   SELECT pgmq.send(
     queue_name => 'tasks',
     msg => '{"url": "https://example.com"}'::jsonb
   );

   The message will be processed immediately and you should see the following output:

   [Info] worker_id=<uuid> [ExecutionController] Scheduling execution of task 1
   
   [Info] Scraped website! { url: "https://example.com", status: 200 }

   Queue Name

   By default, your worker uses a queue named tasks. You can change this by setting the queueName option - useful when running multiple workers or integrating with existing queues.