Web Scraping Copilot

Web Scraping Copilot is a free Visual Studio Code extension by Zyte that helps you generate web scraping code with GitHub Copilot. It streamlines working with Scrapy projects and includes optional integration with Scrapy Cloud, making it easier to deploy and monitor your web scraping jobs.

Requirements

Visual Studio Code 1.104+. Web scraping projects must use Python 3.10+ and Scrapy 2.7.0+. GitHub Copilot Pro or better is recommended for AI web scraping tools; the limited requests of the Free plan can run out quickly otherwise. See all requirements.

Quick Start

1. Install uv while all Visual Studio Code windows are closed.

2. Start Visual Studio Code, open File › Preferences › Settings, locate the following settings with the search box, and set their values as follows:

   1. Set chat.mcp.access to all.

   2. Set chat.mcp.autostart to newAndOutdated.

   3. Optionally, enable the chat to-do list so Web Scraping Copilot can keep tasks in sync:

      • Open View › Command Palette… and run Preferences: Open Settings (JSON).

      • Add the following snippet to your user or workspace settings:

        "chat.agent.todoList": {
          "position": "default"
        }
        

3. Install the extension.

4. Open a Scrapy project.

5. To generate code, click on Generate Parsing Code with AI on the Page Objects view of the extension.

   

   For the best code generation experience, configure the MCP server to use GPT-5 mini, for best code generation quality without premium models.

   To configure allowed models, open View › Command Palette… and select MCP: List Servers › Web Scraping Copilot › Configure Model Access.

Follow the tutorial to learn more. If you run into issues, see Troubleshooting.

Features

Generate maintainable web scraping code with GitHub Copilot.

Browse your spiders and page objects with new interactive views. Run your spiders locally with a click, generate new test fixtures for your page objects, and more.

If you use Scrapy Cloud, you can deploy your spiders with a click, and monitor cloud jobs from the spiders view.

FAQ

How much does the extension cost?

The extension itself is free.

To use code generation, you do need a GitHub Copilot plan, and the Free plan is not recommended because you would spend your requests rather quickly.

To use Scrapy Cloud features, you need a Scrapy Cloud account. The free plan is fine, though.

Does the extension use AI from Zyte?

No, your GitHub Copilot AIs are used.

The extension provides instructions and prompts, and the MCP server tools use MCP sampling to start separate chats in the background to handle the different steps of code generation.

To control which models can be used by the MCP server, open View › Command Palette… and select MCP: List Servers › Web Scraping Copilot › Configure Model Access.

Is my code sent to Zyte?

The code generation workflow that the extension facilitates does not send any code to Zyte, only to GitHub Copilot.

Scrapy Cloud deployment, if used, does upload your code to Scrapy Cloud.

Which LLM model works best for code generation?

The model you use in the main chat should be somewhat smart, since workflow management can be hard for smaller models. We recommend something like GPT-5, although GPT-5 mini has shown good results in our tests.

The MCP web scraping tools, to generate expectations and code, are designed to work well enough with models for which GitHub Copilot paid plans (Pro or better) allow unlimited requests, like GPT-5 mini. Given the number of requests that those tools can generate, it could be very costly to use a smarter model.

Documentation

Check out the docs for more information.