Skip to content
| Marketplace
Sign in
Visual Studio Code>Programming Languages>PySpark DataFrame Column AnalyzerNew to Visual Studio Code? Get it now.
PySpark DataFrame Column Analyzer

PySpark DataFrame Column Analyzer

eredzik

|
1 install
| (0) | Free
Static analyzer for PySpark DataFrame column usage in Python files.
Installation
Launch VS Code Quick Open (Ctrl+P), paste the following command, and press enter.
Copied to clipboard
More Info

PySpark DataFrame Column Analyzer

Static analyzer for PySpark DataFrame column usage in Python files.

Features

  • Detects invalid column references for PySpark DataFrame operations:
    • select()
    • withColumn()
    • withColumnRenamed()
    • drop()
    • join()
    • union() / unionByName()
  • Infers DataFrame schemas from Palantir Foundry dataset metadata for the transform:
    • spark.read.table("table_name") uses the schema derived from Foundry metadata
    • The first select() on a DataFrame when no schema is known
  • Incremental analysis on document changes.
  • Nice-to-have features:
    • Column completion inside df.select("...").
    • Hover on a DataFrame variable name to see its inferred schema.

Configuration

Schemas can come from:

  • a Palantir Foundry transform metadata file referenced by pysparkAnalyzer.foundryMetadataPath (extension-side preloaded map), or
  • live Foundry API calls from the Rust sidecar when FOUNDRY_TOKEN and FOUNDRY_API_URL is set.

For live sidecar fetches, set:

  • FOUNDRY_TOKEN (required): bearer token used for authorization.
  • FOUNDRY_API_URL (required): Foundry stack base URL, e.g. https://my-stack.palantirfoundry.com.
  • FOUNDRY_SCHEMA_CACHE_TTL_SECONDS (optional): in-memory cache TTL, default 300.
  • Contact us
  • Jobs
  • Privacy
  • Manage cookies
  • Terms of use
  • Trademarks
© 2026 Microsoft