VS Code Extension for watsonx.data

Spark lab is a Spark-based development environment that enables you to interactively program, debug, submit, and test Spark applications on a live Spark cluster running on the Analytics Engine service.

It is available as a VS Code extension and you can install it in your local system to access Spark IDE using Visual Studio Code. It reduces the time for development and increases usability.

Remote - SSH Extension is required for this extension to work.

IBM watsonx.data Environments

• Software Environment (Cloud Pak for Data): An on-premises or private cloud setup where IBM's watsonx.data engines runs directly on user-managed infrastructure.
• SaaS (Software as a Service) Environment: A cloud-based solution where IBM's watsonx.data engines are hosted by cloud providers like IBM Cloud or AWS.

Features

What you can do in Visual Studio Code with this extension:

• Connect to watsonx.data console
• Create & Delete Spark labs from the extension
• Securely connect to your watsonx.data console's runtimes via SSH over secure WebSockets
• Develop & Debug the files inside your watsonx.data Git project remotely

Installation

Marketplace

You can use VS Code Marketplace to download and install this extension directly to your VS Code.

From extension package

1. Ask your administrator to enable VS Code support for your cluster

2. Download the watsonx.data extension and then install the downloaded .vsix file in VS Code:

   Cmd/Ctrl + Shift + P -> Extensions: Install from VSIX

3. Optional, but recommended: Install the Remote SSH extension in VS Code.

   If you're not using the Remote SSH extension, you'll need an SSH client on your machine.

Set up SSH on your machine

Generating an SSH key pair on Mac and Linux

Important: Before generating a new key pair, check the ~/.ssh directory for any existing key pairs,

To generate an SSH key pair on MacOS and Linux:

1. Open the Terminal
2. Execute this command: ssh-keygen -t rsa -N ""

When you generate a key pair, specify the location of the private key in the watsonx-data.privateKeyPath and public key in the watsonx-data.publicKeyPath settings of the extension.

Generating an SSH key pair on Windows

Important: Check whether your home directory contains the .ssh directory or run dir "%USERPROFILE%/.ssh" to check if any keys exist.

To generate an SSH key pair on Windows:

On Windows 10 or newer with the OpenSSH Client feature enabled:

1. Open a new cmd command prompt and use the `ssh-keygen utility.
2. Execute this command: ssh-keygen -t rsa -N ""

If you are using an older Windows version:

• Install a third party tool, such as Putty, to generate a SSH key.
• If your're using Putty, use the puttygen utility to generate an SSH key pair. Note: PuttyGen does not generate OpenSSH keys by default. To export an OpenSSH key, convert the key: select Conversions -> Export OpenSSH key.

Optional: Using a separate SSH key for watsonx.data

To use a seperate SSH key for watsonx.data, add a new entry to your SSH configuration under ~/.ssh/config:

Host cpdenv
    HostName localhost
    Port 5681 # Same port as watsonx-data.localPort
    IdentityFile ~/.ssh/id_cpd_rsa # Path to your private key

Then enter the host cpdenv under watsonx-data.localHost.

Extension Settings

This extension contributes the following settings:

• watsonx-data.privateKeyPath: (Optional) Path to the private key file for SSH connection. May be left empty as it is optional from 2.0.3 onwards.
• watsonx-data.publicKeyPath: (Optional) Path to the public key file for SSH connection. May be left empty as it is optional from 2.0.3 onwards.
• watsonx-data.localHost: The SSH host to which Remote SSH will connect to. Can be any host specified in the SSH client config file. The HostName configured in your SSH client config should always point to localhost and the port specified in localPort. This allows you to use your own SSH settings, e.g. a specific private key.
• watsonx-data.localPort: The port to listen on locally.
• watsonx-data.logLevel: The log level of the extension.
• watsonx-data.verifySSL: (Optional) Whether to verify the certificate of the SSL connection or not. Default: true.

Manage Connection Page

There are two input formats: JSON Inputs and Form Inputs. You may use either for connection.

For Form Inputs:

• Host Address of watsonx.data console: You can find it in the address bar of your browser when you log in to Software or SaaS Environment. For example: cpd.xxx.xxx.xxx.com.
• Environment Type: Environment where watsonx.data is running. Software or SaaS. Refer to IBM watsonx.data Environments section.
• Cloud Resource Name (CRN): Only visible and used for SaaS. More here: https://cloud.ibm.com/docs/account?topic=account-crn
• Username: Username that you use to log in to Software or SaaS Environment.
• API Key: API Key used to authenticate to your Software or SaaS Environment.

For JSON Inputs, navigate to:

1. IBM watsonx.data console
2. Go to Configurations on the left side bar
3. Click on Connection information
4. Then select VS Code
5. View VS Code connection configuration

Official Documentation

• Software Documentation: For detailed guides for Software, visit the Software Documentation.
• SaaS Documentation: For detailed guides for SaaS, visit the SaaS Documentation.