DataFlint Copilot

Intelligent Apache Spark optimization and debugging assistant for VSCode

DataFlint Copilot integrates with the DataFlint platform to provide real-time code analysis, performance insights, and visual highlights for Apache Spark applications directly in your development environment.

✨ Features

• 🔍 Real-time Code Analysis: Analyze Spark applications and get instant feedback on performance bottlenecks
• 🎯 Visual Code Highlights: Interactive code lens and hover tooltips show optimization opportunities
• 📊 Performance Insights: Detailed metrics and suggestions for improving Spark job performance
• 🤖 AI Assistant Integration: Works seamlessly with AI assistants through Model Context Protocol (MCP)
• 🔐 Secure Authentication: OAuth2/Auth0 integration for secure DataFlint platform access
• 📈 Job Monitoring: Track Spark job execution and identify issues in real-time

📦 Installation

1. Install the extension from the VSCode marketplace
2. Open VSCode and go to Extensions (Ctrl+Shift+X)
3. Search for "DataFlint Copilot"
4. Click Install

🚀 Quick Start

1. Configure Authentication

1. Open VSCode Settings (Ctrl+,)
2. Search for "DataFlint"
3. Configure your DataFlint credentials:
   • Auth Domain: Your DataFlint Auth0 domain
   • Client ID: Your DataFlint client ID
   • Server URL: Your DataFlint server URL (default: http://localhost:8000)

2. Start Using

1. Open a Spark application project
2. The extension will automatically start and authenticate
3. Code highlights will appear above relevant lines
4. Hover over highlights for detailed information
5. Use the command palette (Ctrl+Shift+P) to access DataFlint commands

🛠️ Configuration

The extension can be configured through VSCode settings:

{
  "dataflint.auth.domain": "https://your-auth-domain.auth0.com/",
  "dataflint.auth.clientId": "your-client-id",
  "dataflint.auth.audience": "http://localhost:8000",
  "dataflint.server.baseUrl": "http://localhost:8000",
  "dataflint.mcp.port": 11223
}

📋 Commands

Access these commands through the Command Palette (Ctrl+Shift+P):

• DataFlint: Toggle Highlight - Show/hide code highlights
• DataFlint: Restart MCP Server - Restart the MCP server
• DataFlint: Check Server Status - Check if the server is running
• DataFlint: Logout - Clear authentication and logout

🤖 AI Assistant Integration

DataFlint Copilot includes MCP (Model Context Protocol) support for AI assistants:

1. The extension automatically configures MCP settings
2. AI assistants can access DataFlint tools for Spark analysis
3. Get intelligent suggestions and optimizations through natural language

Available MCP Tools

• get_dataflint_prompts - Get system prompts and available tools
• get_applications_by_name - Find Spark applications by name
• get_application_jobs - Get job history for an application
• get_job_alerts - Get alerts and issues for specific jobs
• get_prompts - Get detailed optimization prompts
• get_sql_queries_plan - Analyze SQL query execution plans
• add_dataflint_code_highlight - Add visual highlights to code

🔍 Usage Examples

Basic Code Analysis

1. Open a Spark application file (Python, Scala, Java)
2. Code lens will appear above lines with optimization opportunities
3. Click on highlights to open detailed analysis
4. Hover over code for instant insights

Working with AI Assistants

User: "Analyze my Spark application for performance issues"
AI: Uses DataFlint tools to analyze your code and provide specific recommendations

📋 Requirements

• VSCode: Version 1.99.0 or higher
• DataFlint Account: Access to DataFlint platform
• Spark Applications: Python, Scala, Java, or SQL files

🔐 Security

• All authentication uses OAuth2/Auth0 with PKCE flow
• Tokens are stored securely in VSCode's secret storage
• No sensitive data is logged or transmitted insecurely

🐛 Troubleshooting

Common Issues

Authentication Failed

• Check your Auth0 domain and client ID in settings
• Ensure you have access to the DataFlint platform
• Try the "DataFlint: Logout" command and re-authenticate

MCP Server Not Starting

• Check if port 11223 is available
• Try "DataFlint: Restart MCP Server" command
• Check VSCode Developer Console for error messages

Code Highlights Not Appearing

• Ensure your file is a supported Spark application file
• Check that the DataFlint server is running
• Try "DataFlint: Toggle Highlight" to refresh

Getting Help

1. Check the GitHub Issues
2. Review the VSCode Developer Console for error messages
3. Contact DataFlint support for platform-specific issues

🤝 Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🔗 Links

• DataFlint Platform
• Apache Spark
• VSCode Extensions
• Model Context Protocol

📊 Changelog

0.0.1 (Initial Release)

• Initial release with core functionality
• OAuth2 authentication
• Code highlighting system
• MCP server integration
• AI assistant support

Made with ❤️ for the Apache Spark community