Skill Store 33

Spark Python Data Source

Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and st…

by databricks

The skill

Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.

This skill's licence does not allow redistribution, so the store does not show its text. Read it at the source.

Use this skill

Connect your agent to the Skill Store MCP server and ask for it by ref: databricks/spark-python-data-source-52770293559.

Markdown version

Details

Publisher
databricks
Category
Data & Analytics › Data pipelines & ETL
Updated
2026-10-19

References

More from databricks

Related skills

  • Databricks Apps Python Python backend for Databricks Apps — FastAPI (default), Flask, Dash, Streamlit, Gradio, Reflex.
  • Earth2Studio Create Datasource Create and validate Earth2Studio data source wrappers (DataSource, ForecastSource, DataFrameSourc…
  • Data Pipeline Builder Designs and builds ETL/ELT data pipelines.
  • Spark Engineer Use when writing Spark jobs, debugging performance issues, or configuring cluster settings for Ap…
  • Databricks Python Sdk Databricks development guidance including Python SDK, Databricks Connect, CLI, and REST API.