Snowflake SnowPark API — An Introduction

Soumak Das
5 min readSep 18, 2023

The Snowpark library offers a user-friendly toolkit for querying and processing data at scale within the Snowflake platform. With Snowpark, you can develop applications that handle data within Snowflake without the need to move data to an external system where your application code resides. This allows you to efficiently process data at scale, leveraging the power of the elastic and serverless Snowflake engine.

Snowflake currently provides Snowpark libraries for three programming languages: Java, Python, and Scala.

  1. Machine Learning with Snowpark Python
  2. Data Engineering Pipelines with Snowpark Python
  3. Getting Started With Snowpark for Python and Streamlit
  4. Building an Image Recognition App in Snowflake using Snowpark Python, PyTorch, Streamlit, and OpenAI
  5. Getting Started With Snowpark Scala

You can make the most of Snowpark libraries for the languages mentioned in the table below:

  • Java
  • Python
  • Scala

Key Features

Snowpark offers several distinctive features that set it apart from other client libraries:

Benefits Compared to the Spark Connector Developing with Snowpark, as opposed to using the Snowflake Connector for Spark, provides the following advantages:

  • Support for interacting with data within Snowflake using language-specific libraries and patterns tailored for different programming languages, all while maintaining performance and functionality.
  • The ability to write Snowpark code using familiar local tools such as Jupyter, VS Code, or IntelliJ.
  • Comprehensive support for pushdown operations, including Snowflake User-Defined Functions (UDFs). This means that Snowpark pushes down data transformation and heavy computation tasks to the Snowflake data cloud, enabling efficient data processing of any scale.
  • No need for a separate external cluster for computations. All computations are executed within Snowflake, with scale and compute management seamlessly handled by the platform.

Ability to Build SQL Statements with Native Constructs The Snowpark API offers language constructs for building SQL statements. For example, instead of writing ‘select column_name’ as a string, the API provides a select…

--

--

Soumak Das

Sr. Data Engineer @EY & Snowflake/Airflow/Databricks/AWS writer