Creating and Managing Fabric Lakehouses with Notebooks: Reading External Files from OneLake, Writing Delta Tables, and Browsing Results in the Lakehouse Explorer
Learn how to use Spark notebooks in Microsoft Fabric to read CSV and Parquet files from OneLake, write clean Delta tables to the lakehouse Tables zone, and validate your results in the Lakehouse Explorer — the complete data engineering loop from raw files to queryable data. This foundation-level lesson explains every step from first principles with realistic examples and working PySpark code.
🌱 Foundation17 min readSep 22, 2026Updated Sep 22, 2026
Creating and Managing Fabric Lakehouses with Notebooks: Reading External Files from OneLake, Writing Delta Tables, and Browsing Results in the Lakehouse Explorer
Introduction
Picture this: your organization just landed a batch of sales CSV files from a regional distributor. They're sitting in your OneLake Files zone, raw and unprocessed. Your job is to read those files, clean them up, and write the results as a proper Delta table that analysts can query and report against. You know the data needs to land somewhere structured — but you're not sure exactly how notebooks, lakehouses, and OneLake all fit together to make that happen.
This is the workflow that sits at the heart of modern data engineering in Microsoft Fabric. Notebooks are the workbench where you read, transform, and write data. The lakehouse is the home base — a storage structure that organizes your raw files and managed Delta tables under one roof. And the Lakehouse Explorer is your live browser, letting you see your results appear the moment your code finishes running. Together, these three tools give you a fast, flexible, code-first path from raw files to queryable data.
By the end of this lesson, you'll be able to attach a lakehouse to a notebook, read CSV and Parquet files from OneLake using PySpark, write the results as Delta tables, and verify your work using the Lakehouse Explorer — all without leaving the Fabric workspace.
What you'll learn:
How lakehouses organize data into Files and Tables zones in OneLake
How to attach a lakehouse to a Fabric notebook and why the default path matters
How to read CSV and Parquet files from OneLake into a Spark DataFrame
How to write a DataFrame as a managed Delta table using both shorthand and explicit path approaches
How to browse, preview, and validate results in the Lakehouse Explorer
No PySpark experience required — we'll explain every line of code from scratch
Understanding the Lakehouse Layout Before You Write a Single Line of Code
Before you open a notebook, it pays to understand exactly what you're working with. A Fabric lakehouse has two primary storage zones, and knowing the difference changes how you write your code.
The Files zone is an open staging area. Think of it like a shared drive — you can drop anything in here: CSV files, JSON, Parquet, images, text files. Nothing is enforced. This is where raw and landing data typically lives.
The Tables zone is where managed Delta tables live. Delta Lake is the open-source table format that adds ACID transactions, schema enforcement, and versioning on top of Parquet files. When you write a Delta table here, Fabric automatically makes it queryable through the SQL Analytics Endpoint, and it becomes available to Power BI via Direct Lake mode. That automatic promotion is what makes Delta tables so powerful — you write once, and multiple consumers can read.
Both zones physically live inside OneLake, Microsoft Fabric's unified storage layer. OneLake is the one place every Fabric workload reads from and writes to — no copying, no connectors, no syncing. When your notebook writes a Delta table to the Tables zone, it's writing directly into OneLake, and that data immediately becomes visible to the SQL Analytics Endpoint and Power BI.
Key insight
The Files zone and Tables zone aren't just organizational folders — they represent two fundamentally different relationships with the data. Files are raw bytes; tables are structured, versioned, and queryable. The notebook's job is often to move data from one zone to the other.
Creating a Notebook and Attaching Your Lakehouse
Open your Fabric workspace and navigate to your lakehouse. Inside the lakehouse, look at the top ribbon and click the Open notebook dropdown, then select New notebook. This creates a new notebook with your current lakehouse already attached — which saves you a step.
Alternatively, you can create a notebook from the workspace home by clicking New item → Notebook, but you'll need to attach the lakehouse manually afterward. To do that, look at the left-side panel inside the notebook — there's an Explorer pane. At the top of that pane, you'll see a section called Lakehouses. Click Add lakehouse, choose Existing lakehouse, and select your lakehouse from the list.
Once a lakehouse is attached, Fabric sets it as the default lakehouse for the notebook session. This is an important concept: the default lakehouse determines what the shorthand path Files/ and Tables/ resolve to in your Spark code. It also means you can use simplified mount paths rather than full abfss:// URIs.
Tip
You can attach multiple lakehouses to a single notebook — useful when you're reading from a raw landing lakehouse and writing to a curated one. The first lakehouse you attach becomes the default, but you can change this by right-clicking a lakehouse in the Explorer pane and selecting Set as default.
Notebooks in Fabric run on an Apache Spark cluster that spins up automatically when you execute your first cell. You'll notice a status bar showing "Starting Spark session" the first time you run code — this usually takes 30–60 seconds. Subsequent cell runs in the same session reuse the same cluster, so they're faster.
Reading Files from the OneLake Files Zone
Let's assume you've uploaded a file called sales_2024_q1.csv into the Files zone of your lakehouse. You can do this by clicking Get data → Upload files from within the lakehouse interface, or by dragging and dropping files into the Files folder in the Lakehouse Explorer panel.
Your CSV has the following columns: order_id, customer_id, product_sku, quantity, unit_price, order_date, and region.
Now, in your notebook, add a new code cell and read the file:
spark.read opens a Spark DataFrameReader — your gateway to reading data
.option("header", True) tells Spark the first row contains column names
.option("inferSchema", True) tells Spark to look at the data and automatically guess column data types rather than treating everything as a string
.csv("Files/sales_2024_q1.csv") uses the simplified mount path that works because your lakehouse is set as default
df.printSchema() prints a tree showing column names and inferred types — always check this before writing
df.show(5) displays the first five rows as a quick sanity check
When you run this cell, Spark reads the CSV in parallel across its worker nodes, returns a DataFrame (an in-memory table with named columns and rows), and prints your schema and preview.
Warning
inferSchema works well for small files, but for production workloads with large files it requires an extra scan of the data to guess types. In those cases, define the schema explicitly using StructType to avoid the performance hit and prevent type inference surprises.
Reading Parquet Files
Parquet is a compressed, columnar binary format that's much more efficient than CSV for analytics. If your file is already in Parquet format (say, sales_2024_q2.parquet), reading it is even simpler:
Parquet files embed their schema internally, so there's no need for inferSchema. Spark reads the metadata from the file footer and builds the DataFrame schema automatically.
Reading an Entire Folder of Files
In real pipelines, data often arrives as a folder of partitioned files rather than a single file. For example, a vendor might drop a new CSV into Files/sales_raw/ every day. You can read the whole folder in one shot:
Spark treats the entire folder as one logical dataset and parallelizes the read across all files in it. This pattern is fundamental to scalable data ingestion.
Transforming the Data Before Writing
Reading raw data is only half the job. Before writing, you almost always need to clean, filter, or enrich it. Let's do some light transformation on the sales DataFrame.
from pyspark.sql.functions import col, to_date, round as spark_round
df_clean = (
df
.filter(col("quantity") > 0) # Drop bad rows with zero or negative quantity
.withColumn("order_date", to_date(col("order_date"), "yyyy-MM-dd")) # Parse date strings properly
.withColumn("line_total", spark_round(col("quantity") * col("unit_price"), 2)) # Add a calculated column
.dropDuplicates(["order_id"]) # Enforce uniqueness on order_id
)
df_clean.printSchema()
df_clean.show(5)
Here's what each transformation does:
.filter() removes rows where quantity is zero or negative — a common sign of bad data in transactional exports
.withColumn("order_date", to_date(...)) converts the date string into a proper Spark DateType, which enables date math and partitioning later
.withColumn("line_total", ...) adds a derived column so downstream consumers don't have to compute it themselves
.dropDuplicates(["order_id"]) ensures each order appears only once, even if the source file had duplicates
Note
Spark DataFrames are immutable — you never change the original df. Instead, each transformation returns a new DataFrame. This is why we assign to df_clean. The original df still exists unchanged if you need to go back.
Writing the DataFrame as a Delta Table
Now for the payoff: persisting your clean DataFrame as a Delta table in the Tables zone. The simplest approach is:
.format("delta") specifies Delta Lake as the table format — this is what adds versioning, ACID transactions, and schema enforcement
.mode("overwrite") replaces the table entirely if it already exists; use "append" to add rows to an existing table
.saveAsTable("sales_q1_clean") creates a managed table named sales_q1_clean in the default lakehouse's Tables zone
After running this cell, the table physically exists as Parquet files plus a _delta_log folder inside Tables/sales_q1_clean/ in OneLake. The _delta_log is the transaction log that makes Delta tables ACID-compliant — every write operation is recorded here as a JSON entry.
Using an Explicit Path Instead
Sometimes you want more control over exactly where the files land. You can write to an explicit OneLake path using the full abfss:// URI:
You'd replace <workspace-id> and <lakehouse-id> with the actual GUIDs from your workspace and lakehouse properties. This approach gives you precise control but is more verbose. The simpler saveAsTable() approach is preferred for most notebook workflows because it registers the table in the Hive Metastore automatically, making it accessible via SQL.
Tip
After using saveAsTable(), you can immediately query the table with Spark SQL in any subsequent cell: spark.sql("SELECT * FROM sales_q1_clean LIMIT 10").show(). This is a fast way to verify the write succeeded without switching tools.
Once your write completes, you don't need to run another query to confirm it worked. The Lakehouse Explorer — the panel on the left side of your notebook — gives you a live view of your lakehouse contents.
Look at the Explorer pane. Under your attached lakehouse, you'll see two expandable sections: Files and Tables. Click the refresh icon (or right-click and choose Refresh) and your new sales_q1_clean table should appear under Tables.
Click the table name to expand it and see the column list. Each column is listed with its data type — this is a quick visual confirmation that your schema landed correctly.
To preview the data without writing any code, right-click the table name and select Load data → Preview. This opens a small data preview panel showing the first few rows directly in the notebook interface. It's not a full query tool, but it's perfect for a quick spot check.
You can also switch from the notebook interface to the full lakehouse view. Click the lakehouse name in the Explorer pane — it's a clickable link — and you'll be taken to the lakehouse's main UI. From there, click the Tables section in the left navigation to see your Delta tables listed with row counts, column counts, and created timestamps. Click any table name to open a richer row-level preview with pagination.
Key insight
The Lakehouse Explorer panel in the notebook and the Tables view in the lakehouse UI both reflect the same underlying state in OneLake. There's no separate database to sync — whatever you write from a notebook is immediately visible everywhere else in Fabric that touches that lakehouse.
Validating Your Table with SQL
The Lakehouse Explorer is great for spot checks, but sometimes you need to run an actual query. You have two options from inside the notebook.
Option 1: Use a %%sql magic cell.
At the top of a new cell, type %%sql on its own line, then write standard SQL below it:
%%sql
SELECT region, COUNT(*) as order_count, ROUND(SUM(line_total), 2) as total_revenue
FROM sales_q1_clean
GROUP BY region
ORDER BY total_revenue DESC
This runs the query through SparkSQL and renders results as a formatted table directly in the notebook output. No need to switch tools.
Option 2: Use the SQL Analytics Endpoint.
Every Fabric lakehouse comes with an automatically provisioned SQL Analytics Endpoint — a serverless SQL engine that lets you run T-SQL against your Delta tables without Spark. To access it, navigate to your lakehouse and look for the mode switcher in the top right of the screen. Switch from Lakehouse mode to SQL analytics endpoint mode. You'll see the same tables listed, and you can open the built-in SQL query editor to run T-SQL.
Work through this sequence from start to finish in your own Fabric environment:
Setup: Create a CSV file on your local machine called product_inventory.csv with these columns: product_id, product_name, category, stock_count, unit_cost, warehouse_location. Add at least 20 rows of realistic-looking data, including a few rows with stock_count of 0 to test your filter.
Step 1: Upload the file to the Files zone of your lakehouse using the Get data → Upload files option in the lakehouse UI.
Step 2: Open a new notebook attached to that lakehouse. In the first cell, read the CSV file and print the schema:
Step 5: Refresh the Lakehouse Explorer and confirm the table appears under Tables. Right-click the table and preview the data.
Step 6: Add a %%sql cell and run a query that groups by category and sums total_value. Verify the results match what you'd expect.
Common Mistakes and Troubleshooting
"Path does not exist" error when reading files.
This almost always means either the file isn't uploaded yet, or you've mistyped the path. Check the Files section in the Lakehouse Explorer to confirm the exact filename and folder. Note that paths are case-sensitive — Sales_2024_Q1.csv and sales_2024_q1.csv are different files.
Table appears in Files zone, not Tables zone.
If you used .save("Files/my_table") instead of .saveAsTable("my_table"), you wrote Delta files into the Files zone as unregistered data. The table won't show up under Tables and won't be queryable through the SQL Analytics Endpoint. Use saveAsTable() for managed tables, or move to the Tables zone and register manually.
Schema errors when writing with .mode("append").
Delta tables enforce schema consistency. If you try to append a DataFrame whose schema doesn't match the existing table (different column names, different types), Spark will throw an AnalysisException. Check your transformations and make sure the column names and types match the existing table's schema exactly. You can inspect the existing table schema with spark.sql("DESCRIBE sales_q1_clean").show().
Spark session takes too long to start.
This is normal for the first cell run — Fabric provisions a Spark cluster on demand. If it consistently takes more than 3 minutes, try stopping the session from the toolbar and starting a new one. On shared trial capacities, cluster startup can be slower during peak hours.
inferSchema guesses the wrong type for a date column.
CSV files don't carry type metadata, so Spark has to guess. Date columns often get inferred as strings. Always check printSchema() output after reading a CSV, and use .withColumn() with to_date() or to_timestamp() to cast date columns explicitly.
Warning
If you're working toward building a full medallion architecture — with bronze, silver, and gold layers — the notebook patterns you've learned here are the building blocks. Be intentional about which lakehouse and which zone each layer writes to. Read Implementing the Medallion Architecture in Microsoft Fabric: Bronze, Silver, and Gold Layers for a complete walkthrough of that pattern.
Summary and Next Steps
You've now completed the core loop of data engineering in Microsoft Fabric: read raw files from OneLake, transform them with PySpark, write the results as Delta tables, and verify everything in the Lakehouse Explorer. This is not a toy workflow — this is exactly how production pipelines in Fabric process data at every scale.
Here's what you covered:
The structural difference between the Files zone (raw storage) and the Tables zone (managed Delta tables) in a lakehouse
How attaching a lakehouse to a notebook sets the default path context for simplified file references
Reading CSV and Parquet files with spark.read, including folder-level reads for batches of files
Applying filters, type casts, and derived columns before writing
Writing Delta tables with saveAsTable() and understanding what the _delta_log does
Using the Lakehouse Explorer and %%sql cells to validate results without switching tools
The notebook skills you've built here are also the foundation for everything in a proper data pipeline. When you're ready to automate and schedule this kind of processing, Orchestrating Loads with Fabric Data Pipelines: Copy Activities, Parameters, and Schedules shows you how to wrap notebooks in pipelines that run on a schedule and respond to upstream events.
