ingesting-parquet-to-postgres-duckdb

Steps Install duckdb:

pip install duckdb psycopg2

Use DuckDB to read Parquet and write to PostgreSQL:

import duckdb
import psycopg2
 
# Read Parquet and send to PostgreSQL
con = duckdb.connect()
parquet_file = '/path/to/data.parquet'
pg_conn = "dbname=mydb user=postgres password=secret"
 
con.execute(f"""
    COPY (SELECT * FROM parquet_scan('{parquet_file}')) 
    TO '{pg_conn}' (FORMAT 'postgres');
""")

Pros:

• Very fast for large files.
• Efficient handling of Parquet.

Cons:

• Requires DuckDB.