ingesting-parquet-to-postgres-pyarrow

Use Arrow’s pg_copy which is fast.

1. Install the required libraries
   1. pip install pyarrow psycopg2-binary
2. Ingest the Parquet file

import pyarrow.parquet as pq
import psycopg2
 
table = pq.read_table("path/to/data.parquet")
 
# connect to PostgreSQL
conn = psycopg2.connect("dbname=my_database user=postgres password=mysecret")
#cur = conn.cursor()? needed?
 
with conn.cursor() as cur:
	cur.copy_expert("COPY my_pg_table FROM STDIN WITH (FORMAT BINARY)", table.to_pandas().to_csv(index=False, header=False))
	conn.commit()