EJ

Data

1 min read

6 items under this folder.

  • apache-parquet-files

    Column storage format -> easier to run analytics on columns (features) Works well with Hadoop, Spark and Hive For example, say you have an S3 with Parquet files then you could use AWS Glue to catalog the Parquet files then you could use Amazon Athena to run queries on these files.

    • ingesting-parquet-to-posgres-parquet_fdw

      install apt install postgresql-17-parquet-fdw create foreign table CREATE EXTENSION parquet_fdw; CREATE SERVER parquet_server FOREIGN DATA WRAPPER parquet_fdw OPTIONS (root_dir '/path/to/parquet_files/'); CREATE FOREIGN TABLE parquet_table ( id INT, name TEXT, value FLOAT ) SERVER parquet_...

      • ingesting-parquet-to-postgres-duckdb

        Steps Install duckdb: pip install duckdb psycopg2 Use DuckDB to read Parquet and write to PostgreSQL: import duckdb import psycopg2 # Read Parquet and send to PostgreSQL con = duckdb.connect() parquet_file = '/path/to/data.parquet' pg_conn = "dbname=mydb user=postgres password=secret&...

        • ingesting-parquet-to-postgres-pd-sqla

          small to medium size import pandas as pd from sqlalchemy import create_engine # Load Parquet into DataFrame df = pd.read_parquet("data.parquet") # Connect to PostgreSQL engine = create_engine('postgresql://user:password@localhost:5432/mydb') # Ingest into PostgreSQL df.to_sql(&#0...

          • ingesting-parquet-to-postgres-pg_parquet

            Setup If you are using a docker image (PostGIS/PostgreSQL) you will need to setup the environment in the following way (the docker images don’t have these installed): CMD: Log into the container and install the required packages to build cargo: podman exec -it spatialytics-postgis bash apt-get updat...

            • ingesting-parquet-to-postgres-pyarrow

              Use Arrow’s pg_copy which is fast.