Folder: notes

Column storage format → easier to run analytics on columns (features) Works well with Hadoop, Spark and Hive For example, say you have an S3 with Parquet files then you could use AWS Glue to catalog the Parquet files then you could use Amazon Athena to run queries on these files.

Great historical timeline: snarky.ca/how-the-heck-does-async-await-work-in-python-3-5/ Also implements an event loop PEP 492: peps.python.org/pep-0492/ Introduces await for awaitable objects A Curious Course on Coroutines and Concurrency by David Beazley.

Once you have created a database Running PostGIS in Silverblue you may want to import data to it.

install apt install postgresql-17-parquet-fdw create foreign table CREATE EXTENSION parquet_fdw; CREATE SERVER parquet_server FOREIGN DATA WRAPPER parquet_fdw OPTIONS (root_dir '/path/to/parquet_files/'); CREATE FOREIGN TABLE parquet_table ( id INT, name TEXT, value FLOAT ) SERVER parquet_...

Steps Install duckdb: pip install duckdb psycopg2 Use DuckDB to read Parquet and write to PostgreSQL: import duckdb import psycopg2 # Read Parquet and send to PostgreSQL con = duckdb.connect() parquet_file = '/path/to/data.parquet' pg_conn = "dbname=mydb user=postgres password=secret&...

small to medium size import pandas as pd from sqlalchemy import create_engine # Load Parquet into DataFrame df = pd.read_parquet("data.parquet") # Connect to PostgreSQL engine = create_engine('postgresql://user:password@localhost:5432/mydb') # Ingest into PostgreSQL df.to_sql(&#0...

Setup If you are using a docker image (PostGIS/PostgreSQL) you will need to setup the environment in the following way (the docker images don’t have these installed): CMD: Log into the container and install the required packages to build cargo: podman exec -it spatialytics-postgis bash apt-get updat...

Use Arrow’s pg_copy which is fast.

Converts data between formats.

Create volumes podman volume create postgis_data Then run a container: podman run --name spatialytics-postgis \ -e PG_PASSWORD="$POSTGIS_PASSWORD" \ -p 5432:5432 \ -v postgis_data:/var/lib/postgresql/data \ -d postgis/postgis A bind mount is a mount that points to a directory in the host m...

Check Available Databases: Confirm that the database you want to write to is present.

Running with podman Run using podman but you can replace docker and it will work the same.

Core vs ORM SQLAlchemy Core is lower-level, closer to raw SQL. You can use text() to run queries.

Working with DB Metadata SQL Expression Language allows composable construction of SQL queries.

full-duplex: both workstations / servers can send data at the same time (like a telephone) half-duplex: workstations / servers have to take turns to send data (like a walkie-talkie) simplex mode: only transmit one way (like a radio) References www.techtarget.com/searchnetworking/definition/full-dup...

Start libvirtd (check status if needed): systemctl status libvirtd sudo systemctl start libvirtd Authenticate Run a polkit /usr/libexec/polkit-gnome-authentication-agent-1 Authenticate Run virt-manager Do your VM thing References www.reddit.com/r/i3wm/comments/ih989j/i3wm_cant_start_polkitgnome/.

Protocol that provides a two-way (full-duplex ?) communication over a single, long-lived TCP connection.

thermal comfort Definitions according to ASHRAE 55 : thermal comfort: that condition of mind that expresses satisfaction with the thermal environment and is assessed by subjective evaluation.

After trying different alias types (such as in [0] and [1]) I was unable to add an alias with an argument that also worked with autocompletion.

Source: Combine Cross-references Source: Formatting Cross-refrences When working with cross-references, you might need different formats.

Source Source 2 Assuming that the final graph will fit within a single column in a two-column-per-page format.

I use Linux, OSX and Windows on a daily basis, changing between OSes depending on the task at hand.