EJ

Picking Up Django #26355, Part 1: Understanding the Problem

4 min read

This is from #26355 Add support for PostgreSQL’s array_append to ArrayField.

Understanding the problem

Before working on a solution I have to understand what the problem is. Then I can define what an acceptable solution is (as per my understanding) and finally, share my understanding and my proposal to more experienced Django developers to see if I’m on the right track.

The good thing about this ticket is that there have been two previous implementation attempts so that gives me a lot to work with.

Initial request

The problem:

I am using an ArrayField to store a list of dates. I would like to use update() to add a value to the list, only if the list doesn’t contain it already.

The contains part is already supported by Django, but not the array_append.

​http://www.postgresql.org/docs/9.1/static/arrays.html#ARRAYS-MODIFYING

Eventually I’d like to be able to do something like this:

User.objects.exclude(active_in__contains=[current_month]).update(active_in=F('active_in') + [current_month])

which should result in this SQL:
UPDATE user_user SET active_in = array_append(active_in, [%s]) WHERE NOT(active_in @> ARRAY[%s]::varchar[]) ['2012-01']

PS: I want to use update instead of get() and save() for performance reasons.

I want to add that in this particular use case I can get around using it (because I already have the user object anyway), but it would nevertheless be nice of have this feature, I think.

The problem is pretty clear and the great thing about this ticket is that the developer expressed what they hope to see in the end. This gives me a clear acceptance criteria (which is rare).

In my own words: the developer wants to append to an ArrayField in a single update() query.

The developer was kind enough to link the relevant PG docs. So I took a look at what it said about ARRAYS-MODIFYING.

My take away from the docs is that we should be using the append operator instead of array_append. The docs say “In fact, these functions primarily exist for use in implementing the concatenation operator” but besides that, I’m assuming it is useful for consistency and it is more idiomatic SQL. Another thing I found is that chaining is more clean with the append operator. Compare:

-- Using the operator
SELECT ARRAY[1, 2] || 3 || 4;
 
-- Using the function is harder to read
SELECT array_append(array_append(ARRAY[1, 2], 3), 4);

So for now my thought is to use the append operator ||.

Moving onto reading the thread to see what other developers have said.

Thread context

  • Nick Pope proposed broader scope: ||/array_append, array_cat, | set-union, slice assignment, array_remove; nested arrays are the hard part.
  • Mariusz: wants a better dynamic-dispatch interface for combined expressions rather than special-casing.
  • Old PR #6395 became #11310 and appears stalled (last comment”will open a new PR when I can resume the effort again”)

Previous attempts

flowchart LR
    A["#6395 (2016-04)<br/>mrzechonek<br/>multipledispatch registry"] -->|"stalled 1.11 / 2.0"| B["#11310 (2019-05)<br/>auvipy<br/>rebase of #6395, WIP"]
    A -.->|"charettes: defer dispatching to #26355"| C["#11359 (2019-05 -> merged 2020-07)<br/>charettes, Fixed #30446<br/>Value output_field + _connector_combinators"]
    C --> D["1efea11808 (2022-03)<br/>Luke Plant, Refs #33397<br/>register_combinable_fields()"]
    D --> E["contrib.postgres.search<br/>uses it for SearchVector ||"]
    B -->|"closed inactivity 2020-07-22"| X[ticket back to New]
    A -->|"closed: superseded by #11310"| X

#6395: “Added API to register different expressions for operator/field combo”

  • Added multipledispatch registry
  • Superseded by #11310 but closed. It did not address the trac #26355 original query.

#11310: “[WIP] Fixed #26355 — Added API to register different expressions for operator/field combo.” (abandoned)

  • Rebase of #6395
  • Partial port that can’t run as committed and dropped the array part

#11359: “Fixed #30446 — Resolved Value.output_field for stdlib types.” (merged)

  • Added typed Value() for stdlib types, plus a combinable-type table that later became register_combinable_fields(). contrib.postgres already registers into it.
  • Could possibly extend that registry to carry an expression class and dispatch. When the other side is an ArrayField, coerce an untyped Value(list) to that field.
  • to-do: list isn’t mapped to a field, and the registry only infers the result type, not the SQL.

Next steps

I’m coming to the conclusion that this splits into two parts:

  1. Create an explicit function (eg: ArrayAppend) which is useful on its own and reviewable without settling the dispatch question.
  2. Operator sugar. So F('arr') + [x] resolves to ArrayAppend through the registry that came out of #11359, not through a special case.

I’ll write up a proposal with this in mind and see if I can get a test to pass first. I’ll also share this with my Djangonaut team and the ticket itself.

Graph View