Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,12 @@
# CHANGELOG

## v0.2.6 - 2026-05-01

* Fix grammar and typos in the documentation
* Fix example code in `PgLargeObjects.export/3` API docs
* Fix admonition title in `PgLargeObjects.LargeObject.size/1` API docs
* Fix potential resource leak in `PgLargeObjects.import/3` and `export/3`.

## v0.2.5 - 2026-04-27

* Fix `PgLargeObjects.export/3` not honoring `:bufsize` option if `:into`
Expand Down
18 changes: 9 additions & 9 deletions CONSIDERATIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ PostgreSQL 4.2 was released on Jun 30th, 1994. The large objects facility has
been around for a long time!

Yet, it is fairly unknown to many programmers - or considered too unwieldy to
use for productive usage. This is not by accident - there are various trade
offs to consider when deciding if large objects are a good mechanism for
storing large amounts of binary data.
use for productive usage. This is not by accident - there are various trade-offs
to consider when deciding if large objects are a good mechanism for storing
large amounts of binary data.

This document attempts to collect and discuss some of these considerations. If
you feel there are other aspects to highlight, or if any of the items below
Expand Down Expand Up @@ -41,14 +41,14 @@ mechanisms for storing large objects.
For example, the [AWS RDS
documentation](https://aws.amazon.com/rds/postgresql/pricing/) (RDS is Amazon's
managed database offering) explains that at the time of this writing, 1GB of
General Purpose storage for a in the us-east-1 region costs $0.115 per month
General Purpose storage in the `us-east-1` region costs $0.115 per month
for a PostgreSQL database. The [AWS S3 documentation](https://aws.amazon.com/s3/pricing/) (S3 is Amazon's
object storage offering) documents, at the time of this writing, that storing
1GB of data in the us-east-1 region is a mere $0.023 per month!
1GB of data in the `us-east-1` region is a mere $0.023 per month!

I.e. when using Amazon cloud services in the us-east-1 region, storing data in
RDS is five times as expensive as storing it in S3. Depending on the amount of
data and your budget, this might be a significant difference.
I.e. when using Amazon cloud services in the `us-east-1` region, storing data
in RDS is five times as expensive as storing it in S3. Depending on the amount
of data and your budget, this might be a significant difference.

Make sure to check the pricing (if applicable) for storage used by your
PostgreSQL database and consider the change in the decision whether to use
Expand All @@ -73,6 +73,6 @@ frerich@Mac ~ % pg_dump --help
```

Consider your current backup mechanism and see if it's configured to include or
exclude large objects. Decide on the important of large objects for your use
exclude large objects. Decide on the importance of large objects for your use
case and include that in your decision on how often large objects should be
included in backups.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ options for doing so:
2. A separate cloud storage (e.g. AWS S3) could be used. This permits streaming
but requires complicating the tech stack by depending on a new service.
Bridging the two systems (e.g. ‘Delete all uploads for a given user ID’)
requires Elixir support.
requires custom application code.

PostgreSQL features a ‘large objects’ facility which enables efficient
streaming access to large (up to 4TB) files. This solves these problems:
Expand All @@ -39,9 +39,9 @@ streaming access to large (up to 4TB) files. This solves these problems:
with the tables referencing them, operations like ‘Delete all uploads for a
given user ID’ are just one `SELECT` statement.

However, there are trade offs. See the [Considerations](CONSIDERATIONS.md)
However, there are trade-offs. See the [Considerations](CONSIDERATIONS.md)
document for aspects to take into account when deciding if large objects
are good choice for your use case.
are a good choice for your use case.

## Installation

Expand Down Expand Up @@ -108,7 +108,7 @@ end

Use the high-level APIs `PgLargeObjects.import/3` and `PgLargeObjects.export/3`
(exposed as `import_large_object/2` and `export_large_object/2` on the
applications' repository module) for importing data into or exporting data out
application's repository module) for importing data into or exporting data out
of the database:

```elixir
Expand Down
37 changes: 21 additions & 16 deletions lib/pg_large_objects.ex
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,12 @@ defmodule PgLargeObjects do
case data do
binary when is_binary(binary) ->
{:ok, buffer} = StringIO.open(binary, encoding: :latin1)
result = import(repo, IO.binstream(buffer, opts[:bufsize]), opts)
StringIO.close(buffer)
result

try do
import(repo, IO.binstream(buffer, opts[:bufsize]), opts)
after
StringIO.close(buffer)
end

enumerable ->
with {:ok, lob} <- LargeObject.create(repo) do
Expand All @@ -57,15 +60,15 @@ defmodule PgLargeObjects do
Export data out of large object.

This exports the data in the large object referenced by the object ID `oid`.
Depending on the `:into` option, the data is returned a single binary or fed
into a given `Collectable`.
Depending on the `:into` option, the data is returned as a single binary or
fed into a given `Collectable`.

To treat a large object as an `Enumerable` and pass it around as a stream,
reach for the lower-level API in `PgLargeObjects.LargeObject`, e.g.:

```elixir
def stream_object!(object_id) do
{:ok, object} = PgLargeObject.LargeObject.open(object_id)
def stream_object!(repo, object_id) do
{:ok, object} = PgLargeObjects.LargeObject.open(repo, object_id)
object
end
```
Expand Down Expand Up @@ -96,17 +99,19 @@ defmodule PgLargeObjects do
{:ok, buffer} = StringIO.open("", encoding: :latin1)

result =
with :ok <-
export(repo, oid,
into: IO.binstream(buffer, opts[:bufsize]),
bufsize: opts[:bufsize]
) do
{_input, output} = StringIO.contents(buffer)
{:ok, output}
try do
with :ok <-
export(repo, oid,
into: IO.binstream(buffer, opts[:bufsize]),
bufsize: opts[:bufsize]
) do
{_input, output} = StringIO.contents(buffer)
{:ok, output}
end
after
StringIO.close(buffer)
end

StringIO.close(buffer)

result

collectable ->
Expand Down
8 changes: 4 additions & 4 deletions lib/pg_large_objects/large_object.ex
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ defmodule PgLargeObjects.LargeObject do
> #### Transactions Required {: .info}
>
> All operations on `LargeObject` values *must* take place within a database
> transactions since the internal handle managed by the structure is only
> transaction since the internal handle managed by the structure is only
> valid for the duration of a transaction.
>
> Any large object value will be closed automatically at the end of the
Expand Down Expand Up @@ -186,7 +186,7 @@ defmodule PgLargeObjects.LargeObject do

Calculates the size (in bytes) of the given large object `lob`.

> #### Enum.count/1 vs. Enum.size/1 {: .info}
> #### Enum.count/1 vs. LargeObject.size/1 {: .info}
>
> Note that this is not the same as using `Enum.count/1`; `Enum.count/1`, by
> virtue of the `Enumerable` implementation, will return the number of _chunks_
Expand Down Expand Up @@ -246,8 +246,8 @@ defmodule PgLargeObjects.LargeObject do
@doc """
Read data from large object.

Reads a `length` bytes of data from the given large object `lob`, starting at
the current iosition in the object. Advanced the position by the number of
Reads `length` bytes of data from the given large object `lob`, starting at
the current position in the object. Advances the position by the number of
bytes read, or until the end of file. The read position will not be advanced
when the current position is beyond the end of the file.

Expand Down
5 changes: 3 additions & 2 deletions test/pg_large_objects/large_object_test.exs
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ defmodule PgLargeObjects.LargeObjectTest do
end

test "fails given invalid object ID" do
assert {:error, :not_found} == LargeObject.open(TestRepo, 12_345)
assert {:error, :not_found} == LargeObject.remove(TestRepo, 12_345)
end
end

Expand Down Expand Up @@ -208,7 +208,8 @@ defmodule PgLargeObjects.LargeObjectTest do
assert {:ok, 1} = LargeObject.seek(lob, 0, :current)
assert {:ok, "B"} == LargeObject.read(lob, 1)

# Seeting to 0 bytes from the end moves the cursor one past the last byte.
# Setting to 0 bytes from the end moves the cursor one past the last
# byte.
assert {:ok, 7} = LargeObject.seek(lob, 0, :end)
assert {:ok, ""} == LargeObject.read(lob, 1)
end)
Expand Down