Large uploads
A server that receives an image as a stream of chunks does not have to hold the whole body to decode it. Hokusai(spooling:options:memoryLimit:spoolDirectory:) consumes an AsyncSequence of Data (the ByteBuffer form is in HokusaiNIO) and returns a pipeline.
// An upload that arrives in 16 KB chunks, as a server would receive it.let bytes = try Data(contentsOf: URL(fileURLWithPath: "scene.png"))let chunks = stride(from: 0, to: bytes.count, by: 16_384).map { bytes[$0..<min($0 + 16_384, bytes.count)] }
struct Chunks: AsyncSequence { let parts: [Data] struct Iterator: AsyncIteratorProtocol { var index = 0 let parts: [Data] mutating func next() async -> Data? { defer { index += 1 }; return index < parts.count ? parts[index] : nil } } func makeAsyncIterator() -> Iterator { Iterator(parts: parts) }}
// Below memoryLimit the upload stays in memory; above it, it moves to a private temporary file.let image = try await Hokusai(spooling: Chunks(parts: chunks.map { Data($0) }), memoryLimit: 64 * 1024)let output = try await image.resize(width: 400).webp().data()print("spooled", chunks.count, "chunks →", output.info.width, "×", output.info.height)
// maxBytes applies while the body is read, so an oversized upload stops early.do { _ = try await Hokusai(spooling: Chunks(parts: chunks.map { Data($0) }), options: .init(limits: .init(maxBytes: 50_000)))} catch { print("over maxBytes →", error)}How it works
Section titled “How it works”- The upload stays in memory while it is within
memoryLimit(default 1 MiB;0spools everything). Past that it is moved to a private temporary file, so the encoded upload never takes more thanmemoryLimitplus one chunk of memory, however large it is. libvips reads the image from the file with random access. - That bounds the encoded upload only. The memory libvips uses to decode and transform the image is a separate matter, bounded by
InputLimits.maxPixelsandInputOptions.access. InputLimits.maxBytesapplies while the body is read: the chunk that crosses it throwsresource(encodedBytes) and the rest of the body is never requested. Above, 19 chunks of at most 16 KiB withmaxBytes: 50_000stopped withactual: 65536, the chunk that crossed the limit.- The pipeline owns the file. It is removed when the last copy of the pipeline, and any evaluation running on it, is released; a failed or cancelled read removes it at once. A process that is killed first leaves its file: files are named
hokusai-upload-…, readable only by their owner, inspoolDirectory(the system temporary directory by default, which must exist), so a server can sweep that directory at startup. - Writes to the spool run on background queues while the calling task is suspended, so a slow disk cannot delay unrelated async work.
Errors
Section titled “Errors”| Situation | Result |
|---|---|
| An empty upload | invalidInput |
Over maxBytes |
resource (encodedBytes) |
A bad option (e.g. negative memoryLimit) |
invalidOption |
| The spool file cannot be created or written (missing directory, permissions, full disk) | io |
| Whatever the sequence throws | rethrown unchanged |
Hokusai Vapor and Hokusai Hummingbird call this for you (Integrations).