Skip to content
Hokusai2.2.2
ivantokar.com

Large uploads

A server that receives an image as a stream of chunks does not have to hold the whole body to decode it. Hokusai(spooling:options:memoryLimit:spoolDirectory:) consumes an AsyncSequence of Data (the ByteBuffer form is in HokusaiNIO) and returns a pipeline.

// An upload that arrives in 16 KB chunks, as a server would receive it.
let bytes = try Data(contentsOf: URL(fileURLWithPath: "scene.png"))
let chunks = stride(from: 0, to: bytes.count, by: 16_384).map { bytes[$0..<min($0 + 16_384, bytes.count)] }
struct Chunks: AsyncSequence {
let parts: [Data]
struct Iterator: AsyncIteratorProtocol {
var index = 0
let parts: [Data]
mutating func next() async -> Data? { defer { index += 1 }; return index < parts.count ? parts[index] : nil }
}
func makeAsyncIterator() -> Iterator { Iterator(parts: parts) }
}
// Below memoryLimit the upload stays in memory; above it, it moves to a private temporary file.
let image = try await Hokusai(spooling: Chunks(parts: chunks.map { Data($0) }), memoryLimit: 64 * 1024)
let output = try await image.resize(width: 400).webp().data()
print("spooled", chunks.count, "chunks →", output.info.width, "×", output.info.height)
// maxBytes applies while the body is read, so an oversized upload stops early.
do {
_ = try await Hokusai(spooling: Chunks(parts: chunks.map { Data($0) }), options: .init(limits: .init(maxBytes: 50_000)))
} catch {
print("over maxBytes →", error)
}
  • The upload stays in memory while it is within memoryLimit (default 1 MiB; 0 spools everything). Past that it is moved to a private temporary file, so the encoded upload never takes more than memoryLimit plus one chunk of memory, however large it is. libvips reads the image from the file with random access.
  • That bounds the encoded upload only. The memory libvips uses to decode and transform the image is a separate matter, bounded by InputLimits.maxPixels and InputOptions.access.
  • InputLimits.maxBytes applies while the body is read: the chunk that crosses it throws resource (encodedBytes) and the rest of the body is never requested. Above, 19 chunks of at most 16 KiB with maxBytes: 50_000 stopped with actual: 65536, the chunk that crossed the limit.
  • The pipeline owns the file. It is removed when the last copy of the pipeline, and any evaluation running on it, is released; a failed or cancelled read removes it at once. A process that is killed first leaves its file: files are named hokusai-upload-…, readable only by their owner, in spoolDirectory (the system temporary directory by default, which must exist), so a server can sweep that directory at startup.
  • Writes to the spool run on background queues while the calling task is suspended, so a slow disk cannot delay unrelated async work.
Situation Result
An empty upload invalidInput
Over maxBytes resource (encodedBytes)
A bad option (e.g. negative memoryLimit) invalidOption
The spool file cannot be created or written (missing directory, permissions, full disk) io
Whatever the sequence throws rethrown unchanged

Hokusai Vapor and Hokusai Hummingbird call this for you (Integrations).