core

Core of the package.

class bream.core.Batch(data: Any, read_to: int)

A batch returned by a data source.

Parameters:: data (Any) – The data for this batch. The source offset that was read to for this batch.

class bream.core.BatchRequest(read_from_after: int | None, read_to: int | None): A request for a batch sent to a data source.

class bream.core.Batches(batches: dict[str, Batch | None])

A collection of batches.

Parameters:: batches (dict[str, Batch | None]) – A map of source name to batch from that source.

bream.core.Pathlike = 'Path | _AnyPathTypingWrapper': A pathlib.Path object or any type of path provided by cloudpathlib

class bream.core.Source

Abstract base class for bream source definitions.

name: str: Name of the source. This is used by bream as a source identifier.

abstract read(batch_request: BatchRequest) → Batch | None

Read data from the source between the given offsets.

Bream source definitions must implement this.

Parameters:: batch_request – A request for a batch of data, as prepared by bream.
Returns:: The batch of data if data was available, otherwise None.
Return type:: Batch | None

class bream.core.Stream(sources: Sequence[Source], stream_path: Pathlike, stream_options: StreamOptions | None = None)

A stream of batches of data.

Parameters:

sources – The sources of data batches that this stream will be used to process.
stream_path – Path to a location this stream will use to tracks its progress.

start(func: Callable[[Batches], None], min_batch_seconds: float) → None

Start the stream in a background thread.

Parameters:

func – The batch function that will process each batch of data.
min_batch_seconds – The minimum number of seconds between each attempt to fetch a batch.

property status: StreamStatus

Status of the stream.

stop(*, blocking: bool = True) → None

Mark the running stream to be stopped gracefully.

Parameters:: blocking – Whether the current thread should wait until the stream is stopped before proceeding.

wait() → None: If the stream has been started, block until the stream terminates.

class bream.core.StreamOptions(repeat_failed_batch_exactly: bool = True)

Options for a stream.

repeat_failed_batch_exactly: bool = True: Whether to repeat a failed batch exactly on a stream restart, regardless of source configuration.

class bream.core.StreamStatus(started: bool = False, active: bool = False, error: Exception | None = None)

Status information of a stream.

Parameters:

started (bool) – Whether the stream has been started.
active (bool) – Whether the stream is currently active.
error (Exception | None) – The error, if the stream terminated with an error.