Package commons :: Module seqs

[frames] | no frames]

Module seqs

source code

Sequences, streams, and generators.

Classes

[hide private]

safe_pickler

ClosedError

PersistentConsumedSeq
I generate [0, 1, ...], like count, but I can also save my state to disk.

PersistentSeq
I generate [0, 1, ...], like count, but I can also save my state to disk.

test_seqs

Functions

[hide private]

(object, str)

read_pickle(read, init='', length_thresh=100000)
Given a reader function read, reads in pickled objects from it. source code

read_pickles(read)
Reads all the consecutively pickled objects from the read function. source code

write_pickle(obj, write)
Write obj using function write, in a safe, pickle-able fashion. source code

streamlen(stream)
Get the length of a stream (e.g.

source code

chunkify(stream, chunk_size=8192)
Given an input stream (an object exposing a file-like interface), reads data in from it one chunk at a time.

source code

int

total(iterable)
Counts the number of items in an iterable.

source code

pairwise(iterable)
s -> (s0,s1), (s1,s2), (s2, s3), ...

source code

argmax(sequence, fn=None)
Two usage patterns: argmax([s0, s1, ...], fn) argmax([(fn(s0), s0), (fn(s1), s1), ...]) Both return the si with greatest fn(si) source code

argmin(sequence, fn=None)
Two usage patterns: argmin([s0, s1, ...], fn) argmin([(fn(s0), s0), (fn(s1), s1), ...]) Both return the si with smallest fn(si) source code

concat(listOfLists)

source code

flatten(stream)
For each item yielded by stream, if that item is itself an iterator/generator, then I will recurse into flatten(gen); otherwise, I'll yield the yielded item. source code

grouper(n, iterable, padvalue=None)
grouper(3, 'abcdefg', 'x') --> ('a','b','c'), ('d','e','f'), ('g','x','x')

source code

chunker(n, iterable, in_place=False)
Like grouper but designed to scale for larger n. source code

countstep(start, step)
Generate [start, start+step, start+2*step, start+3*step, ...].

source code

take(n, seq)

source code

delimit(sep, xs)

source code

interleave(xs, ys)

source code

span(pred, xs)
Returns (successes, failures), where successes is the sequence of any consecutive elements at the head of xs that satisfy the predicate, and second is everything else. source code

group_as_subseqs(xs, key=<function <lambda> at 0x1d7de60>)
Takes a sequence and breaks it up into multiple subsequences, which are groups keyed on key. source code

Variables

[hide private]

default_chunk_size = 8192
The default chunk size used by chunkify.

HIGHEST_PROTOCOL = 2

__package__ = None
hash(x)

compatible_formats = ['1.0', '1.1', '1.2', '1.3', '2.0']

format_version = '2.0'

Function Details

[hide private]

read_pickle(read, init=`''`, length_thresh=100000)

source code

Given a reader function read, reads in pickled objects from it. I am a generator which yields unpickled objects. I assume that the pickling is "safe," done using safe_pickler.

Parameters:

read (function) - The reader function that reads from a stream. It should take a single argument, the number of bytes to consume.

Returns: (object, str)

A tuple whose first element is the deserialized object or None if EOF was encountered, and whose second element is the remainder bytes until the EOF that were not consumed by unpickling.

streamlen(stream)

source code

Get the length of a stream (e.g. file stream or StringIO). Tries to restore the original position in the stream.

chunkify(stream, chunk_size=8192)

source code

Given an input stream (an object exposing a file-like interface), reads data in from it one chunk at a time. This is a generator which yields those chunks as they come.

Parameters:

stream (stream) - The input stream.
chunk_size (int) - The size of the chunk (usually the number of bytes to read).

total(iterable)

source code

Counts the number of items in an iterable. Note that this will consume the elements of the iterable, and if the iterable is infinite, this will not halt.

Parameters:

iterable - The iterable to count. @type iterable

Returns: int

The number of elements consumed.

flatten(stream)

source code

For each item yielded by stream, if that item is itself an iterator/generator, then I will recurse into flatten(gen); otherwise, I'll yield the yielded item. Thus, I essentially "flatten" out a tree of iterators.

I test whether something is an iterator/generator simply by checking to see if it has a next attribute. Note that this won't include any iterable, so things like lists are yielded like any regular item. This is my author's desired behavior!

I am useful for coroutines, a la DeferredGenerators from Twisted.

chunker(n, iterable, in_place=False)

source code

Like grouper but designed to scale for larger n. Also, does not perform padding. The end of the stream is reached when we yield a chunk with fewer than n items.

group_as_subseqs(xs, key=<function <lambda> at 0x1d7de60>)

source code

Takes a sequence and breaks it up into multiple subsequences, which are groups keyed on key.

>>> map(lis, group_as_subseqs(range(10), lambda x: x/3))
[[0, 1, 2], [3, 4, 5], [6, 7, 8], [9]]

Module seqs

read_pickle(read, init='', length_thresh=100000)

streamlen(stream)

chunkify(stream, chunk_size=8192)

total(iterable)

flatten(stream)

chunker(n, iterable, in_place=False)

group_as_subseqs(xs, key=<function <lambda> at 0x1d7de60>)

read_pickle(read, init=`''`, length_thresh=100000)