kiel.protocol.messages
¶kiel.protocol.messages.
MessageSet
(messages)[source]¶Bases: object
Class representing a set of Message
instances.
Kafka’s compression scheme works by taking a set of messages, compressing them with the chosen compression scheme, and then wrapping the result as the value of an envelope message, called the “message set”.
compressed
(compression, msgs)[source]¶Returns a MessageSet
with the given messages optionally compressed.
If compression is used, the message set is rendered, compressed, and then a new message set is created with the raw output as a value in a single message.
If no compression is set, the returned instance will have a (<offset>, <message>) tuple for each given message, where the offset is -1.
render
()[source]¶Returns a tuple of format and data suitable for struct.pack
.
Each (<offset>, <message>) tuple in self.messages
is render()
-ed
and the output collected int a single format and data list, prefaced
with a single integer denoting the size of the message set.
parse
(buff, offset, size=None)[source]¶Given a buffer and offset, returns the parsed MessageSet
and offset.
Starts by determining the size of the raw payload to parse, and
continuously parses the Int64
offset and Int32
size of a
message then the Message
itself.
If a parsed message’s flags denote compression, parse()
is called
recursively on the message’s value in order to unpack the compressed
nested messages.
kiel.protocol.messages.
Message
(**kwargs)[source]¶Bases: kiel.protocol.part.Part
Basic Part
subclass representing a single Kafka message.
Message =>
crc => Int32
magic => Int8
attributes => Int8
key => Bytes
value => Bytes
parts
= (('crc', <class 'kiel.protocol.primitives.Int32'>), ('magic', <class 'kiel.protocol.primitives.Int8'>), ('attributes', <class 'kiel.protocol.primitives.Int8'>), ('key', <class 'kiel.protocol.primitives.Bytes'>), ('value', <class 'kiel.protocol.primitives.Bytes'>))¶parse
(buff, offset)[source]¶Given a buffer and offset, returns the parsed Message
and new offset.
Parses via the basic Part
parse method, but with added
decompression support.
If a parsed message’s attributes denote that compression has been used,
the value is run through the corresponding decompress()
method.