vpack

package
v0.0.0-...-136b97a Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 27, 2026 License: AGPL-3.0 Imports: 6 Imported by: 0

README

vpack wire format

This document specifies the byte‑level (on‑wire) layout for vote compression. The goal is to minimize vote size while retaining a 1‑to‑1, loss‑free mapping to the canonical msgpack representation of agreement.UnauthenticatedVote. The canonical msgpack representation we rely on is provided by agreement/msgp_gen.go, generated by our custom msgpack code generator which ensures fields are generated in lexicographic order, omit empty key-value pairs, and use specific formats for certain types as defined in our specification.

Compression Layers

Vote compression uses two layers:

  1. Stateless compression (StatelessEncoder/StatelessDecoder): Removes msgpack formatting and field names, replacing them with a bitmask. This is always applied and has no memory overhead.

  2. Stateful compression (optional) (StatefulEncoder/StatefulDecoder): Further compresses by replacing frequently repeated values with references to LRU tables and a sliding window. This layer requires per-connection state (configurable size, e.g., ~224KB per direction for 1024-entry tables) and is optional. When used, it operates on the output of the stateless layer.

Both layers use the same 2-byte header format, with byte 0 used by the stateless layer, and byte 1 used by the stateful layer.

1. High‑level structure

+---------+-----------------+---------------------+--------------------------+
| Header  | VrfProof ("pf") | rawVote ("r")       | OneTimeSignature ("sig") |
| 2 bytes | 80 bytes        | variable length     | 256 bytes                |
+---------+-----------------+---------------------+--------------------------+

All fields appear exactly once, and in the fixed order above. The presence of optional sub‑fields inside rawVote are indicated by a 1‑byte bitmask in the header. No field names appear, only values.

2. Header (2 bytes)

Offset Description
0 Presence flags for optional values (LSB first, see §2.1).
1 Stateful compression flags (zero if not using stateful layer, see §2.2).
2.1 Stateless bit‑mask layout (byte 0)
Bit Flag Field enabled Encoded size
0 bitPer r.per (varuint) 1 - 9 bytes
1 bitDig r.prop.dig (digest) 32 bytes
2 bitEncDig r.prop.encdig (digest) 32 bytes
3 bitOper r.prop.oper (varuint) 1 - 9 bytes
4 bitOprop r.prop.oprop (address) 32 bytes
5 bitStep r.step (varuint) 1 - 9 bytes

Binary fields are represented by their 32-, 64-, and 80-byte values without markers. Integers use msgpack's variable-length unsigned integer encoding:

  • fixint (≤ 127), 1 byte in length (values 0x00-0x7f)
  • uint8 2 bytes in length (marker byte 0xcc + 1-byte value)
  • uint16 3 bytes in length (marker byte 0xcd + 2-byte value)
  • uint32 5 bytes in length (marker byte 0xce + 4-byte value)
  • uint64 9 bytes in length (marker byte 0xcf + 8-byte value)
2.2 Stateful compression flags (byte 1)

When stateful compression is used, byte 1 encodes which values have been replaced by references. When stateful compression is not used, byte 1 must be zero.

Bits Flag Field(s) Meaning
0-1 rndDelta r.rnd delta encoding 00=literal, 01=+1, 10=-1, 11=same as last round
2-4 propRef r.prop window reference 000=literal, 001-111=sliding window index (1-7)
5 sndRef r.snd reference 0=literal (32 bytes), 1=LRU table reference (2 bytes)
6 pkRef sig.p + sig.p1s reference 0=literal (96 bytes), 1=LRU table reference (2 bytes)
7 pk2Ref sig.p2 + sig.p2s reference 0=literal (96 bytes), 1=LRU table reference (2 bytes)

The stateful layer uses:

  • Round delta encoding: Most votes reference the same round as before, with some interleaving votes between the current and next round, as voters gradually observe consensus and move on to the next round. A 2-bit encoding specifies whether r.rnd is the same as the previous vote, or has increased or decreased by one, and can be omitted from the message for these cases. If rndDelta is 00, the literal value appears in the message.
  • Proposal sliding window: A 7-entry HPACK-style window tracks recent proposal values. In a typical round, all votes should be for the same proposal value, compressing previously-seen proposal values from ~96 bytes down to a 3-bit reference.
  • LRU tables: Three 2-way set-associative hash tables cache recently seen values for sender addresses, and two tiers of (public key, signature) pairs. Since some consensus participants vote in rounds more often than others, these tables record the most common field values re-used across votes by the same participant, and replace them with references:
    • The snd table tracks participating addresses, which are re-used across all votes by a participant;
    • the pk table tracks sig.pk and sig.p1s values, which are re-used across votes in the same round by a participant;
    • and the pk2 table tracks sig.p2 and sig.p2s values, which are re-used across all votes in a batch by a participant (typically thousands of rounds), under Algorand's hierarchical consensus signature scheme.

3. Field serialization order

After the 2-byte header, the encoder emits values in the following order. If stateful compression is enabled, the "stateful encoding" column specifies how each field is additionally transformed or omitted.

Field Type Encoded size Presence flag Stateful encoding
pf VRF credential 80 bytes Required Unchanged
r.per Period varuint bitPer Unchanged
r.prop.dig Proposal digest 32 bytes bitDig Omitted if propRef set
r.prop.encdig Digest of encoded proposal 32 bytes bitEncDig Omitted if propRef set
r.prop.oper Proposal's original period varuint bitOper Omitted if propRef set
r.prop.oprop Proposal's original proposer 32 bytes bitOprop Omitted if propRef set
r.rnd Round number varuint Required Omitted if rndDelta set
r.snd Voter's (sender) address 32 bytes Required 2-byte reference if sndRef set
r.step Step varuint bitStep literal (if present)
sig.p Ed25519 public key 32 bytes Required 2-byte reference if pkRef set
sig.p1s Signature over offset ID 64 bytes Required Omitted if pkRef set
sig.p2 Second-tier Ed25519 public key 32 bytes Required 2-byte reference if pk2Ref set
sig.p2s Signature over batch ID 64 bytes Required Omitted if pk2Ref set
sig.s Signature over vote using p 64 bytes Required Unchanged

Documentation

Index

Constants

View Source 
const (

	// MaxMsgpackVoteSize is the maximum size of a vote, including msgpack control characters
	// and all required and optional data fields.
	MaxMsgpackVoteSize = msgpFixMapMarkerSize +
		len(msgpFixstrCred) + msgpFixMapMarkerSize +
		len(msgpFixstrPf) + msgpBin8Len80Size +
		len(msgpFixstrR) + msgpFixMapMarkerSize +
		len(msgpFixstrPer) + maxMsgpVaruintSize +
		len(msgpFixstrProp) + msgpFixMapMarkerSize +
		len(msgpFixstrDig) + msgpBin8Len32Size +
		len(msgpFixstrEncdig) + msgpBin8Len32Size +
		len(msgpFixstrOper) + maxMsgpVaruintSize +
		len(msgpFixstrOprop) + msgpBin8Len32Size +
		len(msgpFixstrRnd) + maxMsgpVaruintSize +
		len(msgpFixstrSnd) + msgpBin8Len32Size +
		len(msgpFixstrStep) + maxMsgpVaruintSize +
		len(msgpFixstrSig) + msgpFixMapMarkerSize +
		len(msgpFixstrP) + msgpBin8Len32Size +
		len(msgpFixstrP1s) + msgpBin8Len64Size +
		len(msgpFixstrP2) + msgpBin8Len32Size +
		len(msgpFixstrP2s) + msgpBin8Len64Size +
		len(msgpFixstrPs) + msgpBin8Len64Size +
		len(msgpFixstrS) + msgpBin8Len64Size // sig.s: bin8(64)

	// MaxCompressedVoteSize is the maximum size of a compressed vote using StatelessEncoder,
	// including all required and optional fields.
	MaxCompressedVoteSize = headerSize +
		80 +
		maxMsgpVaruintSize*4 +
		32*6 +
		64*3 // sig.p1s, sig.p2s, sig.s (sig.ps is omitted)
)

Variables

View Source 
var ErrBufferTooSmall = fmt.Errorf("destination buffer too small")

ErrBufferTooSmall is returned when the destination buffer is too small

View Source 
var ErrLikelyUncompressed = fmt.Errorf("data appears to be uncompressed msgpack")

ErrLikelyUncompressed is returned when vpack decompression detects what appears to be uncompressed msgpack data from a peer claiming vpack support.

Functions

This section is empty.

Types

type StatefulDecoder 

type StatefulDecoder struct {
	// contains filtered or unexported fields
}

StatefulDecoder decompresses votes by using references to previously seen values from earlier votes.

func NewStatefulDecoder 

func NewStatefulDecoder(tableSize uint) (*StatefulDecoder, error)

NewStatefulDecoder creates a new StatefulDecoder with initialized LRU tables of the specified size

func (*StatefulDecoder) Decompress 

func (d *StatefulDecoder) Decompress(dst, src []byte) ([]byte, error)

Decompress reverses StatefulEncoder, and writes a valid stateless vpack format buffer into dst. Caller must then pass it to StatelessDecoder.

type StatefulEncoder 

type StatefulEncoder struct {
	// contains filtered or unexported fields
}

StatefulEncoder compresses votes by using references to previously seen values from earlier votes.

func NewStatefulEncoder 

func NewStatefulEncoder(tableSize uint) (*StatefulEncoder, error)

NewStatefulEncoder creates a new StatefulEncoder with initialized LRU tables of the specified size

func (*StatefulEncoder) Compress 

func (e *StatefulEncoder) Compress(dst, src []byte) ([]byte, error)

Compress takes a vote compressed by StatelessEncoder, and additionally compresses it using dynamic references to previously seen values.

type StatelessDecoder 

type StatelessDecoder struct {
	// contains filtered or unexported fields
}

StatelessDecoder decompresses votes that were compressed by StatelessEncoder.

func NewStatelessDecoder 

func NewStatelessDecoder() *StatelessDecoder

NewStatelessDecoder returns a new StatelessDecoder.

func (*StatelessDecoder) DecompressVote 

func (d *StatelessDecoder) DecompressVote(dst, src []byte) ([]byte, error)

DecompressVote decodes a compressed vote in src and appends it to dst. To re-use dst, run like: dst = dec.DecompressVote(dst[:0], src)

type StatelessEncoder 

type StatelessEncoder struct {
	// contains filtered or unexported fields
}

StatelessEncoder compresses a msgpack-encoded vote by stripping all msgpack formatting and field names, replacing them with a bitmask indicating which fields are present. It is not thread-safe.

func NewStatelessEncoder 

func NewStatelessEncoder() *StatelessEncoder

NewStatelessEncoder returns a new StatelessEncoder.

func (*StatelessEncoder) CompressVote 

func (e *StatelessEncoder) CompressVote(dst, src []byte) ([]byte, error)

CompressVote compresses a vote in src and writes it to dst. If the provided buffer dst is nil or too small, a new slice is allocated. The returned slice may be the same as dst. To re-use dst, run like: dst = enc.CompressVote(dst[:0], src)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.