Items, events and commits

You can use Synck for years without reading this chapter, just as you can use Git without knowing what a tree object is. But Synck’s behavior follows directly from a small data model. Once you know it, the rules in the rest of the documentation stop being rules to memorize and become consequences you could have predicted.

This chapter goes from the bottom up:

  1. Items, events and commits (this page): what Synck stores.
  2. Commit identity and the three coordinates: how things are named, including computing a commit ID by hand.
  3. Inside .synck: how it is laid out on disk.
  4. System design: how the pieces work together under concurrency, failure and copying, and why.
  5. Trade-offs and alternatives: what the design costs, and when to use something else.

We will use the sample repository throughout. Its whole history fits on one screen.

The whole history of a small project

Here is every event in the sample, oldest first, from synck repository log. Each line is a commit’s position, its author, the event’s position in its own item’s history, and the event type:

$ synck repository log launch-project --limit 512 | jq -r '[.commits[]] | reverse | .[] | .repo_seq as $s | .author as $a | .events[] | "\($s)|\($a)|\(.seq)|\(.event_type)"'
1|Maria|1|FOLDER::CREATE
1|Maria|2|FOLDER::SET_METADATA
1|Maria|3|FOLDER::WORKFLOW_RUNTIME
1|Maria|4|FOLDER::WORKFLOW_RUNTIME
2|Maria|1|CHAT::CREATE
2|Maria|2|CHAT::SET_PROFILE
3|Maria|1|FILE::CREATE
3|Maria|2|FILE::SET_CONTENT
4|Sam|3|CHAT::MESSAGE::CREATE
5|Maria|1|FILE::CREATE
5|Maria|2|FILE::SET_CONTENT
6|Maria|3|FILE::SET_CONTENT
7|Maria|4|CHAT::MESSAGE::CREATE
8|Sam|5|CHAT::MESSAGE::CREATE

Fourteen events in eight commits, by two authors, across four items. Everything Synck knows about this project, whether current files, old versions, conversation or authorship, is derived from these rows and the content they point to. Let us take them apart.

Items

An item is anything with an identity and a history. Every item has:

  • an item ID, a UUIDv7 that never changes, such as 01a174b7-8f4e-76f1-8ac4-440db922fb52;
  • an item type, such as FILE, FOLDER, CHAT, EMAIL, TASK_LIST or CALENDAR;
  • a name, and optionally a parent folder;
  • a lifecycle state: LIVE, TRASH or TOMBSTONED;
  • its own ordered list of events.

The engine enforces a small set of structural rules for every item: a parent must be a folder in the same repository, an item has at most one parent, folders cannot contain themselves, and two live items in the same folder cannot have the same name. Names are compared after Unicode normalization and locale-independent lower-casing, so Report.txt and report.txt collide on every operating system, not just on some.

Tree and non-tree items

Folders, files and some application items such as emails and task lists are tree items: they have a place in the folder tree, and the rules above give them paths. Chats, posts, calls, tasks, calendars, structured databases and time-tracking records are non-tree items: their parent is always empty, so they have no path and never appear inside a folder. The engine stores both kinds in one namespace. Non-tree items get generated internal names, such as chat-01a174b7-8a7d-74e0-b0d7-81d946ca6792, so they can never collide with a file name. What people see is a title kept in the item’s metadata. Here is how SynckHub creates the sample’s chat, in commit 2:

{"create_item":{"item_type":"CHAT","name":"chat-01a174b7-8a7d-74e0-b0d7-81d946ca6792"},"event_type":"CHAT::CREATE","item_id":"01a174b7-8a7d-74e0-b0d7-81d946ca6792","set_metadata":{"chat_title":"Launch review","non_tree":true}}

Older SynckHub chats were tree items inside chat folders. Their history still contains that placement, and readers understand both forms.

Workspaces are folders with a role

A SynckHub workspace is an ordinary FOLDER in the engine. What makes it a workspace is in its own history. Commit 1 created Launch with four events: the creation, a metadata event that sets its subtype, and two FOLDER::WORKFLOW_RUNTIME events, one recording the workspace’s state and one logging its creation:

{"event_type":"FOLDER::SET_METADATA","item_id":"01a174b7-8a7d-74e0-b0d7-81cf3866d6b9","set_metadata":{"subtype":"STANDARD_CASE","subtype_fields":{}}}
{"event_type":"FOLDER::WORKFLOW_RUNTIME","item_id":"01a174b7-8a7d-74e0-b0d7-81cf3866d6b9","set_metadata":{"kind":"PROJECTION","payload":{"item_subtype":"STANDARD_CASE","lifecycle_mode":"status_only","lifecycle_version_id":null,"pipeline_key":null,"stage_key":null,"status":"open"},"projection_kind":"CASE_RECORD_STATE_SET"}}

Synck preserves, verifies and copies these events, but the engine attaches no meaning to them. Interpreting a workspace’s state, and associating chats and other non-tree items with it, is SynckHub’s job.

Events

An event records one change to one item. Here are the two events that created proposal.txt in commit 3, exactly as stored:

{"create_item":{"item_type":"FILE","name":"proposal.txt","parent_folder_id":"01a174b7-8a7d-74e0-b0d7-81cf3866d6b9"},"event_type":"FILE::CREATE","item_id":"01a174b7-8f4e-76f1-8ac4-440db922fb52"}
{"event_type":"FILE::SET_CONTENT","item_id":"01a174b7-8f4e-76f1-8ac4-440db922fb52","set_content":{"blob_id":"sha256:fdfdf64e89e31bbe76af72e4df33950552d618c1d77b2b2e76b79e857c6e8043"},"set_metadata":{"content":{"size":91}}}

And the revision in commit 6, and Sam’s first message in commit 4:

{"base_seq":2,"event_type":"FILE::SET_CONTENT","item_id":"01a174b7-8f4e-76f1-8ac4-440db922fb52","set_content":{"blob_id":"sha256:65d1490153c828ac1dde577ddd8a9806f0b4288e47e0f4541f0f8545a2ff51f8"},"set_metadata":{"content":{"size":110}}}
{"base_seq":2,"event_type":"CHAT::MESSAGE::CREATE","item_id":"01a174b7-8a7d-74e0-b0d7-81d946ca6792","set_content":{"content_id":"sha256:91a93c8cd9ee5a004de7f179ff1b90e21a9ef254be46dddd3c6ba692d746e01e"}}

Every event has the same small vocabulary of fields. Each one is present only when it means something:

Field Meaning
item_id The item this event changes
event_type What kind of change, namespaced by item family: FILE::SET_CONTENT, CHAT::MESSAGE::CREATE
create_item For a creation: the new item’s type, name and parent
set_name, set_parent Structural changes: a rename or a move. set_parent: null means “move to the root”
set_content A reference to content: a blob_id for file bytes, or a content_id for an encoded body such as a message’s text
set_metadata Small structured data, such as the size of the content
base_seq The item’s last event that this change was based on, so that a stale change is rejected

Events do not contain file bytes or message text. They contain references: SHA-256 hashes of content stored separately. That keeps the history small and lets identical content be stored once. It also means a complete copy must carry the content as well as the events. The closure covers that.

Item sequences

Each event takes the next position in its item’s history, the item sequence (item_seq). In the listing above, the proposal’s events are numbered 1, 2, 3. The chat’s are 1 to 5, and they interleave with the file’s in the repository’s order without affecting them. Each item has its own counter and its own linear history.

That is what base_seq refers to. Commit 6 says “this new content follows event 2 of the proposal”. If someone else had added a version of the proposal in the meantime, the proposal would be at 3, and the change would be rejected instead of silently overwriting theirs.

Event types and profiles

The engine treats event types mostly as names. It enforces the structural rules above, but it does not know what a “message” is. Meaning comes from profiles layered on top:

  • The FILE/FOLDER profile defines exactly which events a file operation produces: creating a non-empty file is FILE::CREATE followed by FILE::SET_CONTENT, a move is SET_PARENT, and so on. Everyone writing files produces identical events, which is what makes histories interoperable.
  • The shared item definitions describe every family SynckHub stores: chats and their messages, reactions and attachments; email; posts; tasks; records; calendars; calls; time tracking. They define how to rebuild a readable item from its events. That is what synck item show uses.

Commits

A commit groups one or more events into a single change that is accepted atomically: all of its events are recorded, or none are. Commit 1 in the sample created a folder and made it a workspace. Commit 3 created a file and gave it content. There is never a moment when the workspace exists without its role, or the file without its content.

A commit has an author, an optional message, a time, and an ID that is a hash of its exact content. It does not have a parent commit. That is a sharp difference from Git, and the subject of the next page.

A commit can span many items. A folder move that renames three files and creates a fourth is one commit. Because each item has its own sequence, two commits that touch different items do not conflict with each other, even when they happen at the same moment. System design explains what that buys.

A second ledger for labels

Some history is about an item without being a change to it. The clearest case is named versions. Labelling version 2 of a file “approved” does not change the file’s content, so it should not add a file version.

For this, each item has a second, separate ledger: the ref ledger, with its own counter (ref_seq). Here is the label Synck adds automatically when you commit a new file, from the commit that created notes.txt in the first tutorial:

2 | FILE_VERSION::CREATE | item 01a1748d-2155-7312-aa41-44fec4246700 | item-seq 1
    {"base_ref_seq":0,"event_type":"FILE_VERSION::CREATE","owner_item_id":"01a1748d-2155-7312-aa41-44fec4246700",
     "set_content":{"blob_id":"sha256:0acafc69…"},"set_metadata":{"kind":"MINOR","source_item_seq":2,...}}

It belongs to the file (owner_item_id), points at file version 2 (source_item_seq), and is the first entry in the file’s ref ledger. Promoting or removing the label adds more ref events. The file’s own item sequence never moves. Item events and ref events can share one atomic commit, and complete copies preserve both ledgers.

Lifecycle

Deleting an item does not delete its history. It changes the item’s lifecycle:

            delete                    tombstone
   LIVE  ────────────▶  TRASH  ─────────────────────▶  TOMBSTONED
         ◀────────────          (final: no further events)
            recover

A trashed item is absent from listings of what is live, but can be recovered, keeping its ID and history. A tombstoned item can never receive another event. But tombstoning, like trashing, is itself an event. The item’s earlier history, and the content it refers to, is still part of the repository and still travels in complete copies. Nothing in Synck physically erases accepted history.

What the engine does not do

The engine at the center of Synck is deliberately small. It is responsible for identity, events, commits, ordering, structural rules, verification and historical state. Everything else belongs to the application that embeds it:

The engine does The application does
Assign item and ref sequences; reject stale changes Decide who may make a change: authentication and permissions
Group events into atomic, content-addressed commits Decide what a change means: editors, workflows, business rules
Enforce tree and naming rules Locks for editing, notifications, presence, search, previews
Track content references; verify history and content Store and deliver content in a service; send email

The synck CLI is one small application on top of the engine, for local repositories. SynckHub is a much larger one. Both write the same events, which is why a SynckHub export can be read by the CLI.

Previous
Storage space
Next
Commit identity