Internet-Draft DARE Data Container August 2018
Hallam-Baker Expires February 28, 2019 [Page]
Stream:
Internet Engineering Task Force
Series:
Internet-Draft
Status:
Informational
Published:
Expires
Authors:
Phillip Hallam-Baker (Comodo Group Inc.)

Data At Rest Encryption Part 2: DARE Container
draft-hallambaker-dare-container-02

Abstract

This document describes DARE Container, a message and file syntax that allows an append-only sequence of data frames to be represented with cryptographic integrity, signature and encryption enhancements. The format supports data integrity checks using digest chains and Merkle trees. The simplest supports efficient write operations and efficient read operations in either the forward or reverse direction. Support for efficient random-access reads may be provided through the use of binary trees or index records appended to the end of the file.

This document is also available online at http://mathmesh.com/Documents/draft-hallambaker-dare-container.html.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at http://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time.It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on February 28, 2019

Table of Contents

1. Introduction

DARE Container is a message and file syntax that allows a sequence of data frames to be represented with cryptographic integrity, signature, and encryption enhancements to be constructed in an append only format. DARE Container was developed in response to needs that arose out of the design of the Mathematical Mesh [draft-hallambaker-mesh-architecture]. It is built on the binary encodings of JSON data objects, JSON-B and JSON-C [draft-hallambaker-jsonbcd] and the DARE Message format [draft-hallambaker-dare-message].

The format is designed to meet the requirements of a wide range of use cases including:

1.1. Container Format

A DARE Container consists of a sequence of variable length frames. Each frame consists of a forward length indicator, the framed data and a reverse length indicator. The reverse length indicator is written out backwards allowing the length and thus the frame to be read in the reverse direction:

Figure 1 : JBCD Bidirectional Frame

Each frame contains a single DARE Message consisting of a Header, Payload and Trailer (if required). The first frame in a container describes the container format options and defaults. These include the range of encoding options for frame metadata supported and the container profiles to which the container conforms.

All internal data formats are 64 bit clean allowing for containers of up to 18 exabytes to be written.

Five container types are currently specified but this may be reduced if the Digest and Tree types are withdrawn. These are:

Simple
The container does not provide any index or content integrity checks.
Tree
Frame headers contain entries that specify the start position of previous frames at the apex of the immediately enclosing binary tree. This enables efficient random access to any frame in the file.
Digest
Each frame trailer contains a PayloadDigest field. Modification of the payload will cause verification of the PayloadDigest value to fail on that frame.
Chain
Each frame trailer contains PayloadDigest and ChainDigest fields allowing modifications to the payload data to be detected. Modification of the payload will cause verification of the PayloadDigest value to fail on that frame and verification of the ChainDigest value to fail on all subsequent frames.
Merkle Tree
Frame headers contain entries that specify the start position of previous frames at the apex of the immediately enclosing binary tree. Frame Trailers contain TreeDigestPartial and TreeDigestFinal entries forming a Merkle digest tree.

1.2. Write

In normal circumstances, DARE Containers are written as an append only log. As with DARE Messages, integrity information (payload digest, signatures) is written to the message trailer. Thus, large payloads may be written without the need to buffer the payload data provided that the content length is known in advance.

1.3. Read Access

The use of reverse length indicators allows DARE containers to support efficient sequential access in either the forward or reverse directions.

Random access to any part of a file MAY be supported by means of a binary tree index and/or an index record providing direct access to any part of the file.

1.4. Encryption and Authentication

Frame payloads and associated attributes MAY be encrypted and/or authenticated using the approach described in [draft-hallambaker-dare-message].

Incremental encryption is supported allowing encryption parameters from a single public key exchange operation to be applied to encrypt multiple frames. The public key exchange information is specified in the first encrypted frame and subsequent frames encrypted under those parameters specify the location at which the key exchange information is to be found by means of the ExchangePosition field.

The only restriction on the use of incremental encryption is that the frame containing the key exchange information MUST precede the frames that reference the exchange parameters.

To avoid cryptographic vulnerabilities resulting from key re-use, the DARE key exchange requires that each encrypted sequence use an encryption key and initialization vector derived from the master key established in the public key exchange by means of a unique salt.

Each DARE Message and by extension, each DARE Container frame MUST specify a unique salt value of at least 128 bits. Since the encryption key is derived from the salt value by means of a Key Derivation Function, erasure of the salt MAY be used as a means of rendering the payload plaintext value inaccessible without changing the payload value.

1.5. Integrity and Signature

Signatures MAY be applied to a payload digest, the final digest in a chain or tree. The chain and tree digest modes allow a single signature to be used to authenticate all frame payloads in a container.

The tree signature mode is particularly suited to applications such as file archives as it allows files to be verified individually without requiring the signer to sign each individually. Furthermore, in applications such as code signing, it allows a single signature to be used to verify both the integrity of the code and its membership of the distribution.

As with DARE Message, the signature mechanism does not specify the interpretation of the signature semantics. The presence of a signature demonstrates that the holder of the private key applied it to the specified digest value but not their motive for doing so. Describing such semantics is beyond the scope of this document and is deferred to future work.

1.6. Redaction

The chief disadvantage of using an append-only format is that containers only increase in size. In many applications, much of the data in the container becomes redundant or obsolete and a process analogous to garbage collection is required. This process is called redaction.

The simplest method of redaction is to create a new container and sequentially copy each entry from the old container to the new, discarding redundant frames and obsolete header information.

For example, partial index records may be consolidated into a single index record placed in the last frame of the container. Unnecessary signature and integrity data may be discarded and so on.

It is also possible but not necessarily advisable to perform such a redaction in-place provided that the redaction process does not increase the length of any individual frame and that appropriate provision is made for file locking to prevent conflicts and to provide for safe resumption should an interruption occur during the process.

1.7. Alternative approaches

Many file proprietary formats are in use that support some or all of these capabilities but only a handful have public, let alone open, standards. DARE Container is designed to provide a superset of the capabilities of existing message and file syntaxes, including:

Attempting to make use of these specifications in a layered fashion would require at least three separate encoders and introduce unnecessary complexity. Furthermore, there is considerable overlap between the specifications providing multiple means of achieving the same ends, all of which must be supported if decoders are to work reliably.

1.8. Efficiency

Every data format represents a compromise between different concerns, in particular:

Compactness
The space required to record data in the encoding.
Memory Overhead
The additional volatile storage (RAM) required to maintain indexes etc. to support efficient retrieval operations.
Number of Operations
The number of operations required to retrieve data from or append data to an existing encoded sequence.
Number of Disk Seek Operations
Optimizing the response time of magnetic storage media to random access read requests has traditionally been one of the central concerns of database design. The DARE Container format is designed to the assumption that this will cease to be a concern as solid state media replaces magnetic.

While the cost of storage of all types has declined rapidly over the past decades, so has the amount of data to be stored. DARE Container represents a pragmatic balance of these considerations for current technology. In particular, since payload volumes are likely to be very large, memory and operational efficiency are considered higher priorities than compactness.

2. Definitions

3. Container Navigation

Three means of locating frames in a container are supported:

Sequential
Access frames sequentially starting from the start or the end of the container.
Binary search
Access any container frame by frame number in O(log2(n)) time by means of a binary tree constructed while the container is written.
Index
Access and container frame by frame number or by key by means of an index record.

All DARE Containers support sequential access. Only tree and Merkle tree containers support binary search access. An index frame MAY be written appended to any container and provides O(1) access to any frame listed in the index.

Two modes of compilation are considered:

Monolithic
Frames are added to the container in a single operation, e.g. file archives,
Incremental
Additional frames are written to the container at various intervals after it was originally created, e.g. server logs, message spools.

In the monolithic mode, navigation requirements are best met by writing an index frame to the end of the container when it is complete. It is not necessary to construct a binary search tree unless a Merkle tree integrity check is required.

In the incremental mode, Binary search provides an efficient means of locating frames by frame number but not by key. Writing a complete index to the container every m write operations provides O(m) search access but requires O(n2) storage.

Use of partial indexes provides a better compromise between speed and efficiency. A partial index is written out every m frames where m is a power of two. A complete index is written every time a binary tree apex record is written. This approach provides for O(log2(n)) search with incremental compilation with approximately double the overhead of the monolithic case.

3.1. Tree

Binary search is supported by means of the TreePosition parameter specified in the FrameHeader. This parameter specifies the value of the immediately preceding apex.

Calculation of the immediately preceding apex is most easily described by representing the array index in binary with base of 1 (rather than 0). An array index that is a power of 2 (2, 4, 8, 16, etc.) will be the apex of a complete tree. Every other array index has the value of the sum of a set of powers of 2 and the immediately preceding apex will be the value of the next smallest power of 2 in the sum.

For example, to find the immediately preceding apex for frame 5, we add 1 to get 6. 6 = 4 + 2, so we ignore the 2 and the preceding frame is 4.

The values of Tree Position are shown for the first 8 frames in figure xx below:

Figure 2 : Merkle Tree Integrity check

An algorithm for efficiently calculating the immediately preceding apex is provided in Appendix C.

3.2. Position Index

Contains a table of frame number, position pairs pointing to prior locations in the file.

3.3. Metadata Index

Contains a list of IndexMeta entries. Each entry contains a metadata description and a list of frame indexes (not positions) of frames that match the description.

4. Integrity Mechanisms

Frame sequences in a DARE container MAY be protected against a frame insertion attack by means of a digest chain, a binary Merkle tree or both.

4.1. Digest Chain calculation

A digest chain is simple to implement but can only be verified if the full chain of values is known. Appending a frame to the chain has O(1) complexity but verification has O(n) complexity:

Figure 3 : Hash chain integrity check

The value of the chain digest for the first frame (frame 0) is H(H(null)+H(Payload0)), where null is a zero length octet sequence and payloadn is the sequence of payload data bytes for frame n

The value of the chain digest for frame n is H(H(Payloadn-1 +H(Payloadn)), where A+B stands for concatenation of the byte sequences A and B.

4.2. Binary Merkle tree calculation

The tree index mechanism describe earlier may be used to implement a binary Merkle tree. The value TreeDigest specifies the apex value of the tree for that node.

Appending a frame to the chain has O(log2n) complexity provided that the container format supports at least the binary tree index. Verifying a chain has O(log2 n) complexity, provided that the set of necessary digest inputs is known.

To calculate the value of the tree digest for a node, we first calculate the values of all the sub trees that have their apex at that node and then calculate the digest of that value and the immediately preceding local apex.

5. Reference

TBS stuff

5.1. Container Headers

TBS stuff

5.1.1. Structure: ContainerHeaderFirst
Inherits: ContainerHeader
DataEncoding: String (Optional)
Specifies the data encoding for the header section of for the following frames. This value is ONLY valid in Frame 0 which MUST have a header encoded in JSON.
5.1.2. Structure: ContainerHeader
Inherits: DAREHeader

Describes a container header. A container header MAY contain any DARE Message header.

Index: Integer (Optional)
The record index within the file. This MUST be unique and satisfy any additional requirements determined by the ContainerType.
ContainerType: String (Optional)
Specifies the container type for the following records.
IsMeta: Boolean (Optional)
If true, the current frame is a meta frame and does not contain a payload.
Note: Meta frames MAY be present in any container. Applications MUST accept containers that contain meta frames at any position in the file. Applications MUST NOT interpret a meta frame as a data frame with an enpty payload.
UniqueID: String (Optional)
Unique object identifier
ContentMeta: ContentMeta (Optional)
Content meta data.
TreePosition: Integer (Optional)
Position of the frame containing the apex of the preceding sub-tree.
IndexPosition: Integer (Optional)
Specifies the position in the file at which the last index entry is to be found
ExchangePosition: Integer (Optional)
Specifies the position in the file at which the key exchange data is to be found
ContainerIndex: ContainerIndex (Optional)
An index of records in the current container up to but not including this one.
PayloadDigest: Binary (Optional)
If present, contains the digest of the Payload.
ChainDigest: Binary (Optional)
If present, contains the digest of the PayloadDigest values of this frame and the frame immediately preceding.
TreeDigest: Binary (Optional)
If present, contains the Binary Merkle Tree digest value.
Event: String (Optional)
Unique object identifier
Labels: String [0..Many]
List of labels that are applied to the payload of the frame.
KeyValues: KeyValue [0..Many]
List of key/value pairs describing the payload of the frame.
First: Integer (Optional)
Frame number of the first object instance value.
Previous: Integer (Optional)
Frame number of the immediately prior object instance value

5.2. Content Metadata Structure

TBS stuff

5.2.1. Structure: ContentMeta

Information describing the object instance

ContentType: String (Optional)
The content type field as specified in JWE
Paths: String [0..Many]
List of filename paths for the payload of the frame.
UniqueID: String (Optional)
Unique object identifier
Created: DateTime (Optional)
Initial creation date.
Modified: DateTime (Optional)
Date of last modification.

5.3. Index Structures

TBS stuff

5.3.1. Structure: ContainerIndex

A container index

Full: Boolean (Optional)
If true, the index is complete and contains position entries for all the frames in the file. If absent or false, the index is incremental and only contains position entries for records added since the last frame containing a ContainerIndex.
Positions: IndexPosition [0..Many]
List of container position entries
Metas: IndexMeta [0..Many]
List of container position entries
5.3.2. Structure: IndexPosition

Specifies the position in a file at which a specified record index is found

Index: Integer (Optional)
The record index within the file.
Position: Integer (Optional)
The record position within the file relative to the index base.
5.3.3. Structure: KeyValue

Specifies a key/value entry

Key: String (Optional)
The key
Value: String (Optional)
The value corresponding to the key
5.3.4. Structure: IndexMeta

Specifies the list of index entries at which a record with the specified metadata occurrs.

Index: Integer [0..Many]
List of record indicies within the file where frames matching the specified criteria are found.
ContentType: String (Optional)
Content type parameter
Paths: String [0..Many]
List of filename paths for the current frame.
Labels: String [0..Many]
List of labels that are applied to the current frame.

5.4. Signature

Payload data MAY be signed using a JWS [RFC7515] as applied in the DARE Message format [draft-hallambaker-dare-message].

Signatures are specified by the Signatures parameter in the content header. The data that the signature is calculated over is defined by the typ parameter of the Signature as follows.

Payload
The value of the PayloadDigest parameter
Chain
The value of the ChainDigest parameter
Tree
The value of the TreeDigestFinal parameter

If the typ parameter is absent, the value Payload is implied.

A frame MAY contain multiple signatures created with the same signing key and different typ values.

The use of signatures over chain and tree digest values permit multiple frames to be validated using a single signature verification operation.

6. Security Considerations

7. IANA Considerations

8. Acknowledgements

9. Appendix A: Examples and Test Vectors

The data payloads in all the following examples are identical, only the authentication and/or encryption is different.

For conciseness, the raw data format is omitted for examples after the first, except where the data payload has been transformed, (i.e. encrypted).

9.1. Simple container

the following example shows a simple container with first frame and a single data frame:

f4 5d 
f0 59 
f0 00 
5d f4 
f5 01 40 
f0 0f 
f1 01 2c 
40 01 f5 

Since there is no integrity check, there is no need for trailer entries. The header values are:

Frame 0

{
  "Index":0,
  "ContainerType":"List",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Frame 1

{
  "Index":1}

[Empty trailer]

9.2. Payload and chain digests

The following example shows a chain container with a first frame and three data frames. The headers of these frames is the same as before but the frames now have trailers specifying the PayloadDigest and ChainDigest values:

Frame 0

{
  "Index":0,
  "ContainerType":"Chain",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Frame 1

{
  "Index":1}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9
  tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Frame 2

{
  "Index":2}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9
  tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Frame 3

{
  "Index":3}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "ChainDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9
  tn_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

9.3. Merkle Tree

The following example shows a chain container with a first frame and six data frames. The trailers now contain the TreePosition and TreeDigest values:

Frame 0

{
  "Index":0,
  "ContainerType":"Merkle",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Frame 1

{
  "Index":1,
  "TreePosition":0}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t
  n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Frame 2

{
  "Index":2,
  "TreePosition":319}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"7fHmkEIsPkN6sDYAOLvpIJn5Dg3PxDDAaq-ll2kh8722kokkFnZ
  QcYtjuVC71aHNXI18q-lPnfRkmwryG-bhqQ"}

Frame 3

{
  "Index":3,
  "TreePosition":319}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t
  n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Frame 4

{
  "Index":4,
  "TreePosition":1451}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"vJ6ngNATvZcXSMALi5IUqzl1GBxBnTNVcC87VL_BhMRCbAvKSj8
  gs0VFgxxLkZ2myrtaDIwhHoswiTiBMLNWug"}

Frame 5

{
  "Index":5,
  "TreePosition":1451}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t
  n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Frame 6

{
  "Index":6,
  "TreePosition":2586}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"WgHlz3EHczVPqgtpc39Arv7CFIsCbFVsk8wg0j2qLlEfur9SZ0m
  dr65Ka-HF0Qx8gg_DAoiJwUrwADDXyxVJOg"}

9.4. Signed container

The following example shows a tree container with a signature in the final record. The signing key parameters are:

{
  "PrivateKeyRSA":{
    "kid":"MA3HR-W4WTI-QFMMQ-SRXKG-XR5L3-T3BQD-A",
    "n":"sv6WHejtoDKzpM2WrLqYywiBSB2zWU_dQ38YRv2Q8Q5Du5QrzP5hPaY1N6
  XsK-jXIXtLkUphwNNnR4FYj5KBGvlHXWtx326CNIVexsWzJDNXix-HEeGFl6zuFJe
  o99-iHtUGfu6tJbLVEzZO0VhRnjRV4qri88I1CbFq2Q_QUJg4USF6ma26ZC7cjYDD
  D4OtZ1tsgcs1gE8ji9koa8LGIwojD6NIIe8Pmi0WPqZI-z6NRqnKRmPY7x_02cofq
  ezQoqVf2QmAWbP3x7vZRu3PolxezQecTlhqtjyBjEcdbyOmS0xykWp7uElyzJZovC
  g11fcTnY16qZuKJ8B-oKNx_Q",
    "e":"AQAB",
    "d":"UsvWzrEPMhoh5OrAdte9k7U0cBJlArwhonYo2bWzFP4_MIaCmElW27iFR1
  911n3W_YLV2rgobZScYoQfXhIrrDmA3GK2At9QVVEYHSQYEHe7V_xWc1v5esagPmB
  E5ZvJly982Lw41YZab4-gd8-9uUuGKtpXLuA3vmsV_KAHATARoBxloeJDZF-Bi-sf
  J5hX9lh41houcODRntP3fLQyNcaJfH4YULytBdIV3be1N_O0kZmSU-a4VNL2AtORA
  C1xfsjCTU0hGd6nrRMncT0lSU6Njh1Yw2tPE__FVwkx2F1NRYmkK_mqoSpMJ5NWk_
  sTi05IrKG5g8ajlDu3tGbVVQ",
    "p":"xHElx_ETa4Q0V8C_7QtYWs6o3EX-hb3s-OeTRVt7IAKFIj1iq5AhyESX8J
  9Dmt6GWV8uKPP5K0y76CbJM6cM3z1khiAUJsFUy2d6ZHQQUVns2xxSN2qBhzFIxv6
  Lbbk-euhPRqI9YA81iSXfc2T3WlB46Cx7yCzneKnbao6xmb8",
    "q":"6UNAAkoXSWw4UUZ8ZDFrg6d2rBzvJQek99VH5w-KF5-hefWyvAKsXy-ww3
  9MobOPDCk7XevL9580Z_zjuMNB4w33lXa5JywQBFNCwso60-ZLBrQe_LofRa-beqe
  Na0dYSCI4F63D5A_tDDMNad2mU6kUgp3dXUjgj22aQlqzC0M",
    "dp":"KOinRYwezzUo8V1imqrSK5b982FgL9ntj6gqMzdp3LCpqQe_lJrfbiVav
  vFr0sSkldgMdNrZ-Ppx51asb-VJ-POAC66Cg8-fKPzTsDe7bjWb7cVoGcJEVzMcRN
  SdlKwPJulaGQuQXIDOVVFdx1ovaM95XmBlJrqzOZ2l4_rzYWE",
    "dq":"tpQ16NTXb1fuqOcz_DSWhAEzKTqokOJv5LCjTB9kpmMDQHPpR7IL-6FXi
  V39vF_QauNnHXewOgz6Nx7Bm9_xlWhcTzgTaq7ZhpkpG1CxRSTvurqHuqdwTNpMJ3
  A4iCza0_M81oPjXYJ9t9vJ9wBP7RHeBPLNzvN5GarfYGqf1AM",
    "qi":"Ay1lgoJtsIWN_7Htm7x95Xr1MHRNocW3H3uZQ8_o6HdpSxR7LoqkKOOmL
  ZbjJYlJ7fdeGypRnGQTBB9g-bdi5Yhlx2N_aT0l1rAuktUMouRqhDLLERTZDs_RE2
  bQ1ZdD4XXD7iX9LugoSVoFUbMZeLLeQZlxKeUBjbhnPin32Ek"}}

The container headers and trailers are:

Frame 0

{
  "Index":0,
  "ContainerType":"Merkle",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Frame 1

{
  "Index":1,
  "TreePosition":0}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"T7S1FcrgY3AaWD4L-t5W1K-3XYkPTcOdGEGyjglTD6yMYVRVz9t
  n_KQc6GdA-P4VSRigBygV65OEd2Vv3YDhww"}

Frame 2

{
  "dig":"S512",
  "Index":2,
  "TreePosition":319}

{
  "PayloadDigest":"8dyi62d7MDJlsLm6_w4GEgKBjzXBRwppu6qbtmAl6UjZDlZe
  aWQlBsYhOu88-ekpNXpZ2iY96zTRI229zaJ5sw",
  "TreeDigest":"7fHmkEIsPkN6sDYAOLvpIJn5Dg3PxDDAaq-ll2kh8722kokkFnZ
  QcYtjuVC71aHNXI18q-lPnfRkmwryG-bhqQ"}

9.5. Encrypted container

The following example shows a container in which all the frame payloads are encrypted under the same master secret established in a key agreement specified in the first frame.

Frame 0

{
  "enc":"A256CBC",
  "Salt":"hQr1TtvJC2PTH6r7qlqwiw",
  "recipients":[{
      "kid":"MD7VA-LPCWZ-KNRB2-EZVDE-G4OC2-QQXON-A",
      "epk":{
        "PublicKeyDH":{
          "kid":"MALYI-FZLZK-4DNZ2-QNNQE-6DDMO-7LQQG-A",
          "Domain":"YE6bnq1MlX5ojaJto6PLP_PEwA",
          "Public":"NquIeZImtzWI0fLjPuXhHFeWX7wLhUvO3sVrWTjuaPKKcE8
  wdWsEi4walm2yDWibPYhROtp4KbR6zTqrkLCftOzE_vfEWOiC_LABy-6ZmZOr0koC
  Xxvth-mewl3lG6JZ2KQ_o2SsU9oDxqTnY8PfadKypx_aG26217SM2DWZbkH75TKUn
  -_5ZkqMiD6xlS6wRsFKvRINF2Irp7KnfhVZ_mtxSlUMetUBQF1G_-a96t4FT9o0E2
  UwiQEophBLai5YT4Shor-yuZ099fs60wCMKS5eXdsvc8naB4CBNG948OdcGwhFwD-
  XYdu2k37N_mKJUCYJerL6CPXYsF6fu96FeA"}},
      "wmk":"Km6dDZydvXWbSVPvmIMbGUtKlF4qC1lwD_eenZW1RUusHjSZkHxbxw"}
    ],
  "Index":0,
  "ContainerType":"List",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Frame 1

{
  "enc":"A256CBC",
  "Salt":"sk5YcFtgXaWjXhIcitVN2A",
  "Index":1}

[Empty trailer]

Frame 2

{
  "enc":"A256CBC",
  "Salt":"eeVPNc6H94BedXK698pI9g",
  "Index":2}

[Empty trailer]

Here are the container bytes. Note that the content is now encrypted and has expanded by 25 bytes. These are the salt (16 bytes), the AES padding (4 bytes) and the JSON-B framing (5 bytes).

f5 03 3c 
f1 03 27 
f0 10 
3c 03 f5 
f5 01 7c 
f0 47 
f1 01 30 
7c 01 f5 
f5 01 7c 
f0 47 
f1 01 30 
7c 01 f5 

The following example shows a container in which all the frame payloads are encrypted under separate key agreements specified in the payload frames.

Frame 0

{
  "Index":0,
  "ContainerType":"List",
  "ContentMeta":{},
  "DataEncoding":"JSON"}

[Empty trailer]

Frame 1

{
  "enc":"A256CBC",
  "Salt":"u8HwHEzrmfiMzK3cPoA4pQ",
  "recipients":[{
      "kid":"MD7VA-LPCWZ-KNRB2-EZVDE-G4OC2-QQXON-A",
      "epk":{
        "PublicKeyDH":{
          "kid":"MAUSK-BMU2P-TFILI-YNMPX-U34MH-SWBDD-A",
          "Domain":"YE6bnq1MlX5ojaJto6PLP_PEwA",
          "Public":"rmS1sXafsr3K7YCm6sY8oWBGVmLLCRjh2M11Ug6A_P-L_q0
  jmPpC47YS0p71E2iAFDTyZ6y6m6aR6bZF24pa4yR506i5EHapC_ip8XrwTXQeuHg3
  N34yF31B4m23dPoIvHzC8RTcMVla69QEztQy_wsbXtyDx1KPtX0wzUaVV2ZViA39U
  LrqnaqRXNtX6qd80_vapG2lDrmiIukuEswv-uZJXABvuFFWEBr85PW2i52VJut444
  cMVYsJSBvZBtzvdmfHfUYowfg9Rh3mIqHfmfD9j3utGLhVHJNGYoJ70CICqx8PKf9
  OT-Ytkitx4AvodCGBvaYliXWAQupwA7_2Lw"}},
      "wmk":"6fDkIX4mlSNYbQtq8fvuao34duolZKCaWxfTQBfmuvDsvE5uIQVwOA"}
    ],
  "Index":1}

[Empty trailer]

Frame 2

{
  "enc":"A256CBC",
  "Salt":"1rDpenMyncmfmAlgKhPS7w",
  "recipients":[{
      "kid":"MD7VA-LPCWZ-KNRB2-EZVDE-G4OC2-QQXON-A",
      "epk":{
        "PublicKeyDH":{
          "kid":"MBFUO-3YTZ6-ND4GU-CTFXR-ULYAE-ZRSBI-A",
          "Domain":"YE6bnq1MlX5ojaJto6PLP_PEwA",
          "Public":"c7pcI-LLX2FvDC0UfK18Uzi779ObqrAgR4fSCq71S09WXW0
  W-ic2TYTwdsWlDmID5XnMbM0vqhg4E681QbkHAH5Fq-WyREiKNJRhk2gtwnQok8B2
  UNcd2or0wUiVkwWUxVySBzXZkwBl6fDtVSEZnHAZRDp5IeY4xGgM1n-YObQlATeYQ
  jzeUoVjpyS3TcvU8udpDIjsKcKCHf1GEFMaIXao51v3KqdQunJk_8-Qq0E6xUu6_z
  eTFGMm13ECejIBi_9OykQ4A1KaY_zLLXgj8vhJH4IhzcAPVOkI1PqQngW3Y5-cn0R
  _1AloJWqyjP9Qu1GqY_A4R707zpjxsfCriQA"}},
      "wmk":"3nWqFDha2RNXvtRSvhdsvfJdoJag8CGaJl-6kcaw-ExYnRjDEXpMsQ"}
    ],
  "Index":2}

[Empty trailer]

10. Appendix B

public long PreviousFrame (long Frame) {
    long x2 = Frame + 1; 
    long d = 1; 

    while (x2 > 0) {
        if ((x2 & 1) == 1) {
            return x2 == 1 ? (d / 2) - 1 : Frame - d; 
            }
        d = d * 2; 
        x2 = x2 / 2; 
        }
    return 0; 
    }

References

Normative References

[RFC7159]
T. Bray "The JavaScript Object Notation (JSON) Data Interchange Format" RFC 7159 DOI 10.17487/RFC7159
[RFC7515]
M. Jones and J. Bradley and N. Sakimura "JSON Web Signature (JWS)" RFC 7515 DOI 10.17487/RFC7515
[RFC7516]
M. Jones and J. Hildebrand "JSON Web Encryption (JWE)" RFC 7516 DOI 10.17487/RFC7516
[RFC2119]
S. Bradner "Key words for use in RFCs to Indicate Requirement Levels" BCP 14 RFC 2119 DOI 10.17487/RFC2119
[draft-hallambaker-jsonbcd]
Phillip Hallam-Baker "Binary Encodings for JavaScript Object Notation: JSON-B, JSON-C, JSON-D" Internet-Draft draft-hallambaker-jsonbcd-13 <http://www.ietf.org/internet-drafts/draft-hallambaker-jsonbcd-13.txt>
[draft-hallambaker-dare-message]
Phillip Hallam-Baker "Data At Rest Encryption Part 1: DARE Message" Internet-Draft draft-hallambaker-dare-message-01 <http://www.ietf.org/internet-drafts/draft-hallambaker-dare-message-01.txt>

Informative References

[ZIPFILE]
"APPNOTE.TXT - .ZIP File Format Specification " <https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT>
[BLOCKCHAIN]
"Blockchain Specification " <https://chain.com/docs/1.2/protocol/specifications/blockchain>
[RFC5652]
R. Housley "Cryptographic Message Syntax (CMS)" STD 70 RFC 5652 DOI 10.17487/RFC5652
[draft-hallambaker-mesh-architecture]
Phillip Hallam-Baker "Mathematical Mesh: Architecture" Internet-Draft draft-hallambaker-mesh-architecture-04 <http://www.ietf.org/internet-drafts/draft-hallambaker-mesh-architecture-04.txt>

Author's Address


Phillip Hallam-Baker
Comodo Group Inc.
Email:
Prepared: Rendered: