tsJson Package

Author: thosti.

The tsJson contributed package reads, builds and writes JSON, finds values in a document with JSON Pointer and JSONPath, changes documents with JSON Patch and JSON Merge Patch, validates them against a JSON Schema, and converts ISO 8601 dates. The package describes itself as “Parse, query, patch, validate - one JSON model.” This section documents version 1.0.15.0, published under the MIT licence.

All of it works on one model: a JSON object is a JsonObject, a JSON array is a JsonArray, and the other JSON values are plain Variant values. The Json module is the entry point, with a function for each task. JsonOptions holds the options of a call, and JsonError reports what went wrong.

Dim doc As Variant = Json.Parse("{""store"":{""book"":[{""title"":""A"",""price"":8.95},{""title"":""B"",""price"":12.99}]}}")
If JsonError.HasError Then
    Debug.Print JsonError.ToString()
Else
    Debug.Print Json.QueryOne(doc, "$.store.book[?@.price < 10].title")         ' A
    Debug.Print Json.PointerGet(doc, "/store/book/1/title")                      ' B

    Dim cheaper As Variant = Json.PointerSet(doc, "/store/book/1/price", 9.5)
    Debug.Print Json.Serialize(Json.Diff(doc, cheaper), JsonOptions.Minified)    ' [{"op":"replace","path":"/store/book/1/price","value":9.5}]

    Dim report As JsonValidationResult = Json.Validate(doc, Json.Parse("{""type"":""array""}"))
    Debug.Print report.ToString()                                                ' type / - Type is not 'array'.
End If

The JSON model

A value of JSON has these counterparts in twinBASIC. A function that reads JSON creates them, and a function that writes JSON accepts them.

JSON In twinBASIC Notes
object JsonObject Members in the order they were added. Keys are compared by character code, so a and A are different keys.
array JsonArray Elements numbered from 0.
string String  
number Decimal when read; any numeric type when written A Decimal keeps every digit. The package rejects a number outside the range of Decimal (see Parse).
true, false Boolean  
null Null Not Empty and not Nothing.

A Date is not a JSON value. Json.Serialize writes one as an ISO 8601 string in local time, and Json.Parse never creates one: a string that holds a date stays a string until Json.ParseIso8601 converts it. Empty, Nothing, and any object that is not a JsonObject or a JsonArray, are not JSON values either, and Json.Serialize refuses them with the error UnsupportedType. Json.TypeOfValue classifies a value.

A value that is read from a document is the object or array inside it, not a copy. Changing it changes the document. Json.Clone makes a deep copy, and the functions that return a changed document (PointerSet, PointerRemove, Patch, MergePatch) work on a copy and leave their argument as it was.

Where a function returns a list, the array is unbound when the list is empty, and UBound raises run-time error 9 for it. This applies to Json.Query, Json.QueryHits, JsonObject.Keys, JsonObject.Items, JsonArray.ToVariantArray and JsonValidationResult.Errors. Json.Serialize writes such an array as [].

Errors

The package does not raise run-time errors for bad data or bad arguments. A function that fails returns an empty result (Empty, Nothing, an empty string or False) and sets the last error, which the predeclared object JsonError reports: HasError is True, Code is a short name such as InvalidSyntax, and Message, Origin, the position in the text and the JSON Pointer of the place say more. A function that can fail clears the last error when it starts, so the last error belongs to the latest call. A search that finds nothing, and an invalid regular expression given to a pattern function, are not errors. JsonError lists the codes.

Dim v As Variant = Json.Parse("[1, 2,]")
Debug.Print IsEmpty(v)                                    ' True
Debug.Print JsonError.Code                                ' InvalidSyntax

Debug.Print Json.TryParse("[1, 2,]", v, JsonOptions.Jsonc)   ' True
Debug.Print Json.Serialize(v, JsonOptions.Minified)          ' [1,2]

The package itself raises a run-time error on purpose in one case: an assignment to a property of the predeclared JsonOptions object, which is a factory (error 5). The other run-time errors that a program meets are those of the language, such as error 91 for a call on the Nothing that a failed function returned, error 424 for the assignment of the Empty that a failed Parse returned to a variable of a class type, and error 9 for UBound of an unbound array. One more comes from Serialize, which raises error -2147467259 for a two-dimensional array, and leaves the last error stuck (see Known problems).

The standards

Standard What the package provides
RFC 8259, JSON Parse and Serialize, with limits on size and depth, and the JSONC dialect (comments and a trailing comma) as an option.
NDJSON, JSON Lines One value on each line: ParseLines and SerializeLines, and the same for files.
RFC 6901, JSON Pointer PointerGet, PointerSet and the other pointer functions.
RFC 9535, JSONPath Query and the other query functions, and CompilePath for a path that is used again. The syntax and its departures from the RFC are on the page of JsonPathQuery.
RFC 9485, I-Regexp The regular expressions of JSONPath and of schema patterns, and IsMatchPattern and its two partners.
RFC 6902, JSON Patch Patch, and Diff to make a patch.
RFC 7396, JSON Merge Patch MergePatch.
JSON Schema, draft 2020-12 Validate and CompileSchema, with a local catalogue of documents. JsonSchema lists the keywords that are checked.
ISO 8601 ParseIso8601 and FormatIso8601.

The package reads and writes text, byte arrays (UTF-8, UTF-16 little endian and big endian, with a byte order mark detected) and files, and always writes UTF-8. It has no streaming interface and no mapping of JSON onto user-defined types.

Where the package departs from the standards

Each departure is described where it applies; this is the list.

  • RFC 8259. The reading is strict. A number that a Decimal cannot hold is an error, which the RFC allows an implementation to do (Parse). The package accepts a byte order mark in bytes but not at the start of a String.
  • RFC 6901. The form of a pointer as a URI fragment, #/a/b, is not accepted. An empty token after another token names the wrong place (a defect, in Known problems).
  • RFC 6902. A patch can be a single operation object, which the RFC does not allow. The operations behave as the RFC says in every case that was tested: the add, remove, replace, move, copy and test operations, the index rules for arrays and the handling of the root.
  • RFC 7396. None found.
  • RFC 9535. A comparison with a missing value is false, <= and >= are false for equal values that are not numbers or strings, a call of an unknown function is not an error, a number literal in a filter must be an integer, length counts UTF-16 code units, and the results are values and pointers, not normalized paths. The first three are defects (JsonPathQuery).
  • RFC 9485. The matcher accepts \d, \s and \w, which the RFC leaves out, and does not accept \n, \r, \t, \-, \^ or a hyphen after a single character at the end of a set, which the RFC has. It does not try the other alternatives of a choice once one has matched (Patterns).
  • JSON Schema, draft 2020-12. pattern uses I-Regexp, not ECMA-262 regular expressions; lengths are counted in UTF-16 code units; format is checked loosely, and ipv4 wrongly; some keywords are not evaluated (JsonSchema).
  • ISO 8601. Only the extended forms with a date, or with a date and a time with seconds, are read (ParseIso8601).

Compared with the other JSON sections of this reference: VBA_JSON and the cJson class of TBMANLIB (which uses the VBA-JSON code) return Dictionary and Collection objects, read numbers as Double or, when long, as String, and raise run-time error 10001 for text they cannot read, while VBA-JSON does not check for commas between items and ignores text after the value. tsJson has its own classes, reads numbers as Decimal, accepts only valid JSON unless JSONC is asked for, and reports errors through JsonError.

Classes and modules

Test code

The package also contains its own test code, which has no pages: the modules Tests, TestsCore, TestsParse, TestsSerialize, TestsIo, TestsDates, TestsPointer, TestsPath, TestsPatch, TestsSchema, TestsRegexp, TestsExtNst, TestsExtNstData, TestsSmoke and TestsTour. Tests.RunAll runs twelve areas, Core, Parse, Serialize, Io, Dates, Pointer, Path, Patch, Schema, Regexp, ExtNst and Smoke, and prints the result of each check with Debug.Print, so the lines appear in the DEBUG CONSOLE of the IDE and nowhere else. Each check prints OK, FAIL or SKIP, a failed check does not stop the run, and the last lines give the counts of each area and in total and say that the tests passed or failed. The Io area and the demonstration write small files to the folder named by %TEMP% and delete them. The ExtNst area (TestsExtNst, with its data in TestsExtNstData) runs the accept and reject cases of the public JSONTestSuite against Parse, and skips the cases that the package decides differently: documents with a repeated key, which are an error by default, and numbers outside the range of Decimal. Tests.RunDemo runs TestsTour, which demonstrates the package on one order document and prints each result as pretty-printed JSON, with the calls that made it as comment lines in front. In version 1.0.15.0 a run has 714 checks that pass, none that fail and 6 that are skipped, and it ends with the line === tsJson tests passed followed by = characters. The areas have 96 (Core), 66 (Parse), 22 (Serialize), 23 (Io), 21 (Dates), 29 (Pointer), 38 (Path), 34 (Patch), 62 (Schema), 38 (Regexp), 278 (ExtNst) and 7 (Smoke) checks that pass; the six skipped checks are in ExtNst. The tests pass although the problems below exist, because no check covers them.

Internal modules

Nineteen more modules and classes are part of the package’s implementation and have no pages. The package describes all of them except JsonLimits as not part of its public API: JsonBuf (the string builder), JsonEncoding (the UTF-8 and UTF-16 codecs), JsonErrorStore (the storage behind JsonError), JsonIo (the file access), JsonIso8601 (the date conversion behind ParseIso8601), JsonLexer, JsonParser and JsonText (the JSON reader), JsonSerializer (the writer), JsonValues (helpers shared by JsonObject and JsonArray), JsonPointer, JsonPatch, JsonPath with its node class JsonPathNode, and JsonSchemaEngine, which implement the functions of Json that are named after them, JsonReNode (a node of a compiled regular expression), JsonUnicodeCats (the table of Unicode categories for \p{...}) and JsonIRegexp, the I-Regexp matcher that IsMatchPattern and its partners call. JsonLimits holds the constants of the default limits, which JsonOptions lists with their values. None of them is declared Private, so a project that references the package can name them.

Debug.Print JsonLimits.DEFAULT_MAX_DEPTH       ' 1000
Debug.Print JsonLimits.DEFAULT_MAX_BYTES       ' 16777216

The test modules are public in the same way, so their names, such as Tests, are visible to the whole project. A module of the project with the same name takes precedence, with no error: in a project with its own module Tests, a call of Tests.Run reaches the project’s module, and the package’s Tests can no longer be named.

Module Tests
    Public Function Run() As String
        Return "mine"
    End Function
End Module

Debug.Print Tests.Run()    ' mine

Known problems in version 1.0.15.0

Note

The package has three more problems, each of which is an error that the reader cannot miss.

  • Numbers must fit a Decimal. Parse rejects a valid JSON number with the error InvalidNumber when its digits make a whole number above 79228162514264337593543950335, or its scale is outside -28 to 28, so 1e29, 1e-29, a number of 36 digits such as pi, and the largest Double, 1.7976931348623157e308, are errors. Serialize fails the same way for a Double above that range. RFC 8259 allows an implementation to limit the range of numbers.
  • A JSONPath filter takes integer literals only. RFC 9535 allows a fraction and an exponent. [?@.price < 10.5] is the error InvalidPath. Compare with a whole number, or compare strings.
  • Tests.SetContinueOnFail has no effect. A failed check never stops Tests.RunAll, whatever the setting.
Debug.Print IsEmpty(Json.Parse("1e29"))                    ' True
Debug.Print JsonError.Code                                 ' InvalidNumber
Debug.Print IsEmpty(Json.Parse("1.7976931348623157e308"))  ' True
Debug.Print IsEmpty(Json.Parse("1e28"))                    ' False
Debug.Print Json.CompilePath("$[?@.price < 10.5]") Is Nothing   ' True
Debug.Print JsonError.Code                                 ' InvalidPath

The first check of this sample fails on purpose. The second check still runs after SetContinueOnFail False:

Tests.BeginRun
Tests.SetContinueOnFail False
Tests.CheckTrue "first", False
Tests.CheckTrue "second", True
Debug.Print Tests.FailCount()
' Output:
' === tsJson tests ===========================================
' FAIL  first
' expected: True
' actual:   False
' OK    second
' 1

See Also