23ValidationwithJSONSchema

Validations

Outline

Overview
JSON Schema validations
$jsonSchema operator
Validation of any BSON documents
Validation of flat BSON documents
Validation of nested BSON documents

Semistructured and schema-less properties of JSON/BSON data model allow for very flexible manipulation of database structures. It is possible to create the add new documents with very different structure from the documents that have been already inserted into a collection. At any moment in time it is possible to extend a structure of the documents without changing anything in a formal description of a collection just by inserting a new document. Such flexibility in manipulation on the documents' structures is very convenient yet it bring some problems.

A very flexible and uncontrolled manipulations of database structures open the possibilities for pretty easy corruption of database structures and database contents, for example, due to the random mistakes. For example a new document inserted into a collection may have a different structure from the already inserted documents not because such was an intention of a database manager but simply due to an ordinary mistake. A typical mistake is to accidentally skipa fragment of a document description or to use a different names for keys in a document. Then, without any formal definition of document structure it is impossible to detect such mistake.

Practice shows that certain level of verification of database consistency constraints is always required. Such verification is possible when information about all acceptable structures of a document is provided. Such information is usually called as a "schema". For example "relational schema" or "XML Schema" or "hierarchical schema". Sometime a name is a different, fro example "Document Type Definition" (DTD) or class definition in ordinary object-oriented programming languages.

MongoDB provides the capability to validate documents during updates and insertions (and not deletions). There are two validation systems. The older one, implemented within MongioDB and the new one based on a concept of JSON Schema.

It is also possible use collMod command with the validator option to dynamically turn on/off the validation rules.

This presentation shows how to perform validation of BSON documents in MongoDB collections with JSON schemas.

Overview

Semistructured and schemaless properties of JSON/BSON data model allow for very flexible manipulation of database structures
In the same moment very flexible and uncontrolled manipulations of database structures open the possibilities for pretty easy corruption of database structures and database contents, for example, due to the random mistakes
Practice shows that certain level of verification of database consistency constraints is always needed
MongoDB provides the capability to validate documents during updates and insertions (and not deletions)
Validation rules are specified within createCollection() method using the validator option
It is possible use collMod command with the validator option to turn on/off validation rules

Validations

JSON Schema is a BSON document that defines the structure of JSON data for validation, documentation, and interaction control.

Validation of a BSON documents with JSON schema is performed through verification of consistency of the structures and contents of the document with JSON Schema.

A concept of JSON Schema is based on very similar concepts "borrowed" from XML Schema.

The process starts from application of a method createCollection() to create an empty collection and associate a validator with a collection.

If validation with JSON Schema is required then validator is associated with a JSON schema that determines the structures and the contents of the documents in the collection.

If validation is no longer needed or it must be temporarily disabled then it is possible use collMod method with the validator option to turn on/off validation rules.

JSON Schema validation

JSON Schema is a BSON document that defines the structure of JSON data for validation, documentation, and interaction control
JSON Schema validation of a BSON document is performed through verification of consistency of the structures and contents of the document with JSON Schema
JSON Schema is based on the concepts "borrowed" from XML Schema
A method createCollection() creates an empty collection and associates validator with a collection
If validation with JSON Schema is required then validator is associated with a JSON schema that determines the structures and the contents of the documents in the collection
It is possible use collMod method with the validator option to turn on/off validation rules

Validations

Outline

Overview
JSON Schema validations
$jsonSchema operator
Validation of any BSON documents
Validation of flat BSON documents
Validation of nested BSON documents

Whenever we create a collection with a method CreateCollection an operator $jsonSchema is associated with a JSON Schema to be used for validation of BSON documents.

The operator can also be used to query with find command or with $match aggregation stage.

The following application of $jsonSchema operator validates any BSON document in a collection department:

db.createCollection(

"department",

{"validator":{$jsonSchema:{"bsonType":"object"} }

} );

For example, the following BSON document validates well against a validator above.

{"name":"School of Astronomy",

"address":{"street":"Franz Josef Str","bldg":4},

"courses":[{"code":"SOA101","title":"Astronomy for Kids","credits":3} ] }

It is because the document has a correct syntactical structure of a BSON document.

$jsonSchema operator

In MongoDB an operator $jsonSchema is asociated with a JSON Schema to be used for validation of BSON documents
$jsonSchema operator can also be used to query with find command or with $match aggregation stage

The following application of $jsonSchema operator validates any BSON document in a collection department


db.createCollection("department",
                    {"validator":{$jsonSchema:{"bsonType":"object"} }
                    } );

For example, the following BSON document validates well

	      
{"name":"School of Astronomy",
 "address":{"street":"Franz Josef Str",
            "bldg":4},
 "courses":[{"code":"SOA101",
             "title":"Astronomy for Kids",
             "credits":3} ] }

Validations

Outline

Overview
JSON Schema validations
$jsonSchema operator
Validation of any BSON documents
Validation of flat BSON documents
Validation of nested BSON documents

Validation of any BSON documents

The following JSON Schema validates well any document in a collection anyDocument


db.createCollection("anyDocument",
                    {"validator":{$jsonSchema:{"bsonType":"object"} }
                    } );

Note the application of bsonType in MongoDB validation instead of type in standard JSON schema

Validations

Outline

Overview
JSON Schema validations
$jsonSchema operator
Validation of any BSON documents
Validation of flat BSON documents
Validation of nested BSON documents

Assume that we would like to insert the following BSON document into a collection "tiny".

db.tiny.insert({"name":"Harry Potter"});

The document fails validation against the following JSON schema.

db.createCollection( "tiny", {"validator":{$jsonSchema:

{"bsonType":"object",

"properties":{"name":{"bsonType":"string"} },

"required":["name"],

"additionalProperties":false } } } );

A validation fails because a key "additionalProperties" is associated with a value false and a key "_id" included in every BSON document

is not on a "required" list. It means that either a key "_id" is included to a "required" list of keys or "additionalProperties" is associated with a value "true".

It is definitely more consistent with a spirit if database systems to have a key "_id" liste as "required" and exclude any additional properties not

listed in a conceptual schema of a database.

Validation of flat BSON documents

The following BSON document in a collection tiny

fails validation against the following JSON schema


db.createCollection( "tiny",
                    {"validator":{$jsonSchema:
{"bsonType":"object",
 "properties":{"name":{"bsonType":"string"} },
 "required":["name"],
 "additionalProperties":false
} } } );

A validation fails because a key additionalProperties is associated with false and a key "_id" included in every BSON document is not on required list

Validation of flat BSON documents

The following BSON document in a collection tiny


db.tiny.insert({"_id":"HP666","name":"Harry Potter"});

passes validation against the following JSON schema


db.createCollection( "tiny",
                    {"validator":{$jsonSchema:
{"bsonType":"object",
 "properties":{"_id":{"bsonType":"string"},
	       "name":{"bsonType":"string"} },
 "required":["_id","name"],
 "additionalProperties":false
} } } );

A validation passes because a key additionalProperties is associated with false and a key "_id" included in every BSON document is on required list

The following BSON documents in a collection "empty"

db.empty.insert({"_id":"HP666"});

db.empty.insert({"_id":"HP667","name":"Harry Potter"});

db.empty.insert({"_id":"HP668","name":"Harry Potter","occupation":"wizard"});

db.createCollection( "empty",{"validator":{$jsonSchema:

{"bsonType":"object",

"properties":{"_id":{"bsonType":"string"} },

"required":["_id"],

"additionalProperties":true} } } );

In a JSON schema above a key "_id" is included into "required" list and additional properties are allowed.

The first document passes through the validation because it consists of one key:value pair {"_id":"HP666"}.

The second and the third document pass through the validation because both have "_id"key and all additional properties like "name" and "occupation"are allowed.

Validation of flat BSON documents

The following BSON documents in a collection empty


db.empty.insert({"_id":"HP666"});
db.empty.insert({"_id":"HP667","name":"Harry Potter"});
db.empty.insert({"_id":"HP668","name":"Harry Potter","occupation":"wizard"});

validate well against the following JSON schema


db.createCollection( "empty",
                    {"validator":{$jsonSchema:
{"bsonType":"object",
 "properties":{"_id":{"bsonType":"string"} },
 "required":["_id"],
 "additionalProperties":true
} } } );

A validation passes because a key additionalProperties is associated with true and a key "_id" included in every BSON document is on required list

The following BSON document in a collection "flat"

db.flat.insert({"_id":"HP666","name":"Harry Potter","age":NumberInt("100")});

validate well against the following JSON schema

db.createCollection( "flat",

{"validator":{$jsonSchema:

{"bsonType":"object",

"properties":{"name":{"bsonType":"string","maxLength":100},

"age":{"bsonType":"int","maximum":200,"exclusiveMaximum":true} },

"required":["name","age","_id"],

"additionalProperties":false

} } } );

In a JSON schema above a key "_id" is included into "required" list and additional properties are not allowed.

A default type of a value associated with a key "_id" is string. the values associated with the keys "name" and "age" are consistent with the types determined in JSON schema. No other keys are included in the document.

Validation of flat BSON documents

The following BSON document in a collection flat


db.flat.insert({"_id":"HP666","name":"Harry Potter","age":NumberInt("100")});

passes validation against the following JSON schema


db.createCollection( "flat",
                    {"validator":{$jsonSchema:
{"bsonType":"object",
 "properties":{"name":{"bsonType":"string",
		       "maxLength":100},
	       "age":{"bsonType":"int",
		      "maximum":200,
		      "exclusiveMaximum":true} },
 "required":["name","age","_id"],
 "additionalProperties":false
} } } );

A default type of a value associated with a key "_id" is string

The following BSON documents in a collection "department"

db.department.insert({"_id":"Finance","budget":123});

db.department.insert({"_id":"Sales","budget":123,"fee":456});

validate well against the following JSON schema.

db.createCollection("department",

{ "validator":{$jsonSchema:

{"bsonType":"object",

"properties":{"_id":{"bsonType":"string"},

"budget":{"bsonType":"double",

"description":"Budget of department"},

"fee":{"bsonType":"double",

"description":"Fee paid"}},

"required":["_id","budget"],

"additionalProperties":false

} } } );

In a JSON schema above a key "_id" is included into "required" list and additional properties are not allowed. Atype of value associated with "_id" key is consistent in both documents with a type determined in JSON schema.

Apart from "_id" and "budget" no other keys are compulsory. A key "fee" is not in required list and it is why it can be missing from the first document. No other keys are allowed.

Validation of nested BSON documents

The following BSON documents in a collection department


db.department.insert({"_id":"Finance","budget":123});
db.department.insert({"_id":"Sales","budget":123,"fee":456});

pass validation against the following JSON schema


db.createCollection("department",
                    { "validator":{$jsonSchema:
  {"bsonType":"object",
   "properties":{"_id":{"bsonType":"string"},
		 "budget":{"bsonType":"double",
			   "description":"Budget of department"},
                 "fee":{"bsonType":"double",
			"description":"Fee paid"}
	        },
   "required":["_id","budget"],
   "additionalProperties":false
  } } } );

Validations

Outline

Overview
JSON Schema validations
$jsonSchema operator
Validation of any BSON documents
Validation of flat BSON documents
Validation of nested BSON documents

Now we look at validation of nested BSON documents. The following BSON document in a collection nested

db.nested.insert({"_id":"HP667","name":"Harry Potter",

"address":{"city":"Dapto","code":NumberInt("2530") } });

validates well against the following JSON schema.

db.createCollection( "nested",

{"validator":{$jsonSchema:

{"bsonType":"object",

"properties":{"_id":{"bsonType":"string"},"name":{"bsonType":"string","maxLength":100},

"properties":{"city":{"bsonType":"string",

"minLength":5,"maxLength":30},"code":{"bsonType":"int","maximum":9999"exclusiveMaximum":false} },

"required":["city","code"],

"additionalProperties":false } },

"required":["name","address","_id"],

"additionalProperties":false } } } );

The document is nested because a value associated with a key "address" is a BSON subdocument (without identifier associated with "_id"). A JSON schema identifies three properties at the topmost level of the document: "_id", "name" and "address". A fact the "address" must be associated with a subdocument and JSON schema determines it in a line

"address":{"bsonType":"object",

Later on an object associated with a key "address" must be described by the compulsory keys "city" and "code". The types of values that can be associated with the keys "city" and "code" is determined in the same way as the types of values associated with the keys at the topmost level of the document. The same keys "required" and "additionalProperties" may apply fro a description of a nested component of a document.

Validation of nested BSON documents

The following BSON document in a collection nested


db.nested.insert({"_id":"HP667",
	 	  "name":"Harry Potter",
                  "address":{"city":"Dapto",
			     "code":NumberInt("2530") }
	         });

passes validation against the following JSON schema


db.createCollection( "nested",
                    {"validator":{$jsonSchema:
{"bsonType":"object",
 "properties":{"_id":{"bsonType":"string"},
	       "name":{"bsonType":"string",
		       "maxLength":100},
	       "address":{"bsonType":"object",
		          "properties":{"city":{"bsonType":"string",
						"minLength":5,"maxLength":30},
					"code":{"bsonType":"int",
						"maximum":9999,
						"exclusiveMaximum":false} },
                          "required":["city","code"],
                          "additionalProperties":false } }, 
 "required":["name","address","_id"],
 "additionalProperties":false } } } );

In the example on the present slide we look at nesting implemented through an array associated with a key.

The following BSON document in a collection nested

db.nested.insert({"_id":"HP667","name":"Harry Potter",

"cars":["Ferrari","Mercedes","Hyundai"]});

validates well against the following JSON schema

db.createCollection( "nested",

{"validator":{$jsonSchema:

{"bsonType":"object",

"properties":{"_id":{"bsonType":"string"},

"name":{"bsonType":"string", "maxLength":100},

"cars":{"bsonType":"array","items":{"bsonType":"string"},"uniqueItems":true }},

"required":["name","cars","_id"],

"additionalProperties":false

} } } );

Note a value of type array ["Ferrari","Mercedes","Hyundai"] associated with a key "cars". To validate the vales associated with a key "cars" JSON schema provides the following description

"cars":{"bsonType":"array","items":{"bsonType":"string"},"uniqueItems":true }},

A type of values that can be associated with a key "cars" is "array" and the type of "items" included in an array is "string". A key "uniqueItems" associated with a value true means that array must contain only unique items, i.e. no string in an array can be repeated one or more times.

Validation of nested BSON documents

The following BSON document in a collection nested


db.nested.insert({"_id":"HP667",
		  "name":"Harry Potter",
                  "cars":["Ferrari","Mercedes","Hyundai"]
	         });

passes validation against the following JSON schema


db.createCollection( "nested",
                    {"validator":{$jsonSchema:
{"bsonType":"object",
 "properties":{"_id":{"bsonType":"string"},
	       "name":{"bsonType":"string",
		       "maxLength":100},
	       "cars":{"bsonType":"array",
		       "items":{"bsonType":"string"},
		       "uniqueItems":true }
	      }, 
 "required":["name","cars","_id"],
 "additionalProperties":false
} } } );

The example on the present slide is almost identical to an example from the previous slide. It is again validation of a nested BSON document where nesting is implemented as an array of values.

Please note a key "description" in JSON schema below. It is a comment that can be added to a JSON schema to provide more semantics.

The following BSON document in a collection month

db.month.insert( {"_id":"Winter", "months":["June","July","August"]} );

validates well against the following JSON schema

db.createCollection( "month",

{"validator":{$jsonSchema:

{"bsonType":"object",

"properties":{"_id":{"bsonType":"string"},

"months":{"bsonType":"array",

"description":"Winter",

"items":{"bsonType":"string"} } },

"required":["_id","months"],

"additionalProperties":false

} } } );

Validation of nested BSON documents

The following BSON document in a collection month


db.month.insert( {"_id":"Winter", 
                  "months":["June","July","August"]} );

passes validation against the following JSON schema


db.createCollection( "month",
                    {"validator":{$jsonSchema:
{"bsonType":"object",
 "properties":{"_id":{"bsonType":"string"},
	       "months":{"bsonType":"array",
                         "description":"Winter",
                         "items":{"bsonType":"string"} } },
 "required":["_id","months"],
 "additionalProperties":false
} } } );

The example in the present slide BSON document is nested with an array of subdocuments each one representing a single address.

The following BSON document in a collection "addressBook"

db.month.insert( {"_id":"addressBook",

"address":[{"city":"Sydney",

"street":"25 Victoria St."},

{"city":"Wollongong",

"street":"125 Northfields Ave."}] } );

validates well against the following JSON schema

db.createCollection("department",

{"validator":{$jsonSchema:

{"bsonType":"object",

"properties":{"_id":{"bsonType":"string"},

"description":"Cities and streets",

"items":{"bsonType":"object",

"properties":{"city":{"bsonType":"string"},

"street":{"bsonType":"string"} },

"required":["city","street"],

"additionalProperties":false} } },

"required":["_id","address"],

"additionalProperties":false

} } } );

A type of value associated with a key "address" is an array of BSON sub-documents. A document give in the present slide assigns an array of 3 BSON subdocuments with a key "address". Then the items inside an array are of type "object".

Validation of nested BSON documents

The following BSON document in a collection addressBook


db.month.insert( {"_id":"addressBook", 
	          "address":[{"city":"Sydney",
                              "street":"25 Victoria St."},
                             {"city":"Wollongong",
                              "street":"125 Northfields Ave."}] } );

passes validation against the following JSON schema


db.createCollection("department",
   	            {"validator":{$jsonSchema:
{"bsonType":"object",
 "properties":{"_id":{"bsonType":"string"},
	       "address":{"bsonType":"array",
                          "description":"Cities and streets",
                          "items":{"bsonType":"object",
                                   "properties":{"city":{"bsonType":"string"},
                                                 "street":{"bsonType":"string"} },
                                   "required":["city","street"],
				   "additionalProperties":false
				  }
			 }
	      },
 "required":["_id","address"],
 "additionalProperties":false 
} } } );

Validation of nested BSON documents

The following BSON documents in a collection school

db.school.insert(
{ "name":"School of Astronomy",
    "code":"SOA",
    "total staff number":25,
    "budget":10000,
    "address":{"street":"Franz Josef Str",
               "bldg":4,
               "city":"Vienna",
               "country":"Austria"},
   "courses":[ {"code":"SOA101",
                "title":"Astronomy for Kids",
                "credits":3},
               {"code":"SOA201",
                "title":"Black Holes",
                "credits":6},
               {"code":"SOA301",
                "title":"Dark Matter",
                "credits":12}
             ]
  } );

validates well against the following JSON schema ...

Validation of nested BSON documents

db.createCollection("school", { "validator": { "$jsonSchema": {
"bsonType":"object",
"required":["name","code","total staff number","budget","address","courses"],
"properties":{ "name":{ "bsonType":"string",
                        "description":"Name of a department" },
               "code":{ "bsonType":"string",
                        "description":"Code of a department" },
               "total staff number":{ "bsonType":"double",
                                      "description" :"Total staff number" },
               "budget":{"bsonType":"double",
                         "description":"Budget of a department" },
               "address":{ "bsonType":"object",
                           "required":["street","bldg","city","country"],
                           "properties":{ "street":{"bsonType":"string",
                                                    "description": "Street name" },
                                          "bldg":{"bsonType":"double",
                                                  "description":"Building number" },
                                          "city":{"bsonType":"string",
                                                  "description":"City name" },
                                          "country":{"bsonType":"string",
                                                     "description":"Country name" } } },
                "courses":{ "bsonType":"array",
                            "items":{ "bsonType":"object",
                                      "required":["code","title","credits"],
                                      "properties":{"code":{"bsonType":"string",
                                                            "description":"Subject code"},
                                                    "title":{"bsonType":"string",
                                                             "description":"Subject title"},
                                                    "credits":{"bsonType":"double",
                                                               "description":"Total credit points"}}}}}}}});

Validations

Outline

Overview

Validations

Outline

JSON Schema validation

Validations

Outline

$jsonSchema operator

Validations

Outline

Validation of any BSON documents

Validations

Outline

Validation of flat BSON documents

Validation of flat BSON documents

Validation of flat BSON documents

Validation of flat BSON documents

Validation of nested BSON documents

Validations

Outline

Validation of nested BSON documents

Validation of nested BSON documents

Validation of nested BSON documents

Validation of nested BSON documents

Validation of nested BSON documents

Validation of nested BSON documents

References