According to other sources on the web I would rewrite your second address definition: mySchema = “”” { “name”: “person”, “type”: “record”, “fields”: [ {“name”: “firstname”, “type”: “string”}, {“name”: “lastname”, “type”: “string”}, { “name”: “address”, “type”: { “type” : “record”, “name” : “AddressUSRecord”, “fields” : [ {“name”: “streetaddress”, “type”: “string”}, {“name”: “city”, “type”: “string”} ] … Read more
Needs to add confluent repositories in pom.xml Please add below lines in the pom.xml <repositories> <repository> <id>confluent</id> <url>https://packages.confluent.io/maven/</url> </repository> </repositories>
You need to use array type for creating the list. Following is the updated schema that handles your usecase. { “name”: “Parent”, “type”:”record”, “fields”:[ { “name”:”children”, “type”:{ “type”: “array”, “items”:{ “name”:”Child”, “type”:”record”, “fields”:[ {“name”:”name”, “type”:”string”} ] } } } ] }
Protocol Buffers is a very mature framework, having been first introduced nearly 15 years ago at Google. It’s certainly not dead: Nearly every service inside Google uses it. But after so much usage, there probably isn’t much that needs to change at this point. In fact, they did a major release (3.0) this year, but … Read more
Schema evolution can be (very) expensive. In order to figure out schema, you basically have to read all of your parquet files and reconcile/merge their schemas during reading time which can be expensive depending on how many files or/and how many columns in there in the dataset. Thus, since Spark 1.5, they switched off schema … Read more
The default value of a union corresponds to the first schema of the union (Source). Your union is defined as [“long”, “null”] therefor the default value must be a long number. null is not a long number that is why you are getting an error. If you still want to define null as a default … Read more
Avro is a Row based format. If you want to retrieve the data as a whole you can use Avro Parquet is a Column based format. If your data consists of a lot of columns but you are interested in a subset of columns then you can use Parquet HBase is useful when frequent updating … Read more
ASN.1 is an ISO/ISE standard. It has a very readable source language and a variety of back-ends, both binary and human-readable. Being an international standard (and an old one at that!) the source language is a bit kitchen-sinkish (in about the same way that the Atlantic Ocean is a bit wet) but it is extremely … Read more
I think the main difference I can describe relates to record oriented vs. column oriented formats. Record oriented formats are what we’re all used to — text files, delimited formats like CSV, TSV. AVRO is slightly cooler than those because it can change schema over time, e.g. adding or removing columns from a record. Other … Read more