Evolving schemas Suppose intially you designed an schema like this for your Employee class { {“name”: “emp_name”, “type”:”string”}, {“name”:”dob”, “type”:”string”}, {“name”:”age”, “type”:”int”} } Later you realized that age is redundant and removed it from the schema. { {“name”: “emp_name”, “type”:”string”}, {“name”:”dob”, “type”:”string”} } What about the records that were serialized and stored before this schema … Read more
I found a better way to solve this problem. Looking at the Schema generation source in Avro, I figured out that internally the class generation logic uses Velocity schemas to generate the classes. I modified the record.vm template to also implement my specific interface. There is a way to specify the location of velocity directory … Read more
It took me a couple months of reading source code and testing things out. In a nutshell, Spark can only handle String and Binary serialization. You must manually deserialize the data. In spark, create the confluent rest service object to get the schema. Convert the schema string in the response object into an Avro schema … Read more
According to the explanation by Doug Cutting, Avro’s JSON encoding requires that non-null union values be tagged with their intended type. This is because unions like [“bytes”,”string”] and [“int”,”long”] are ambiguous in JSON, the first are both encoded as JSON strings, while the second are both encoded as JSON numbers. http://avro.apache.org/docs/current/spec.html#json_encoding Thus your record must … Read more
Take a look at the Java reflection API. Getting a schema looks like: Schema schema = ReflectData.get().getSchema(T); See the example from Doug on another question for a working example. Credits of this answer belong to Sean Busby.
Fields cannot be renamed in BACKWARD compatibility mode. As a workaround you can change the compatibility rules for the schema registry. According to the docs: The schema registry server can enforce certain compatibility rules when new schemas are registered in a subject. Currently, we support the following compatibility rules. Backward compatibility (default): A new schema … Read more
I think the main difference I can describe relates to record oriented vs. column oriented formats. Record oriented formats are what we’re all used to — text files, delimited formats like CSV, TSV. AVRO is slightly cooler than those because it can change schema over time, e.g. adding or removing columns from a record. Other … Read more
If you want all you string fields be instances of java.lang.String then you only have to configure the compiler: java -jar /path/to/avro-tools-1.7.7.jar compile -string schema or if you are using the Maven plugin <plugin> <groupId>org.apache.avro</groupId> <artifactId>avro-maven-plugin</artifactId> <version>1.7.7</version> <configuration> <stringType>String</stringType> </configuration> […] </plugin> If you want one specific field to be of type java.lang.String then… you … Read more
Yes, it’s possible. I’ve done that in my java project by defining common schema files in avro-maven-plugin Example: search_result.avro: { “namespace”: “com.myorg.other”, “type”: “record”, “name”: “SearchResult”, “fields”: [ {“name”: “type”, “type”: “SearchResultType”}, {“name”: “keyWord”, “type”: “string”}, {“name”: “searchEngine”, “type”: “string”}, {“name”: “position”, “type”: “int”}, {“name”: “userAction”, “type”: “UserAction”} ] } search_suggest.avro: { “namespace”: “com.myorg.other”, “type”: … Read more
you can define the default attribute as undefined example. so the field can be skipped. { “name”: “first_name”, “type”: “string”, “default”: “undefined” }, Also all field are manadatory in avro. if you want it to be optional, then union its type with null. example: { “name”: “username”, “type”: [ “null”, “string” ], “default”: null },