#777

[tribbloid]

experimental prototype for 2-stage encoder derivation for record-like objects:

• generic
• typedRow

[pomadchin]

Could you add more description and re-open PR as a draft?
+ that's way to much of the fmt changes; hard to review // I know that its WIP but just making sure we're on the same page.

[tribbloid]

@pomadchin yep, easy, just submitted the minimalistic version and converted to a draft

In this PR, the new TypedRow[T <: HList] uses the intermediate Record type G in RecordEncoder as compile-time schema. This schema representation is as informative as product type but more flexible.

It has its own encoder which share most of the implementation & implicit evidences with the old RecordEncoder, This allows a TypedDataset[TypedRow[_]] to be converted from/to RDD[TypedRow[_] and a Spark DataFrame at will.

The new test case under RecordEncoderTests demonstrated its usage

[tribbloid]

In the future we can declare TypedDataFrame an alias of TypedDataset[TypedRow[_]], and make the behaviour of functions like withColumn more consistent with their vanilla counterparts

[tribbloid]

@pomadchin can the example be reproduced in your environment? Do you anticipate any difficulty in implementing this roadmap?