Note: This top comment is heavily edited. It was kept up with the current state until this issue closed.
Motivation
Flatbuffers should allow users to choose to use optional types. There has been some interest in distinguishing between default values (which are not stored in the binary) and some notion of None which the user controls.
Here are some links to previous interest in this idea:
- #333
- #3777
- https://groups.google.com/forum/#!topic/flatbuffers/1hrNtBI0BQI
- https://github.com/google/flatbuffers/issues/5875#issuecomment-619737916
Currently, a user can control a field's presence in the binary by specifying "force_defaults" and checking "IsFieldPresent" which is a bit of a hack. This proposal should define proper Flatbuffers optional types, which should be a better way of doing this. Use of this feature is only advisable for new fields, since changing default values is in general backwards-incompatible.
How do we represent this in the schema file?
We will specify it like so
table Monster { mana: int = null; }
This visually implies that optional types are at odds with default values and is "consistent" since the value to the right of the equals sign is what we interpret non-presence to mean.
field_decl = ident : type [ = (scalar | null
) ] metadata ;
~We can add a field tag, e.g. "optional" or "no_default", that triggers this behavior. Hopefully no one is using those tags. Maybe we can make it specifiable to flatc, an "--optional-field-keyword-tag" flag, just in case people are using it and can't stop.~
How do we represent this in the binary?
We are going with option (A).
(A) Non-Presence means None
Instead of omitting zero-like values, the generated code must store them. Non-presence for optional fields no longer means "whatever the default is," now it means None. You can interpret it as "the default value is None". This also means we cannot specify both specify a non-null default and mark the field as optional.
Pros:
- This seems more intuitive.
- It aligns with the "force_defaults" + "IsFieldPresent" hacky manual approximation of this feature.
- If Nones are more common than zero-likes then this will have smaller binaries.
Cons:
- @aardappel thinks this is harder to implement
"making presence an indicator would require we pass this special field status down to the field construction code to override the current val == default check, which means slowdown, codegen and runtime changes in all languages.. whereas my "least likely to be used default" trick requires no changes"
~(B) Some Sentinel value means None~
In this scenario, zero-like values are still not stored. Instead we choose some "sentinel" value which we interpret to be None (e.g. int
can use int_min
and float
can use some kind of Nan
).
Pros:
- @aardappel thinks this is easier to implement
"it requires the schema parser to set default values for you, and no changes anywhere else"
- If zero-likes are more common than None then this will have smaller binaries
Cons:
- Someone might want to use the sentinel value (Hyrum's law).
- This can be mitigated by publishing the sentinels and letting users decide whether they need the sentinels.
- This probably won't work for fields representing raw bits.
How do we represent this in every language API?
We'll need to change the type signature of all generated code (building/reading/mutating/object/etc) around the optional type to signal its optional-ness. I think we should use the language's local standard for optional types. Suggestions:
- Python:
Optional[T]
.
- Rust:
Option<T>
.
- C++17 has
std::optional<T>
but its not obvious what to use for earlier versions. T*
would work.
- Java:
Optional
shows up in Java 1.8 and triggers autoboxing, so idk :/
The exact generated-API for a language should be discussed in the PR implementing this feature in that language.
Out of scope
(I'll add links if you make issues for these feature requests)
- Syntactic types
- Default values for strings and tables
TODO
| task | owner | done
|---|---|---|
| Change flatc to support schemas with optional types and cause an error if they're used in unsupported languages | @CasperN | #6026 ✅
| Implement optional type API in C++ | @vglavnyy | #6155 ✅
| Implement optional type API in Java | @paulovap | #6212 ✅
| Implement optional type API in Rust | @CasperN | #6034 ✅
| Implement optional type API in Swift | @mustiikhalil | #6038 ✅
| Implement optional type API in lobster | @aardappel | ✅
| Implement optional type API in Kotlin | @paulovap | #6115 ✅
| Implement optional type API in Python | @rw?
| Implement optional type API in Go | @rw?
| Implement optional type API in C | @mikkelfj | ✅
| Implement optional type API in C# | @dbaileychess | #6217 ✅
| Implement optional type API in Typescript/javacscript | @krojew | #6215 ✅
| Php, Dart, etc ... | ?
| Update documentation to advertise this feature | @cneo | #6270 ✅
[edits]
- added todo list
- added points from the discussion to each section.
- added out of scope section
- Decision: go with (A) and use
= null
syntax in schema file (cross out alternatives)
- Updated TODO list, finished parser, Rust in progress
- Change to schema grammar, link to swift PR, Note at top
- Added more languages to the TODO
- Lobster support 🦞
- Kotlin and C support
- Java, C#, TS/JS support and docs, issue closed, no longer editing.