// +++ // title = "Why null sucks, even if it's checked" // date = 2021-08-20 // updated = 2022-07-08 // description = "We all know that `null` is a \"billion-dollar mistake\", that it creates a lot of easy ways to make terrible mistakes. But it's only so bad when it's not checked by anyone and the compiler doesn't force you to check it, right? Well, the title might be a spoiler, but let's find out..." // [taxonomies] // tags = ["kotlin", "csharp", "java", "scala", "haskell", "rust", "go", "swift"] // +++ #import "../shared.typ": blog_post, callout #blog_post( "01-why-null-sucks", "Why null sucks, even if it's checked", datetime(year: 2021, month: 08, day: 20), )[ We all know that `null` is a #link("https://en.wikipedia.org/wiki/Null_pointer#History")["billion-dollar mistake"], that it creates a lot of easy ways to make terrible mistakes. But it's only so bad when it's not checked by anyone and the compiler doesn't force you to check it, right? Well, the title might be a spoiler, but let's find out... // = Context / What are you talking about? In this article, I'm specifically talking about the #link("https://kotlinlang.org/docs/null-safety.html")[Kotlin] and #link("https://docs.microsoft.com/en-us/dotnet/csharp/whats-new/csharp-8#nullable-reference-types")[C\# post v8] approach to `null`. I'm more familiar with Kotlin than C\#, so I'll mostly be talking about it, but their approaches are similar, so that doesn't really matter. Basically, you can't assign null to just any reference type in Kotlin (and C\# post v8 with certain settings). For example, this is a compilation error: ```kotlin // compilation error val a: String = null; // ok var b: String = "not null"; // compilation error (again) b = null; ``` And since you can't assign `null` to non-nullable types, `null` can't screw you! You can't get a null pointer (reference) exception, etc. Cool! If you really want to assign null to something, you need to explicitly mark the type of the variable as nullable with `T?` syntax: ```kotlin // ok val a: String? = null; // also ok var b: String? = "not null"; b = null; ``` But when you have a `T?` (nullable) type, you also need to explicitly check for null. E.g. given a class `Mine` with a `cat` field and a variable `mine` of type `Mine?`, you can't access `mine.cat`, it would be a compilation error: ```kotlin val mine: Mine? = /* ... */ val your: Mine = /* ... */ // compilation error val a: Cat = mine.cat // elvis operator, if `mine` is null, // then `your` will be used val b: Cat = (mine ?: your).cat // safe call operator, if `mine` is null, // then so be `c` val c: Cat? = mine?.cat // “trust me, I'm an engineer” operator, // throws an exception if `mine` is null val d: Cat = mine!!.cat if mine != null { // Special Kotlin trick: // in this block `mine` actually has type `Mine`. // perfectly fine (we've checked) val x: Cat = mine.cat } ``` (The same applies for calling a function with an argument of non-nullable type, you need to either check that something is not null or explicitly assume so) The "special kotlin trick" is called "smart cast". Basically, if you check that a variable is not null (or has a particular type), then the compiler changes type of the variable in the block, where the check holds. You can read more about it in Kotlin docs: #link("https://kotlinlang.org/docs/typecasts.html#smart-casts")[typecasts], #link("https://kotlinlang.org/docs/null-safety.html#checking-for-null-in-conditions")[null-safety]. And it's all great, don't get me wrong! This behaviour is a lot better than the plain old "every reference can be null, just crash if it is and it'll be fine". However... == Why it's still meh In my opinion, this comes down to 2 things: generality and extensibility. == Generality In modern languages, we have ways to abstract over types. Usually, the mechanism to do so is called generics. For example, a `HashMap` doesn't care what the key and value types are, it only cares that the key is comparable and hashable. Thus you can have `HashMap`, `HashMap`, etc while only writing `HashMap` once. That's all good and all, but what's the problem? You may ask. Well... Consider this example: the `HashMap` has a method `get` which accepts key and returns the value associated with the key if there is one. But what should it return if there is no value associated with the key? It seems reasonable to return `null` since there is no value. So something like this: ```kotlin class HashMap { // If `key` is present in the hash map, // returns the value associated with it. // Otherwise returns `null`. fun get(key: K): V? { /* ... */ } } ``` But what happens if we use `HashMap<_, T?>`? Well, then `V` is `T?` and so `V?` is `T??` which is the same as `T?` because there is only one `null` value. The crux of the problem is this: you can't distinguish between "absence of value" (no key in the map) and "presence of absence of value" (value is `null`). And if you need to distinguish them, then you need to call `containsKey` or something which is suboptimal. This is just a simple example, but in general, you can't use `null` in generic code because `null` is too special and *unique*. == Extensibility `null` is not extensible. This mechanism is only useful when you have an optional value: either there is a value, or there isn't. It isn't useful when you need to express errors (`null` is used for this anyway, but this is problematic), exclusive or (either you have `A` or `B`, but not both and not neither), etc. It may seem like this isn’t a problem --- we were talking about expressing optional values from the very beginning. But bear with me, and for now just understand that `null` is not extensible and there may be other, more extensible, mechanisms. = The saviour? As I said before, there may be a more general and extensible way to handle optional values. And there is! It has a lot of different names --- sum types, enumerations, tagged unions, discriminated unions, etc. They’re all conceptually similar, but I like the "sum types" name the most, so that’s what I’ll call it. In any case, the concept stays the same --- a type that can represent one out of multiple choices. Here are some well-known examples of definitions of sum types: ```rust enum Option { None, Some(T) } enum Either { Left(L), Right(R) } ``` ```haskell data Maybe a = Nothing | Just a data Either a b = Left a | Right b ``` #callout[ // FIXME: the link is invisible :lolsob: I've shown examples in Rust and Haskell because I'm familiar with them but note that sum types exist in #link("https://en.wikipedia.org/wiki/Algebraic_data_type#Programming_languages_with_algebraic_data_types")[many languages]. ] `Option` (`Maybe`) is exactly what we're looking for! It can be either `None` (`Nothing`) or `Some(value)` (`Just value`). So either there is nothing or there is some value, exactly the same as with `null` so far. But it actually solves all the `null` problems I've mentioned! #callout[ Another nice fact about `Option` (`Maybe`) is that it can be defined in a library. Unlike nullable types, there's nothing special about this type. The compiler doesn't need to know about it, it's just a type. ] == Generality `Option>` is a meaningful type, unlike `T??`. In the same example with `HashMap::get` there isn't any problems if it returns `Option`. ```rust impl HashMap { // If `key` is present in the hash map, // returns the value associated with it. // Otherwise returns `None`. fn get(&self, key: &K) -> Option { /* ... */ } } ``` If the key isn't present in the `HashMap>`, then `None` is returned. If the key is present but the associated value is `None`, `Some(None)` is returned. Otherwise, if the key is present and the value isn't `None`, then `Some(Some(value))` is returned. The use of sum types gives us 3 distinct kinds of values that can be distinguished from one another, instead of an ambiguous null. == Extensibility Sum types can be used for optional values via `Option`-like types. But they are not limited to only this. You can define your own sum types. It's very handy when you need to return errors (See Rust #link("https://doc.rust-lang.org/std/result/index.html")[`Result`] for example), define the errors themselves or just in general when you need to hold different kinds (types) of data in one place. == Explicitness There is one thing you may prefer about `null` over sum types: `T` can be #link("https://en.wikipedia.org/wiki/Type_conversion")[coerced] to `T?`. That means that you don't need to explicitly wrap values in `Some` (`Just`). This may be handy, because you need to type ~6 characters less if you want to pass `T` to a function that accepts `T?`. However, I don't think that wrapping inconvenience outweighs the benefits of sum types. ```kotlin fun f(x: Int?): Int? = x?.let { it + 1 } val res = f(1) ``` ```rust fn f(x: Option) -> Option { x.map(|it| it + 1) } let res = f(Some(1)); // ^^^^^ ^ ``` ~~It would be interesting to see a language with sum types and `T` to `Option` coercion though 👀~~ (see note about Swift down below) = Niche optimization #callout[ I could only find mentions of this optimization in Rust, but I think it's very neat anyway, so I'll talk about Rust in this paragraph. ] There are some types that have unused space in their memory representation --- a *niche*. For example, `bool` is only 1 bit of information, but it uses 1 byte of space, so 7 bits are unused. References (`&T`, `&mut T`) and `NonNull` can never be null, meaning that they have an unused bit pattern --- all zeroes. The same goes with #link("https://doc.rust-lang.org/std/num/index.html")[`NonZero*`]. Enumerations, just as `bool`, can have unused bits/bit patterns. What if we could use this for something actually useful? Well, Rust can. If a sum type has one of its variants being a type with a niche, then it can use it to represent other variants, instead of using a tag. I.e. `Option<&T>` isn't identical to `struct { value: union { Some(&T), None }, tag: u8 }` (pseudo syntax), but instead it's just `union { Some(&T), None }` where the `None` is encoded as `0` (since reference can never be `0` you can distinguish variants without a `tag`). Moreover, this exact optimization of `Option<&T>` is even #link("https://doc.rust-lang.org/std/option/index.html#representation")[guaranteed], so it is fully layout compatible with `*const T`. This allows using it in C-ffi, making `None` on the Rust side a `null` on the C side. This optimization can be observed using `mem::size_of` function: ```rust // `u8`, `bool` and `Option` all occupy 1 byte. // `Option::::None` is encoded as 2. It's fine, since `bool` // can only ever by 1 or 0. size_of::() = 1 size_of::() = 1 size_of::>() = 1 unsafe { transmute::, u8>(None) } = 2 unsafe { transmute::, u8>(Some(true)) } = 1 unsafe { transmute::, u8>(Some(false)) } = 0 // `Option>` behaves similarly, it just needs 2 patterns. size_of::>>() = 1 unsafe { transmute::>, u8>(None) } = 3 unsafe { transmute::>, u8>(Some(None)) } = 2 // On 64 bit machines references take 8 bytes. // `Option::<&_>::None` is encoded as a `0`/`null`. size_of::<&u8>() = 8 size_of::>() = 8 unsafe { transmute::, *mut u8>(None) } = 0x0000000000000000 // `u8` has no niche, so `Option` needs a tag (in this case: first byte) size_of::>() = 2 unsafe { transmute::, Pair>>(None) } = (0, MaybeUninit) unsafe { transmute::, [u8; 2]>(Some(0xAA)) } = [1, 170] // You may have expected 9, instead of 16, // but in Rust, size is always a multiple of the alignment. // References are 8-bytes aligned. The nearest // multiple of 8 bigger than 8 is 16. size_of::>>() = 16 // It seems like the first word is tag and the second // is optional reference. unsafe { transmute::>, Pair>>(None) } = (0, MaybeUninit) unsafe { transmute::>, [usize; 2]>(Some(None)) } = [1, 0] unsafe { transmute::>, [usize; 2]>(Some(Some(&1))) } = [1, 93965956011124] ``` _Edited output of the #link("https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=0166b25cd98164e32879456d65a8e9b8")[test program]._ = A sad note: Kotlin Kotlin *does* support sum types via #link("https://kotlinlang.org/docs/sealed-classes.html")[sealed classes]. Here is an example how you could define and use `Option` (#link("https://pl.kotl.in/XZ1fBjith")[play]): ```kotlin sealed class Option { class None() : Option() {} class Some(val value: T) : Option() {} } fun main() { val a: Option = Option.Some(12); val b = when(a) { is Option.None -> "None" // `a` is smart-casted to `Some`, so `value` is accessible. // IMO pattern matching would be better, but that's ok. is Option.Some -> a.value.toString() } println(b) } ``` #callout[ In my opinion, Kotlin's support of sum types is extremely hacky, but who am I to say that, right? ] Nevertheless, Kotlin doesn't use this for optional values! In fact, it doesn't even have an `Option` class in the standard library. For me, it seems like a big omission. It seems like making `T?` equivalent to `Option` and `Option` (where `N` is not `Option`) be layout compatible with `Java`'s `N` (i.e. `Java`'s `null` being the same as `None`) would be sufficient... = A sad note: Java & Scala Originally I didn't plan to say anything about Java (or Scala) --- after all it is a classic example of why *unchecked* `null` is problematic. Everyone (?) already knows that it's hard to always remember to check for `null` by hand and that it causes `NullPointerException`s all of the time. So it felt like brining up Java here wouldn't be valuable. But! Recently I was reminded that Java has an interesting unsoundness in its type system that is related to `null`. In a nutshell Java (and similarly Scala) assumes that if a value of a type exists, then this type is well-formed. Well-formedness means that the type is sensible, that all its type parameters satisfy their bounds. When coupled with other features like wildcards in Java or path-dependant types in Scala, this allows you to create a type that proves a relationship between two types. I.e. that one type is a subtype of another type. You can then create a function that accepts such a proof of type relationship and coerce one type to another. This is all sensible, because normally to create a value of a type you need to prove that it is well-formed. However there is a flaw in this assumption: `null` (unsurprisingly, given the theme of this post). You see, `null` makes all reference types inhabited, even the ones that are not well-formed. As shown in the #link("https://raw.githubusercontent.com/namin/unsound/master/doc/unsound-oopsla16.pdf")[Java and Scala’s Type Systems are Unsound] paper `null` allows you to falsify the proof of type relationship, allowing to coerce any two types to each other. #callout[ The paper itself is a great read, I recommend reading it, if you haven't already! ] This is another place where implicitness of `null` caused issues, quoting the paper: #quote(attribution: [#link( "https://raw.githubusercontent.com/namin/unsound/master/doc/unsound-oopsla16.pdf", )[Java and Scala’s Type Systems are Unsound]])[ It [implicit-null feature] adds a case that is easy to forget and difficult to keep track of and reason about. Interestingly, here it causes the same problem for the same reasons, but at the type level. The reasoning for wildcards and path-dependent types would be perfectly valid if not for implicit `null` values ] = A sad note: C\# C\# doesn't support sum types. To some extent they can be simulated using inheritance from an interface or abstract class, however, such an approach lacks one of the greatest benefits of sum types, namely the exhaustiveness check. It's especially sad since F\# (which is running on the same VM) supports sum types (F\# docs call them #link("https://docs.microsoft.com/en-us/dotnet/fsharp/language-reference/discriminated-unions")[*Discriminated Unions*]), but C\# doesn't. #{ set raw(lang: "F#") // workaround parsing bug; ```F# doesn't include # in the lang ``` type Option<'a> = None | Some of 'a type Either<'a, 'b> = Left of 'a | Right of 'b ``` } = A sad note: Go Go doesn't support sum types. It's a little sad on its own, but Go also doesn't have exceptions and all reference types are implicitly nullable. This means that if a function wants to return an error, it needs to return a tuple of success and error values. This not only makes checking which is `null` (actually `nil`, but it’s the same thing) pretty annoying, but also leaves the possibility for an invalid state where neither success nor error values are `null`. I think it's inexcusable to have such error-prone design flaws for a language created in 2012. = A happy note: Swift I'm not very familiar with Swift, however I often hear that it has some interesting/ergonomic/nice features. This time is not an exception --- I was told that swift has both sum types and `T` -> `Option` coercion. From what I've gathered from #link("https://developer.apple.com/documentation/swift/optional")[here] and #link("https://www.programiz.com/swift-programming/optionals")[here] in Swift the type is called `Optional`. It has two variants (or cases as they are called in Swift) --- `some(Wrapped)` and `none`. `switch` can be used to pattern match. ```swift let a: Optional = Optional.none let b: Optional = Optional.some(1) switch b { case let Optional.some(x): // ^^^ --- let is needed to introduce new variables (`x`). // honestly, this is quite a good choice. // it allows to use `Optional.some(pat)` // to compare the wrapped value to `pat` variable. print(x) case Optional.none: print(":(") } ``` And then on top of that Swift adds a whole bunch of syntax sugar 1. `T?` can be used instead of `Optional` 2. `T` can be coerced to `T?` (and coercions are transitive it seems, so `T` -> `T???` is also supported) 3. `nil` can be used instead of `Optional.none` (there is a coercion from `nil` literal to `T?`) 4. `.variant` can be used if the type can be inferred (e.g. `.none` and `.some(16)`) 5. `a?.b` can be used to do chaining / map access (`a.map(|x| x.b)` in Rust terms) 6. `a ?? b` can be used to provide a default 7. `if let x = a` assigns `x` to the inner value if `a` is `some` (syntax sugar for `if case let Optional.some(x) = a {`) 8. `guard let x = a else { return }` can be used to early-return if a value is not `some` 9. `a!` can be used to unwrap a variable crashing if it's `none` 10. and more (probably (I'm a little tired of writing this list)) _Personally_, I'd say that's a bit too much sugar and I'm a little concerned how coercions interact with generic code. But overall I think that's a pretty nice feature set and I guess it's nice to use all of this. I wish more languages went in direction "make `Optional` nicer" instead of "make `null`" nicer (yes, I'm still looking at you, Kotlin). #callout[ Also Swift gets it right and calls the `T? -> (T -> U?) -> U?` function `flatMap`. Yes, I'm looking at you, Rust. ] = Conclusion There is no nice conclusion. The reality is painful as usual. We are stuck with a 56-year old abstraction that has proven itself to be error-prone for a while now. We are stuck with it, even though a better solution exists for longer than I have. bye. ]