The problem with null
In programming languages that have the concept of null
, any value in a system can be (for example) a number or null
; a string or null
; an object or null
.
Null
(or nil
in Ruby, undefined
in Javascript, etc.) presents a problem for us when working with untyped sources of data. If we’re not careful null
values can spread throughout our software, causing errors in remote code locations that expect non-null values.
Let’s examine how null values can enter a system and what to do about it. In the following code snippet, we will try to fetch the “Q Score” of a coffee from an array of coffees:
coffees = [ { name: 'Tanzania AAA', roast_date: '2016-09-01', flavour: 'very nice', metadata: { altitude: 1600, q_score: 83.1 } } ] coffees[0][:metadatum][:q_score] # => NoMethodError: undefined method `[]' for nil:NilClass # ~> NoMethodError # ~> undefined method `[]' for nil:NilClassg # ~> # ~> /var/folders/v5/nl_spw2j3qj0_scbfy6gv_sm0000gn/T/seeing_is_believing_temp_dir20160913-70485-je99qk/program.rb:13:in `'
That didn’t work because we misspelled the :metadata
key.
One way to avoid this sort of exception is to explicitly check for keys in the hash, in a defensive style of coding:
coffees = [ { name: 'Tanzania AAA', roast_date: '2016-09-01', flavour: 'very nice', metadata: { altitude: 1600, q_score: 83.1 } } ] if coffees[0].key?(:metadata) metadatum = coffees[0][:metadata] if metadatum.key?(:q_score) metadatum[:p_score] end end # => nil
But that’s a lot of code, suffers from quite a bit of duplication and doesn’t really solve the problem (what if we accidentally typo the key when we access the element, but type it correctly when writing the conditional? As we did above
with :q_score
/:p_score
).
Ruby has a Hash#fetch
method which is useful in these situations:
coffees = [ { name: 'Tanzania AAA', roast_date: '2016-09-01', flavour: 'very nice', metadata: { altitude: 1600, q_score: 83.2 } } ] coffees[0].fetch(:metadatum).fetch(:q_score) # => KeyError: key not found: :metadatum # ~> KeyError # ~> key not found: :metadatum # ~> # ~> /var/folders/v5/nl_spw2j3qj0_scbfy6gv_sm0000gn/T/seeing_is_believing_temp_dir20160913-70032-15zv3x0/program.rb:13:in `fetch' # ~> /var/folders/v5/nl_spw2j3qj0_scbfy6gv_sm0000gn/T/seeing_is_believing_temp_dir20160913-70032-15zv3x0/program.rb:13:in `'
The error we get from that code tells us exactly what’s wrong.
Tolerating nulls
But Ruby also has Array#dig
, Hash#dig
and Struct#dig
, which we could use like this:
coffees = [ { name: 'Tanzania AAA', roast_date: '2016-09-01', flavour: 'very nice', metadata: { altitude: 1600, q_score: 83.2 } } ] coffees.dig(1, :metadatum, :q_score) # => nil
We didn’t get an error for that, just nil
. Depending on the circumstances that might be ok. We also can’t tell whether the value at [0][:metadatum][:q_score]
equals nil
or whether one of those steps failed.
Ruby’s dig
method (introduced in Ruby 2.3) appears to be so named because you are digging for data with uncertainty as to whether it’s all there. For quickly prototyping something, this is no doubt appropriate, but in a production system we’d like more visibility into errors so we can fix our code or investigate API changes.
This leaves us with 2 choices – fetch
and dig
– fetch
gives us specific errors when keys are missing from Hashes, but is rather verbose when reaching deep into data structures (not uncommon when working with 3rd party data); dig
allows us to express the problem quite simply, with a sequence of keys or indexes, but gives us no visibility of missing keys or indexes.
Nulls all the way down
The examples so far have been short so the error has been easy to spot, we get nil
, when we expected a value. The problem of null
causes havoc far beyond the code that introduces the null
. To see how, let’s examine a larger code sample:
all_coffees = [ { name: 'Tanzania AAA', roast_date: '2016-09-01', flavour: 'very nice', metadata: { altitude: 1600, q_score: 83.2 } }, { name: 'Brazil AAA', roast_date: '2016-09-02', flavour: 'chocolate and broken dreams', metadata: { altitude: 1500, q_score: 88.1 } }, { name: 'Decaf whatever', roast_date: '2016-09-03', flavour: 'despair and cherries', metadata: { altitude: 1450, q_score: 72.1 } } ] def coffees_roasted_at(coffees, date) coffees.select { |coffee| coffee[:roasted_at_date] == date } end def lowest_altitude_coffee(coffees) coffees.sort_by { |coffee| coffee[:metadata][:altitude] }.first end lowest_altitude_coffee( coffees_roasted_at(all_coffees, '2016-09-02') )[:name] # => NoMethodError: undefined method `[]' for nil:NilClass # ~> NoMethodError # ~> undefined method `[]' for nil:NilClass # ~> # ~> /var/folders/3l/1lt4f3dx4658n0qw4lpx_6t00000gn/T/seeing_is_believing_temp_dir20160913-59940-dtfhjy/program.rb:39:in `'
The last line of that program has raised a NoMethodError
on nil (we’re calling [:name]
on nil), but where did that nil come from? Now we have to hunt up the stack of data and method calls to find what introduced a null
value. Replacing any call to Hash#[]
with Hash#fetch
will solve most of these issues and result in stacktraces that point to the introduction of the null
value.
Null objects
What happens when you call a method on nil
? In Ruby, you get a NoMethodError
, which you would expect (nil
is an instance of NilClass
and that likely does not implement the method you want). One way to deal with the problem of nulls, instead of sprinkling null checks throughout your code, is to use the Null Object Pattern1. The fundamental idea behind the Null Object pattern is to create Null classes that implement the same interface as your non-null objects do. An example of that would be:
Coffee = Struct.new(:name, :roast_date, :flavour, :metadata) do def self.from_hash(attrs:) coffee = new coffee.members.each do |key| coffee[key] = attrs.fetch(key) end coffee end def altitude metadata.fetch(:altitude) end def q_score metadata.fetch(:q_score) end end class NullCoffee attr_reader :name, :roast_date, :flavour, :altitude, :q_score end def coffee_attrs [ { name: 'Tanzania AAA', roast_date: '2016-09-01', flavour: 'very nice', metadata: { altitude: 1600, q_score: 83.2 } }, { name: 'Brazil AAA', roast_date: '2016-09-02', flavour: 'chocolate and broken dreams', metadata: { altitude: 1500, q_score: 88.1 } }, nil ] end coffee_attrs.map(&:class) # => [Hash, Hash, NilClass] def coffees coffee_attrs.map { |coffee_attributes| if coffee_attributes Coffee.from_hash(attrs: coffee_attributes) else NullCoffee.new end } end coffees.map(&:class) # => [Coffee, Coffee, NullCoffee] def earliest_roast_date coffees.select(&:roast_date).sort_by(&:roast_date).first end earliest_roast_date # => # # name="Tanzania AAA", # roast_date="2016-09-01", # flavour="very nice", # metadata={:altitude=>1600, :q_score=>83.2}> def names_match?(pattern:) coffees.select { |coffee| coffee.name =~ pattern } end names_match?(pattern: /AAA/) # => [# # name="Tanzania AAA", # roast_date="2016-09-01", # flavour="very nice", # metadata={:altitude=>1600, :q_score=>83.2}>, # # # name="Brazil AAA", # roast_date="2016-09-02", # flavour="chocolate and broken dreams", # metadata={:altitude=>1500, :q_score=>88.1}>]
In the above example, using NullCoffee
objects instead of nil
itself allows us to write code that works using the basic principle of Duck Typing2.
coffees.select(&:roast_date).sort_by(&:roast_date).first
could be read as “select all the coffees that have a roast date, sort them and take the first one”. There’s no mention there of nil
values or (crucially for Duck Typing) the type/class of each coffee (Coffee
or NullCoffee
). We haven’t banished null
entirely though as NullCoffee.new.roast_date
returns nil
(that’s what the call to select
relies on to filter for coffees with valid roasting dates), we’ve just prevented calling our object’s methods on a nil
value.
Banishing null using a type system
In statically typed programming languages, types can be used to distinguish between what would be null
and what would be a value. Rust is a programming language that uses types to avoid nulls and it has two main optional types: Option
and Result
.
An Option
type in Rust can be either Some(value)
or None
(the closest thing Rust has to a null
value). Let’s try this type out:
fn operation_that_could_fail1(_in: String) -> Option<String> { Some("that worked".to_string()) } fn operation_that_could_fail2(_in: String) -> Option<String> { None } fn main() { let result = operation_that_could_fail1("input".to_string()) .and_then(|val| operation_that_could_fail2(val)); if let Some(value) = result { println!("We got '{}' back", value); } else { println!("Nothing came back, oh dear."); } }
The idea behind an optional type is that it can either contain a value or can be None
. This isn’t conceptually all that different from null
at first glance, until you realise that code such as let a: i32 = 10
does not allow that variable to ever hold a null
-like value – it can only ever be a 32-bit-wide integer, anything else is a compile-time error.
If the if let Some(value) = result
code above seemed a bit verbose, then Rust’s Option
implementation includes a method unwrap
for situations where you might want to quickly write some code and write more defensive error checking later:
fn main() { let first: Option<&str> = Some("ada"); println!("first name is {}", first.unwrap()); let last: Option<&str> = None; println!("name is {}", last.unwrap()); println!("This line will never be reached"); }
The preceding example will panic on the line that does last.unwrap()
because last
is a None
value (the line that prints the first name will work without issue however). This means you can write quite simple code without checking for None
values, which will crash when something goes wrong.
What about default values? Option
provides a method called unwrap_or
which allows you to provide a default value in case the value is None
:
fn main() { let name: Option<&str> = None; println!("Name is: {}", name.unwrap_or("Unknown")); }
The preceding snippet will fail to find a name, but the default value will be returned instead, it will print out “Name is: Unknown”.
Silent failures
Fundamentally, all this treatment of nulls comes down to how we deal with failures. In certain circumstances, a null
value doesn’t indicate a failure and sometimes it does. Picking the right tools and code patterns to match can be quite challenging. Rust deals with this situation by providing the Result
type. A Result
is either Ok(T)
or Err(Err)
– this is very similar to Option
types except that errors have data associated with them. These error types allow the cause of an error to be provided to the function’s caller. However, the same tricks can be performed on Result
types, including chaining a pipeline of functions together so that data gets passed from function to function (short-circuiting out upon the first error):
fn main() { let numbers = "4".parse::<u32>().and_then(|num| { "hello".parse::<u32>().map(|another| num + another) }); match numbers { Ok(n) => { println!("That went well, the result is: {}", n); } Err(e) => { println!("Oh dear! Something bad happened: {:?}", e); } } }
In the above example, when the string "hello"
fails to parse as a number, the entire expression returns an Err
value, short-circuiting the call to map
which would otherwise add the 2 numbers together. This example is quite small and contrived, but hopefully the pattern is clear. The way Option
and Result
types can be chained together is quite similar, but if you need to know the cause of a failure (such as a number failing to parse), then Result
is useful.
Conclusion
The situation with null
is a tricky one. For many programming languages, both static and dynamic, any variable can be null
and they require extra care when writing code. For those statically typed languages that don’t provide null
values, we are always encouraged by the compiler to handle all eventualities. The end result of strict and explicit treatment of None
or Error
values is that we never see simple programming errors related to unexpected null
in production code.
A tool that prevents whole classes of errors that can occur in production is something we should all be seriously considering and weighing up with other factors around their use. This is why the Elm3 programming language makes such a big deal of the fact that code written in Elm never throws exceptions in production code. Compared to the runtime experience with Javascript, that’s a very compelling advantage.
The story on the backend is not as straightforward – I’ve illustrated the advantages to static languages without null
using Rust and although there are other options here (Haskell, Swift, Scala, etc.) for one reason or another these languages are not in widespread use yet (for backend services). Up until recently, if you wanted a good static programming language for your backend service, your choices were limited and involved seemingly overwhelmingly academic concerns. For better or worse, choosing a programming language for a system often has less to do with the semantics of the language itself and more to do with the tooling, developer experience (a highly subjective criteria) and community (involving everything from the available libraries and frameworks, to the inclusiveness and friendliness of prominent programmers and their approach to engaging newcomers). For all of those criteria, I see Rust as an excellent future option for teams that value stability, speed and excellent tooling. Currently, the ecosystem is young and there isn’t a clear set of mature libraries that teams could use to build backend services in the same fashion as they can currently with Ruby or Javascript, but it is definitely a technology to keep an eye on.
Header image courtesy of #WOCinTech Chat (CC BY 2.0)
Footnotes:
Null Object Pattern: https://en.wikipedia.org/wiki/Null_Object_pattern
Duck Typing: https://en.wikipedia.org/wiki/Duck_typing
Elm: http://elm-lang.org
`nil` and `undefined` are different. In Ruby, you have a concept of `nil` but you also have a concept of un`defined?`
`nil` has a place in languages that embrace it and just like any other tool you must use it properly.
You can use the similar `optional` pattern in Ruby: `if coffee_metadata = coffees.first&.fetch(:metadata)`
However, you are right that it’s not enforced like in other languages… but of course there are plenty of other ways to shoot yourself in the foot 🙂
LikeLike