Causerie

How to Bake a Deep Network

Illustrated by Fauzy Lukman

Like many people from my generation, I remember returning to our small apartment to my mother slaving away over a hot GPU. If you're like me, you probably get nostalgic over the smell of hot PCB and the ping of notifications as Mum excitedly chatted to her friends over IRC.

I'm going to teach you a great recipe for a deep network, just like my mother used to make. You'll be a hit in user groups, at science fairs, and at potlucks if you whip out one of these classics.

A lot of people these days get their models premade from cloud or SaaS and that's OK (hey, we all get busy sometimes), but you can make a classic model at home, and isn't that more satisfying?
You will need:

  • 50,000 data examples (with labels)
  • 1 GPU, ~2 Terra-flops
  • 1 deep learning framework

The key to a great neural network is good ingredients. That's why I always use high-quality, hand-labeled data. Call me a traditionalist, but as my mother would say: "A great model needs great inputs! Simple as that!"

Start by splitting your data into validation, training, and test. A lot of people ask me how to split it up. Traditionally it depends on the school and region you're from, but go with a 60/20/20 split and you should be fine!

Now normalize your data. This is a small step but a lot of people forget this nowadays. Just like salting an eggplant, you want to get that nasty, high-variance bitterness out and ensure a nice, smooth zero mean. Mm, mm, that's deep learning!

Layers, oh boy. Back when I was in university there was one kind of layer we all used and it was fine. Nowadays, kids have their Inceptions, ResNet, SqueezeNet, YOLOs...it's hard to keep up! I like a classic multilayer perceptron (MLP) myself, nothing more fancy than matrix multiplication and sigmoids. Stack five to seven layers with sigmoid activation, but feel free to use ReLu if you want something a bit more spicy.

Bake for 20–40 epochs at a high learning rate, then reduce learning rate and let the model simmer for another 20–40 epochs. Checking validation performance regularly. Add regularization to taste.

Now invite four friends over and build a web interface or API, and you're halfway to being a startup. I leave you, dear reader, with some parting advice from my mother's wisdom:

"Son, wash behind your ears and always make off-site backups."

Bonne epoch!

Andy Kitchen

author

Andy is a startup CTO, loves all things AI, Machine Learning and Computer Science. Never wants to stop learning and building things.

Fauzy Lukman

illustrator

Fauzy Lukman is a multidisciplinary designer based in Indonesia who work on graphic, branding, illustration and also UI for Web & Mobile app. Check his portfolio on Behance and dribbble.