What can you learn from a side project ⁉️ LZ78, emoji and google cloud.
A 6 minutes story written on Dec 2017 by Adrian B.G.
One of the most recommended technique to improve your software development skills is the to “do a side project”, but not everyone will tell you exactly what and why you actually can learn from one. I decided to share my experience in building a pet project and its results.
I wrote this article for junior/mid-level developers to demonstrate the benefits of a side-project (pet project).
The trick is to find a proper balance between the known/unknown things you want to build. I mean you can create a side-project with 100% new technologies (for you) but it could be frustrating to make it work and you may quit (EX choosing a different project paradigm, language, IDE, platform etc).
I suggest doing a pet project with a 50⁄50 ratio of things you master and things you want to learn.
️Once upon a time (last week), I had an idea, to modify an encoding algorithm to use emoji. I think this came after I wrote about encoding and held a meetup with my students about encoding in general and protocols.
The next step was to choose the technologies, this was the easy part, I just had to read my to-learn list (it is quite large). I am currently learning Golang (a “cloud” programing language), and I also needed some practice writing unit tests. so this covered the main technologies. The libraries also required a working demo, so the next items from the to-learn list were AppEngine and Jekyll, you can read more details in the next section.
The 3rd step was to seek collaborators, so I involved 1 (out of 2) junior web developers I (try to) mentor, it was a good opportunity to show them some workflow and coding techniques.
The project took around … 30h or so, but the goal wasn’t the end project rather the journey. Another 10h or so were used for the library demo project, which consists of a website with a backend, custom domain, and hosting. The end result can be seen here:
Unicode is hard ✍🏽
One of the first walls I hit was to deal with Unicode. I haven’t worked with it at a byte level since … forever or never? A nice advantage was that Go literals are UTF-8, so it allowed a more “friendly” code to be written. What have I learned?
- more about UTF-8 and Unicode
- that characters don’t exist anymore
- without the proper libs, dealing with Unicode is painful
- that Unicode has a normalizedform
- to appreciate developers that already implement Unicode (like IDE’s)
- to appreciate newer programming languages
Before choosing which compression algorithm to port, I had to do some research. I quickly decided that a dictionary-based algorithm is more suited to this problem and narrowed down the list. After a few other tens of Wikipedia pages, I decided to choose the LZ78 algorithm. Learnings so far:
- there are lossy and lossless compressions
- many dictionary-based algorithms are used in web development
- some methods focus on the speed of the compress, decompress or the size of the archive, each used to solve different problems
- smart devs from Facebook, Google and Microsoft created their own algorithms
- Snappy is used at Google, Hadoop, MongoDB, Cassandra …
- to appreciate simpler algorithms
Google cloud was on my to-learn list for a while. My only hands-on experiences with it were ~7yrs ago when it launched the PHP support and 1 month ago, at the Google Cloud OnBoard conference. I wanted AppEngine in particular because AWS doesn’t have a similar service (it is somewhere between the EC2 container and Lambda services). What I have learned:
- GCloud is … different than AWS in many ways
- the Standard environment is very limited
- it is fast and easy to work with (1 deploy command __ after the initial setup)
- you get many features for free, that in a normal (non-cloud) env you would have to do yourself (rollback, multiple version traffic split, auto-scaling with or without limits, monitoring, tasks, etc …)
- I liked the admin UI & docs way more than the AWS ones, they are more user-friendly, it’s a subjective statement ofc
- “serverless” is perfect for prototypes, small/medium projects, very small teams, limited scope projects.
- it’s mostly free for small projects (even after the 1y trial)
- it was a free and good opportunity the check this hosting, as a prospect for a future more larger side-project
I have also come across many horror-user-stories about Google cloud, most of them involving huge invoices, but I think most of the problems can be avoided if you just use the price calculator (their staff uses it too) and do not forget that Google is a profit based business.
Static website 👹
For the demo example I made an interactive website. Jekyll was on my to-learn list since I used Hugo and Github Pages was a no-brainer choice. What have I learned:
- Github is awesome, free hosting with a custom domain is a bliss
- liquidis not HTML
- that I have to invest more time if I want to deliver a static website as a commercial project, for example I haven’t managed to make the config variables functional, in HTML or JS. Most probably I would search a different generator, with some more advanced features.
- if webpack or create-react-app has a too big overhead for a project (like a landing page), most probably you will want a Static Website Generator like Hugo or Jekyll.
Vanilla JS6 💀
- jQuery era is long gone, in a web component project I cannot find its purpose
- the new HTML APIs are quiteinteresting
- it is good that I have already switched to JS6(with a different, larger side project)
Visual Studio Code 🎶
Side projects are a good opportunity to learn a new IDE or new features of the one you are already using. I have been using VSCode for a while now, but with this project I encountered a lot of pesky small issues, only a few of which I remember:
- nothing works if your go files cannot compile
- Liquid (HTML) support is … bare minimal (no auto format)
- I wouldn’t use it for a medium/large project (>50 source code files), it’s simply too unreliable (eg: the autocomplete stopped working many times) compared to a full IDE (I have an IntelliJ license)
Nevertheless, it will remain my favorite multi-language text editor.
Unit testing 💉
Unfortunately, my unit testing skills are lacking, and side-projects are a perfect way to improve any skill. We wrote some tests for the library and the demo web server. Good practice makes it better!
Resources I have used 📚
- Strings, bytes, runes and characters in Go - The Go Blog
- Text normalization in Go - The Go Blog
- The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and…
- LZ77 lecture: https://www.cs.helsinki.fi/u/tpkarkka/opetus/12k/dct/lecture07.pdf
- Data compression - Wikipedia
- Data Compression - LZ-78
- 😃 Every Emoji by Codepoint
- Code point - Wikipedia
- Dictionary coder - Wikipedia
- The App Engine Standard Environment | App Engine Documentation | Google Cloud Platform
- Always Free Usage Limits | Google Cloud Platform Free Tier | Google Cloud Platform
- AWS vs. Azure vs. Google Cloud: Which free tier is best?
I curate a list of articles, talks and papers for one/two times per month. They are mostly related to computer science, distributed systems, databases, Go, containers and Cloud solutions.