khalilstemmler.com RSS Feed

The Power of Composition

Tue, 25 Nov 2025 15:04:10 GMT

Quick question.

How often do you see this type of code?

If the answer is "a lot", that's okay - we've all been there.

Especially in the Code-First Phase of Craftship.

(Learn more about the 5 Phases in this free guide here)

However, if you're seeing this a lot, it's a good sign that you may be missing Composition, which is one of 12 Essentials that -- when balanced properly, tends to lead to testable, flexible, maintainable code...

... and when out of balance, tends to lead to significant issues that are REALLY hard to fix later on down the road.

In my estimation, nearly every problem you face in your codebase can be traced back down to at least one of the 12 Essentials.

This took me a long time to figure out, so I'll do my best to explain in a short email.

What is composition?

Composition is simply:

How your application “comes together.” How objects get created. How dependencies get wired. How environments boot.

You use composition every time you boot your app:

Boot in dev mode → you need composition

Boot in test mode → you need composition

Boot integration tests → you need composition

Boot in production → you need composition

Boot a CLI tool? → you need composition

Boot a worker? → you need composition

Composition is the grand configurer-er. The mechanism everything else sits on.

You know those little Russian Dolls? (they're called Matryoshka dolls).

Well, that's kinda how I see composition. Your application is composed from parts and wholes, coming together to fulfill a larger responsibility.

One environment wrapping the next, cleanly composed, cleanly booted.

It's quite beautiful, really.

And if structured correctly, you can design your application to spin up the right way, with the right parts, for the right context, in a simple way, from a single location (typically called the Composition Root).

This is immensely powerful, and also immensely debilitating if you don't have a composition root.

What most devs do

Most developers at the Code-First phase aren't yet aware of how important this essential is downstream, so they miss it.

Instead, you'll end up seeing things like:

imports scattered all over the place
mutating global state
using environment variable checks
hacking dependencies in place
coupling everything in a single controller or Express.js route

And yeah — it works…

until the moment you try to test it.

Or deploy it.

Or run it in a different environment.

Or swap a dependency.

Or add a new feature.

Then it turns into pain.

Ask me how I know 😅

This is one of the root causes behind fragile systems.

It's also at the root of why you can't write tests.

I honestly think that

Developers who say they “hate testing” don’t actually hate testing. They hate the mess they have to fight through just to test anything. But it comes from a lack of Composition and Horizontal Decoupling (another essential).

The benefits of composition

Clean composition gives you:

real testability
clean dependency inversion
multiple composable boot modes (dev, prod, ci, xyz)
more confidence when refactoring
the heart of a stable architecture
the ability to scale your codebase in a modular way (just add new modules)

For example, one of my favourite things is the ability to just boot my application the exact same way every time, simply specifying the context, and allowing the application to wire itself up properly behind the scenes.

Like this.

How to fix it

It takes a few course modules to fully explain how to fix this without breaking everything, but in summary, the answer is to actually start using Abstraction properly.

A good place to start?

Start encapsulating stuff.

This is a big topic to explain, and there are a lot of smaller topics underneath it (see the below diagram)

But you can begin to untangle this problem by implementing a Layered Architecture, slowly, and being more intentional about the entry points to your various execution environments.

My recommendation is to implement a single bootstrap location, like a "poor man's version of composition".

You'll notice that we lean heavily on Dependency Inversion / Injection for this.

From here, we can begin to branch into more sophisticated forms of Composition.

But in my opinion, if you're at the Code-First level, this is a great start, in order to start to create some semblance of structure.

FAQ

“What if I’m not doing composition?”

You are. Everyone is to some degree. You can’t not compose your app.

The real question is:

Are you doing it intentionally or accidentally?

To what degree of consciousness are you composing your application?

“How important is this?”

It’s one of the 12 Essentials. Miss it and everything downstream gets harder. Just like all of the other ones.

“What are the levels?”

All Essentials have a scale of mastery from 1 to 10.

Sometimes the problem calls for a simpler solution, sometimes a more robust one. Discernment and wisdom to know the difference comes from awareness and practice.

Level 1: Everything in one file (Express controller, Next.js route, etc). It works — but costs you flexibility.

Level 10: Full-blown DI container, composable boot modes, multi-env testing layers, modules, compositional roots within modules, extremely modular.

Final notes

A student in the community built a great DI container. He's been using it in production with his team for over a year without any issues. Check it out 👉 https://github.com/webiny/di
Take a look at this solution that one of my mentorship students put together demonstrating how to use composition to compose your backend for testing and running production. https://github.com/trikitirem/ddd-forum-be. I especially like how he uses abstraction to abstract the composition root based on the context we run the application. Really, really nice implementation.

Thanks for reading,

Khalil

As a reminder,

You can get The Software Essentialist @ 30% off right now
Testing Mastery is 50% off until December 1st
And if you want direct help, I’ve got one more slot open for my 6-month mentorship container. 👉 Apply here

Temporal Decoupling: Why We Use Events & Messages

Thu, 14 Nov 2024 15:04:10 GMT

A few big projects happening right now in essentialist.dev-land.

November's theme has been finishing solidbook (finally), pattern-first in The Software Essentialist, and... moving!

Going through and tidying up my bookshelf, I dusted off this one. Have you read it?

Written by Gregor Hohpe and Bobby Woolf, it's a book that explains how messaging systems work, how to connect applications together, the types of problems you'll encounter, and the vast array patterns that help with solving 'em.

And seriously, there are lot of patterns.

To be completely honest, when I tried reading it about 6 or so years ago, it didn't really make much sense to me.

At that time, yes, I found this topic of messaging fascinating, especially because I was just learning how to use Domain-Driven Design and the Clean Architecture to split my application into logical subdomains (which was a big step for me at the time)..

But straight up - I was:

1) not at the level of awareness to appreciate the contents, and

2) not dealing with problems that needed me to use messaging yet

Even through I tried to read this book back then, the topic was just way too far out and away from my reality.

At that time, my world involved working in a frontend consultancy building frontend applications, still in that phase struggling to write useful tests (which you can move past pretty quickly in Testing Mastery).

Nonetheless, the seed was planted in my mind.

It would take me a number of years to run into higher quality problems, focusing more on backend, fullstack, and then architecture to realize the real burning necessity of messaging.

Temporal Decoupling

In The Software Essentialist, one of the 12 Essentials I teach is called Temporal Decoupling.

The idea is this: every feature you add, change, remove, test - they all happen after or before a policy, command, or something else happens.

Because your job is entirely about solving problems...

And the solution to problems can be described as either steps or past tense events ...

And because reality is cyclical in nature...

It's clear that it only makes sense to design in most cases to assume an architecture with the capacity to decouple features from each other using events.

It's exactly why Redux, mobx, and any 1-directional data flow system you've built on the frontend can scale so well..

And it's why CRUD often falls short in the long run, as I learned many years ago..

Why am I telling you all of this?

You want to feel confident in your skills as a developer, but maybe you're worried you're not focused on the right things.

Here's my best take.

I've said this before, and I will say it a million times...

Your job is not to write code, it is to:

help customers make money & save money
help users get back time & have their time enriched
write code in such a way that it makes it easy for you and the other developers on your team do the previous two things.

Focus on that and you'll automatically, as a result, end up mastering what matters.

What you'll notice as you continually take on challenges that drive you to learn new things, is that your perspective will continue to expand and rise.

You'll be able to both see into the details, and you'll be able to zoom out to see the system as a whole.

That's why I've modelled The Phases of Craftship process and curriculum in The Software Essentialist based on the natural process I've noticed myself and others on their journey have taken as well.

So in essence, the way to become a confident crafter is to embody a systems thinking perspective, but you have to go through certain experiences to develop it.

There's so much more I'd like to say about this, but I'll leave it here for now.

Power & love to you, And as always,

To Mastery Khalil

Tension Seeking Resolution (The Law of Polarity)

Fri, 25 Oct 2024 15:04:10 GMT

He holds out a rubber band, expands it until it snaps and says,

“Tension seeking resolution. That’s it. That’s the key. The fundamental principle.”

The instructor’s name was William Whitecloud.

I thought about this lecture for months, which was a testament to the teaching itself.

How it began

After the initial early adopter launch of The Software Essentialist, I was pushing my mind to its limits to produce the initial batch of material for around 300 new students to play with.

Yes, I wanted to do my best work, but I was not pacing myself properly.

My family could feel it.

My now ex girlfriend could feel it.

And much like the rubber band, my mind was going to snap eventually.

And it did.

It felt like I had been hit by a car.

I took a few weeks off to recover, because my body and my mind were clearly telling me to.

When I returned, I studied leadership, self development, psychology, NLP, high performance, and spirituality.

I learned more about myself during that period than any other time in my life.

At some point along my path, I came across an image I hadn’t seen in a long time.

I’m sure you know it.

The yin-yang.

Well that’s profound, I thought.

I printed it out and kept it with me, because I knew there was something aptly important here.

The law of polarity

Upon returning to recording the videos for the first version of what would today become Testing Mastery and Best Practice Phase of Craftship material, I took note that of the fact that my work occurred differently to me.

As I wrote a test, I began to notice something…

When I wrote an empty test, I could feel a sort of tension.

It was different from the psychological tension I experienced writing the code first, full well knowing that I’d be creating a temporary mess to later clean up.

Instead, this tension generated when I wrote an empty test. It seemed to create a vacuum — asking to be collapsed.

A sort of curiosity gap.

You know, because when you write an empty test… when you Program By Wishful Thinking (my favourite Best Practice-First technique), sometimes, you have no idea how you’re going to make that test pass.

And while I’ve always known that half of the work is:

upfront
top-down

And the other half, of implementing the working code is:

emergent
bottom-up

…this time, instead of just knowing these truths, I understood them.

The Metaphysics

I've observed something about reality: it's cyclical, multi-dimensional, and multi-perspectival.

This will go above 85% of people's heads, but for any sort of long-term success, we really have to dig into this one a fair amount, because as Ray Dahlio says,

Truth - more precisely, an accurate understanding of reality - is the essential foundation for producing good outcomes.

Understand: every time you ask a question, write a test, make a plan, declare an estimate, commit to a new habit, or take on a goal to improve your life… you simultaneously commit to a temporary period of struggle.

You encounter the inevitable tension seeking resolution.

What a beautiful game of chaos and order we get to play.

The art of life.

When expectation meets reality.

To all you magicians who zig the zag and zag the zig.

To Mastery, I say to you.

The way of the middle path

Imagine you're an entrepreneurial developer.

You're ready to go all in again.

So you quit your job with 12 months of savings, you identify a problem in the world that you want to solve, then you start to throw some code at it.

You’re not guarding any of it with tests, but that’s okay. You know how this game goes.

You stack more and more code.

In other words, you stack more and more tension.

Then you get your first customer.

Relief, bliss, and then more tension. Why?

Well, it’s really on now.

You keep on building.

You hear the little voice say, will this little project of mine work?

It seems like it will, because you just got another customer.

Over the following weeks and months of building and refining, you start to notice:

it’s getting harder to add features
the code is starting to feel overwhelming
you have a fear of breaking something in a distant area of the codebase by changing code in another

So what do you do?

Well, you’ve validated your project already.

So now you decide it’s time to cover that ground you’ve trodden with tests.

You know the “right thing to do” would have been to have started with tests, but hey — this wasn’t guaranteed. You had no idea if any of this was going to amount to anything.

If you were in an office and could expect a paycheque every month, you’d have probably taken a different path.

Some BDD and TDD practitioners will tell you to always start with the tests. Every scenario.

But this is a very different game you’ve chosen to play.

It requires a certain type of maturity to know how to keep the balance.

But one thing is for certain: not knowing how to write tests is a lot different from deciding not to.

It took Eckhart Tolle's The Power of Now for me to realize that reality is nothing but the present moment.

In actuality, reality is but a continuous, perfect and harmonious hum of energy - an energy which continually balances itself out.

There is no past, no future. That's all mind-generated stuff that the ego requires to maintain frames of perception and synchronize present moment behaviour.

It has also become clear to me that we achieve outcomes within the universe using cycles.

And when you write code, you start the cycle of tension (of stepping out of line with this continual stream of balanced energy), and you collapse that tension with a test.

My friend. This is the creative process of divergence and convergence.

In my eyes, until you learn how to test as well as you can code, you are continually at odds with the law of polarity.

You can sling messy code all you want, that's okay. Sometimes the situation calls for it.

But expect the eventual slingshot of entropy. Expect that your code will come back to bite you.

Because believe me, reality does not discriminate.

My advice then is to learn how to test.

Master it, then move on to the next skill to you need to collect to play the game of balance only accessible to those who know the rules.

Because nothing in life is perfect.

Seek not perfection. Seek the middle path.

Of wu wei.

Of effortless action.

And dance with tension seeking resolution.

Enjoy your week, and as always.

To Mastery

Two Categories of Tests: High Value vs. Typical Tests

Fri, 16 Aug 2024 15:04:10 GMT

Before we begin, You can join The Software Essentialist for an additional 15% off before we ship the Pattern-First phase of craftship. Use the code TESTING-MASTERY before August 19th to join us in the course & community.

By now, you’re probably aware there’s a lot of different types of tests.

Sure, we have unit, snapshot, component, so on & so forth.

But speak to anyone and you’ll hear that categorizing the types of tests you write in the field can get kinda hairy.

Some developers say “don’t worry about the types of tests, just write tests”.

Others say “write mostly integration tests” or “write e2e tests”.

Getting the definitions down is a good place to start to answer “how TDD works in the real world”.

What we'll cover

In this letter, we’re going to:

✨ Learn what the main types of tests are & why we need different ones anyway
✨ Learn why test intent matters (& why I first break tests into high value customer-oriented vs. typical developer type tests)
✨ Understand the little known high value (acceptance) tests, how they work & why they’re extremely powerful

What are the different types of test categories?

The main test types that everyone is familiar with are unit, integration, and e2e.

This is the scope at which most developers debate back and forth about tests, but I see it a bit differently.

I believe it’s not enough to look at them like this — we have unit, integration, and e2e, of course, yes — but when we change our intent behind the test, something interesting happens.

For example, we can use unit tests, which we’ll learn are tests that run really fast — and which are typically just used to test something not all that life-changing or substantial like simple methods or utility classes...

OR we can use them to test very valuable things, like features.

For this reason, I group tests based on intent.

"Are we writing tests for the customer or for the developer?"

And the answer to this tells us if our tests are typical tests or if they are high value tests.

Category #1: Typical (developer-oriented) tests

Typical tests: These are your e2e, unit & integration tests — generally, every type of test can fit into one of these 3 broad types of tests. You’re probably somewhat familiar with these in at least some capacity.

Developer-oriented tests verify technical details

What do I mean when I refer to developer-oriented tests?

Well, these are tests that are purely focused on ensuring that the internals or some technical aspect of our systems work properly.

For example, tests like these are developer-oriented tests.

// Testing a date formatting utility
const formatDate = require('./formatDate');

test('formatDate', () => {
  // Test case 1: Simple date formatting
  const testDate = new Date(2021, 2, 12);
  const expectedResult = 'March 12, 2021';
  expect(formatDate(testDate)).toBe(expectedResult);

  // Test case 2: Formatting with time
  const testDateWithTime = new Date(2021, 2, 12, 16, 30);
  const expectedResultWithTime = 'March 12, 2021, at 4:30 PM';
  expect(formatDate(testDateWithTime)).toBe(expectedResultWithTime);

  // Test case 3: Handling invalid input
  const invalidDate = 'not-a-date';
  const expectedErrorMessage = 'Invalid date provided';
  expect(() => formatDate(invalidDate))
	  .toThrowError(expectedErrorMessage);
});

// Testing that we can connect to the database*
const models = require('./models');

describe('Database connection', () => {
  it('should connect to the database', async () => {
    try {
      await models.sequelize.authenticate();
      expect(true).toBe(true);
    } catch (error) {
      expect(error).toBeNull();
    }
  });
});

// Verifying we can fetch data from an API (not a great test)
const jobsAPI = require('./jobsClient');

describe('Jobs API client', () => {
  it('fetches data from the jobs API', async () => {
    const response = await api.fetchJobs();
    expect(response.status).toBe(200);
    expect(response.data).toHaveProperty('items');
  });
});

If you guessed that:

the first one was a unit test
the second one was an integration test
and the third one was also an integration test

… then you’d be right.

And if you guessed wrong, that’s ok — we’ll learn about them next.

But most importantly, what’s the intent behind all of these tests?

Who are we really supporting by writing these tests?

Ultimately, these are tests which are meant to validate technical things — utilities, clients, classes, methods, functions, etc — and that makes them developer-oriented tests.

Coming back to Behaviour-Driven Design and the Abstraction Prism, developer-oriented tests are more-low level.

Yes, we still need them, but they’re low level.

And because they’re low level, they’re not as inherently valuable as the tests which verify valuable outcomes.

What do I mean by a valuable outcome?

That’s it. The outcome is the valuable thing — not the code. It’s the acceptance criteria. The features. The user stories. And these come from the customer.

There’s a fair amount of distance between these typical tests and these valuable abstraction layers.

What’s the closest type of test to the user stories and acceptance criteria?

Acceptance tests. In reality, customer-oriented tests are acceptance tests.

Category #2: High value (acceptance) tests

High value (acceptance) tests: These are tests which verify valuable customer/business outcomes. Commonly implemented as E2E tests, because they’re a subset of your typical tests, all typical can be written as high value acceptance tests. Done properly, a unit test can be a high value unit test. Done properly, an integration test can be a high value infrastructure test or an integration test.

Customer-oriented tests verify features/acceptance criteria

If developer-oriented tests support the developer, then acceptance tests are customer-oriented tests which support the customer.

How? By giving them what they want. The features.

For example, if the customer needed to synchronize their Notion Tasks to their Google Calendar, we could express the requirement with an acceptance test specification like so:

# useCases/syncTasksToCalendar/SyncTasksToCalendar.feature

Feature: Sync Notion tasks to Google Calendar

  Scenario: Sync new tasks
    Given there are tasks in my tasks database
    And they don't exist in my calendar
    When I sync my tasks database to my calendar
    Then I should see them in my calendar

And the test code would look something like this:

// useCases/syncTasksToCalendar/SyncTasksToCalendar.ts

import { defineFeature, loadFeature } from 'jest-cucumber';
import path from 'path'
import { SyncTasksToCalendar } from './SyncTasksToCalendar'
import { TasksDatabaseBuilder, TasksDatabaseSpy } from '../../testUtils/tasksDatabaseBuilder'
import faker from 'faker'
import { SyncServiceSpy } from '../../testUtils/syncServiceSpy'
import { ICalendarRepo } from '../../services/calendar/calendarRepo';
import { CalendarRepoBuilder } from '../../testUtils/calendarRepoBuilder'
import { FakeClock } from '../../testUtils/fakeClock'
import { DateUtil } from '../../../../shared/utils/DateUtil';

const feature = loadFeature(path.join(__dirname, './SyncTasksToCalendar.feature'));

defineFeature(feature, test => {
  letcalendarRepo: ICalendarRepo;
  lettasksDatabaseSpy: TasksDatabaseSpy;
  letsyncTasksToCalendar: SyncTasksToCalendar;
  letsyncServiceSpy: SyncServiceSpy;
  let fakeClock = new FakeClock(DateUtil.createDate(2021, 8, 11));

  beforeEach(() => {
    syncServiceSpy = new SyncServiceSpy();
  })

  test('Sync new tasks', ({ given, and, when, then }) => {

    given('there are tasks in my tasks database', () => {
      tasksDatabaseSpy = new TasksDatabaseBuilder(faker, fakeClock)
        .withAFullWeekOfTasks()
        .build();
    });

    and('they dont exist in my calendar', () => {
      calendarRepo = new CalendarRepoBuilder(faker)
        .withEmptyCalendar()
        .build();
    });

    // Act
    when('I sync my tasks database to my calendar', async () => {
      syncTasksToCalendar = new SyncTasksToCalendar(
        tasksDatabaseSpy, calendarRepo, syncServiceSpy
      )
      await syncTasksToCalendar.execute();
    });

    // Assert
    then('I should see them in my calendar', () => {
      expect(syncServiceSpy.getSyncPlan().creates.length)
        .toEqual(tasksDatabaseSpy.countTasks())
      expect(syncServiceSpy.getSyncPlan().updates.length).toEqual(0);
      expect(syncServiceSpy.getSyncPlan().deletes.length).toEqual(0);
    });
  });
});

Don’t worry about the various things in here you might not yet get like Builders, Spies and whatnot.

Just focus on the intent.

Hopefully it’s clear that if this works, it’s going to be a much, much more valuable sort of test to the customer than a test against a text util class, right?

Acceptance tests as the shared, single source of truth for what to implement on any side of the stack

“But Khalil, aren’t acceptance tests are supposed to verify the system from the perspective of the user? Does that mean we have to write them End to End?”

Not necessarily.

You can write acceptance tests at the:

e2e scope
unit testing scope
integration testing scope

AND you can use the same test and execute it on the frontend, backend, desktop, mobile, wherever.

Check out this beautiful image again. You’ll need to enlarge it to see it properly.

Allow me to explain.

First, it starts by building the acceptance testing rig — which is the common denominator.

The acceptance test rig

The acceptance test rig is a necessary component we need to write our high value tests of any sort.

Whenever we cross architectural boundaries, we need 4 layers to set up our work in cohesive way.

Those 4 layers are the following:

The Acceptance Test Layer — this is where we write our acceptance tests
The Executable Specification Layer — this is your test code (ie: jest)
The Domain Specific Language Layer — this is where you focus on expressing what, not how
The Protocol Driver Layer — this is where we express the how to implement the what (ie: when we cross architectural boundaries, we often need this final layer to translate the domain language layer instructions into HTTP calls, GraphQL calls, console instructions, or browser clicks or button presses)

These 4 layers work together to spread out the process involved in translating an English looking acceptance test into real-life interactions.

Acceptance tests are the single source of truth

Regardless of the scope, the single source of truth is the acceptance test for all scopes.

Yet again, this is the power of contracts.

If a customer asks for a new feature, well, both the frontend and the backend can treat that acceptance test as a contract and implement the desired behaviour — albeit in different ways, with different tasks, but they implement it.

Why is it so important to know how to write high value tests?

"We are what we repeatedly do. Excellence, then, is not an act, but a habit." - Aristotle

I used to pride myself in the ability to make something work once, but that doesn't impress me at all anymore.

Building products and writing code in such a way that you can continue to stack value on top of value...

Past a certain threshold, it is HARD.

And if you know my story, you know I had to learn the hard way.

As I've covered in "Why You Have Spaghetti Code", value creation is a zig-zag act of divergence-convergence.

It's a game of vector dynamics if you will.

You set a target, not entirely sure how you'll get there, and then you bridge the gap with code. That's what we're doing with tests. Your tests are the vector conditions. Your code is the bridge.

Developers that know how to do this, either at the start of a feature, or after the fact (ie: producing value for a company by cleaning up technical dept in a slow, consistent manner)...

I think these devs are those that stand the most likely chance of not only standing out to get the best jobs and opportunities, but to actually thrive in them as well.

So yeah, all the easy stuff is... easy 😂.

Anyone can write code.

But solving business problems?

That's a metaphysical game. It's a whole different level, man.

How to get started?

In my opinion, the best type of testing you should practice are the tests against features.

But you kinda need to master the basics - the physical mechanics of TDD first.

In The Software Essentialist, we go through a ton of exercises to really drill this skillset down to prepare you for the hard stuff.

So start with basic TDD exercises first.

Once you feel like you're comfortable there, move onto the more complicated high value tests that we use to refactor legacy code and contractualize the complex test states that comes with the territory.

There's so much more, but just take the first step and start figuring it out.

Because the paradox of mastering testing, is that once you master testing, you master vector dynamics.

And if you master vector dynamics, you're mastering goal achievement.

And if you master goal achievement, well then nothing can stop you, really.

Summary

In summary, there are a number of different types of tests we can use (on any side of the stack).

I generalize them as the typical tests and the High Value (Acceptance) Tests.

We write tests primarily for the customer or for the developer, and differentiate tests based on the concerns/abstraction layers they validate.

—

Power & love to you, my friend.

And as always,

To Mastery.

Using Builders to Model Complex Test States

Thu, 15 Aug 2024 15:04:10 GMT

In a recent letter, I explained that edge cases can be an absolute nightmare to deal with when testing testing features.

To deal with this, we learned that there are 2 general techniques at your disposal:

randomness (which sometimes works)
resetting and constructing the test state (which always works)

Admittedly, #1 is super easy, and that’s why it only sometimes works for simple cases.

But #2… well this is hard, but it always works.

In this letter, I’ll gift you the gift of how to use #2 — you’ll learn how to use my favourite testing pattern — the builder pattern.

But first, why exactly is it so hard to set up test states?

Why is it so hard to set up test states?

To illustrate, let’s return to the School domain.

And then let’s imagine we wanted to test the create class room feature.

Feature: Create Class Room

    As an administrator
    I want to create a class
    So that I can add students to it

    Scenario: Successfully create a class room
        Given I want to create a class room named "Math"
        When I send a request to create a class room
        Then the class room should be created successfully

Recall that it’s easy to test success scenario because to set up the test state, all we have to do is wipe out the database or use randomness to make our tests idempotent again.

For example, we don’t want to trigger the unique constraint on the name field (because you can’t have two classes named “Math”).

But what about the following scenario? How would you test this?

Feature: Create Class Room

...
        
    Scenario: Classroom already exists
        Given a classroom already exists
        When I send a request to create a class room of the same name
        Then the class room should not be created

Ah, the classroom already exists…

Well, of course, like always, you’d start by representing it in the executable specification, of course.

test("Classroom already exists", ({ given, when, then }) => {
  let classroomName = "Science";
  let requestBody: any = {
    name: classroomName
  };
  let response: any = {};

  given("a classroom already exists", () => {
    // ??
  });

  when("I send a request to create a class room of the same name", async () => {
    response = await request(app).post("/classes").send(requestBody);
  });

  then("the class room should not be created", () => {
    expect(response.status).toBe(500);
    expect(response.body.success).toBeFalsy();
    expect(response.body.error).toBe("ServerError");
  });
});

But how do you model the fact that the classroom already exists?

Do we just call the create classroom API twice?

Do you write a function to “create the classroom” beforehand?

It seems reasonable, but when you follow that line of thinking and working to extremes, it actually results in a lot of confusing code prone to duplication.

Honestly, this is something I used to find extremely difficult.

We want to know how to compose the database test state properly. I prefer to hold a root level understanding, when I can, so let me introduce you to what I believe is a foundational concept — a first principle, if you will.

It may very well have a different name, but I call it The Data Model Tree.

The Data Model Tree

In a vital lesson on composition in The Software Essentialist, I explain that your application is a web of objects (or a tree).

We see this when we link dependencies and bootstrap our applications.

But the phenomenon actually extends far beyond just how we connect classes and functions.

The Data Model Tree is the relationship between all of your data models in your database.

And they too, come together to form a tree in their sequence of creation.

You have to reconstruct the tree to test scenarios

Here’s what makes setting up your database for your tests so complex: you have to re-create the entire tree.

Wait, what?

Look: say I want to test a feature in our School domain called gradeAssignment. To write this test, all of the side-effects of grade assignment to have already happened.

To begin this process, I’d first ask: “what is the state of the data models at the point I want to grade an assignment?”

Well, in order for a teacher to grade an assignment, it would need to have first been assigned to a student, right?

And to get all the way up to this point in the sequence of creation, it’d mean:

first, the classroom and the student need to exist
then, once this is true, I need to then link the student to the classroom using a student enrolment record
then I’d create an assignment for the classroom, because it directly relied upon the existence of a classroom
from here, I’d now assign the assignment to the student, linking together the enrolledStudent and the assignment, creating a studentAssignment record
then, the student has to submit an assignment, creating an assignmentSubmission record
then, and ONLY then, can I, the teacher, grade the assignment, creating a grade record

OH MY GOD.

So your head doesn’t explode: look.

Here’s the sequence of data models required to run this test.

Sound like a lot of work?

I’m making it out to sound worse than it is.

It’s not that bad.

But you know how I talk about aligning your actions with the nature of reality?

Yeah, well, this is where data comes from.

It emerges in little chunks and pieces like this.

The plus side: This is about as hard as it gets. Everything is so much easier (especially aggregate design in domain-driven design) when you understand this and think about your data as if you’re building bits and pieces of trees with each API call and operation.

As a side note, you can join The Software Essentialist for an additional 15% off before I ship the Pattern-First phase of craftship. Use the code TESTING-MASTERY before August 19th to join us in the course & community.

You have to reconstruct the tree in reverse

Listen, I know you know how trees work.

I’ve been in leetcode hell, myself 😂

In case you forgot, we’ve got a root, leaf nodes, intermediate nodes, branches, and traversals.

“Why are you triggering my PTSD right now, Khalil. Are you that sick and twisted?”

Not intentionally..

Look. Here’s why I bring up trees:

Each test state has at least one Target Data Model in focus, and to prepare your database state for the test, you have to create the entire structure of the tree in reverse, from that Target Data Model.

Pause on that one and read it again.

It can be confusing, but it’s the key.

To illustrate, let’s see some examples.

Example #1: If I was testing “assign student to a class” (ie: to create an enrolment), the Target Data Model is “EnrolledStudent” means I first need to have:

created the student and the class

Therefore, your tree setup should look like this:

Example #2: If I were testing “createAssignment”, the Target Data Model is “Assignment”, which that means I need to have first:

created a classroom

And thus, like this:

In this case, you don’t need to create students or anything else because you don’t need to create a student to get up the tree to the classroom.

Can you see how an understanding of the data model is required to do this work?

Another example.

Example #3: If I were testing “grading an assignment, the Target Data Model is “GradedAssignment”, which means I need to have first:

created the student and the class
enrolled the student to the class
created the assignment
created a student assignment
created a student assignment submission

That’s the most complex one.

But if you get this, you are golden, my friend.

If you don’t get it yet, that’s cool. There’s a massive difference between knowledge & experience. You need practice. We do a lot of that in the course.

“That’s cool and all, Khalil. But how do you ACTUALLY set this up for a test? It seems like it’s going to be some sort of complex graph and trees traversal stuff.”

It’s actually very straightforward if you follow a few design principles and patterns.

I’ll show ya the pattern first.

Let’s see the tremendous Builder Pattern.

What are builders?

As I said, The Builder Pattern is one of my favourites.

Builders help you construct values, implement relationships, and set up your test states in an extremely readable, expressive, declarative and domain-driven way. And that’s what I’m all about — the link between TDD, DDD, great design and great DX.

So, for example, let’s say we wanted to create a student as a pre-condition for a test.

While we can design these to construct just pure objects, we’re going to need to design our builders to manipulate the database for us. Usage of such a builder that sets up the database state might look like this from the outside.

const studentBuilder = new StudentBuilder();
await studentBuilder
	.withName('Jackson')
	.withRandomEmail()
	.build();

And on the inside, it might look like the following.

class StudentBuilder {
  private props: Partial<StudentProps>;

  constructor() {
    this.props = { name: '', email: '' }
  }
    
  withName(name) {
    this.props.name = name;
		return this;
  }
  
  withRandomEmail() {
    this.props.email = faker.internet.email();
    return this;
  }
  
  async build() {
    const student = await prisma.student.create({
      data: {
        name: this.props.name as string,
        email: this.props.email as string
      },
    });

    return student;
  }
}

Pretty nifty, eh?

It’s the chaining that I like the most.

How to design builders? (practical tips)

This letter is running pretty long again, so let’s get practical.

What I’ve found is this: the best way to build builders is to follow the laws of The Data Model Tree, and to work in reverse starting from target data model, constructing the test state using the SPECIFIC keywords: build, from, and and with.

Here’s what I mean.

Demonstration

Let’s do “assign a student to an assignment”.

This is a real tricky one.

But it’s easy if we focus on one layer of abstraction at a time.

First layer? The acceptance test.

Feature: Assign an assignment to a student

    As a teacher
    I want to assign a student to an assignment
    So that the student can achieve learning objectives

    Scenario: Assign a student to an assignment
        Given there is an existing student enrolled to a class
        And an assignment exists for the class
        When I assign the student the assignment 
        Then the student should be assigned to the assignment

What needs to first exist in order to do this?

the student must first be enrolled to the class
the assignment must first exist

So we need to model those, in the executable specification.

test("Assign a student to an assignment", ({
    given,
    when,
    and,
    then,
  }) => {

  let requestBody: any = {};
  let response: any = {};

  beforeAll(async () => {
    await resetDatabase();
  });

  given("there is an existing student enrolled to a class", async () => {
    // 
  });

  and("an assignment exists for the class", async () => {
    // 
  });
  
  ...
})

And now, for each of these pre-condition clauses, the question is: “what’s the Target Data Model?”

Well let’s see. Let’s do the first one.

For the Given, it appears we’re dealing with an enrollment right? “a student is enrolled to a class”. Yep. And what’s the relationship?

an enrolment
- comes from a student
  - with a name
  - with an email
- and from a classroom
  - with a name

Makes sense, right?

How would you model that using a Builder?

Like this.

given("there is an existing student enrolled to a class", async () => {
    const enrollmentResult = await studentEnrollmentBuilder
      .fromClassroom(classroomBuilder.withClassName("Math"))
      .and(studentBuilder.withName('Johnny').withEmail('student@example.com'))
      .build();
    student = enrollmentResult.student;
  });

And putting it all together?

test("Assign a student to an assignment", ({
    given,
    when,
    and,
    then,
  }) => {

  let requestBody: any = {};
  let response: any = {};
  let student: Student;
  let assignment: Assignment;

  beforeAll(async () => {
    await resetDatabase();
  });

  given("there is an existing student enrolled to a class", async () => {
    const enrollmentResult = await studentEnrollmentBuilder
      .fromClassroom(classroomBuilder.withClassName("Math"))
      .and(studentBuilder.withName('Johnny').withEmail('johnny@example.com'))
      .build();
    student = enrollmentResult.student;
  });

  and("an assignment exists for the class", async () => {
    assignment = await assignmentBuilder
      .fromClassroom(classroomBuilder.withClassName("Math"))
      .build();
  });
  
  ...
})

Boom 💥

Notice the following:

I use the word with to express fields/values
I use the word from and and to express relationships
I use the word build to construct it all
I focus on the end result, the thing I really want to build — which is the enrolment, and I assume that it will work.

That’s the magic, my friend.

And if you like that, we can continue to improve it like so:

test("Assign a student to an assignment", ({
    given,
    when,
    and,
    then,
  }) => {

  let requestBody: any = {};
  let response: any = {};
  let student: Student;
  let assignment: Assignment;

  beforeAll(async () => {
    await resetDatabase();
  });

  given("there is an existing student enrolled to a class", async () => {
    const enrollmentResult = await anEnrolledStudent()
      .from(aClassRoom().withClassName("Math"))
      .and(aStudent().withName('Johnny').withEmail('johnny@example.com'))
      .build();
    student = enrollmentResult.student;
  });

  and("an assignment exists for the class", async () => {
    assignment = await anAssignment()
      .from(aClassRoom().withClassName("Math"))
      .build();
  });
  
  ...
})

Wonderful, wonderful stuff.

I’ll break this down more in future letters.

Happy testing!

And as always, To Mastery

Khalil

PS: As reminder, you can join us in The Software Essentialist for an additional 15% off using the code TESTING-MASTERY before August 19th.

How to Improve Legacy Code w/ Characterization Tests

Mon, 12 Aug 2024 15:04:10 GMT

Ever been so scared to change your code out of fear you’ll break something?

You’re not alone.

Been there.

Over the last months, I’ve been hella busy with The Software Essentialist and the first edition of solidbook, but I wanted to take a minute to address this common question, which is:

How do you deal with legacy code? How can you get started improving it safely?

And it’s a great question.

If you asked me way back when I first started out, I would have thought to look to Domain Driven-Design and how to refactor from unstructured to structured code.

I’d have opted for tools like Event Storming and Event Modelling to think through the design properly and re-write it.

And while that stuff is cool and essential (in a separate phase), there’s actually a more important stage first, and it’s not sexy.

You’ve got to characterize your code, mate.

Which (and I know you don't wanna hear this, because I also didn't want to back then) means learning how to write decent tests first 😂 .

Tests are the key to entropy reversal

If you’ve read the early drafts of solidbook, you’ll recall that I talk about entropy — a naturally occurring phenomenon whereby the universe has a tendency towards complexity. Chaos. Disorder.

You know, the dishes piling up in the sink.

Your plans with friends getting cancelled because someone caught the flu.

That’s life. Things get messy.

And as a wise man with a long grey beard probably once said,

“It is what it is.”

Well, over the years on my search for the answer to “how to write scalable, testable, flexible code”, I paid more attention to metaphysics and how reality tends to work.

Why?

It’s like what Ray Dahlio says in his excellent book Principles,

“Truth — more precisely, an accurate understanding of reality — is the essential foundation for producing good outcomes.”

Before I discovered my own set of principles which would eventually become The 12 Essentials that guide literally everything we do in The Software Essentialist, I learned that tests (or more generally — contracts) are the initial key to reversing entropy.

In other words, if you want to make your code better, it’s always going to start with a layer of code that documents what’s coming in the next layer of code.

And that’s a test.

Characterization tests

First of all, let’s clarify.

What’s legacy code?

I’m paraphrasing Michael Feathers when he says that “legacy code is code without tests”.

That’s a sharp distinction, but it makes sense to me.

What is a characterization test?

Simply, it’s a test that you write after the fact.

Basically, it’s a test against legacy code.

And what do we do?

We determine the characteristics of the codebase as it is, and we lay down tests that express the behaviour.

Why? It stabilizes the codebase so that you can do more invasive things moving forward.

It’s kinda like a tailor putting pins in your jacket and checking with you before cutting into the fabric.

“Let’s just make sure it’s right before we continue.”

Characterization tests merely document what the code already does.

Lemme show you how it works.

Example code

Let’s consider an example shall we?

Imagine a Classroom System domain. And imagine it was all written in a messy index.ts or something.

Here’s a submitAssignment API call.

// POST student submitted assignment
app.post("/student-assignments/submit", 
	async (req: Request, res: Response) => {
  try {
    if (isMissingKeys(req.body, ["assignmentId", "studentId"])) {
      return res.status(400).json({
        error: Errors.ValidationError,
        data: undefined,
        success: false,
      });
    }

    const { studentId, assignmentId } = req.body;

    // check if student assignment exists
    const studentAssignment = await prisma.studentAssignment
	    .findUnique({
	      where: {
	        studentId_assignmentId: {
	          assignmentId,
	          studentId,
	        },
	      },
	    });

    if (!studentAssignment) {
      return res.status(404).json({
        error: Errors.AssignmentNotFound,
        data: undefined,
        success: false,
      });
    }

    if (studentAssignment.status === "submitted") {
      return res.status(400).json({
        error: Errors.AssignmentAlreadySubmitted,
        data: undefined,
        success: false,
      });
    }

    const studentAssignmentUpdated = await prisma
	    .studentAssignment.update({
	      where: {
	        studentId_assignmentId: {
	          assignmentId,
	          studentId,
	        },
	      },
	      data: {
	        status: "submitted",
	      },
	    });

    res.status(201).json({
      error: undefined,
      data: parseForResponse(studentAssignmentUpdated),
      success: true,
    });
  } catch (error) {
    res
      .status(500)
      .json({ 
	      error: Errors.ServerError, 
	      data: undefined, 
	      success: false 
	    });
  }
});

Aside from the barebones design, there’s a lot of logic here.

Some application logic. Some business logic. Some validation, some errors & exceptions.

Lots going on here, and lots of room for things to go wrong. And they will, and without tests, it’s going to feel pretty painful to keep your code in check over time without a lot of manual testing.

Let’s not go that road.

Creating a test rig

If you’re new to high value acceptance testing, know that there are 3 types of excellent tests we can write which will give you a MASSIVE amount of confidence in your code, if done correctly:

high value e2e
high value unit
high value integration

But to write any of these, you’ll first need to set up a test rig.

What is a test rig?

I love this concept.

Admittedly, if you’re only used to writing a few Jest unit tests here and there, this might feel a bit odd to you.

But a test rig is a specific architectural component you create which is responsible for running your high value tests against either your frontend, backend, or both.

Comprised of 4 layers:

acceptance test specification layer
executable specification layer
domain specific language layer
protocol driver layer

... this is the first thing I teach to developers in the Best Practice-First phase of craftship once they’re past the basics of testing and they’re ready to get serious.

It’s a little bit different on the frontend vs. the backend, but those 4 layers are always present.

From user stories to high value e2e acceptance test

Using high value e2e tests

Once we set up this 4 tier architecture, we’ll be ready to start characterizing our code.

Now, I recommend for most scenarios, to use high value e2e or high value integration tests to test your features.

If you’re characterizing a backend, there’s little value in choosing integration over e2e.

If you’re characterizing a frontend, high value integration tests can be extremely powerful, but for you to do that, you need to know how to decouple from React/the view layer (but that’s another email).

The user story

We’ll do one for now, but imagine you had a ton of features you wanted to make sure work safely first before you change anything.

You’d start by gathering up those user stories.

You can write 'em rough. Here's what I gather from this code.

As a student
    I want to submit an assignment
    So that I can get a grade

Dope.

What’s next? The test.

Writing the acceptance test

I’m a huge fan of Gherkin, a way to express your functional using Given-When-Then in a super human-readable way.

Gherkin is great, and it’s pretty much a BDD staple for testing.

To continue, I’d express that story as a Feature using jest-cucumber, document the success case, and then document the many edge cases I think I’m noticing as well. I’ll just do one for now.

Feature: Submit an assignment

    As a student
    I want to submit an assignment
    So that I can get a grade

    Scenario: Successfully submit an assignment
        Given I was assigned to an assignment
        When I submit the assignment
        Then It should be marked as submitted

    Scenario: Fail to submit an assignment twice
        And I submitted the assignment
        When I submit the assignment again
        Then I should see an error message

Writing the executable specification

Then comes the executable specification — the actual test file itself.

Most of us are somewhat familiar with these, although maybe not using the Gherkin/acceptance test specification strategy. Here’s what that test could look like:

...

const feature = loadFeature(
  path.join(__dirname, "../features/submitAssignment.feature")
);

defineFeature(feature, (test) => {
  afterEach(async () => {
    await resetDatabase();
  });

  test("Successfully submit an assignment", ({ 
	  given, when, then 
	}) => {
    let requestBody: any = {};
    let response: any = {};
    let studentAssignment: StudentAssignment;

    given("I was assigned to an assignment", async () => {
      studentAssignment = await studentAssignment
	      .fromStudent(studentBuilder)
	      .andAssignment(assignmentBuilder)
	      .build();
    });

    when("I submit the assignment", async () => {
      requestBody = {
        studentId: studentAssignment.student.id,
        assignmentId: studentAssignment.assignment.id,
      };

      response = await request(app)
        .post(`/student-assignments/submit`)
        .send(requestBody);
    });

    then("It should be marked as submitted", async () => {
      expect(response.status).toBe(201);
      expect(response.body.data.status).toBe("submitted");
    });
  });

The domain specific language

Notice the language that I use to set up the test:

studentAssignment = await studentAssignment
  .fromStudent(studentBuilder)
  .andAssignment(assignmentBuilder)
  .build();

This stuff.

Ah, the wonderful Builder Pattern. I’ll explain more in a coming email later this week, but I love these.

Extremely powerful. Extremely declarative. Extremely versatile pattern.

The protocol driver

The protocol driver is what actually crosses the threshold, taking us from one system (our test rig) to the actual system itself (the backend, in this case).

I opted to use supertest here, but sometimes I’ll actually build my own apiClient for many, many reasons which I’ll also get to another time.

when("I submit the assignment", async () => {
  requestBody = {
    studentId: student.id,
    assignmentId: assignment.id,
  };

  response = await request(app)
    .post(`/student-assignments/submit`)
    .send(requestBody);
});

How to continue?

If you characterize your features first using the 4-tier acceptance test rig, you’ll be in great shape.

That’s the main idea.

I recommend spending some time getting this blanket of tests down first before you make any invasive changes/refactorings, because honestly — you could have some bleed-over between your Vertical Slices (your API calls).

While I could go on, this email is already pretty long.

I'll wrap it here for now.

Enjoy the rest of your week ~

As always, To Mastery

Khalil

PS: You can join The Software Essentialist for an additional 15% off before we ship the Pattern-First phase of craftship. Use the code TESTING-MASTERY before August 19th to join us in the course & community.

The Metaphysical Mental Models: Abstraction & The Feedback Loop

Tue, 26 Mar 2024 15:04:10 GMT

Recently, I've been thinking about those early days of when I first started out coding.

There were a bajillion things to learn, right?

And today, we have new tools, libraries, frameworks… and AI tech.

I remember that feeling. Like I was on a never-ending treadmill, always learning new stuff, but still writing spaghetti messes and untestable code in my codebases.

Eventually I had enough of that and went on a journey to discover how to master what matters.

Today, after all that time and what I've seen, I realize something key – simplicity is golden.

And that goes for most things in life actually.

But if I were to write a letter to myself when I was first starting out, I’d tell him this:

“Khalil. Keep it simple. Your entire job as a programmer is to perform problem decomposition. Break valuable problems down into bite-sized pieces and find solutions with code."

I’d also tell him that:

“Programming and problem decomposition are mostly about mental models. Your code is only as good as if it solves a problem or not, and if it uses language and symbols that other developers understand. To help you, there are 2 main mental models you need to master: Abstraction and The Feedback Loop. Keep learning new tools and improving your stack, yes, but primarily focus on how to use these concepts while coding. Everything will come into place if you do.”

Allow me to explain what they are.

Abstraction

How aware are you of what’s going on behind the scenes when you press “POWER” on your tv remote? Do you know how the sequences of 0’s and 1’s sent from the remote to the receiver are encoded, transmitted & decoded?

Or is it simply enough to think “power button means the tv turns on”?

Second answer, right? Yeah, second answer.

Abstraction is one of the least understood concepts… but also the most important.

It’s all about taking a big, complex problem, breaking it apart and creating new, easier to understand chunks or layers of the problem that interact with each other to solve the problem.

Do it right and our codebases are valuable, readable, understandable — glorious.

Do it incorrectly and our code starts to look like the inside of a turtle (I have no idea what that might look like, nor do I want to, but I imagine it’s not pleasant).

The Feedback Loop

I used to spend so much time changing lines of React code, refreshing the UI, re-walking through a set of steps to see if it worked and - nope, didn't work, time to try again.

These days I just write a test, the minimum amount of code I have to, and avoid all of that.

If I could get those hours back.

I'd probably have had so many more hours to go pro Junkrat in my Overwatch career back then.

There are so many more effective ways to work. In the course, we talk about something called The Ideal Developer Workflow and FA²STR but really, simplicity-wise - it comes down to introducing Feedback Loops.

For example, when we learn how to talk to customers, use things like Impact Maps, DDD techniques like Event Storming & Modelling to clarify before we build, and when we work outside-in by driving your code with tests first (yes, tests, y'all - I know... trust me though, they're not as scary as they seem)… well, then we’re using feedback loops conceptually.

How so?

We’re making a guess, taking action based on that guess, and then we check to make sure we’re OK before continuing deeper and deeper into abstraction-land.

Do this right and we build the right thing, the right way, and with useful tests that’ve got our backs (to protect against accidentally changing some code that breaks an existing feature).

So next time you're coding, if you care about long-term quality (you probably do if you're subscribed to this newsletter - you absolute class act), think about abstraction & the feedback loop.

How might you use these ideas better?

Framing your focus this way makes a huge difference.

And hey, nobody gets it perfect from the start.

Actually, I’ve learned that there are about 5 Stages of Phases of Craftship (a coaching tool I use to make sense of a developer's skill level) that we tend to go through as we improve.

This happens naturally if we're exposed to the right paradigm shifts at the right times, where each one involves better use of each of the essential mental models!

But more on that later.

Looking forward to hearing your thoughts on this.

Why You Have Spaghetti Code

Sun, 24 Mar 2024 15:04:10 GMT

Before we begin: Doors open for the latest version of The Software Essentialist on March 31st. Join the waitlist for perks, bonuses & 55% off.

If I had to call out the definitive Code-First experience, it's that feeling when you realize you're writing code that gets worse instead of better all the time. Code where no one understands it, no one wants to touch it because they're afraid they'll break something...

And as a result, you get this pretty unfulfilling working your codebase.

It feels disheartening knowing that you tried your best, yet still: spaghetti.

Been there.

I talk about this in the Phases of Craftship - that Code-First developers need to go through this experience at least once (or twice) to realize that code isn't enough. Only then when you experience this do you typically realize it's time to look for better practices.

But why does this happen in the first place?

What's at the heart of this un-maintainability problem?

I believe the answer lies in the ideas of divergence and convergence, two fundamental laws of abstraction and creation (of anything).

Divergence & convergence

To create anything in the world relies upon these two creative forces: divergence and convergence.

Divergence is about:

expansion
emergence

Where convergence is about:

contraction
contracts

For well-designed creative works, we're going to need to use a mixture of these two sorts of forces: these two forms of design (upfront design & emergent design).

Divergence involves making a mess

For example, when I'm in the initial brainstorming phase of writing newsletters and course modules, I fully expect a mess. You'll see messy whiteboards, papers, voice notes - all sorts of things. I don't give too much focus to anything other than getting the ideas out & following the trains of thought.

Or when I sit down to write a song. Sometimes I'll beatbox the drums, make noise, add parts randomly, use whatever I have available, or ramble the lyrics. I don't give too much focus to anything other that following what's interesting to me.

This is all an act of divergence. It's un-boundaried, flowing, and it can basically go forever (if you let it).

Divergence is a crucial part of any creative process, but any time we write code without boundaries, this is where the spaghetti comes from.

(Well, not where REAL spaghetti comes from 🍝 - that's from Italy, but you know what I mean).

Convergence involves using boundaries

If divergence is all about expanding and building up, the convergence is all about bringing things to a completion.

In the writing process, this is where I look at the brainstorm and select the goals, write the headers & the summary before writing the content.

In the music production process, this is where I'd select the instrumentation, take it to the studio and get notes from the producer before re-recording it.

Convergence is all about starting from the end. It's all about defining exactly what success will look like, before we even start, and then every action we take next is to bring the work to a close.

It's a focus on outcomes, not steps.

And it's something we contractualize -- typically, in programming using acceptance tests, like so:

You need both

Acceptance tests are contracts that help us converge. They give us the absolute bare minimum set of things our code has to do.

And when we configure an acceptance testing rig, we can actually balance a mixture of test-code-test-code-test-code moving inwards into the codebase, writing as minimal code as possible.

Play the zig-zag

It's incredible. Every creative process involves this zig zag between the two.

Writing a newsletter

Brainstorm (divergence)
Select goals (convergence)
Write summary (divergence)
Write the headers (convergence)
...

Making a song

Recording an idea (divergence)
Selecting the arrangement (convergence)
Recording the demo (divergence)
Producer helps suggest revisions for the next recording (convergence)
...

FAQ

What about refactoring?

Refactoring is a form of convergence. You're cutting down on what was divergently created.

But consider the problem with refactoring after having performed way too many divergence cycles in a row.

It's like asking a painter to completely change a painting to a new one.

At that point, it'd just be way easier to create a new painting.

And this is precisely what Code-First developers often feel called to do.

Ah yes, The Classic Rewrite™️.

The costly and expensive rewrites.

Better solution is to avoid this altogether, start with valuable tests and zig-zag your features to fruition.

How can I fix my spaghetti code?

OK, so what to do if you're facing spaghetti right now?

(Eat it. 🍝 Spaghetti is delicious. Ha, kidding.)

For real though, in this situation, the experts suggest you focus on SAFELY refactoring your code.

And the first step to SAFELY refactor your code is to characterize it.

What do I mean by this?

Functionally, a characterization test is the same thing as an acceptance test; meaning: it documents the feature and the acceptance criteria, but from a high level.

So after you've already made a mess (or you're getting ready to clean up some legacy code), start with convergence.

Write a characterization test before you do anything else.

Just like how a tailor sets pins in place before cutting any fabric, you want to clarify the Vertical Slice of functionality - the feature that you're about to adjust - and you want to do this before you do anything else.

This can vary in terms of challenge and complexity.

For example, your features are all coupled together, you might have to characterize 'em all upfront, otherwise you could break one by fixing another.

I know developers love the phrase "it depends". That's sarcasm.

But really, it does depend.

It depends primarily on how coupled your features are to each other AND how easy it is to work with or mock out any connected infrastructure to test infrastructure.

I'll share more on this in future letters.

In summary

What's the main takeaway of all this?

When you write a test, you're converging.

When you write code, you're diverging.

When you only write code, it'll continue to diverge and diverge until it's extremely hard to continue to stack new features and functionality on top.

The solution is to play the zig-zag of divergence and convergence. Cutting and adding.

And finally, when you start with a valuable test, you actually set up a vector of success. You get a guardrail which helps you avoid accidental complexity, and helps you converge only on what matters. On what's valuable.

So learn to do 'em both. Convergence & divergence. Tests & production code.

This is how we get out of Code-First spaghetti territory.

That's all for this letter, folks.

Enjoy the rest of your weekend.

As always, To Mastery

Khalil

PS: A reminder that the waitlist for TWSE 55% opens March 31st at 8:00AM EST. We've already reached 300 waitlist submissions with only 200 spots. Get notified early when doors open here.

Reality → Perception → Definition → Action (Why Language Is Vital As a Developer)

Wed, 10 Jan 2024 04:05:26 GMT

It was only a matter of time before I fell in love with Domain-Driven Design.

Not only did it clarify how to organize my code and untangle the spaghetti…

The truth is: I’m a word nerd.

I’ve always loved books.

I’ve always loved writing.

I’ve always found magic in accents, languages, blogging, poetry, songwriting, copywriting..

Anything and everything within the realm of words, I’m in.

While I’d love to “TED talk” your inbox full of word-related self development content like affirmations, NLP, subconscious programming & so on…

I ain't Tony Robbins.

If you’re getting this email, it’s because you’re a software developer.

So let's focus in.

What we'll cover

In this letter, I’d like to take you on a little journey and:

discuss why language matters so much as a developer
explain how I’ve noticed code-first developers think & talk about code
discuss how refining my language and definitions has helped me improve my ability to develop The Software Essentialist
share an update on the course & book progress
let you know about an upcoming sale for new course members

Why language matters as a developer

Say I told you to go look for stones and bring ‘em back.

Why? Ah, because we’re going to cast ‘em in the water and make pretty ripples.

Nice way to spend a Sunday afternoon, no?

Cool, so unconsciously what’d you’d do is formulate a definition for rounded, cast-able stones in your mind.

In neuroscience, there’s something called the Reticular Activating System — basically the part of your brain that helps you notice stuff that’s relevant.

Based on this instruction, you’d constrain your perception to small, tiny, throwable stones, probably.

—

Now if I said that we were going to get some stones again, but this time, it was to forge some weapons.

Why?

Because we’re gonna head into the woods to hunt bears, bring it home & make a big ol’ bear steak burger.

You’d probably start looking for a different type of stone. Maybe pointy, jagged, or even larger rocks to be quite honest, right?

Basically, what I’m trying to convey to you is that your definitions (conscious or unconscious) affects your perception.

And the way you think and talk about about code affects how you write code.

This is crucial.

It’s the way I make sense of how most developers move through the Phases of Craftship (ie: this is what I call the 5 stages of software development mastery).

Code-First developers think in terms of tools and libraries

At the end of the day, your job as as developer is one single thing: problem decomposition

It’s how well you go from problem to solution.

And developers who are just getting started or experiencing a lot of friction in the way that they work? I call these Code-First developers

This is the first phase of craftship.

Code-First developers think about code in terms of libraries, tools, “frontend” and “backend”.

Because this isn’t a very nuanced approach to problem decomposition, generally, the quality isn’t great. And when we’re at this phase, we run into basically all the problems.

Spaghetti code, fighting the framework, unknown unknowns, refreshing and changing a line of code over and over, debuggers all over the place.

Been there.

Totally normal.

Painful, but necessary when you’re first starting out.

Compared to later stages like the **Responsibility-First Phase of Craftship, your definitions of code — the way you think about it — it’s more within the domain of roles, responsibilities, collaborations and stereotypical objects.

You can kinda see into the matrix a bit here because you’re looking at code from a much deeper, much more refined vantage point. It’s almost like taking a magnifying glass to a codebase. You see all the little itty-bitty atomic aspects of it that make it work.

From this perspective, you have way more options, control, and mastery over the way you work.

You can zoom all the way out and look at problems from a code-first “blocky” api + database + frontend perspective…

And you can zoom all the way in and look at problems and say things like: “ah, this library (like Redux) carries out the coordinator object stereotype responsibilities, and that stereotypically behaves this way and talks to these other stereotypical objects this way”.

At this level, you can identify patterns, tie abstractions together, design custom abstractions at an object-oriented level, and make much better decisions for how to select libraries and frameworks (to name a few).

Ultimately, when you can think deeper - you can decompose problems in more strategic, formless ways that the situation calls for.

Does that make sense? When your language is more refined, your problem-solving capabilities are more refined.

(But that’s obvious, right?)

How I used Domain-Driven Design principles to refine my course development philosophy

A tough pill I had to remember to swallow was that you tend to take a lot of L's when you start a new business.

It's just the way it goes.

See: Ray Dalio's framework for doing anything difficult

Personally, I had to undergo a period of looking at my own definitions because honestly…

On a positive note: after launching the early adopter version of The Software Essentialist not only was last year the most fulfilling, growth-oriented, and illuminating year of my life…

(Some students got jobs at well-known companies, others had major aha moments that personally took me years to grasp)

... but on a more realistic note: it was also the most difficult and challenging one I’ve faced yet as well.

Outside of the personal development growing pains of running a new company, I faced a number of course development challenges and drawbacks like:

doing a lot of re-work
big chunks of the curriculum had to get re-designed
video & audio editing snags
being “forced” to work 10-15+ hour days

Now, as a first time course creator, in a new domain of mastery, I realized much later on, that I was basically the equivalent of a Code-First developer as a course creator.

Entirely new things to learn. Tons of mistakes around every corner.

Alex Hormozi calls this “paying your tuition to the market”.

Fair enough.

But in December 2023, after a year of Q&As, videos, learning new skills & publishing content, took my foot off the gas for a while to question my definitions.

To analyze the way I think and work.

Basically, here’s what I did:

I collected all my learnings over the year
I asked myself every question I could ask myself about how I work, manage time, structure content, and structure my work
I answered those questions with what I currently knew to be true

And with simple questions and answers - this process of diagonis - I watched as my own personal philosophy of course development formulated before my eyes.

Some of you might just be thinking: “dude, sounds like you’ve just discovered SOPs (standard operating procedures)”.

Fair enough (x2).

But upon further reflection, I realized that this is exactly what we do every time we perform Domain Discovery (one of the 12 Essentials of software design, architecture, testing & strategy.

After spending some time condensing down, defining and dividing my concepts into domain language and philosophies…

Discovering the language of this domain of “Software Essentialist Course Development”…

I watched as epiphanies, new initiatives, and a renewed level of clarity for how to produce high-quality material emerged. With a lot less effort.

For example, I started working with one of our esteemed community members, Lucas, on Interactive Visual Summaries for each of the submodules you walk through in The Phases of Craftship.

This helps to untangle the maze of topics and present a path for you to learn them in a contextual, practical way.

He’s built out a v1 of this with d3.

And I gotta say, it’s pretty incredible so far.

And since we’re doing show and tell here… take a look at this.

This is a screencap of some of the the course elements that I’ve invented and named. These are used all throughout the course and as inputs and outputs of the workflows that I use to develop course material.

It’s amazing how much of this was unconscious, and oftentimes is unconscious, largely invisible to me, until I sat down and decided to define and divide — to name things & processes properly.

If you haven’t already, take this as a prompt to learn go Domain-Driven Design.

It’s really important for any sort of problem decomposition. Even outside of software development!

What’s next and what’s new?

Here’s what I’m up to:

I’m currently over here in Sydney, Australia happily away from the Toronto weather, doing some leadership training while working on the rest of TSWE.
Later last year we rolled out a new curriculum based around what I call The Metaphysics and the Phases of Craftship in which there are 5 Phases.
The Metaphysics is up; I’m in the middle of publishing the refined versions of the first 2 (Code & Best Practice-First).
A number of you have asked me to do a Black Friday sale, Christmas Sale and so on — I was pretty hesitant to that idea; I wanted to time things right for y’all to go through this new content. Many of you know I’m using the opportunity to work with you directly as a feedback mechanism to make solidbook.io’s final version the most useful book it possibly can be.
I’m going to offer a discount for the next 50 new early adopter students to join the program to test out the new Code & BP-First content & give feedback. Stay tuned for an update on this.

Chat soon, my friends!

And as always,

To Mastery

The Code-First Developer

Wed, 10 Jan 2024 04:05:26 GMT

"Wait, is that Sean? Hold on a sec," I exclaimed, interrupting a friend mid-conversation.

I had just seen him speeding past the famously cursed McDonald's on the corner of Queen and Spadina in downtown Toronto: Sean, the programmer who had opened my eyes to what great code could look like.

Walking alongside a fellow Software Essentialist Course member, we strolled past the McDonald's when I spotted him — this key developer who had played a pivotal role in shaping my career.

"Whoa, no way... Sean!" I called out, quickening my pace to catch up with him.

Sure enough, it was him.

"Oh, hey Khalil, funny running into you down here. How's it going?" Sean greeted me warmly.

In that moment, memories flooded back, reminding me of the time we crossed paths in 2017.

I had been working at a small frontend consultancy, seeking a job that would allow me to focus on my startup, Univjobs, in my spare time without constantly lingering thoughts of work in the evenings.

I was assigned to collaborate with Sean on a magnetometer software project, exploring the depths of ocean research and exploration.

Until then, my experience had been marred by messy codebases no one wanted to work with. My first job involved:

Injecting SQL code directly into a massive Java file.
Totaling over 2000 lines.
Determining its correctness based on whether it returned any data.

Welcome to the industry.

Fast forward to that consultancy where Sean and I crossed paths. During a lunch and learn session, Sean prepared a talk about Domain-Driven Design. I could feel the cogs in my mind change irreversibly.

Assigned to work with him on the UI portion of an application, I was astonished by the elegance of his C# code. Despite the complexity of the problem we were tackling, I found myself grasping the essence of the domain simply by examining his code.

But something didn't quite sit right with me while at that consultancy. The conversations between other consultants seemed… way too surface-level.

My colleagues argued about things like what Dan Abramov said about single-file React components and why Vue is better than React.

It felt like we were stuck in a repetitive cycle, building the same app for the same 2 clients that funded 80% of the business year after year, just continually refining the implementation with the latest libraries and frameworks.

It felt like the kind of cautionary situation Robert Greene or Patrick Bet-David would write entire chapters about.

I couldn't help but question the value we were truly providing. Many of the other frontend developers were deeply curious and wanted to learn backend, but instead of taking that into their own hands, I noticed most were silently waiting for our employer to fund their learning efforts.

I never expected that to happen, and guess what? It never did.

Instead, for the next 8 months, I spent my mornings and evenings writing buggy backend code, trying to learn how to make it better.

To some, that might seem extreme, but I really wanted to figure it out.

And after everything I'd experienced thus far in my college, university, and early professional experience, I had very little trust that things would just get sorted out by themselves.

And during that time of work & study, the industry certainly did shift.

In a few years, the advent of AI was upon us. With GitHub Copilot and ChatGPT, gone was the inherent value of being a mere code-first frontend developer.

Laws from timeless texts like The Mythical Man-Month (1995, Fred Brooks) such as the law of least developers, whereby "adding more developers to a project typically makes it take longer" - well, these laws locked into place.

Suddenly, the value of a single full-stack Pattern-First Developer with access to ChatGPT became greater than 5+ frontend developers all combined.

From 2021 to present day, I've seen a ton of low-skilled frontend and hyper-specialized developers lose their jobs due to this phenomenon I've been calling the Value Differential - the teeter-totter of what's valuable and what isn't (more on this later).

Back to Sean.

With him in front of me, I asked him what he was up to these days.

When he told me the consultancy we used to work for died and that he was being made CTO at the client's company, I was not surprised at all.

Excellent work always prevails. Mastery is truly recession-proof.

At some point in our conversation, we spoke of Bruce Lee.

"You must be shapeless, formless, like water. When you pour water in a cup, it becomes the cup. When you pour water in a bottle, it becomes the bottle. When you pour water in a teapot, it becomes the teapot. Water can drip and it can crash. Become like water my friend." — Bruce Lee

That encounter with Sean on the street brought back a flood of memories, reminding me of the path I had taken and the lessons I had learned. It solidified my conviction that mastering the first principles and possessing key mental models in this evolving landscape is critically important because it simply is not enough to be a mere Code-First Developer anymore.

In The Software Essentailist, we walk through what I call the 5 Phases of Craftship:

Code-First
Best Practice-First
Pattern-First
Responsibility-First
and Value-First

Each of these phases marks a distinct shift in your value system as a developer, how you think about code, and how you decompose and solve problems.

Because definition creates your perceptual reality, the more you refine your thinking (ie: your definitions, beliefs & knowledge about what code is and how you decompose and solve problems with it), the more complex the problems you can confidently decompose and solve.

Let's discuss the first Phase: the Code-First Phase.

(I sometimes call this "The Pain Phase", the "Imposter Syndrome Phase", the "First Phase", or the "Necessary Phase" of craftship).

What we'll discuss

In this short article, here's what we're going to do:

We'll do a quick recap of who the Code-First Developer is
We'll review the traps that prevent you from progressing
Learn the next steps to transition from Code-First to Best Practice-First

What is the Code-First Developer?

Alright, so the Code-First Developer.

Who is this again?

Ultimately, this is the type of developer that has learned to code but has yet to learn to craft.

As we discussed in a previous module, the 4 pillars that every developer needs to focus on design, testing, architecture, and strategy. We call these The 4 Pillars of Software Development.

And these 4 pillars have a chief purpose.

And that is to act as assistive tools for us to help the 3 roles (customers, developers, users) achieve their goals.

What are all those goals you might ask?

In essence, you need to learn how to work in ways that will prove to be valuable. If you do it right, when you work, you'll keep the 3 roles in benefit by either creating:

TimeGivenBack (to pursue other human desires)
or TimeEnriched (to enjoy the present moment)

That may be too abstract, so let's look at an example that should make sense to you.

We'll use the developer - actually a frontend developer since when we're at the Code-First phase, most developers tend to be frontend developers.

So, what are some common problems for frontend developers at the code-first phase?

What might you be asking yourself?

how do I write E2E tests that aren't flaky? → because it takes time to write these, watch them run, and I hate the feeling of having to go back and re-do them every time the smallest thing changes
how should I be writing unit tests? → because I've tried to figure it out, but it was confusing, and taking too much time to know what to do
what exactly should I test in the first place? → because I literally have no idea, and I know tests are important but it doesn't seem very useful to test something like Redux
how should I organize my code so that it's scalable? → because I notice that I spend a lot of time flipping between abstractions and others seem to not really know what to do
how do I do things the Angular / React / Vue way? → and so on...
how can I get better at architecture? I don't know how to use in a testable way? → and so on...

Key questions.

Each hints at a tiny slice of pain (wasting time and feeling frustrated).

And each with very little good answers at this level.

That's because when you're at the code first phase, you don't know much of what matters.

Yes, you can use the basics of your tools to solve problems but… testing, design, architecture, strategy? The stuff which actually allows you to keep those 3 roles in benefit? Not likely at this phase.

We're talking about understanding the skills that allow you to:

Make testing fun and enjoyable?
Approach your work consistent?
Adapt to the changing requirements flexibly?
Select a pragmatic testing architecture that will allow you to feel confident in your code given the time constraints?

These thing are really important. But they don't come until later phases.

So we better move through 'em.

The Value Differential has changed

I won't spend too much time on this here, and you already know, but being Code-First used to be fine.

Thinking back to the early days of React, when things were kind of a mess, the API was changing every few weeks, and Webpack was the worst thing on the planet… this was when the most valuable thing you could be WAS a React Specialist.

Back then, in 4 to 6 months, with access to FreeCodeCamp and some 12-hour YouTube video compilations, you could possess an incredibly valuable, in-demand skill nearly everyone needed.

But now?

What do we need now to get the same result?

Internet access and a ChatGPT account.

Yep.

"Hey, ChatGPT, can you generate a React Hook that helps me synchronize XYZ?"

Sure, here you go.

Thank you, artificial intelligence.

So what does that mean for all the freshly graduated frontend bootcamp devs looking for jobs?

It means we need a different strategy, of course.

Your biggest problem is the sheer amount of unknown unknowns.

When I backtracked the path I was walking in solidbook, I realized that the biggest problem all Code-First developers face is that you don't even know what you need to know, and you don't know how to BEGIN learning what you need to know.

It's the wild west.

If you listen to what everyone else tells you to do, you'll learn new libraries and frameworks every few weeks. NOT leading yourself towards their depths and getting to the next Phases. New tools.

The Expert Junior Developer

This trap of taking a breadth-first approach keeps you stuck at the surface and can potentially turn you into the Expert Junior Developer.

You may have been around for quite some time, but like many of the consultants I worked with in the past at that now dead frontend consultancy, when you get back into the industry, you're still asking yourself:

how do I write E2E tests that aren't flaky?
how should I be writing unit tests?
what exactly should I test in the first place?
how should I organize my code so that it's scalable?
how do I do things that Angular / React / Vue way?
how can I get better at architecture? I don't know how to use in a testable way?

These are the same questions.

And we all know what comes next: burnout.

Breadth > depth leading to burnout

I interpret burnout as when the level of effort and energy you spend doesn't match the level of fulfillment you should experience.

I believe the key to lifelong fulfillment is Mastery.

To be able to set a goal, chart the path, and consistently walk that path confidently, regardless of what it pertains to, is one of the most fulfilling experiences, in my opinion.

But the key, as I've probably said twelve times in different ways, is depth, not breadth.

Do less, more.

🌀 Focus on the main idea: Responsibilities. The metaphysical, atomic unit of matter that describes what software is comprised of is called a Responsibility (for doing and knowing). Let's continually refine our understanding of this. (You can take a look at Responsibility-Driven Design if you'd like to jump ahead a bit).

How to progress to Best Practice-First (what to do instead)

So let's discuss what we gotta do to move toward the next phase.

Step 1: Shift your focus to Responsibilities instead of Code

We want to get to Value-First so that you can do life-changing stuff.

But doing so involves developing a deeper, more mature perception of code and how to code properly.

First step: start to pay attention to the various responsibilities your tools handle.

Start to notice the things that your tools do and know.

What are the things that your web server does and knows?

What about React? What does it do and what does it know?

It's funny because you can use React every day for five years and become an expert, but when you expand your perception of what it is you're dealing with...

When you no longer see React as the thing you use to "build the frontend"...

But instead, you see it as a tool which allows you to "build the UI while coding outside-in" @ the Best Practice-First Phase.

Or you'll see React as the tool you use to "build your View Components and Page Components" separately from how you organize other concerns using more robust frontend patterns like View Models, Domain Objects, Gateways and so on.

Step 2: Commit to becoming a Value-First developer (which means you have to become a fullstack developer)

You want to be able to produce as much value as possible.

That means you need the ability to create information systems from end to end.

But not just any information system. I'm not just talking about using CRUD and REST.

These need to be information systems that are Software Artifacts of Value.

This means they help users, help customers achieve their goals, and developers can carry out their common development use cases.

What you place your awareness on, you manifest.

Let's focus on that.

It's fun!. The world rewards you for that the value you put out.

Step 3: Build an information system from end to end, making all the mistakes

In the first Phase of Craftship, you want to focus on building information systems from end to end.

That means frontend and backend.

Build a React app that talks to a backend and a database.

Build a Gatsby or Next.js app that uses a CMS.

Doesn't matter. Just build something that a real human being can log onto an accomplish a task with.

While you're building it, don't worry about best practices or if you're doing things right.

Just make it work.

Make it work and survey the landscape of tools you needed to use along the way.

How do they help you cross the gap?

Step 4: Validate your understanding

As I said, the main challenge at the Code-First level is unknown unknowns.

Focus on building something - anything - along the lines that it could serve others and a customer could possibly benefit from it.

And while you do this, zoom all the way out, and then focus on what your tools are really for, why you use them, and what they do well.

I will give later give you some strategies for ensuring you understand the role your tools play, the responsibilities, and how they either help or hurt the three roles in achieving their goals.

Summary

The Code-First developer might be technically skilled at the implementation side of things, but the Value Differential says that code is not enough.
The biggest challenge Code-First developers face is the sheer amount of unknown unknowns — knowing what to learn, what to prioritize, and how to learn it effectively without suffering from overwhelm or burnout.
The goal is to be able to think and work responsibility-first because that's what will allow you to focus on what matters (creating value for the three roles) later on.
So, let's take it one step at a time. To get to the next Phase of Craftship, you must commit to becoming full stack, build a real, basic information system that does something valuable, and ensure that you understand the purpose of the Code-First tools and topics used to do so.

"Tests Are a Waste of Time"

Wed, 08 Mar 2023 15:04:10 GMT

I want to tackle a hot take that happens to be one of the biggest ones holding developers back from ~~endless puppy videos~~ becoming truly indispensable software designers.

We're on a mission here to learn the first principles of design, testing & architecture.

That's going to open up a LOT of doors for us.

Now, here's the thing I hear all the time:

"Writing the test first (TDD) is a COMPLETE waste of time".

Ah man, that's a shame.

Front-end developers, I'm looking at y'all.

But, I do know how we got there...

I've been there myself.

You've tried TDD and realized — compared to the way you currently do things — it's pretty annoying
And since you can still make your code work without writing the test first...
It just seems unnecessarily challenging to approach your work this way

Totally.

And if you're a front-end developer, you may have heard things like: "write tests, not many; mostly integration".

Or that functional programming doesn't need tests (and that you're doing FP by using the function keyword...)

Or that all you need is E2E tests.

Ahhh... geez.

Many crafters ah've met.

And many I've lost to the most unfortunate Bad Testing Advice rabbit holes.

Most devs follow the Best Practices ™️ of what the official docs for their framework say to do (such as using snapshot tests — don't do that).

And then there are tools like Jest and Cypress, which are fantastic, but to be fair, they can't really give you the foundational design, architecture & testing wisdom you need in order to know how to use them correctly.

That's why we place 90% of our energy on skills > tools in the Phronesis Labs and Software Essentialist communities.

But just like you want to ensure you've got some good grip strength before doing a 200lb deadlift...

We need to get our mindset right before we embark on this path of skill development — where we'll master The Lost Art of Writing Scalable, Testable Code.

To explain why this take is a massive wrench in your growth plan (though there is a caveat), I'll give you a three-part response.

Ready?

Here we go.

1. The Entire Purpose of Professional Software Development is to Guess

What do I mean by this?

Well, just like there are a ton of different ways you can play Super Mario Bros (you can Speed Run it like a manic, play for "leisure" with your partner — yeah right, or collect the coins on each level)...

There are also a number of different ways you can play The Game of Software Development.

You can play:

the Code-First Mode: where you just care about building & making things work (as they occur)
the Exploratory Mode: where you're just trying out new libraries & frameworks, having fun with them, and it's pretty light
the Professional Software Development Mode: where the focus is on scalably, sustainably producing artifacts of value

I'm willing to bet that you want to get paid for your efforts, so that means you'd like to see yourself as a Professional Software Developer.

Me too.

What is Value Anyway?

Very simply, the way I think about value is this.

Value is anything that makes you feel good or gives you time back (so that you can spend that time in the pursuit of trying to feel good).

This is a key part of Essential #1: Software Essentialism, because setting up these variables that we'll use throughout the entirety of craftship journey allows us to move strategically and make good (design, architecture, testing & overall product) decisions.

Another key thing to recognize is that everything is a state machine.

Including Humans (that's you - I hope).

David K. Piano was right, everyone.

Here's how this works.

From Pain → Benefit = Value

It's simple. We don't like pain. We like benefits.

You create value when you take someone from pain to benefit.

From Benefit → Benefit

It is also staying IN benefit.

Example: Thank you Facebook for reminding me that Grandma's birthday was today.

Example: Your email service continuing to chime along and work swimmingly, as it did last week.

Negative Value

But software can have the opposite effect as well.

It can make you frustrated, feel dumb, silly, and bad about yourself.

This effect is NEGATIVE VALUE.

Example: Your email service all of a sudden stops working the way it used to.

Example: The UX to order a bowl from Chipotle on UberEats is so confusing that you accidentally order literally just chicken, nothing else - and you do that twice in a row by accident and then they ban you from the store because they think you're trolling them.

Example: Instagram changes the placement of the buttons for the 13th time this month.

This sucks.

The Guess (it's Theoretical & Physical)

Know what this means?

That there are at least 2 things that we don't know for sure.

If what we want to build is valuable (theoretically, does someone even want this?)
- Ie: "Hey everyone, I'm going to build an Anime Debate App so you can all stop using Discord now, and use my app instead :) ...anyone?"
If it will work, and won't result in Negative Value (physical)

These two planes of Guesses have a ton of sub-guesses within them.

We make Guesses at:

The Theoretical Plane: with at least 3 different Guess Points
The Physical Plane: with at least 8 different Guess Points

And the only way for us to verify that our guesses are correct, is to perform Essential #2: The Feedback Loop.

This is also known as Three Ways.

This is just TDD at the 8 Physical Levels, but holistically, this is the entire philosophy for how Professional Software Developers play the game of Professional Software Development. This is also business owners, product people - essentially, anyone involved in playing the role of "creating software artifacts of value".

Strategic, Pragmatic Guessing

Now, I know what you're going to say:

"You can't possibly expect me to write the test first for every single one of those feedback loops. that doesn't make any sense. I'll just E2E test the most important features. That's enough."

And you're absolutely, 100% right.

Sometimes, it's non-strategic to spend time Feedback Looping over more inner-layer Guess Points.

For example, if your app doesn't have any customers (no value just yet), it doesn't really make sense for you to TDD your way through. You should be focused more on patching through to creating value for the right people.

At that point, with a fresh, new product and a ton of uncertainty, yeah, go right ahead -- go wild with whatever frameworks you want, because your focus is NOT on The Physical just yet.

It's on The Theoretical.

Focus on getting at least ONE Theoretical Guess correct.

In Part 5 of The Software Essentialist titled "Bridging The Gap (Theoretical → Physical)", we apply this process and learn how to identify, design, and build your own product of value, from scratch.

After one of your Theoretical Guesses connects, and you've created real value for someone out there, your goal should shift.

It should shift to the balance between:

Finding New Value
Maintaining Existing Value
Preventing Negative Value

Sounds straightforward, right?

Until it's not.

Seriously.

It's much easier to be wrong than it is to be right.

And there are so many ways to be wrong.

Take Discord. When I was doing a live AMA in the Phronesis Labs Discord with over 100+ members, it suddenly just decided to cut out and drop everyone from the call.

N E G A T I V E V A L U E

I call this Feedback Loop Level 8: The Execution Guess Point.

A particularly tricky one.

When this happens, expect your customers to start looking for solutions in other places (I'm thinking about switching to Zoom).

But if it could happen to Discord, it'll probably happen to us too.

That's why we Lead With The Guess, make no assumptions, and prioritize covering as many Guess Points with tests as humanly and reasonably possible within the time frame we have.

BTW: Remember when I said that learning this stuff opens up a lot of doors for you? If you understand how to identify Value Gaps in the real world AND you learn how to put software there, there's just one thing left. Selling it.

But that's a topic for another time.

Let's press on.

2. To write the tests first, you need a testable architecture

At this point, you might be thinking:

"That's cool, Khalil - but I can always just add those tests later."

Actually, you can't.

Know what? Yeah, you can — but you're going to hate your life trying to carve out the ability to add those tests later on.

Here's why.

Ever heard of the hexagonal architecture? (Similar or comparable to the clean, onion, or ports & adapters one).

Well, the entire purpose of the hexagonal architecture is to give you the ability to reach those other Guess Points.

For example, most front-end developers don't have the baseline design foundation to write tests at Feedback Loop Level 3: The Application Guess Point.

This is an incredibly missed opportunity.

If you learn how to decouple from infrastructure here, you can:

force your code down various code paths using stubs
cleanly test side-effects using test doubles
have fun writing super fast, high-value developer tests that exhaust the majority of your edge cases

The thing is, unless you've started with a layered architecture, it's likely going to be a major pain to refactor towards it.

It's still possible, but it's better for you to start that way - assuming you're going to be making Guesses, because it's as they say:

Architecture is about the decisions you make at the beginning of a project. Architecture is the stuff that matters, whatever that is.

But it's never too late.

If you want to learn this, do like we do in The Software Essentialist. Start from the innermost feedback loop levels, and work your way towards the outer ones.

There are some foundational skills you'll need to pick up along the way to get pretty good at this, like:

Red-Green-Refactoring
Programming by Wishful Thinking
Using the 3 forms of Subject Verification to test anything

3. Testing Requires a High Level of Mastery (But It's The Key)

Hopefully by now you're beginning to see how critical it is for us to prioritize developing the capacity to Lead With The Guess and Write The Test First.

We may not always do that if it doesn't fit into our strategy, but we definitely need the capacity to exercise a Feedback Loop at each of these Guess Points.

In actuality though, I think a huge reason why most developers don't bother with testing is because this:

It is hard to learn.

I was just chatting with one of your fellow Phronesis Labs community members about this recently.

We both agreed that in order to learn this skill properly, you need to know:

the basics of object design
dependency injection
dependency inversion
hexagonal architectures & decoupling from infrastructure
how to use test doubles
how to create great abstractions
how to know when your abstractions aren't great

It's true. It's traditionally been a really hard thing to learn.

It took me years to get really good at, even after having built DDDForum (see it on my GitHub) and having written the first version of SOLID BOOK.

But it's the key.

When you learn how to test properly, you learn how to:

Design, instead of just mixing
Win the game of Professional Software Development
Build your own SaaS products
Mitigate negative value in your codebase
Think & code backwards, producing the minimal amount of accidental complexity possible
Get a safe grounds to test out designs
Always produce the highest quality designs possible
Create abstractions that others love to work with
Develop scalable, maintainable code
- Imagine you wanted to implement a new feature -- and all you had to do was add a few lines of code. This is what us OOP nerds call Open-Closed Principle, or extensibility for short.
Ultimately, you become a benefit to employers
- Did you know that over $85 billion dollars goes right into the dumpster fire every year trying to maintain badly written software?
- Want a piece of that pie?
- Learn to Lead with the Guess and create truly scalable, value adding artifacts

Good stuff, y'all.

That does it for this one.

As always,

To Mastery

Khalil

No title

Sat, 26 Nov 2022 04:05:26 GMT

🌱 This blog post hasn't fully bloomed. It's very likely to change over the next little while.

What is this?

Stateful tests are obviously tests that rely on state
What makes these different than other tests?
- You have to set up the state of the world
- You usually have to test that the new state of the world is correct
- You usually have to test that the commands that changed the state of the world were, in fact called, and were called correctly.

State changes

What are state changes?
A successful command-like operation is a state-change.
Examples:
- A successful save, saving a user aggregate to a userRepo
- A successful makePurchase call to a Stripe API
- A successful addRole on a user aggregate

State machines

Your application is fundamentally a state machine. We move people from A to B - problem to solution. If your app had 9 different states, from beginning to end, and you wanted to test the transition from state 6 to 7, you'd need to load up the state of the world from 1-6, in order to test that transition. Or at the very least, load up what the state of the world looked like at 6.

Questions

Why don't you test queries?
- Because queries don't change the state of the world.
- Yes, they might return different data which informs the behaviour, sending it down different code paths, but ultimately, there's no point in testing queries for stateful tests.
What's the role of queries in tests then?
- Queries merely return the current state of the world.
- And depending on that state of the world, our application (a state machine), has to act upon that data in different ways.
- This is what we care about.
- "Is my feature/use case behaving the way it should based on this state of the world? Do I get a failure result when I try to do something (ie: perform a transition from one state to the next) at an invalid state? Do I get a failure when I don't provide the appropriate data to perform the transition (the behaviour)?"
What's the difference between stateful and stateless tests?
- State, of course.
- That is, with stateless tests, we are testing functions or objects which typically live in total isolation. They have no stateful collaborators.
- Why? Because there is no state to them. Stateless because they themselves have no state. There is no state to them.
- We are often just testing purely behaviour.
- Data, behaviour, namespaces
  - Let us remember that the essence of software is solve problems. And that is achieved by gathering data, performing behaviour against that data, and organizing the units of data and behaviour into cohesive namespaces.
- Do you see how this is much different from a stateless test?

How to Find and Kill a Background Node Process

Thu, 03 Nov 2022 15:04:10 GMT

I'm sure you've run into this issue before.

You try to npm run start a project up but you get an error that looks something like the following:

project git:(master) ✗ npm run start

> gatsby develop

/Users/khalilstemmler/Documents/Personal/current/solid-book-wiki/projects/wiki/node_modules/yoga-layout-prebuilt/yoga-layout/build/Release/nbind.js:53
        throw ex;
        ^

Error: listen EADDRINUSE: address already in use 127.0.0.1:8000
    at Server.setupListenHandle [as _listen2] (node:net:1372:16)
    at listenInCluster (node:net:1420:12)
    at GetAddrInfoReqWrap.doListen (node:net:1559:7)
    at GetAddrInfoReqWrap.onlookup [as oncomplete] (node:dns:73:8)
Emitted 'error' event on Server instance at:
    at emitErrorNT (node:net:1399:8)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  code: 'EADDRINUSE',
  errno: -48,
  syscall: 'listen',
  address: '127.0.0.1',
  port: 8000
}

What's going on here?

You have another process running on port 8000 and whatever you're trying to spin up also needs to utilize that port.

You can tell this from the EADDRINUSE error constant and the fact that it says 127.0.0.1:8000 is alreaady in use.

What to do?

Here's a quick an easy fix to kill the process running so you can start something else up on that port.

1. Find what's running on the port

First thing is first. Use the lsof command to see what's running.

We'll use it like this: lsof -i:<port>.

wiki git:(master) ✗ lsof -i:8000
COMMAND   PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
node    49516 khalilstemmler   27u  IPv4 0xb0855a2b3b9a7a2b      0t0  TCP localhost:irdmi (LISTEN)

Just as I thought. Some node process running.

2. Take note of the process number

Take a look at the PID column. That marks the process number.

wiki git:(master) ✗ lsof -i:8000
COMMAND   PID           USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
node    49516 khalilstemmler   27u  IPv4 0xb0855a2b3b9a7a2b      0t0  TCP localhost:irdmi (LISTEN)

Copy it. You'll need it for the next step.

3. Kill whatever's running on that port by using the process number

Now, let's kill the process.

You can do so with kill -9 <process-number>.

For example:

kill -9 49516

4. Confirm the process has been killed

Good.

Lastly, let's make sure that worked.

We can run the lsof command again to see if there's a process running.

➜  wiki git:(master) ✗ lsof -i:8000
➜  wiki git:(master) ✗

Nothing! Looks like we did it.

5. Re-run whatever you were trying to run

And finally, re-run whatever you were trying to run on that port.

Done-zo.

Now get back to it.

Summary

Here's a summary.

Use lsof -i:<port> to find the offending process id
Take note of the process number in table
Use kill -9 <process-number> to kill the process at the port
Confirm it worked with lsof -i:<port> again. You should not see the same process.

3 Steps to Solve Most Design Problems | How to Find Abstractions Effectively

Thu, 08 Sep 2022 15:04:10 GMT

When I was in university, I decided to build my first startup with a couple of buddies that helped students find jobs.

Having spent a year learning JavaScript and flying through Tyler McGinnis’ courses on React and Redux, I went in.

Slinging code like wild, I eventually encountered a challenge: to design a complex page-routing system that would re-route users to the correct pages depending on a variety of different factors (state, user type, account verification).

Since I couldn’t find a library to help me do this, I had to build it myself.

Feeling overconfident, I over-engineered the heck out of it with no tests.

Over the rest of the lifetime of the project, that routing system became the key area to cause bugs, frustrating customers and humbling me in the process.

Ah, the art of creating good abstractions.

How I wish I knew back then what I know now.

Let’s talk about how to solve most design problems and find abstractions effectively.

The Responsibility-Driven Design method

I’ve explored many different ways to design software but if you ask me, I’d say that learning Responsibility-Driven Design was the last thing that I really needed to deeply understand the nature of design.

The Responsibility-Driven Design process is pretty straightforward, but the magic is in the details.

Here’s how it works.

Start with a requirement (functional or non-functional), convert it into responsibilities, assign those to roles, then find the collaborations.

Allow me to demonstrate using a call I had with a dev last week as an example.

Step 0: Identify the requirement

Say you’re working on some back-end code.

You’ve been asked to add a feature.

The feature is: if you perform a request while logged in, the system should update your account’s lastActive time.

Step 1: Decompose the requirement into responsibilities

First things first.

Understand that requirements contain a multitude of smaller responsibilities and responsibilities.

Responsibilities can be for either doing or knowing.

Here’s what we did. At first, we found the following responsibilities.

For doing

Update the last active time
Intercept requests

For knowing

Know if the user is logged in or not

But then, after further questions about performance (ie: like how do we do this really fast if we have to go to the database all the time — wouldn’t that be inefficient), we identified even more responsibilities. Responsibilities involving caching.

The working list of responsibilities changed a bit more:

For doing

Intercept all requests
Update the last active time in the cache
Persist last active time in cache to the database periodically

For knowing

Know if the user is logged in or not
Know when to persist to the database

Not bad so far.

Step 2: Look for common object (role) stereotypes

The next thing we do is take your list of responsibilities, and for each one, assign them to roles.

If you forgot, a role is a collection of responsibilities. For example, a Repository usually has the responsibility to save and find domain objects.

Cool, but how do we come up with roles? How do we know which responsibilities to assign Is there a structured way to do this?

There is.

You use the six Object Stereotypes.

Check that article out and come back.

Back?

Sweet.

So yeah, object stereotypes are probably the best part of RDD. They act as building blocks — stereotypical elements — of any design.

After we got up to speed on object stereotypes, when we looked at the intercept all requests responsibility, I asked:

“What kind of stereotype do you think handles something like intercepting requests?”

Almost immediately, we realized this behaviour as something that a Coordinator (the fourth stereotype) would do.

Coordinators typically only have one real purpose: to pass information to other objects so that they can do work.

Next question. Do we need to create a new coordinator or is there already one present we could merely extend?

Me: “So, do you have already anything in your architecture that acts like a coordinator?”

Him: “Hmm. Actually, yeah. I’m using Express. You know, as my router. It has this middleware feature in it. I could probably use that, right?”

Me: “Absolutely.”

Amazing. We already have a coordinator. It can handle the intercepting responsibility.

That’s 1 out of 5 responsibilities assigned to roles.

We continue going down the list and assigning responsibilities to roles.

Step 3: Collaborations

After you’ve assigned your responsibilities to roles (ideally based on Object Stereotypes), you’re going to need to figure out how to connect your new or existing roles together.

It’s time to identify the collaborations.

There are number of ways to do this, but the most obvious way is to — again, look back to the stereotypes.

If you know:

The object stereotype
Where it lives within in a layered architecture

Then you can usually figure out who its neighbour's are. We can ask ourselves:

Who does it need help from?
Who does it help?

After assigning the responsibilities to roles, we started fleshing out the collaborations:

For intercepting all requests → We identified the Express Middleware (Coordinator) and Coordinators typically communicate with Controllers. Therefore, we realized we needed UpdateLastActive use case (Controller).
For updating the last active time in the cache → We had to invent a Cache (both Interfacer & Structurer stereotypes). This was to be used by the UpdateLastActive use case (Controller).

A tricky one was the responsibility of persisting the last active time in cache to the database periodically.

For this, we had to do break it down even further. The idea of periodic updates implies schedules which implies scheduling responsibilities. We had to:

Find all the scheduling responsibilities
and assign them to new and existing roles

Eventually, we ended up with:

a Scheduler (both the Service Provider and Coordinator stereotypes)
a ScheduledTask (Controller)
and of course the UserRepo (Interfacer) to actually update the user’s info

Boom. Done.

A quick little drawing to make sure things actually line up and we’re off to writing the first tests and coding it out.

Patterns, communication & abstraction

Do you see how powerful this is?

By adopting a design method and levering the fact that we, as humans, are pattern-matching machines, you not only gain the ability to solve problems with well-known solutions, you find it incredibly easy to communicate your design ideas with others.

Without needing to get into the details, as soon as we established what a Coordinator was, the developer was able to identify it within his architecture instantaneously.

Abstraction. It’s what Alan Watts taught us.

Archetypes. It’s what Carl Jung knew.

If you were a writer, you’d leverage the Jungian personality archetypes to write engaging stories and plays.

But you’re a developer, so familiarize yourself with object stereotypes and learn to write scalable software.

Learn to cook, don’t just mix

Once, my mom visited me in university to check in on me and how I was cooking for myself.

Noticing the empty Kraft dinner boxes, I heard her say:

“Kraft Dinner isn’t cooking; that’s just mixing.”

I apologize to any Kraft dinner chefs out there reading this, but she’s right.

Doesn’t matter if you’re frontend or backend, when you’re stuck in library and framework-land, you don’t learn how to design and you don’t learn how to architect.

You just mix.

When all you have are the things that libraries and frameworks give you, you’re pretty darn limited in your thinking.

It’s always gonna be roles, responsibilities, and collaborations

I presented an image earlier. I’ll show it again here.

This image depicts pretty much everything you’ll ever come across in building web applications.

I recommend you screenshot it or save it.

Doesn’t matter if you’re building robots, video games, or something else — your task is always the same: assign responsibilities to roles and design collaborations.

Even if you’re doing pure (and proper) functional programming. Same task.

To me, Responsibility-Driven Design is as deep as you need to go. It contains foundational ideas that help you:

Develop a second sense for how to solve design problems
Learn how to design scalable frontend and backend architectures
Identify the patterns and stereotypes within your favourite libraries and frameworks, rely on them less, use them more effectively, and fill in design gaps with your own abstractions when they try to lead you astray to do things that would render your code untestable
Venture into how to design distributed systems

I hope that was helpful.

You’ll hear more about the important parts of RDD and how to design scalable frontend architectures from me in the future.

Let me know if there’s something you’d like to know specifically.

—

Until then, enjoy the rest of your week.

To mastery,

Khalil

—

What happened this week

The Frontend Architecture Academy — This just in. An 8-week 1-on-1 coaching program where I teach you how to build scalable frontend applications using the best parts of DDD, TDD, BDD, RDD, clean architecture and more. Slots are limited. If you’re interested in joining, want more info, or could simply use some advice, throw some time on my calendar. I’d love to hear what your challenges are and to see if I can point you in the right direction.
Phronesis Labs Discord community — Well folks, I’ve finally done it. The advanced software design and architecture discord community you’ve been asking me to make for about 2 years is finally here.

You can join for free here.
Writing a song on flow, Dionysus, and the creative process (demo) — In The Birth of Tragedy, Nietzsche wrote that the struggle of human nature is that we are both Apollonian (see the Greek god of light and reason) and Dionysian (see the god of wine, ecstatic emotion, and tragedy). Things like music, art, passion, and love pertain to the realm of Dionysus. Sculpture, craftship, and progress pertain to the Apollo. Whenever I sit down to write (music, books, blog posts), I find myself transported to the realm of the Dionysus — perhaps the very same realm that Mihaly Csikszentmihalyi calls Flow. It’s the optimal state of consciousness. Of happiness. You can listen to the rough demo here.

Object Stereotypes

Wed, 07 Sep 2022 22:04:10 GMT

🌱 This blog post hasn't fully bloomed. It's very likely to change over the next little while.

Why the heck is LEGO still so comforting after all these years?

Maybe it's because it feels good to come back to the same baseline elements: the brick, the baseplate, the tile, the mini-fig, etc.

Object stereotypes, a concept from Responsibility-Driven Design, are effectively your programming LEGO.

They’re your starting pieces when you're trying to find the right abstractions to create.

Within your codebase, your libraries, frameworks, and so on - if you look closely, you'll notice that your objects tend to conform to one or more object stereotypes.

How to use object stereotypes

The Responsibility-Driven process involves taking a requirement, extracting out the responsibilities, and the assigning those responsibilities to roles.

How do you find the correct role?

Look to the stereotypes.

They act as hints to the roles we want to introduce.

The 6 Stereotypes

The Information Holder
The Structurer
The Service Provider
The Coordinator
The Controller
The Interfacer

1. The Information Holder

The Information Holder is perhaps the simplest stereotype we know of. Most similar to idea of a Value Object made popular in Domain-Driven Design, Information Holders are usually domain layer concerns, but they can also represent other things as well (such as Data Transfer Objects).

What’s important to know is that the stereotypical responsibility of this role is to hold information. Anytime you notice responsibilities to hold info, consider designing that object candidate as an Information Holder.

2. The Structurer

Structurers are primarily about creating and maintaining relationships between objects. Structurers protect against invariants and can make it easy to find related objects.

Examples of Structurers are Aggregates (from Domain-Driven Design), hash tables (or any mechanism responsible for caching), or objects that pool or handle connections.

3. The Service Provider

Service providers are objects that sit passive and contain useful, specialized computations.

Stable service providers: They typically sit dormant waiting for other objects to call them, but because they’re often relied upon by so many objects, they may contain responsibilities that relatively stable, not changing often.

For example, utility classes like TextUtil, DateUtil, NumberUtil provide respective text, date, and number utilities and probably don’t change dramatically (or at all for that matter). For this reason, we’d say they’re stable. Just think about what would happen if we had to change the JSON.stringify method in JavaScript.

Pure fabrication: Alternatively, service providers are also objects that live in the domain layer yet don’t represent concepts from the problem domain. Instead, they’re objects that are specifically invented to ensure other domain objects maintain low coupling, high cohesion, and encapsulation. In Domain-Driven Design, we call these types of classes Domain Services.

Cross-cutting concern service providers: Lastly, service providers are known to contain responsibilities that are commonly know as “cross-cutting concerns” like logging, security, or tracing (each of which are non-functional requirements). A concern = a functional or non-functional requirement. Non-functional requirements are the ones that are “cross-cut”. This means that Use Cases either rely on these service-providers OR we design the collaboration so that the service providers wrap the Use Case (hence the name “cross-cutting”).

Either solution works, but there are tradeoffs to both.

4. The Coordinator

The Coordinator stereotype has one real purpose: to pass information to other objects so that they can do work. When we look at it like this, Coordinators are one of two objects most responsible for determining how control happens in our applications (the other is the Controller).

5. The Controller

We’ve dealt with this particular type of stereotype before, but we’ve been calling it the Use Case instead. In RDD, the Controller stereotype is the exact same thing as the Use Case pattern.

The stereotypical responsibility held by a Controller is to act as a higher-level (application-layer specifically), declaratively focused object that coordinates the behaviour of lower-level concerns.

6. The Interfacer

The last of the stereotypes is the Interfacer. The Interfacer primary acts as a bridge between object neighborhoods (internal and external).

Internal interfacer: To cross internal object neighborhoods, we need to make the entry-point for our components as simple as possible. The object which sits at the front, relaying messages to objects behind the scenes is the Interfacer.

External interfacer (gateway): To cross external object neighborhoods (as is the case when we send messages out to remote APIs or to libraries which we don’t own), the more common name for the object that encapsulates this access is called a Gateway, however — stereotypically, it’s an Interfacer.

UI-related interfacers: Interfacers which listen to events from the UI and pass them to non-UI related parts of our applications are typically just called Event Listeners (in JavaScript, there is an EventListener API) whereas objects that format data for presentation in the UI are aptly named Presenters. Both are stereotypical Interfacers.

Considerations on stereotypes

Stereotypes make it easier for us to identify objects: This should clear up a lot of headroom for you. Because we know the six general types of objects we’re likely to encounter, we'll find it easier to invent object candidates.

Stereotypes help us emphasize what matters about our objects: We’re constantly striving to ensure that our objects are cohesive and well-defined. If we discover a new object candidate, we can ask ourselves “what stereotype is this most like?” This question helps us determine if we should redistribute misplaced responsibilities to other object candidates to make them more cohesive and focused.

A candidate object can belong to more than one stereotype: Taking it the other direction, there are times when objects have responsibilities that float between multiple stereotypes. For example, consider the Repository pattern. If it were to also cache the objects it retrieves, we’d say it acts more like a combination of the Interfacer and Structurer stereotypes.

Object stereotypes make understanding & communication simpler: It’s easier to talk about design with stereotypes in mind. Once you’re aware of object stereotypes, you’ll start seeing them (or the lack of them) in the frameworks and libraries you’re using. This presents a great opportunity for us to get what we need by creating it ourselves.

98% of Developers Use React as a Framework, Not a Library

Sun, 04 Sep 2022 15:04:10 GMT

🌱 Note: This post is still being updated for clarity and utility. Short on time this week.

What’s the difference between a framework and a library?

A framework is something that you plug your code into.

A library is something that you plug into your code.

If that’s the case, then 98% of developers use React as a framework instead of a library.

Control centers

In Responsibility-Driven Design, there’s this idea of control centers.

Control centers are the parts of your codebase tasked with coordination and control.

Coordination = handling events, reacting to them, relaying events to others
Control = directing the action of lower-level objects, delegating work.

Every application has at least one place that does coordination and often several places that handle control (hence why we tend to have multiple controllers in MVC).

Most developers let React handle coordination and control

The React documentation and thought leadership today urges you to write the majority of your code within React, using custom Hooks as the key place to put things that have to do with control, data, rules, processes, and so on.

This is great for getting started, but essentially, we’re doing the exact same thing we try to avoid doing on the backend: creating an anemic domain model.

Coordination and control are application layer concerns

Just like application service use cases and command handlers are application layer concerns in backend programming, on the frontend, code that determines what happens after a click or a keypress - these too, are application layer concerns.

What happens when we put coordination and control inside of React?

I can think of a few different implications, but the key one is that we have less testing options.

Because you need React to spin up and test your application (and domain) layer code, the idea of testing your core functionality in isolation isn’t possible. In fact, it's borderline impossible.

How do we fix this?

Decouple from the library and framework
Create your own control center — To create your own control center, you need at least one Coordinator and as many Controllers as necessary (these are synonymous to application-layer Use Cases though I’ve also used the word Interactor and describe the type of logic they encapsulate as Interaction Logic).

I will say more on exactly how to do this in another post, but for now, see the following Object Stereotypes.

The Coordinator: Coordinators are one of two objects most responsible for determining how control works in our applications. The Coordinator stereotype has one real purpose: to pass information to other objects so that they can do work. This is effectively the Observer Pattern. While it’s common practice to use something like Redux to create this, it’s not necessary. You can have something like this working in less than 20 lines of code.

The Controller: In RDD, the Controller stereotype behaves the same way as the Use Case pattern. The stereotypical responsibility held by a Controller is to act as a higher-level (application-layer specifically), declaratively focused object that coordinates the behaviour of lower-level concerns. The Controller pulls other objects together and makes them collaborate with each other. They can be modelled as Features.

No title

Thu, 04 Aug 2022 04:05:26 GMT

No title

Thu, 04 Aug 2022 04:05:26 GMT

No title

Thu, 04 Aug 2022 04:05:26 GMT

No title

Thu, 04 Aug 2022 04:05:26 GMT

What Is Hashing?

Tue, 10 May 2022 15:04:10 GMT

Hashing is a technique used to convert a key into another value (typically for cryptographic or data storage purposes). It works by running a mathematical function (called a hash function) called a key to create a new value — the hash (or the hash value).

One-way hashes

Cryptographically secure hash functions are said to be good when they’re one way (one way hashes). This means that you cannot get the original key from the hash value.

Collisions

If a hash function yields the same hash value for two or more keys, then that’s when we end up with a collision — this is non-ideal. There are ways to handle this (see this article).

When we do we hashing?

Implementing hash tables

Hash tables are a data structure that helps us store key/value data. They are comprised of two parts — a storage structure (be it an array, object, etc) and a hash function.

The hash function is responsible for fitting data within the data structure so that it can be retrieved later.

If the storage is bounded (fixed) such as an array, then the hash function must generate a key which can be used as the index, but fits within the size constraints of the array.

For example, a trivial formula to compute the index of a string or integer key is to do:

index = key % sizeOfTable

This would work, but for any fixed size table, we’re likely to run into collisions eventually.

See here for “Collision detection and dynamic array resizing".

When storage is unbounded — like when using a JavaScript object — we don’t have to worry about collisions. We can just use some unique aspect of the data (such as an email address, phone number, or a compound-key) to create the hash value to use as an index.

However, if security is important, we should still use a cryptographic function to create the hash.

That leads to the final point.

Cryptography & data encryption

Hashing is a very common technique when it comes to cryptography. The most common reason we hash is to convert passwords to something more secure (than plain text) before we store them to a database. We do this for two reasons:

If a hacker steals the records from the db, they don’t get the actual passwords — just the hashed ones (which cannot be reversed since they were one way hashes).
To prevent staff from accidentally viewing plain-text passwords in the production database.

To hash passwords, check out MD5 or SHA-2 — two widely-used crypto hashes.

Non-Functional Requirements (with Examples)

Tue, 03 May 2022 22:04:10 GMT

What’s the difference between software that works and software that works well? Non-functional requirements.

Consider the following example.

We’ve been hired to build an online store for a local brick-and-mortar store. We’ve gone ahead and applied the feature/use case-driven approach to design popularized by XP. We’ve done that process of identifying the use cases, planning and estimating them with user stories, and thinking about how we can write those stories as tests with BDD. Looking back at what we’ve done, we’ve acceptance tested the use cases, integration tested the infrastructure code that interacts with external APIs and the outside world, and even did a little bit of end-to-end testing to ensure things work from front to back.

Great job!

Unfortunately, upon going live, we’re surprised hear about a myriad of complaints from the customer.

The customer calls us and says:

“The cart page takes around 10 seconds to load — it’s way too slow! If it’s already this slow under regular traffic, I’m not sure what’s going to happen when we have our boxing day sale. Will the site just go down completely?”
“We keep having to refund orders! Customers are purchasing products that are out of stock, so I’m not sure how you did the sync to our backend inventory but it’s either taking too long to sync or it’s not syncing at all.”
“The site looks weird on mobile.”
“Customers are confused about how to get to the page where they can see the status of their orders.”
“The animations moving between pages feels kind of choppy. I was expecting it to be smoother. Can you fix that?”

Dang. Where did we go wrong?

Well, behind each of these disgruntlements lies a missed requirement — a non-functional requirement. And non-functional requirements are an incredibly important piece of the puzzle that is software design.

What we’ll learn

In this post, we’re going to discuss:

What non-functional requirements are and why knowing them can help us better design systems that meets the needs of the customer and end users.
How non-functional requirements compare to functional ones.
Best practices to handle non-functional requirements.

What is a non-functional requirement?

Where functional requirements describe what the system should do (ie: the use cases or features), non-functional requirements (sometimes called constraints or behavioural requirements) describe what the system should be (ie: the system level properties).

With functional requirements, if we’re following the feature-driven folder structure, it can be relatively easy to trace features down to individual slices of functionality or modules within our application.

Non-functional requirements, however — these are system-level properties. Since they describe the system as a whole it’s a lot harder (and sometimes impossible) to trace them down to a location in the codebase.

Types of non-functional requirements

There’s no one single definitive list of valid non-functional requirements. Some of them can also been seen as related or within umbrellas of each other (for example: we can define Performance or Speed as NFRs within Efficiency).

To get us started, here’s a short list.

Capacity — How much data can we store? Will we ever run out of storage space?
Regulatory — Do local, regional, provincial or state laws or regulations that dictate some aspect of what we’re legally allowed to do?
Portability — Which browsers, operating systems and versions should we expect the software to work on? Is it cross-platform?
Reliability — How often can we expect the system to be available to use? How does it handle critical failures? How long does it take for a system to come back online after a failure?
Performance — How fast should requests process? How many requests can we handle at any given time? What should we expect for response times when the system is under heavy load?
Localization — Will the system adapt to measurement systems, timezones, and other locally specific concepts in other states or provinces?

Non-functional requirement examples

Lets say that a functional requirement in a trading application was to make sure that when users execute the getOffers use case, they’re presented with a list of offers, then a larger-picture non-functional requirement may state:

Scalability, performance, efficiency: “The website should process each request within 4 seconds or less 99% of the time.”
Scalability, performance: “In standard network conditions, the website should fully load in less than 5 seconds when the total number of simultaneous users on the website are greater than 50,000.”
Security, data integrity, capacity: “Every unauthorized request to a resource must be logged and stored for audit over the next 5 years.”

Execution vs. evolution qualities

Because there are so many different non-functional requirements out there, a great way to think about them is to consider them to be either either execution qualities or evolution qualities. Allow me to explain the difference.

Execution qualities: These are aspects which can only be observed at runtime such as security, usability, and efficiency (performance, speed).
Evolution qualities: These are aspects that have to do with how it affects our ability to evolve the code and structure over time such as if the code is testable, flexible, maintainable, and scalable.

Functional vs. non-functional requirements

Let’s make sure we really understand the differences between the two.

	Functional	Non-functional requirement
Description	Describes what the system should do	Constrains the functional requirements and describes “how the system should be”
Mandatory	Yes	No, but some are more important than others
Scope	As a use case / feature - component level	As a system-level property against the system as a whole
Ease of definition	Straightforward	More difficult (it can be difficult to make qualitative properties quantitative — we sometimes have to use indicators instead).
Test types	Unit, integration, acceptance, end-to-end tests	Performance, stress, security, usability testing (for example)
Example	Given a user has not created an account, when they register, a new users should be created and an email should be sent to them.	(1) The site should be able to process 99.9% of requests in less than 6 seconds. (2) Emails should be sent no later than 20 minutes after the triggering event.

Why should we learn about non-functional requirements

My first startup project was used by few hundreds users. It wasn’t until it received a moderate amount of press on a popular website that I learned it was not cut out for real-world production traffic. With thousands of people signing up at the same time, the website — hosted on a tiny AWS EC2 micro instance, unable to stand up to the performance, efficiency, and reliability non-functional requirements — died a tragic death and left many users annoyed... never to return again.

It’s one of the main reasons why projects fail

Often times, even if we get all the features working, non-functional requirements are the key factor that dictate if a project will succeed or fail. It’s taken me some time to realize this, but it’s very much true.

What good is a slow social networking website? Who’s going to play a game that crashes all the time? Is the project really a success if it’s written without tests and every time we add or change a feature, we fear breaking an existing feature in a distant module? What if people don’t even know how to use it?

It has a massive influence on architecture (execution qualities)

Non-functional requirements provide hints to what sorts of architectural components (like queues and caches) and patterns (design patterns and architectural patterns) we’ll need in order to successfully get the system to be a certain way — to have the design criteria we need.

For example, a media-heavy site like Instagram or YouTube allows users to upload media. That’s the functional requirement. Non-functionally though, we’d like to afford users the capability to continue to move around the site while their media is being uploaded.

This simple usability non-functional requirement introduces the need for some component capable of decoupling the user-facing command of UploadMedia from the work to be done (such as processing the media, encoding it, annotating it, adding subtitles, etc). Therefore, it’s likely we’d benefit from an event-based architecture and the addition of a Job/Task Queue as an important architectural component.

Filling in the non-functional gaps with Object Design: Do I just automatically know what sorts of components we need? While a familiarity of patterns, tools, and services and the sorts of problems they solve helps, there is a structured design method to discover what sorts of components need to either be invented or integrated into our application architecture. It’s called Object Design (or as I like to say how to do object-oriented programming properly). Object design teaches us how to discover and divvy up the necessary requirements (both functional and non-functional requirements) into neighbourhoods of collaborating objects. In Part V — Object Design with Tests from solidbook.io, we learn Responsibility-Driven Design and foundational skills for system design.

Makes good design techniques necessary (evolution qualities)

A large reason why we use things like:

Tests (acceptance, E2E, integration, unit)
Design patterns
Design principles
Architectural styles
Architectural patterns
Domain-driven design

... is because they lead to more testable, flexible, and maintainable code. The evolution qualities. That’s pretty much the goal for us web developers focused on taming backend complexity. As a result, following the many good coding practices listed above leads to code can that can be more easily evolved, lives longer, is more understandable, and so on.

Best practices for dealing with non-functional requirements

Uncover non-functional requirements early: Because of the impact NFRs have on architecture, it’s important to discover them early. Once you know the functional requirements, ask the customer (or yourself) probing questions to determine what non-functional requirements exist to constrain the functional ones. How fast should it be? How do users expect this to behave? What will happen if users have a lot of data in their account? Are there auditing requirements? What could make this project fail? What does affordability look like for you? How easy will it be to change? Can we expect users to know what to do?
Make requirements as measurable as possible: Here’s an example of a non-functional requirement that you can’t really test or measure: “it should be fast as hell”. What if we made that quantitative and said “it should respond to 95% of requests within 3 seconds”? That’s better, isn’t it?
Use popular products as a quality baseline: Every couple of years, users come to expect better website experiences. Faster, more real-time, more of that wow-factor and so on. To get a baseline for the non-functional requirements, ask customers to tell you what they like from other websites. It could be just that it feels consistent or has a lot of whitespace. These are all constraints on the functional requirements.

Summary

Non-functional requirements describe what the system should be instead of what the system should do. It’s about system-level properties.
Non-functional requirements can be divided into two rough categories: execution qualities and evolution qualities.
We should learn non-functional requirements because failing to meet them is one of the main reasons why projects fail, they have a massive influence on architecture, and they make a solid case for good coding practices like TDD and DDD and more advanced concepts like design patterns and architectural styles.

References

JavaScript forEach | Looping Through an Array in JS

Tue, 03 May 2022 15:04:10 GMT

For many developers, JavaScript acts as introduction to the functional programming paradigm. And if you've never encountered callback functions before, the JavaScript for each function might look a little funny.

In this quick article, we're going to see how to iterate over an array of data using JavaScript's forEach function. We'll look at different ways to call it and discuss when to use it vs. the traditional for loop.

Using the JavaScript for each function

In JavaScript, the array object contains a forEach method. This means that on any array, we can call forEach like so:

let fruits = ['apples', 'oranges', 'bananas'];

fruits.forEach(function (item, index) {
  console.log(item, index)
})

This should print out the following.

apples 0
oranges 1
bananas 2

Let's explain how this works.

We pass an anonymous function to the forEach method.
Within that anonymous function, we include parameters to get access to the current item and the numerical index in the array that the item represents.

The function we provide to the forEach function has three parameters (two of which are optional).

(Required) The current element - Represents the current element
(Optional) Index - Returns the current index
(Optional) Array - Returns the entire array for each loop

Alternate ways to call it

Option 1: An anonymous function

The first of the alternate way to call this function is to utilize the anonymous function. This is what we just saw.

arr.forEach(function (item) { // Anonymous function
  // Write what you'd like to happen here
})

Option 2: Using arrow functions

The second option is to utilize an arrow function.

arr.forEach((item, index) => console.log(item, index))

Try this out. It allows you to write the loop on a single line.

Option 3: A callback function

The last option is to utilize a callback function, which is really just a function defined separately that we pass in as an argument to the forEach function.

let fruits = ['apples', 'oranges', 'bananas'];

function printPretty (item, index) {
  console.log(`${index} - ${item}`)
}

fruits.forEach(printPretty);

This should print out the following:

0 - apples
1 - oranges
2 - bananas

When to use the for loop vs. a for loop

Most of us first learn about loops the traditional way - you know, the loops with counters that get incremented? Like this:

let fruits = ['apples', 'oranges', 'bananas'];

for (let index = 0; i < fruits.length; i++) {
  console.log(fruits[i]);
}

The truth is, you can use either traditional for loops or forEach loops. It doesn't matter.

However, using forEach loops results in code that looks a lot more declarative and easy to read, especially if you're dealing with nested loops (which I opt to try to avoid in the first place).

Responsibility-Driven Design

Sat, 16 Apr 2022 22:04:10 GMT

🌱 This blog post hasn't fully bloomed. It's very likely to change over the next little while.

“Understanding responsibilities is key to good object-oriented design” — Martin Fowler

Do you struggle with object-oriented programming? Does it sometimes feel like it's overly complex for no reason? When you're writing object-oriented code, do you sometimes feel like you're slowly watching it turn into a spaghetti mess? Do you sometimes feel confused as to when you should create new classes? Do you feel like it might be better to just use functions instead?

What if I told you that there's more to building software with objects than just classes, inheritance, polymorphism and encapsulation?

What if I told you that there's a design method - a way of turning requirements into object-oriented designs - that actually

And what if I told you that it was the programming world's most influentual design method (that all the expert object designers are using) that you've likely never heard about.

About Responsibility-Driven Design (RDD)

Allow me to introduce you to Responsibility Driven Design (RDD): the programming world’s most influential, effective, and intentional approach to designing object-oriented software. I know, big claim. But it’s true.

My copy of the legendary Object Design book. My second favourite blue programming book. This book is so rare that it's nearly $200 USD nowadays!

RDD, originally invented by Rebecca Wirfs-Brock in the late 80s and modernized in her 2002 book titled Object Design is a learnable and repeatable design method for building object-oriented applications.

Where most of us today write and think in procedural style code using classes and functions because it's often what the languages dictate, Responsibility-Driven Design presents an approach to actually design objects with purpose and intention.

Roles, responsibilities, collaborations

Object software is neighborhoods of collaborating objects which (polymorphically) come and go, each filling in for roles with related responsibilities.

Role: A collection of related responsibilities
Resoponsibility: An obligation to do (some task/behavior) or know something (some data)
Collaboration: An interaction of objects or roles (or both)

...

To be continued...

Check out the wikipedia page on Responsibility-Driven Design and feel to leave comments or questions for when I finish this post.

Highly recommend reading this page and watching this talk.

4 Principles of Object-Oriented Programming

Sat, 16 Apr 2022 22:04:10 GMT

Even though I studied object-oriented programming in university, I still struggled with it once I got out into the real world and tried using it to develop software.

I think this has to do with the fact that I was unaware that object design is actually comprised of three parts:

analysis: where we learn a domain, discover functional and non-functional requirements, and turn them into use cases
design: where we utilize a design method like Responsibility-Driven Design to convert requirements into roles, responsibilities, and collaborations, we consider design patterns and their tradeoffs, we make architectural decisions - deciding on archetectural styles and patterns
programming: where we map designs into tested, flexible, maintainable code

Object design is comprised of three parts: Analysis, Design, and Programming

Only learning the programming aspect, I found myself confused about what good design looked like, how to turn requirements into objects, and when and why to use concepts like abstraction, encapsulation, inheritance, and polymorphism. Yes, I could somewhat remember what these concepts were, but knowing when and why to use them was unclear to me.

The four main principles of object-oriented programming (abstraction, inheritance, encapsulation, and polymorphism). The core principle is abstraction. Without it, the others couldn't exist.

In this post, I want to revisit these four main ideas — these principles of object-oriented programming — discuss why they’re beneficial and explain them with simple, relatable, and practical explanations.

Abstraction

Are you really aware of how your TV turns on when you press the ON button on the remote? Do you, as a user, need to know the specific sequence of 0’s and 1’s that your remote control emits to signal to the television’s receiver that it should turn on? Or is pressing the ON button good enough?

Are you really aware of how your car starts when you turn your key in the ignition? Do you, as a driver, need to know about the ignition switch, how the voltage from the battery hits the ignition coil, how an engine spark gets directed to the spark plugs, and how that ignites the fuel to make the car run? Or is turning the key and hearing a ding-ding enough?

Abstraction makes technology easier to use. Most of us know that pressing the ON button on the remote will turn on a TV. And that's good enough for us. Imagine that you needed to know the low-level electronic details in order to turn your TV on to watch your favourite HBO show. The learning curve would be tremendous. Very few people would watch TV if that were the case.

It should be clear that we maneuver the world using abstractions — conceptual models that give us a good enough understanding of inventions that are much more complex behind the scenes. And in object-oriented programs, the inventions that we aim to simplify and make easy to use are objects.

Abstraction is a key principle of design. Since object-oriented programs can get quite large, it’s just not humanly possible to expect others to be aware of the inner-workings of each and every class or component within them. Therefore, we strive to abstract objects in a way that makes them intention-revealing, simple to use, and simple to understand.

Consider the abstraction of a washing machine.

// Options for the wash cycle
type WashOptions = {
  dryLevel: 'low' | 'medium' | 'high'
  temperature: 'cold',
  duration: 'hour',
  ecoEnabled: false
}

// The abstraction
class WashingMachine {
	
  // Private instance variables
  ... 

  public startCycle (options: WashOptions): void {
	  // Parse the options
    // Get access to the physical layer
    // Convert options into commands
    // Lots of low-level code
    // And so on...
    ...
  }
	
  // More methods
  ...

}

Assuming that the only public method right now is the startCycle method, what else does the client need to know? Does it need to know how startCycle works? Does it need to know about the private instance variables within the class? Does it need to know about any other private methods?

No, absolutely not. Just like the power-button on the TV remote, it’s simplified for public use. For someone to use this abstraction, all they need to know is the existence of the startCycle method and how to call it. That’s it. All other details are abstracted away within the class.

// Obtain access to the abstraction
import { WashingMachine } from '../washer'
 
// Create an instance
const washer = new WashingMachine();

// Usage
washer.startCycle({
  dryLevel: 'medium',
  temperature: 'cold',
  duration: 'hour',
  ecoEnabled: false
});

By keeping what’s not necessary for the client to know abstracted away, we reduce confusion and make it easy for other developers to know how to use our code and extend it.

Abstraction is the root principle of design

Abstraction has many uses — for both the designer and the client. There is much more that can be said on abstraction and abstraction tools like interfaces, abstract classes and types, such as the fact that they give us:

The ability to design using contracts vs concretions.
The ability to separate the what from the how.
The ability to focus on the declarative separately from the imperative.
The ability to focus on the high-level ideas separately from the low-level details.

But for now, I’ll say that it’s the root principle of design and a central point of focus for many other important topics like design patterns, principles, responsibility-driven design and more.

Bonus: Alan Watts on Abstraction (and Thought)
+

From Alan Watts' lecture titled "The Silent Mind"

"To think, we use a series of images or very simple grasps of the world, and these are abstractions."

Also see

Abstraction is the macro-level principle. Here are a few micro-level topics which are directly related.

Humans & Code: How can we tell if our projects are pleasant to work with? We need to understand how we understand. This section of solidbook introduces the principles of Human-Centered Design and techniques to develop more human-friendly codebases.
Maintain a Single Layer of Abstraction at a Time: Keep your methods cohesive and readable by maintaining a single layer of abstraction at a time. Understand what responsibilities for doing and knowing your class has and stick to it.
Minimal Interface: Boil the public API (of a class, component, or service) down to the smallest set of operations possible that lets people do what they need to do. May not be the most human-friendly, but operations can be combined in ways that let clients accomplish their goals.
Humane Interface: Find out what people want to do and then design the interface in a way that represents it in the most human-centered language possible.
Thin Interfaces, thick implementations: Stanford Computer Science Professor John Ousterhout's idea is design thin interfaces with thick implementations. This idea further reinforces that abstraction is about making interfaces easy to understand and use while hiding away complex details.

Inheritance

Wouldn’t it be nice if we could reuse code or extend it for more specific use cases? That’s what inheritance is about.

To go about code reuse, in TDD, we have this technique called the Rule of Three which specifies that upon seeing duplication three times, we should refactor it into an abstraction.

The classic example I use to demonstrate is one where we rely on several different API endpoints from a front-end application but we duplicate the fetching logic across each API adapter (think /users, /forum, /billing, etc).

Instead, what we could do is refactor the data-fetching logic into a common place. The abstract class is a good tool for this use case.

// Base API
export abstract class API {
  protected baseUrl: string;
  private axiosInstance: AxiosInstance;
   
  constructor (baseUrl: string) {
    this.baseUrl = baseUrl
    this.axiosInstance = axios.create({})
    this.enableInterceptors();
  }
  private enableInterceptors () {    // Here's where you can define common refetching logic  }
  // Common "get" logic
  protected get<T> (url: string, requestConfig?: RequestConfig): Promise<AxiosResponse<T>>{    return this.axiosInstance.get<T>(`${this.baseUrl}/${url}`, {      headers: requestConfig?.headers ? requestConfig.headers : {},      params: requestConfig.params ? requestConfig.params : null,    });  }
  // Common "post" logic
  protected post<T> (url: string, requestConfig?: RequestConfig): Promise<AxiosResponse<T>>{    return this.axiosInstance.post<T>(`${this.baseUrl}/${url}`, {      headers: requestConfig?.headers ? requestConfig.headers : {},      params: requestConfig.params ? requestConfig.params : null,    });  }}

And then, for each API adapter we need, we can merely subclass the base API abstract class to get access to the common functionality (public or protected methods and properties).

// Users API
export class UsersAPI extends API {
  constructor () {
    super('http://example.com/users');
  }
 
  // High-level functionality (extending it)
  async getUsers (): Promise<Users> {
    let response = await this.get('/'); // Use the common logic from the base class    return response.data.users as Users
  }
  
  ...
}

// Forum API
export class ForumAPI extends API {
  constructor () {
    super('http://example.com/forum');
  }
 
  // High-level functionality (extending it)
  async getPosts (): Promise<Posts> {
    let response = await this.get('/'); // Here, as well    return response.data.posts as Posts;
  }
  
  ...
}

// Billing API
export class BillingAPI extends API {
  constructor () {
    super('http://example.com/billing');
  }
 
  // High-level functionality (extending it)
  async getPayments (): Promise<Payments> {
    let response = await this.get('/'); // And here ;)    return response.data.payments as Payments;
  }
  
  ...
}

Inheritance relies on the principle of abstraction; with it, we gain the ability to abstract away (duplicated) low-level details to the base class (API) so that the subclasses can focus on the (unique) high-level details (UsersAPI, ForumAPI, BillingAPI).

A good reason to refactor using inheritance is to clean up duplication and make generic functionality easier to use for more specific objects.

It’s about reuse, not hierarchies

When most developers learn about inheritance, they’re presented with trivial Animal-Dog or Person-Staff-Teacher examples which map nicely to the real world but manifest real problems with the complex hierarchies they create.

It is very important to note that concepts which exist in the real world often do not translate into useful software objects. For example, it’s probably not important to model the hierarchy of a car’s speed, dimensions, upgrades, parts, and window tint within the context of an online oil change booking form. However, in the design of a multiplayer racing game, they may very well be useful and necessary.

Why do we invent abstractions anyway? What's the point? To help us express what we need to express to meet the requirements, and to do so in a maintainable way — that’s it. As I've written about in solidbook.io, the goal of software design is to write code that:

Meet the needs of the customer and

Is cost-effectively changed by developers

Inheritance used properly helps #2. Therefore, inheritance is a tool we use upon finding opportunities for reuse, not to express real-world hierarchical similarities.

"At the heart of object-oriented software development there is a violation of real-world physics. We have a license to reinvent the world, because modeling the real world in our machinery is not our goal."

— Rebecca Wirfs-Brock via Object Design [2002]

Composition & delegation

If inheritance is really just about reuse (and not about world-building), then that means we can skip the hierarchy entirely and use the techniques of composition and delegation to implement inheritance.

That means we could also refactor the duplication of the HTTP logic like this.

class UsersAPI {
	
  // Compose using dependency injection
  constructor (http: API) {
    this.http = http;
    this.baseURL = 'https://example.com/users'
  }
  
  getUsers () {
    return this.http.get(this.baseURL)
  }

  ...

Therefore, if we encounter duplication, we have a lot of refactoring techniques:

Inheritance (Extract superclass): When you have two or more classes with common methods or fields, create a superclass and inherit the shared behavior.
Composition (Extract class): When one class does work for two or more classes, extract the related methods to its own class and compose your reliant classes with the new one.

Rule of thumb — only inherit from contracts, not concretions: If you’re going to use inheritance the classic OO way, I generally advise that you only inherit or extend from contracts (interfaces, abstract classes, types) and not from concretions (classes). There’s a design method which explains why this rule makes sense (see Responsibility-Driven Design), but as a quick rule of thumb, you’ll know if you’re using inheritance poorly if you find yourself needing to subclass a concrete class.

Encapsulation

Objects contain state, make decisions based on that state, and relay messages to other objects asking them for help to fetch or compute things, like a network of routers or a web of atoms.

Because objects make decisions (and enact business logic) based on their state, it’s important that each object manage its own state and protect against mutations from outside classes. Failure to do so can lead to strange behavior, bugs, and an anemic domain model.

Encapsulation is the technique of making state private. Encapsulation means that the class which owns the state decides the degree to which state can be accessed and changed. This is done through the use of public methods.

Let’s continue to demonstrate with the washing machine example. Depending on the washing machine’s current state, some behaviours are valid while others are invalid.

For example, if the machine is currently ON , it is valid to call turnOff() to turn the machine OFF. If the machine is in the WASH state, it’s OK to call pause() to put the machine into the PAUSED state. But if the machine is OFF (or ON for that matter) it would not be valid to call PAUSE.

Why? Because a washing machine can’t go from being OFF to PAUSED. It doesn’t make sense.

Encapsulation is the mechanism that governs these rules (also known as class invariants) for us.

type WashState = 'OFF' | 'ON' | 'WASH' | 'PAUSED' | 'DONE';
type Cycle = undefined | 'COOL' | 'WARM';

type MachineState = {
  washState: WashState;
  currentCycle: Cycle;
}

class WashingMachine {
  // Encapsulation of state is achieved here by making
  // state private and inaccessible directly.
  private state: MachineState;
	
  // Access is provided via public methods
  public getWashState () : WashState {
    return this.state.washState;
  }
  
  public getCurrentCycle (): Cycle {
    return this.state.currentCycle;
  }

  // State is changed only through the use of public
  // mutator methods
  public pause (): void {
    // Note this important business logic which is encapsulated
    // to the correct place. The related data and behavior live
    // together.

    // We ONLY do the pause logic if the current state is
    // 'WASH'
    if (this.state.washState === 'WASH') {
      // Stop washing
      ...
      // Cue pause sound
      ...
      // Set new state
      this.state.washState = 'PAUSED'
    }
  }

	...
}

Given this, client usage may look as follows:

washer.startCycle({
  dryLevel: 'medium',
  temperature: 'cold',
  duration: 'hour',
  ecoEnabled: false
});

console.log(washer.getWashState()); // 'WASH'
washer.pause();
console.log(washer.getWashState()); // 'PAUSED'

Encapsulation combines together the data and related behavior which — like abstraction — simplifies usage, but also acts as a single place for us to put relevant logic. With this technique, we invent single sources of truth and obtain appropriate places to locate logic once and for all.

Polymorphism

Polymorphism is about “taking on many shapes”. It's about designing your use cases and algorithms in such a way that we always get the same high-level behavior, but we allow for dynamic low-level behavior at runtime.

I still think the best real-world examples I can provide are either:

Regardless, let's demonstrate with an example that will let me introduce you to the idea of roles from Responsibility-Driven Design (the actual way to do OO correctly as far as I know). Think about the role of a clerk at a grocery store. What’s a clerk supposed to do? Answer questions, scan products, bag your items, and prompt you for payment, right? So you could say those are the responsibilities of a clerk.

A clerk CRC card that defines the role of a clerk, the related responsibilities and the collaborations it has.

Now let’s say that there are 4 different employees that work at this grocery store. Each one can take on the role of a clerk. There’s James who is generally pretty snappy, Mariah who can be somewhat brooding, Max who is always on time (and the friendliest person you’ve ever met), and Mark who is actually the manager but will sometimes fill in when they’re short staffed.

It’s clear that each of the employees does their job slightly differently, but they still perform the responsibilities that a clerk should perform. That is, they are concrete classes which implement the contract of a Clerk.

Let's represent the contract using an interface.

interface Clerk {
  answerQuestion (question: string): Answer;
  scanProduct (product: Product): void;
  bagProduct (product: Product): void;
  promptForPayment (customer: Customer): Promise<PaymentResult>
}

Then, we could realize the concretions by implementing the Clerk interface, handling things with slighly different behavior if we wish.

class ClerkJames implements Clerk {

  answerQuestion (question: string): Answer {
    ... // Implement uniquely to James
  }

  scanProduct (product: Product): void {
    ... // Implement uniquely to James
  }

  bagProduct (product: Product): void {
    ... // Implement uniquely to James
  }

  promptForPayment (customer: Customer): Promise<PaymentResult> {
    ... // Implement uniquely to James
  }

}

class ClerkMariah implements Clerk {

  answerQuestion (question: string): Answer {
    ... // Implement uniquely to Mariah
  }

  scanProduct (product: Product): void {
    ... // Implement uniquely to Mariah
  }

  bagProduct (product: Product): void {
    ... // Implement uniquely to Mariah
  }

  promptForPayment (customer: Customer): Promise<PaymentResult> {
    ... // Implement uniquely to Mariah
  }

}

// Implement the others

And then finally, in the code that relies on a Clerk, we'd provide a Clerk and a Customer to check out.

function checkoutCustomer (clerk: Clerk, customer: Customer): Promise<PaymentResult> {
  for (let question of customer.getQuestions()) {
    clerk.answer(question);
  }

  for (let product of customer.getProductsInCart()) {
    clerk.scanProduct(product);
    clerk.bagProduct(product);
  }

  await clerk.promptForPayment(customer);
}

Note that this function relies on a Clerk, not a ClerkJames or a ClerkMariah (or so on). Therefore, it doesn’t matter who shows up to work to fill the role. It could be James, Mariah, Max, or Mark — doesn’t matter. All that matters is that some object with the role of clerk is supplied and that fully implements the requirements of the contract for a Clerk:

let customer: Customer = { ... };

let clerkJames: ClerkJames = new ClerkJames();
let clerkMariah: ClerkMariah = new ClerkMariah();

checkoutCustomer(clerkJames, customer); // Valid
checkoutCustomer(clerkMariah, customer); // Also valid since it's a `Clerk`

Do you see how this works? By relying on a contract instead of a concretion, we gain the ability to sub in for different possible implementations.

This is what polymorphism is about. Dynamic runtime behavior. Substitutability.

What I want for you to take away from this example is that the idea of relying on roles/contracts can be very effective. Within the context of a clean/hexagonal/ports and adapters architecture, the most common reasons you'll do this is to:

Keep core code separate from infrastructure code using dependency inversion
Keep application layer use cases pure so that they can be unit tested. Swap out an infrastructure dependency (StripeAPI, PaypalAPI) with a test double (MockPaymentAPI) so that we don't make real calls to external services or infrastructure during testing.

Also see

React components are an example of polymorphism: I've mentioned it before, but in React, the Component class is an example of polymorphism. By extending the component class and implementing the render() method, we create dynamic runtime behavior (where the dynamic behavior is how our components get rendered). Read more here.

Rule of thumb — switch statements (conditionals) can be refactored to polymorphism: Most of the time, if we’re dealing with three or more conditionals in a particular block (and we expect to add more conditions), it could be a good idea to refactor to polymorphism. Extract the generic/common behavior to a contract (abstract class or interface) and put the specific behavior in subclasses. Then program against the contract and allow dynamic runtime binding to work its magic.

Conclusion

Abstraction, encapsulation, inheritance, and polymorphism are four of the main principles of object-oriented programming.

Abstraction lets us selectively focus on the high-level and abstract way the low-level details.
Inheritance is about code reuse, not hierarchies.
Encapsulation keeps state private so that we can better enforce business rules, protect model invariants, and develop a single source of truth for related data and logic.
Polymorphism provides the ability for us to design for dynamic runtime behavior, easy extensibility, and substitutability.

What are Dependencies, Composition, Delegation, and Aggregation in Object-Oriented Programming?

Sat, 16 Apr 2022 22:04:10 GMT

Dependency: The Address object comes from outside, it's allocated somewhere else. This means that the Address and Employee objects exists separately, and only depend on each other.

Composition: Here you see that a new Engine is created inside Car. The Engine object is part of the Car. This means that a Car is composed of an Engine.

Composition is about the relationships between objects.

Delegation is about passing work from one object to another.

These are actually different (but sometimes related) concerns.

What you've got is B composed of A (B refers to A).

Aggregation & composition: An aggregation is an object that represents a concept as a whole and handles managing how its parts are accessed and changed. Composition is related but it adds an additional meaning to aggregation. Composition says that parts of the aggregation can’t exist independently (ie: a wheel can’t exist unless it belongs to a car).

So dependency == aggregation? stackoverflow.com/questions/11881552/… – danihodovic Jan 9, 2014 at 15:04 @dani-h No, aggregation and composition describe how things are build/structured, while dependency is more a property of a certain structure. See @TheLostMind's great answer

Is DDD Overrated? | Domain-Driven Design

Fri, 08 Apr 2022 15:04:10 GMT

Is Domain-Driven Design (DDD) overrated?

In my opinion, you can only say DDD is overrated once you've achieved mastery over it. At this point, you can toss it aside. All the tactical patterns, techniques, approaches — you've already absorbed the wisdom. You're the boss. You don't need rules, cerimonies, other peoples' best practices. You set your own rules. Because you understand design, you know what's good, bad, too much design, too little, what the correct amount thereof is (see the Golden Mean) — you now have room to experiment and get creative.

If you're an experienced developer (perhaps like the author of this excellent post of the same title), you've likely witnessed the rise of use cases, responsibility-driven design, and have experience with various approaches to object-oriented and functional software.

For you, yes. DDD may very well be overrated.

But for everyone else — the junior developers, the fresh developers, the devs who don't know software design, the devs still struggling to learn where to put their business logic — DDD is a game-changer.

To quote Eric Elliot,

"99% of fresh CS grads don't know the basics of software design".

Since DDD is a very human approach to design, developers w/o software design experience who study it gain two things:

How to write code for humans
A shortcut to some of the most important design concepts

That being said, we should eventually leave DDD and look beyond it to fully understand software design, otherwise we treat DDD as a sort of dogma.

A developer will read the DDD material and think, "Well that's it. I've learned the secrets." When all they've learned is the rudiments of one angle on software design. - via Scott Bellware

Other great approaches exist as well, like Responsibility Driven Design... and if you look closely, you'll see a lot of similarities to DDD (look into Object Stereotypes and see how a Value Object or an Entity is essentially an Information Holder object stereotype).

What do these similarities tell us? That we're close to some ineffable truths or laws of design. Or at least a little further into our discovery of the mystery that is software design.

Summary

In summary,

Every developer should learn DDD
Every developer should then learn the truths of software design that make DDD effective
Every developer should then abandon anything cerimonious about DDD and develop their own style based on their own experience

Update: I'm Teaching Software Design & Architecture Full-time @ khalilstemmler.com

Wed, 06 Apr 2022 15:04:10 GMT

Well, everybody. The time has come.

March 31st, 2022 was my last day at Apollo GraphQL. After about 3 years, countless blog posts, 4 talks, 3 trips, I'm stepping away from my Apollo fam to focus on teaching devs how to write testable, flexible, maintainable code.

I'm going to miss the team; and while I believe that the supergraph is seriously the future, duty calls. I'm returning to focus primarily on my book, courses, blog, videos, and open source work.

Why I started this blog

I started this blog in 2019 after having successfully tanked the quality of the code for my own startup company called Univjobs.

Freshly graduated from university, I was surprised at how ill-prepared I actually was in the craft of software design.

Not knowing how to organize business logic, write tests, or write code in a way that it gets better instead of worse after time, I sought out answers. I decided that I was going to study, practice, and master the craft of software design.

Best hits

The most popular articles I've written from 2019-2021 are:

solidbook.io

Documenting my learnings here on the blog and within solidbook.io (a guide to software design and architecture), over 2500 developers have purchased the book!

I've been blown away by the feedback I've received even though it's only about halfway done (and only a first draft)! Not only that, but the open source project we build in the book, DDDForum, has over 1200 stars and 240 forks.

Thank you everyone for your support, feedback, and PATIENCE - especially you early book readers. I appreciate you tremendously.

To mastery

“Rule of thumb: The more important a call or action is to our soul's evolution, the more Resistance we will feel toward pursuing it.” ― Steven Pressfield, The War of Art: Winning the Inner Creative Battle

Complexity is chief antagonist of well-crafted software. A naturally occuring phenomenon, if we do nothing about it, it will only get worse. To you, the reader:

who is learning how to write code that fans the flame of complexity
who is thinking strategically about feature-development because you know that "just making it work" once != good code
who is learning how to use tests to gain consistency and reliability in their work

To you developers - you who feel the pull...

... where are we going?

To mastery.

Hash Tables | What, Why & How to Use Them

Wed, 19 Jan 2022 15:04:10 GMT

On this site, we're primarily concerned with software design, architecture, and the many problems that occur later on in your developer journey, like keeping code clean, handling complexity, and writing testable code.

Early on, when you're primarily concerned with learning the ins and outs of a language, you'll find yourself also need to master the algorithmic basis of software development — code in methods and functions. One of the most integral tools to solving these problems? Data structures.

I want to talk about one of the most important data structures: the hash table.

In this article, we’ll begin by learning what hash tables are and why you'd want to use them. Then we’ll look at different hash tables implementations and closely related data structures. Finally, we wrap with a discussion of which hash table implementation you should reach most of the time and discuss some of the most common use cases for hash tables.

What is a hash table?

A hash table is a data structure that you can use to store data in key-value format with direct access to its items in constant time.

Hash tables are said to be associative, which means that for each key, data occurs at most once. Hash tables let us implement things like phone books or dictionaries; in them, we store the association between a value (like a dictionary definition of the word "lamp") and its key (the word "lamp" itself).

We can use hash tables to store, retrieve, and delete data uniquely based on their unique key.

Why use hash tables?

The most valuable aspect of a hash table over other abstract data structures is its speed to perform insertion, deletion, and search operations. Hash tables can do them all in constant time.

For those unfamiliar with time complexity (big O notation), constant time is the fastest possible time complexity. Hash tables can perform nearly all methods (except list) very fast in O(1) time.

Algorithm	Average	Worst case
List	O(n)	O(n)
Search	O(1)	O(n)
Insert	O(1)	O(n)
Delete	O(1)	O(n)

Because of this efficiency, you'll find hash tables to be pretty dang useful for many use cases. And if you look carefully, you’ll notice that they’re actually implemented in a variety of places throughout your tools like your databases, caches, data-fetching libraries, and so on.

We’ll expand on some of the uses cases for hash tables later on in the article.

How do hash tables work?

There are four distinct aspects to discuss how hash tables work:

storage
key-value pairs
a hashing function
table operations

Storage

A hash table is an abstract data type that relies on using a more primitive data type (such as an array or an object) to store the data. You can use either, but slight implementation implications occur depending on what you choose. We'll discuss those nuances in detail later on.

Key-value pairs

Regardless of the underlying primitive data structure, you use to store data, to store data as a value in a hash table, we need some way to identify it uniquely. We need a key.

Sometimes data contains an individual property that can very naturally assume responsibility for the key. For example, if we assume that a user has an email and a userId, then either the email or the userId could be used as the key if they are guaranteed to be unique.

{
  email: 'khalil@khalilstemmler.com', // highlight
  name: 'Khalil Stemmler',
  userId: '2',                        // or this
  phone: '555-123-4567'
}

Other times, we're working with data where uniqueness is a little more ambiguous. Take, for example, the case of a dictionary again. Consider the dictionary definition for the word chair looks like this:

{
  word: 'chair',
  definition: `a separate seat for one person, typically with a back 
   and four legs.`
}

What can we use for uniqueness here?

If we wish to store this in a hash table to quickly access the dictionary definition for the word, we'd have to use the word chair as the key itself.

However, based on this design, we cannot be sure to have uniqueness. Why? Because within the English language exists the idea of homonyms: words spelled (and sometimes pronounced) the same but with different meanings.

For example, there is another dictionary definition for the word chair.

{
  word: 'chair',
  definition: `the person in charge of a meeting or of an organization 
   (used as a neutral alternative to chairman or chairwoman).`
}

See the problem here? If we merely use word as the key, then we cannot be sure it's unique. And uniqueness is important for retrieval to get access to the definition by providing the word. If we have two definitions for the same word, this probably has to influence how we shape our data.

We could solve this problem by changing how we choose to model our data.

{
  word: 'chair',
  definition: [
    `a separate seat for one person, typically with a back 
	   and four legs.`, 
    `the person in charge of a meeting or of an organization 
	   (used as a neutral alternative to chairman or chairwoman).`
  ]
}

If we wanted, it also wouldn’t hurt to improve the data structure even more by giving each definition a type and example property. This means that the data we store within the definition array would actually need to become an object.

{
  word: 'chair', // the key to all of this data  definition: [
    { 
      type: 'noun',
      example: 'no one occupied the seat, so I sat in it',
      value: `a separate seat for one person, typically with a back 
        and four legs.`,
    }, 
    {
      type: 'noun',
      example: 'a three-year term as the board’s deputy chair',
      value: `the person in charge of a meeting or of an organization 
        (used as a neutral alternative to chairman or chairwoman).`
    }
  ]
}

Great, so we have some data which can be uniquely identified, and we have some data that we’d like to store so that we can retrieve quickly.

Hash functions

Once we have the key-value pair, we pass them to the hash table to store the data for later retrieval. Hash tables need a hash function to determine how the table should store the data, and this is one of the standard hash table operations.

The hash function requires both key and the value.

The key contains the logic that determines what index the value will live at within the underlying data structure (array or object).

When we use objects as storage, the index is often just some string or integer version of the key.
When we're a fixed-size array as storage, we need to generate a hash code: a value that maps the key to a valid index/position in the array.

After deciding on an index, depending on how we've implemented it, the hash function can insert or merge the value at the position specified by the index.

For example, suppose our dictionary hash table had saved only one of the possible definitions for the word chair so far. In that case, we could configure the hash function to merge additional non-duplicate definitions to the chair value's definition when we try to hash it.

{
  'cat': [...],
  'chai': [...],
- 'chair': { 
-    type: 'noun',
-    example: 'no one occupied the seat, so I sat in it',
-    value: `a separate seat for one person, typically with a back 
-      and four legs.`,
-  }
+ 'chair': [
+    { 
+     type: 'noun',
+     example: 'no one occupied the seat, so I sat in it',
+     value: `a separate seat for one person, typically with a back 
+	      and four legs.`,
+    }, 
+    {
+     type: 'noun',
+	  example: 'a three-year term as the board’s deputy chair',
+     value: `the person in charge of a meeting or of an organization 
+	      (used as a neutral alternative to chairman or chairwoman).`
+    }
+  ]
}

Alternatively, we could also configure it to throw an error if the hash function encounters a duplicate. In implementations where we're not merging data, this is generally referred to as a collision.

Methods/operations (what can it do?)

We know that one of the operations a hash table should perform is to hash an item — that is, to create the unique index based on a key. What other things should it be capable of?

At the bare minimum, a hash table should be able to perform:

add — add a key-value pair
get — get a value by key
remove — remove a value by key
list — get all the keys (or key-value pairs)
count — count the number of items in the table

How to choose the key in a hash table

To determine what should be the key in the key-value pair, it must be:

Unique (e.g., yellow is a unique word — and though it has many definitions, we can store those definitions at the index created by the yellow key).
known by the client in the world so that data can be re-retrieved (e.g., in storing the value, we must remember the key to re-retrieve it in constant time)

Basic JavaScript hash table examples

In this section, we’ll look at various ways to implement hash tables using JavaScript (and TypeScript). There are differences in languages like Java and Python (for example, in Java, you can use the HashMap interface as a part of the Java collection package). Still, ultimately, most general-purpose languages have a few different ways to implement a hash table.

Let's start with the simplest one.

Object implementation

The object implementation of a hash table in JavaScript is to merely lean on the usage of the plain ol' JavaScript object.

let table = {};

There are a few ways the API allows us to add a key-value pair to our table. The first is to use the dot-property.

let table = {};
table.yellow = 'color between green and orange';
console.log(table); // { yellow: 'color between green and orange' }

Another way is to use object brackets (which is useful when our key is numeric).

let table = {};
let pairOne = { key: 'yellow', value: 'color between green and orange' }
let pairTwo = { key: 2, value: { name: 'Bob' } }

// Text key
table[pairOne.key] = pairOne.value;
console.log(table); // { yellow: 'color between green and orange' }

// Numeric key
table[pairTwo.key] = pairTwo.value 
console.log(table); // { 
                    //    yellow: 'color between green and orange', 
                    //    2: { name: 'Bob' } 
                    //  }

To see if a value is present, we can use the hasOwnProperty method that’s a part of every object in JavaScript.

let table = {};
table.yellow = 'color between green and orange';

table.hasOwnProperty('yellow'); // true

To get the value of an item, we can call upon it using the key within either object brackets or with object dot notation.

let table = {};
table.yellow = 'color between green and orange';

console.log(table['yellow']); // 'color between green and orange'
console.log(table.yellow);    // 'color between green and orange'

When you're solving one-off algorithmic problems, you can use objects to get the functionality you need from a hash table quickly. However, there are better ways to design hash tables to be easier to use, and one such route is through encapsulation.

Encapsulated hash table

When you have related functionality, it's generally a good idea to encapsulate it within a cohesive object or a collection of functions. Here, we create a HashTable class that uses a plain object for storage and exposes all the necessary methods for a hash table.

class HashTable {
  constructor() {
    this.storage = {};
  }

  // Hash function
  add(key, value) {
    if (this.exists(key)) {
      throw new Error(`${key} already exists`);
    }

    this.storage[key] = value;
  }

  get(key) {
    if (!this.exists(key)) {
      throw new Error(`${key} not found`);
    }

    return this.storage[key];
  }

  exists(key) {
    return this.storage.hasOwnProperty(key);
  }

  remove(key) {
    delete this.storage[key];
  }

  list() {
    return Object.keys(this.storage).map((key) => ({
      key,
      value: this.storage[key],
    }));
  }

  count() {
    return Object.keys(this.storage).length;
  }
}

let hashTable = new HashTable();
hashTable.add(1, "Bill");
hashTable.add(2, "Mary");
hashTable.add(3, "Samson");

console.log(`The hashtable has ${hashTable.count()} elements`);
console.log(`This is ${hashTable.get(2)}`);

hashTable.remove(1);

console.log(`The hashtable has ${hashTable.count()} elements`);
console.log(`Here are all the elements: ${hashTable.list()}`);

The console output would look like the following:

The hashtable has 3 elements
This is Mary
The hashtable has 2 elements
Here are all the elements: [object Object],[object Object]

Other similar data structures

Lots of languages have hash table-like data structures already defined into the language.

You may hear about:

Hashmaps (Java)
Maps
Sets
Dictionaries (Python)

Each of these are (more or less) the same thing as a hash table.

Associative array JavaScript hash table example (advanced)

Let's now look at a more advanced way to create hash tables using arrays as the backing data structure instead of objects.

We create a HashTable class and initialize a fixed-size array to 200 elements.

class HashTable {

  constructor() {
    this.table = new Array(200);
    this.size = 0;
  }

}

Hash code

Next, let's implement the hash method. Assuming this is a private method used by all of the other public methods, its job is to create a hash code from a key.

A hash code is a value computed from the key which designates the index which the value should be saved at within the array.

In this implementation, the hash function expects a string key and returns a numeric hash code. Looking at each element in the string, we add the character codes to create a hash code.

// Assume the key is a 'string'
_hash(key) {
  let hash = 0;
  for (let i = 0; i < key.length; i++) {
    hash += key.charCodeAt(i);
  }
  return hash;
}

But how can we make sure that the hash value stays within the bounds of the array? One solution is to use the modulo operator to have the number roll around and never surpass the table length.

_hash(key) {
  let hash = 0;
  for (let i = 0; i < key.length; i++) {
    hash += key.charCodeAt(i);
  }
  return hash % this.table.length;
}

Implementing the operations

Within the add method, we create a hash code to use as an index, and then we store both the key and the value within the array. We also increment the array size so we can tell how fully utilized the array storage is. We may want to resize it later when we get to a decent size utilization.

add(key, value) {
  const index = this._hash(key);
  this.table[index] = [key, value];
  this.size++;
}

For the get method, we'll rely on the hash function again to get the index for the value.

get(key) {
  const index = this._hash(key);
  return this.table[index];
}

And then to remove an item, we merely remove the item from the array and decrement the size counter.

remove(key) {
  const index = this._hash(key);
  const itemExists = this.table[index] && this.table[index].length

  if (itemExists) {
    this.table[index] = undefined;
    this.size--;
  } 

}

A full associative array hash table implementation looks like the following:

class HashTable {
  constructor() {
    this.table = new Array(200);
    this.size = 0;
  }

  _hash(key) {
    let hash = 0;
    for (let i = 0; i < key.length; i++) {
      hash += key.charCodeAt(i);
    }
    return hash % this.table.length;
  }

  set(key, value) {
    const index = this._hash(key);
    this.table[index] = [key, value];
    this.size++;
  }

  get(key) {
    const target = this._hash(key);
    return this.table[target];
  }

  remove(key) {
    const index = this._hash(key);
    const itemExists = this.table[index] && this.table[index].length;

    if (itemExists) {
      this.table[index] = undefined;
      this.size--;
    }
  }
}

Let’s go ahead and test it out now.

const ht = new HashTable();
ht.set("Bill", '519-429-2212');
ht.set("Kit", '647-232-1246');
ht.set("Susan", '754-664-2397');

console.log(ht.get("Bill")); // [ 'Bill', '519-429-2212' ]
console.log(ht.get("Kit")); // [ 'Kit', '647-232-1246' ]
console.log(ht.get("Susan")); // [ 'Susan', '754-664-2397' ]

console.log(ht.remove("Kit")); 
console.log(ht.get("Kit")); // undefined

Looks good! If we were also to inspect the console afterward, here’s what the table could look like.

Collisions

A collision is when the hash function maps two or more key-value pair to the same storage index. As we populate the array, we’re bound to run into scenarios where collisions occur.

For example, if we were to run the following, we’d end up with a collision.

const ht = new HashTable();
ht.set("Susan", '754-664-2397');
ht.set("z", '555-123-5555');

console.log(ht.get("Susan")); // ['z', '555-123-5555']
console.log(ht.get("z"));     // ['z', '555-123-5555']

It appears we’ve overridden the “Susan” key with the “z”. This is very bad because it means we’ve lost data. And we don’t want that.

How does this even happen? Well, the reason why this happened is because the ASCII character for “z” is equal to the added ASCII characters in “Susan”. It’s 122.

To fix this, we’ll need collision detection (and optionally, array resizing) logic.

Collision detection and dynamic array resizing

It turns out that this is quite a challenging problem to solve. There are trivial approaches like just storing the same data in the same array index position and then merely traversing through it, but that increases the time complexity to O(n). As you'll discover down the road of writing your own hash table with associative arrays, there is no perfect approach to collision detection and dynamic array resizing.

There is one easy thing we can do, however. Using a prime number for the array (like 257, for example), we drastically reduce the likelihood of a collision.

If we had set the array size to 257 instead of 200, we’d get:

const ht = new HashTable();
ht.set("Susan", '754-664-2397');
ht.set("z", '555-123-5555');

console.log(ht.get("Susan")); // ['Susan', '754-664-2397']
console.log(ht.get("z"));     // ['z', '555-123-5555']

One technique for collision detection and resizing is called Prime Double Hash Table and it relies on the use of one smaller and one larger hash table. You can read here for more information on how this works.

Which implementation should I use?

For simplicity, I generally advocate you use one of the language's hash tables or encapsulate an object for storage instead of rolling your own associate array style hash table. Dealing with collisions and resizing is messy, and there are other more important things you need to spend your time coding.

When to use hash tables (use cases)

Not only are hash tables one of the first types of data structures you should reach for when solving problems in programming interviews, but their fast retrieval time makes them great for a variety of other use cases. Here are some common ones.

Databases (indexing)

Probably the most common use for hash tables is databases. To retrieve and increment through data in a database in a performant way, we need some way to identify it uniquely, and we need some way to index it.

In databases, we use indexes to locate data fast without needing to search through every single row. If a database has records from 0 to 100 and it knows that we're looking for 88, it doesn't need to start from 0 and traverse up to 88. It can begin the search from a much closer index.

Caching (backend)

Caching is a technique we use to speed up requests.

We separate read-like commands from write-like ones in Command Query Response Segregation (CQRS). We do this so that we can scale both sides independently.

On the read side, if we could tell that data hasn't changed yet (there haven't been any recent write-like operations — no commands), then there's an opportunity to optimize reads to the database.

In computing, reading from disk (database) takes longer than reading from memory (cache). Therefore, if we used an in-memory data store like Redis, we could speed up requests by offloading some of the reads to Redis instead.

Here's a Gherkins Given-When-Then (BDD-style) story to describe what should happen.

Feature: Caching

	Scenario: Cache miss 
		Given a resource is not cached  
		When a read request comes in for the resource 
		Then the controller fulfils the request by going to the database
		And the controller saves the resource to the cache for the next time

	Scenario: Cache hit
		Given a resource is cached
		When a read request comes in for the resource
		Then the controller fulfils the request by going to the cache

  Scenario: Cache invalidation
		Given a resource is cached
		When a write request on the resource is performed
		Then the cache invalidates the resource by removing it from the cache

Caching (front-end state management)

In the state management layer of a client-side application, we often find ourselves needing to persist, remove, and update data in storage.

Using tools like Apollo Client, Redux, or React hooks, the work we do to uniquely identify data and normalize it within our client-side cache is very similar to hashing function and persistence logic we write in hash tables.

Finding all unique elements

One other great use for a hash table is to find all unique elements among a list or collection of unorganized data.

Using some common aspects as the key, we can ensure that an item is only inserted once. Once that's done, we can merely iterate through the keys to get the entirety of the hash table's unique items.

Hash tables are a prevalent technique used in algorithm-based interview questions. As mentioned in Cracking the Coding Interview, hash tables should be one of the first data structures that come to mind for solving a potential challenge.

Storing transitions (transposition table)

This one is cool. Think about chess or other games where it may be essential to keep track of the entire sequence of moves made.

We can figure out any move's previous move in constant (O(1)) time with a transposition table. Assume a player's piece is at [1, 3]. How can we know what the previous move was? If we store moves like [xPosition, yPosition, moveNumber, player], we can use that as a key that maps to the previous position as a value.

For example:

[1, 3, 5, 1] → [1, 4, 4, 1]

This says that player 1 is now at position [1, 3] as their 5th move. It also says that before their 5th move (their 4th), they were at [1, 4].

To retrieve the previous move, the hash function could easily find the index based on the player's piece's current position and the turn number. This is just an example; of course, there may be better ways to do this, but it should further illustrate the power of hash tables.

To learn more, read about transposition tables here.

FAQ

Hashtable vs. hashmap — what’s the difference?

Hash tables and hash maps are essentially the same things, and it just comes down to the methods you use and what you choose to use as your backing data structure.

People typically equate the idea of a hash table with that of the associative array approach. But people often associate the idea of a HashMap with the object approach.

Right? They do the same thing and ideally have the same time complexity, but it comes down to the methods and storage.

How do you create a hashmap in JavaScript?

We technically already did this with the object implementation example we first explored in this article. HashMaps are a specific type of data structure that exists in Java but (and hackily in JavaScript).

Without a HashMap abstraction, we can achieve similar behavior like this:

function createHashCode (object){
  return obj.someUniqueKey; // Just an example
};

let dict = {};

dict[key(obj1)] = obj1;
dict[key(obj2)] = obj2;

There is, however, the Map abstraction in JavaScript.

let map = new Map();
let obj1 = { value: 1 }
let obj2 = { value: 2 }

map.set(1, obj1);
map.set(2, obj2);

console.log(map.get(1));
// expected output: { value: 1 }

map.set(3, { value: 'billy' });

console.log(map.get(3));
// expected output: { value: 'billy' }

console.log(map.size);
// expected output: 3

map.delete(2);

console.log(map.size);
// expected output: 2

Summary

In this article, we learned:

That hash tables an abstract data structure that stores data in key-value pairs and can be used for fast data retrieval
Hash tables can be implemented to have either an object or an array as the backing storage structure
A key must be unique and readily queryable by the client
The object implementation of a hash table is perhaps the simplest but languages like Java, Python, and TypeScript also have their own language-defined implementations of hash tables which can be used instead of rolling your own
If using an array for the backing data storage of a hash table, one must be ready to perform collision detection and dynamic array resizing logic
Hash tables are a common data structure in software engineering. They can be found or influenced in the design and implementation of databases, caching strategies, and more.

When to Use TypeScript Abstract Classes

Sun, 09 Jan 2022 15:04:10 GMT

If you're just getting started with object-oriented programming, the notion of an abstract class may seem a little foreign. Abstraction is one of the key ideas from object-oriented programming, and abstract classes are one of the two tools that OO provides to implement abstraction.

In this article, we'll learn all about abstraction, abstract classes, and everyday use cases for using them in your projects.

Let's start by understanding abstraction: perhaps the most crucial concept from object-oriented programming.

What is abstraction?

Abstraction is perhaps the most important element of design.

By selectively choosing which details to include, we can make concepts appear less complex, increase understandability, communicativeness, readability, and so on.

As programmers, we rely on abstraction to solve advanced architectural challenges. Abstraction removes the need to implement all the low-level details upfront; instead, we can focus on the high-level and figure out the specifics later.

You could also say that abstraction allows us to spend more time on the declarative (asking for what we want) and less on the imperative (specifying exactly how it'll work).

Concretions & contracts

We deal primarily with two types of classes in object-oriented programming languages: concretions and contracts.

Concrete classes are real, and we can construct objects from them to use at runtime using the new keyword. They're what most of us end up using when we first start learning object-oriented programming.

// Concrete class
class User {
  private name: string;

  public getName (): string {
    return this.name;
  }

  constructor (name: string) {
    this.name = name;
  }
}

const user = new User('Khalil'); // Creating an instance 
                                 // of a concrete class

On the other hand, contracts are blueprints that specify the properties and methods that a concretion should have if we want to create one from it. We can use them to contractualize the valid structure of an object or a class.

// Interface (abstraction)
interface Box {
  length: number;
  width: number;
}

const boxOne: Box = { length: 1, width: 2 }; // ✅ Valid! Has all props
const boxTwo: Box = { length: 1 };           // ❌ Not valid, missing prop

// Interface (abstraction)
interface Box {
  length: number;
  width: number;
}

// Concrete class implementing Box abstraction
class MobileBox implements Box { // ✅ Valid! Implements all necessary props
  public length: number;  
  public width: number;

  constructor (length: number, width: number) {
    this.length = length;
    this.width = width;
  }
}

let boxThree = new MobileBox(1, 2);

How to create contracts in OO languages

In the previous examples, we used the interface keyword to create contracts. However, when we speak of contracts in object-oriented programming, we're generally referring to one of two tools:

interfaces (i.e.: the interface keyword) or
abstract classes (i.e.: the abstract keyword in front of a class)

Type systems as a contract tool: We should also note that we can use types to implement contracts if we use a language with a type system (such as TypeScript).

Why are contracts necessary?

Perhaps the most foundational ideas are that:

you are always diverging or converging when you work
in solving any problem, you always have to utilize a bit of upfront (convergent) design and a bit of emergent (divergent) design
contracts are the tool we use to keep these two worlds in sync
we use contracts to cross architectural boundaries, synchronize components...

There are so many powerful things we can use contracts to do.

So many problems we can solve.

A key one is that it allows us to create a plugin architecture, separating the high-level from the low-level.

In traditional structured programming, we couldn't safely implement what's known as dynamic binding — the ability to refer to a contract (something that doesn't concretely exist) and feel confident that the dependency would work with our code.

For example, imagine we were writing the code for an OS that made USB ports work. We’d need to ensure that every device plugged in can function and do its job, but in reality, there are a lot of different things that can be plugged in. What should we do? Write some code for each and every single one of those?

switch (deviceType) {
  case 'MOUSE':
		mouseSubroutine();
	case 'HEADPHONES':
		headphonesSubroutine();
	case 'MIDI_KEYBOARD':
		midiKeyboardSubroutine();
	case 'WEBCAM':
		webcamSubroutine();
	...
}

That won't scale. What happens when there's a new type of USB device? Come in and write more code? No, this is architecturally poor. Without contracts, we'd continue to increase the complexity of our USB port code as we add support for more devices over time. We'd see our simple program blow up into an enormous and ever-growing control-flow graph of subroutines on subroutines.

What if there was a way to invert this dependency relationship? What if we could focus on defining a contract containing all the necessary things that a USB device needs to provide if they want it to work with the OS — and leave it to future USB device developers to implement them?

We are asking: what if we could program against an contract instead of an implementation (a concretion)?

We can!

This is the idea of ports and adapters. It's probably the simplest way to understand this phenomenon. Consider the contract as the port. It defines a what a valid adapter is. With this design, we create a plugin architecture. This removes the need for us to need to define every single last USB device and leave it to future developers to implement compatible adapters based on our contract.

Abstraction is deeply related to the ideas of:

Liskov Substitution Principle — since we define valid subtypes, each implementation should work and be interchangeable as long as it implements the contract.
Dependency Inversion — we do not directly depend on the concretions; instead, we rely on the abstraction that the concretions depend on; this keeps our core code unit testable.
Inversion of Control — we can give the client developer the ability to customize behavior by inverting program control to the client developer at hook points.

Next, let's take a look at the mechanics behind creating and using abstract classes in TypeScript.

Abstract class in TypeScript

The most common use of abstract classes in TypeScript is to locate some common behavior to share within related subclasses. However, it's essential to know that you cannot instantiate an abstract class. Therefore, the only way to access shared behavior is to extend the abstract class with a subclass.

For a simple demonstration, we'll imagine that we're selling digital books (PDF, EPUB, Kindle, etc.) — all of which have some common logic and their specific logic as well. To start, we'll build the base Book contract to put the common logic.

To declare an abstract class, we use the abstract keyword.

abstract class Book { 
  // .. 
}

Defining common properties

Within this Book abstract class, we can then decide on the contract for a Book. Let us say that all Book subclasses must have author and title properties. We can define them as instance variables and accept them as input using the abstract class constructor.

abstract class Book { 
  private author: string;  private title: string;  
  constructor (author: string, title: string) {
    this.author = author;
    this.title = title;
  }
}

Defining common logic

We can then can place some of the common logic within the Book abstract class using regular methods.

abstract class Book { 
  private author: string;
  private title: string;

  constructor (author: string, title: string) {
    this.author = author;
    this.title = title;
  }

  // Common methods
  public getBookTitle (): string {    return this.title;  }
  public getBookAuthor (): string {    return this.title;  }}

Remember that an abstract class is still a contract, after all.

Even though it looks like a class, really, it's mostly comparable to an interface.

That means we can't instantiate contracts directly (because they don't really exist, they're theoretical).

Sure, we're putting some logic on the abstract class, and this is what makes it special and unique from an interface...

Imagine levels of realness. You'd have:

interfaces/types = least real / theoretical (defines the 'what')
abstract classes = midpoint (can define some of the 'how', but can't create)
(concrete) classes = completely real, defines the 'how'

Trying anyway for demonstration purposes, you'd notice that we'd we get an error that looks like this:

let book = new Book (          // ❌ error TS2511: Cannot create an 
  'Robert Greene',             // instance of an abstract class. 
  'The Laws of Human Nature'
);

So then, what shall we do? We'll introduce one of our specific types of Book concretions — the PDF class. We extend/inherit the abstract Book class as a new subclass to hook it up.

class PDF extends Book { // Extends the abstraction
  
  private belongsToEmail: string;
  
  constructor (author: string, title: string, belongsToEmail: string) {
    super(author, title); // Must call super on subclass
    
    this.belongsToEmail = belongsToEmail;
  }
}

Since PDF is a concrete class, we can instantiate it. The PDF object has all of the properties and methods of the Book abstraction, so we can call getBookTitle and getBookAuthor as if it were originally declared on the PDF class.

let book: PDF = new PDF(
	'Robert Greene', 
	'The Laws of Human Nature', 
	'khalil@khalilstemmler.com'
);

book.getBookTitle();  // "The Laws of Human Nature"
book.getBookAuthor(); // "Robert Greene"

Defining mandatory methods

Abstract classes have one last important feature: the notion of the abstract method. Abstract methods are methods that we must define on any implementing subclass.

In the abstract class, we define them like so:

abstract class Book { 
  private author: string;
  private title: string;

  constructor (author: string, title: string) {
    this.author = author;
    this.title = title;
  }

  abstract getBookType (): string; // No implementation}

Notice that there is no implementation for the abstract method? That's because we implement it on the subclass.

Demonstrating with the PDF and an additional EPUB subclass, we add the required abstract method to both of them.

...

class PDF extends Book {
  ...
  getBookType (): string { // Must implement this    return 'PDF';  }}

class EPUB extends Book {
  
  constructor (author: string, title: string) {
    super(author, title);
  }
  
  getBookType (): string { // Must implement this    return 'EPUB';  }}

Failing to implement the required abstract methods will fail to make the class complete and concrete — this means we'll run into errors when trying to compile our code or create instances.

There are good reasons for using this technique. The best way to understand is to consider real-world scenarios. We'll explore this in the next section.

Use cases (when to use abstract classes)

Now that we know the mechanics behind how abstract classes work, let's talk about real-life use cases for it — scenarios you are likely to encounter.

There are two primary use cases for needing to use abstract classes:

Sharing common behavior and
Template method pattern (framework hook methods)

A prevalent scenario is the need to perform HTTP data-fetching logic within a front-end application that relies on multiple backend APIs endpoints.

Let’s say that we definitely need to fetch data from:

a Users API located at example.com/users
a Reviews API located at example.com/reviews

What’s the common logic?

Setting up an HTTP library (like Axios)
Setting up interceptors to set up common refetching logic (say, like when an access token expires)
Performing HTTP methods

Using an abstract class, we can define a BaseAPI as an abstraction with some of the implementation already provided.

export abstract class BaseAPI {
  protected baseUrl: string;
  private axiosInstance: AxiosInstance | any = null;
   
  constructor (baseUrl: string) {
    this.baseUrl = baseUrl
    this.axiosInstance = axios.create({})
    this.enableInterceptors();
  }

  private enableInterceptors () {
    // Here's where you can define common refetching logic
  }

  protected get (url: string, params?: any, headers?: any): Promise<any> {
    return this.getAxiosInstance({
      method: 'GET',
      url: `${this.baseUrl}${url}`,
      params: params ? params : null,
      headers: headers ? headers : null
    })
  }

  protected post (url: string, params?: any, headers?: any): Promise<any> {
    return this.getAxiosInstance({
      method: 'POST',
      url: `${this.baseUrl}${url}`,
      params: params ? params : null,
      headers: headers ? headers : null
    })
  }
}

Here, we've placed the low-level behavior within a BaseAPI contract. Now we can define the high-level behavior — the rich stuff — in the subclasses.

export class UsersAPI extends BaseAPI {
  constructor () {
    super('http://example.com/users');
  }
 
  // High-level functionality
  async getAllUsers (): Promise<User[]> {    let response = await this.get('/');    return response.data.users as User[];  }  
  ...
}

This pattern is widespread. Apollo's REST DataSource API builds on the same abstract class approach.

2. Template method design pattern (front-end libraries & frameworks example)

The template method design pattern (also known as the template pattern) is a behavioral design pattern that works by defining the skeleton of the algorithm in an abstract class and deferring some of the steps off to subclasses.

Because it allows for customizability, the template method pattern is widespread in libraries and frameworks like Angular, Vue, or React.js.

To demonstrate, let's consider older versions of React.js, the popular library for building front-end interfaces.

When you create a React component using class-based components, what’s the one method you absolutely positively need to implement? That’d be the render method, right?

So then, a simplistic version of the abstract class for a component could look like the following:

abstract class Component {
  private props: any;
  private state: any;

  abstract render (): void; // Mandatory
}

And then to use it, we’d need to extend the Component contract and implement the render method.

class Box extends Component {
  render () {
    // Must implement this method
  }
}

We need to implement the render method because it is a critical part of the work involved with deciding what gets created on screen. It's up to the client developer (us) to determine what HTML & CSS the browser should show. The library can't decide on that for us, so we need to provide it.

An abstract class is a suitable tool for this problem is because there is an algorithm running behind the scenes — an algorithm in which the render step is but a single step amongst many.

We can see that there are three distinct phases to React: mounting, updating, and unmounting. React's abstract class further gives us (the client developer) the ability to customize our components by connecting the ability to hook into various lifecycle events. For example, you can perform some behavior as soon as your component mounts (componentDidMount), when it updates (componentDidUpdate), and when it's about to unmount (componentWillUnmount).

Another vastly simplistic (and not entirely accurate or complete) depiction of what that may look like behind the scenes is like this:

abstract class Component {
  private props: any;
  private state: any;

  abstract render (): void;
	
	// Algorithm for mounting
	constructor () {
    this.render();
    this.componentDidMount();
  }

	// The general algorithm for updating. Called when new props occur, 
	// when setState is called or when forceUpdate is called
  onUpdate () {
    if (this.componentShouldUpdate()) {
      this.render();
      this.componentDidUpdate(); 
    }
  }
  
  // Algorithm for unmounting
  onUnmount () {
    this.componentWillUnmount();
    // Complete unmounting
  }
	
  public componentDidMount (): void {
    // No implementation lifecycle method, allow the client to override
  }

  public componentDidUpdate (): void {
    // No implementation lifecycle method, allow the client to override
  }

  public componentWillUnmount (): void {
    // No implementation lifecycle method, allow the client to override
  }

  private componentShouldUpdate (): boolean {
    // Determines if the component should update
    ...
  }
}

And now, a customized component subclass can plug in behavior within those key lifecycle hook methods. This is the concept of Inversion of Control.

class Box extends Component {
  constructor () {
    super();
  }
  
  componentDidMount () {
    // (Optional) Perform custom logic
  }
  
  componentDidUpdate () {
    // (Optional) Perform custom logic
  }
  
  componentWillUnmount () {
    // (Optional) Perform custom logic
  }
  
  render () {
    // Must implement this method
  }
}

So as a framework designer, the Template Method design pattern is attractive when:

You need the client to implement a step of the algorithm → so you make that step an abstract method
You want to provide the ability for clients to customize behavior at various steps of the algorithm → so you expose optional lifecycle methods to the client

In summary

Object-oriented programs contain concretions and contracts. Abstract classes are one of two ways we can implement contracts in object-oriented programming.
Use abstract classes to define shared behavior or implement the Template Method pattern.

Frequently asked questions

What is an abstract class constructor for?

The abstract class constructor is used to set up properties or run other setup logic from within the abstract class so that the low-level abstract class methods can use them. We invoke the constructor by calling super on a subclass (derived class).

What does the abstract method do?

The abstract method exists to force clients to implement specific functionality necessary for the abstract class to work. We abstract away the exact details to a subclass.

What’s the difference between an interface vs. abstract class?

Interfaces and abstract classes are both examples of contracts in object-oriented programming. Where an interface can only define the properties and methods of an contract, an abstract class can define the properties and methods in addition some common behavior.

We also use the implements keyword to inherit from an interface but use the extends keyword to perform inheritance against an abstract class.

When you need an contract, default to an interface (or a type in typed languages) but opt for an abstract class when you need to define common behavior or generalize an algorithm using the Template Method design pattern.

Camel Case vs. Snake Case vs. Pascal Case — Naming Conventions

Tue, 21 Dec 2021 15:04:10 GMT

The tldr

Naming conventions dictate the way you write names (for methods, folders, variables and so on). The three most common are camel case, pascal case, and snake case.

Camel case always starts out lowercase with each word delimited by a capital letter (like personOne, textUtil, thingsToDo)
Pascal case is similar to camel case, but the first letter is always capitalized (like PersonOne, TextUtil, ThingsToDo)
Snake case means that we delimit words with an underscore (like person_one, text_util, things_to_do)

Continue reading for more information on usage best practices.

Introduction

There are many different ways to improve the maintainability of your codebase. One of the first things developers learn how to do is to decide upon and enforce a set of coding conventions to create consistency in the formatting and style of your code. Under the umbrella of coding conventions exists the idea of naming conventions.

Camel case, snake case, and pascal case are three of the most common types of programming language naming conventions found across all programming languages.

In this article, we discuss why naming conventions matter; we look at camel case, snake case, and pascal case — and for each, we demonstrate with examples and learn when to use each in a variety of programming languages.

Naming conventions

What are they?

Naming conventions are rules that dictate the way you name the various files, folders, and tokens in our code (such as variables, functions, classes, methods, objects, and so on).

Naming conventions based on language

While a team can decide upon naming conventions for any project, each programming language has standard naming conventions for different tokens.

For example, in C-styled languages like Java, JavaScript, and TypeScript, while it's conventional to use the camel case convention for variables, functions, method names, and object properties, it's common practice to use the pascal case convention for classes, interfaces, and namespaces. That's not to say that you can't use snake case or other unconventional naming conventions for a language like Java or JavaScript. In most cases, your code will still compile and run, but it's good to generally try to avoid going against the typical conventional naming practices for the language. Because there are usual language conventions, adhering to those rules can help improve the ease of future maintenance.

Why naming conventions?

The entire idea behind naming conventions is to help make code more maintainable for the reader. By enforcing a convention and naming tokens consistently, we accomplish two things: discoverability and understanding.

Discoverability: How quickly can someone find a folder, file, feature, class, or method they need to change?
Understanding: How fast can someone look at our code and understand what it is they're looking at?

We want to make it very easy for developers on our team to find what they're looking for, understand what it is, and understand how to use it as quickly as possible.

The most common naming conventions

Across all programming languages, the most common naming conventions are the following:

Camel case (ex: someVariable)
Snake case (ex: some_variable)
Pascal case (ex: SomeVariable)

Let's explore each one.

Camel case

Camel case is perhaps the most common naming convention of all three. A camel case word is a single or compound word that uses capital letters to make different parts of the word easier to read.

It adheres to the following rules:

The word doesn't end on a capital letter (like someVariablE)
Capital letters can only appear at the start of the second word (like someVariable).
No dots, underscores, numbers, dashes, or any other special characters are allowed within the word (like som.eVar-1able)

In most C-style languages, we use camel case to denote variables, properties, attributes, methods, and functions.

Some make a distinction between lowerCaseCamel and UpperCaseCamel, but most developers refer to the former as merely camel case and the latter as pascal case (which we'll talk about later).

Camel case examples

someVariable
eventDispatcher
eventsRepo
slackService
parentComment

Camel case usage in TypeScript

In TypeScript & JavaScript, the camel case convention is used to signify that a token is a variable, function, method, parameter, or property.

type UserProps = {   
  firstName: string; // Used to signify properties
  lastName: string;
  email: string
}

interface Serializable { 
	toJSON (): string;  // A method also camel cased
}

class User implements Serializable {  
  private props: UserProps;   // Camel cased property
	
  // constructor and parameters are camel cased
  constructor (firstName: string, lastName: string) { 
    this.props = { firstName, lastName };
  }
  
  public toJSON (): string {
    return JSON.stringify(this.props);
  }
}

// Variables are camel case
const userOne = new User('Khalil', 'Stemmler');
const userTwo = new User('Nick', 'Cave');

Snake case

Snake case (also referred to as underscore case) is when all the letters of the word are lower case but delimited by an underscore.

We seldom use the snake case coding convention in C-style languages like Java, JavaScript, and TypeScript. They're typically only used in programming languages like these to denote constants. To denote a constant, we typically implement screaming snake case: snake case with all of the letters capitalized.

You'll most commonly see snake case in the Python scripting language as is popularized in the Python style guide.

Snake case examples

some_variable
event_dispatcher
events_repo
slack_service
parent_comment

Snake case usage in TypeScript

Let’s look at the same TypeScript example and amend it ever-so-slightly to demonstrate when we may utilize the snake case naming convention.

type UserProps = {   
  firstName: string; 
  lastName: string;
  email: string
}

interface Serializable { 
	toJSON (): string; 
}

class User implements Serializable {  
  private props: UserProps;  
  private static MAX_USERS: number = 10; // Screaming snake case                                          // to depict a constant	
  constructor (firstName: string, lastName: string) { 
    this.props = { firstName, lastName };
  }
  
  public toJSON (): string {
    return JSON.stringify(this.props);
  }
}

// Variables are camel case
const userOne = new User('Khalil', 'Stemmler');
const userTwo = new User('Nick', 'Cave');

As you can see, snake case isn't a prevalent naming convention in C-styled languages like TypeScript. We only tend to use it when we're declaring constants.

Snake case usage in Python

In Python, however, the snake case naming convention is much more common. Taking the same TypeScript example translated to Python, look at the widespread usage of snake case.

import json

class User:
  props = {
    "first_name": "", # Properties of dictionary objects
    "last_name": ""
  }

  MAX_USERS = 10 # Screaming snake case on constants
  
  # Used on method parameters
  def __init__(self, first_name, last_name):  
    self.props["first_name"] = first_name
    self.props["last_name"] = last_name
  
  # Usage on method names
  def to_JSON(self):
    return json.dumps(self.props)

# We even use snake case to for naming variables
user_one = User("Khalil", "Stemmler")
user_two = User("Nick", "Cave")

Pascal case

Pascal case follows the same camel case naming convention rules — all but one: we capitalize the first letter (e.g., SomeVariable).

In object-oriented languages like Java and TypeScript, we use pascal case to denote classes, namespaces, and abstractions like interfaces.

In scripting languages like Python, you'll rarely find camel or pascal case code except when defining a class's name.

Pascal case examples

SomeVariable
EventDispatcher
EventsRepo
SlackService
ParentComment

Pascal case usage in TypeScript

Continuing with the same example, let’s look at where pascal case is used in TypeScript code.

type UserProps = {    // Types are also abstractions
  firstName: string; 
  lastName: string;
  email: string
}

interface Serializable {   // Interfaces use pascal case
	toJSON (): string; 
}

class User implements Serializable {  // Class names use pascal case
  private props: UserProps;  
  private static MAX_USERS: number = 10;
                                         
  constructor (firstName: string, lastName: string) { 
    this.props = { firstName, lastName };
  }
  
  public toJSON (): string {
    return JSON.stringify(this.props);
  }
}

const userOne = new User('Khalil', 'Stemmler');
const userTwo = new User('Nick', 'Cave');

Comparison of naming conventions in other programming languages

Here’s a table that depicts all of the typical naming conventions for the languages: Python, C#, TypeScript, and Java.

	Python	C#	TypeScript	Java
functions & methods	snake_case()	PascalCase()	camelCase()	camelCase()
classes	PascalCase	PascalCase	PascalCase	PascalCase
interfaces	N/A	PascalCase	PascalCase	PascalCase
namespaces	N/A	PascalCase	PascalCase	PascalCase
constants	snake_case	SCREAMINGSNAKE\CASE	SCREAMING_SNAKE_CASE	SCREAMING_SNAKE_CASE

Conclusion

In summary,

Coding conventions are patterns, techniques, styles, and approaches to improve the maintainability of your code, and naming conventions are one of such techniques.
Naming conventions improve maintainability by using consistency to optimize for discoverability and understanding.
Naming conventions depend on the language. Each language has its own naming conventions that specify the use of programming tokens.
Camel case (ex: camelCase) is usually for variables, properties, attributes, methods, and functions.
Snake case (ex: snake_case) is commonly used in scripting languages like Python and as a constant in other C-styled languages.
Pascal case (ex: PascalCase) is mainly reserved for class names, interfaces, and namespaces.

What’s next?

If you're setting up a new project or improving an existing one, I recommend you check out the series on enforcing conventions with Husky.

Additionally, I'm going to assume you're interested in learning about ways to improve the maintainability of your code. In that case, I recommend learning TDD, writing acceptance tests, and adopting a feature-driven approach to software design.

How to use Prettier with ESLint and TypeScript in VSCode

Sun, 19 Dec 2021 15:04:10 GMT

This post is a part of the Clean Code Tooling series.
You may want to read the previous post.
1. How to use ESLint with TypeScript

Intro

In the previous article in this series, "How to use ESLint with TypeScript", we learned how add ESLint to our project and configure it to determine if our code was adhering to our project's coding conventions.

This is great, but it can get pretty annoying having to re-run npm run lint everytime we want lint (and optionally fix) our code.

Here's where Prettier, particularly the VS Code extension for Prettier, comes in.

Understanding Prettier

Prettier is an opinionated (yet fully configurable) code formatter. ESLint can kind of format code too, but it's mostly intended to sniff out when we're not following the mandated coding conventions.

Prettier can be configured to format your code (makes it look prettier 😉) after you save a file or manually tell it to, and by default, it comes configured with a set of common code cleanliness rules.

We get the most benefit out of Prettier when we combine it with ESLint though.

Using Prettier with ESLint

Combining both ESLint and Prettier, the responsibility division is this:

ESLint defines the code conventions
Prettier performs the auto-formatting based on the ESLint rules

Now that's a bomb combo.

Goals for this blog post

At the end of this post, you should be able to:

Set up Prettier for a TypeScript or JavaScript project
Decide on the formatting configuration that best suits your style
Configure Prettier to work with ESLint

Prerequisites

A code editor installed (VS Code is the champ, don't @ me)
An existing codebase (if you need to get setup, you can follow "How to Setup a TypeScript + Node.js Project first and then come back to this article)
An understanding of ESLint and a working configuration (see "How to use ESLint with TypeScript" for an example)

Installing Prettier

First thing's first, we'll install Prettier as a dev dependency.

npm install --save-dev prettier

Configuring Prettier

As per the docs, we can expose a JSON file called .prettierrc to put our configuration within.

touch .prettierrc

A basic .prettierrc setting is the following:

.prettierrc

{
  "semi": true,
  "trailingComma": "none",
  "singleQuote": true,
  "printWidth": 80
}

These settings specify the following rules:

semi set to true means that Prettier will add semicolons when necessary.
trailingComma set to none means that Prettier will remove any trailing commas at the end of objects.
singleQuote set to true means that Prettier will prefer single quotes instead of double quotes unless the number of double-quotes outweighs the number of single-quotes. Read more here.
printWidth set to 80 specifies that the printer will wrap any lines that exceed 80 characters.

You can view the rest of the options here and change them as you like! This is just my personal preference.

Testing Prettier using the Prettier CLI

So far, we've locally configured Prettier, and we're in shape to test formatting some code.

Since we're using the code from simple-typescript-starter, the only file we have is src/index.ts, and it looks like this:

console.log('Hello')

When we add a script to format all the code in the folder and execute it, the only change we should notice is an added semicolon.

Add the following script to your package.json.

package.json

{
  "scripts": {
    ...
    "prettier-format": "prettier --config .prettierrc 'src/**/*.ts' --write"  }
}

And then run it.

npm run prettier-format

The console output shows that one file has been changed and that it took 186ms to execute.

npm run format

> typescript-starter@1.0.0 format /simple-typescript-starter
> prettier --config .prettierrc 'src/**/*.ts' --write

src/index.ts 186ms

Looking at src/index.ts, you'll notice that the console.log was, in fact, appended with a semicolon.

console.log('Hello world!');

This is a neat way to use Prettier, but it's not the best, in my opinion. Here's what I prefer.

Using Prettier

One of the most common ways that people use Prettier is to install the VS Code extension that adds it to your editor.

It enables you to, on save, format your code. This is good because the feedback loop is incredibly short, and it turns formatting into something that you don't have to occupy any brain cycles thinking about.

Formatting using VSCode on save (recommended)

Install the Prettier VS Code extension here.

To set the defaults, press CMD + SHIFT + P (on MacOS) or CTRL + Shift + P (on Windows), then type in preferences open settings.

You want to select the JSON option so that we can manually edit the preferences via a JSON file.

Options for typing in "preferences open settings". You want to select the JSON option.

In the JSON file, if it's not already added, add the following lines to the root of the object.

// Default (format when you paste)
"editor.formatOnPaste": true,
// Default (format when you save)
"editor.formatOnSave": true,

These settings will format your code both when you paste new code and when you save code for any file extension that Prettier understands. If the root of the project that the file lives in has a .prettierrc, it will use the settings that live in that file to format your code.

Let's say you're working with multiple languages and automatically formatting code in another language is a no go for you. You can also override the default settings by specifying the language as shown below.

"[typescript]": {
  "editor.formatOnPaste": false,  "editor.formatOnSave": false,},
"editor.formatOnPaste": true,
"editor.formatOnSave": true,

Using the above config, Prettier will not format TypeScript code on paste or save but it will format code on paste or save for any other language that it understands.

Formatting before a commit (recommended)

When working with other developers as a team, it can be challenging to keep the formatting of the code clean constantly. Not everyone will want to use the Prettier VS Code extension. Not everyone will want to use VS Code!

How do we ensure that any code that gets commited and pushed to source control is properly formatted first?

Read the next post, "Enforcing Coding Conventions with Husky Pre-commit Hooks".

Formatting using an filesystem watcher

If you really don't like VS Code or there are too many people on your team not using it, there's another option to tighten the feedback loop of formatting code as you're writing it.

The Prettier docsc suggest using a package called onchange in order to watch the filesystem for when changes are made to your source code, then run the Prettier CLI tool against any changed files.

Here's how that works.

Install onchange.

npm install --save-dev onchange

Then, add this script to your package.json, making sure to change the watched directory if you keep your source code in a location different the src folder.

package.json

"scripts": {
  "prettier-watch": "onchange 'src/**/*.ts' -- prettier --write {{changed}}"  }

Opening a new console, running prettier-watch, and editing code within the src folder will give us the same observable outcome as if we set Prettier up with VS Code or not.

Manually formatting a single file

On MacOS, if I've installed the VS Code extension, I can format the current file by typing SHIFT + OPTION + F.

This might be different for you. You can see what the command is by typing COMMAND + SHIFT + P and entering "format".

The command to format the current file in VS Code is shown here for me.

Configuring Prettier to work with ESLint

Here's where the real magic happens.

As a reminder, you'll want to have completed the previous ESLint tutorial first.

With ESLint and Prettier already installed, install these two packages as well.

npm install --save-dev eslint-config-prettier eslint-plugin-prettier

eslint-config-prettier: Turns off all ESLint rules that have the potential to interfere with Prettier rules.
eslint-plugin-prettier: Turns Prettier rules into ESLint rules.

Lastly, we need to make an adjustment to the .eslintrc.

.eslintrc

{
  "root": true,
  "parser": "@typescript-eslint/parser",
  "plugins": [
    "@typescript-eslint",
    "prettier"  ],
  "extends": [
    "eslint:recommended",
    "plugin:@typescript-eslint/eslint-recommended",
    "plugin:@typescript-eslint/recommended",
    "prettier"  ],
  "rules": {
    "no-console": 1,       // Means warning
    "prettier/prettier": 2 // Means error  }
}

If you're not interested in using the recommended plugins, the most basic .eslintrc with Prettier enabled looks like this:

.eslintrc

{
  "extends": ["prettier"],
  "plugins": ["prettier"],
  "rules": {
    "prettier/prettier": 2 // Means error
  }
}

Running npm run lint (from the previous article) lints the files and tells us all of the unformatted lines in addition to any code convention violations we've protected against through ESLint rules.

typescript-starter@1.0.0 lint /simple-typescript-starter
> eslint . --ext .ts

simple-typescript-starter/src/index.ts
  1:1  error    Replace `⏎console.log('Hello·world!')⏎` with `console.log('Hello·world!');`  prettier/prettier  2:1  warning  Unexpected console statement                                                 no-console

✖ 2 problems (1 error, 1 warning)
  1 error and 0 warnings potentially fixable with the `--fix` option.

That's it! Run npm run prettier-format to format everything.

Conclusion

You can find the code for this post here.

We're pretty lucky that tooling has advanced to the point where enforcing coding conventions and formatting code isn't something we have to personally concern ourselves with anymore.

Even if you're not using VS Code, you should be able to add some basic formatting to your code using the steps outlined in this post.

After that, the next step is to enforce your conventions and formatting within your team by using a tool like Husky.

Read the final post in this series titled "Enforcing Coding Conventions with Husky Pre-commit Hooks".

How to Test Code Coupled to APIs or Databases

Wed, 17 Nov 2021 15:04:10 GMT

In our humble introduction to Test-Driven Development, we learned the basics of TDD. You now know that you should start with a failing test, write the mininum amount of code to make it pass, and refactor it to make it cleaner. Red-green-refactor.

Straightforward, right? Though it may take a while to internalize and start working this way, the physical aspect - the mechanics of it - are pretty easy to grasp. That's the good news.

What's not good news is that when you get out there into the real world, you're going to encounter situations where you need to test code that relies on a variety of infrastructural dependencies: external APIs, services, databases, caches, webhooks, and so on.

It is incredibly common for developers new to TDD to give up their test-driven journey at this point. When I first started with TDD years ago, it just didn't seem practical. I was only able to figure out how to TDD my way though simple things like simple functions or pure React components.

The question then, is this: how do we test code that relies on infrastructural concerns? Do we spin up a database, tear it down, and seed it everytime we want to write a unit test? Do we spend real money everytime we want to test code that relies on a paid external API? (Hint: the answer is no, you don't have to).

In this post, we'll discuss how to test code that relies on infrastructure. And we'll do so by first separating core code from infrastructure code.

Watch on YouTube

Please excuse the first few moments of bouncy video - I learned I can't rest my arm on my desk when recording video :) I'm starting up my YouTube channel! If you prefer this way of teaching, please like and subscribe ❤️.

Core code and infrastructure code

What do I mean by core code and infrastructure code? And why is it important to keep them separate?

Core code

Core code is the heart of your application. This is your family jewels. It's what makes your app special.

This code is also the code that represents the essential complexity. It encapsulates the real-world complexity of the domain and the application.

Consider a createUser use case. You need to:

1. Check that that the user's details like their email and password are valid
1. Check that the account wasn't already created, and
1. Confirm that the user's handle wasn't already taken

Because these rules represent the nature of the application and what makes the use case succeed or fail, we call this application logic. Application logic usually depends on infrastructure, so depending on the way we write it, it may or may not be completely pure and decoupled from infrastructure.

On the other hand, if we think a layer deeper, we get to domain logic. Domain logic is moreso about the data and behavior of the core business objects (think User entities or Email value objects). Because of this, we could benefit by thinking back to state machines and drawing out legal behavior.

For example, let's consider a washing machine. The state machine for a washing machine would illustrate the fact that you can go from OFF to ON and then WASH, but it would restrict you going from OFF to WASH to ON. That'd put the washing machine into an illegal state. Domain logic is about enforcing these rules - typically validation and object behavior.

Domain logic is normally pure. It normally contains zero dependencies to infrastructure and it is written by you. This makes it very easy to unit test.

All of this stuff needs to get tested. As a test-driven developer, your goal is to turn the essential complexity that is your application and domain logic into tested code - using the simplest, most flexible and maintainable way you know how.

Infrastructure code

In the real world, the way to realize a web application is to integrate it with infrastructure. Application and domain logic can't do much if we can't host it somewhere in the cloud, if it doesn't maintain state somehow, or if we can't make requests to the backend API.

On the backend, infrastructure code is denoted by concerns like your database, your cache, your local file system, or your REST or GraphQL web server. On the front-end, this is your view-layer library, your GraphQL or REST client, and your browser APIs.

All of these concerns are things that we don't own. We either download them, connect to them over the network, or they exist as a part of the platform upon which our code executes. They are infrastructural.

Unlike core code, we don't unit test it ¹. Instead, we want to test the way that it integrates — that's a key word: integrates. We want to test how it integrates with our code core.

Therefore, we write integration tests.

For example, one particular type of integration test is called a Contract Test. This ensures that - for example, our database adapter can save and retrieve objects in the way we expect it to.

Therefore, we need to write different types of tests for different aspects of our application.

Integration tests for testing integrations
Unit tests for testing the core application functionality — the most valuable aspects of our software

What's stopping us from getting to the point where we can do this?

The problem: coupled core and infrastructure code

The problem is that this isn't really possible when your core code and your infrastructure code is mixed. Actually, the better word to use is coupled to each other.

If your core code and your infrastructure code is coupled, we can't cleanly isolate parts of our code cleanly in order to write these specific types of tests.

What we end up doing instead is coupling way too much core code in RESTful API controllers or GraphQL resolvers and suddenly, the only way to test the application core (the features, yes - the heart of our software) is to bring a database and a web server along for the ride. This makes things very slow and it makes tests harder to setup and teardown.

Example: Coupled feature

Here's an example of a feature with coupled core and infrastructure code.

You can view the before and after code here on GitHub.

// modules/users/useCases/createUser/index.ts

import * as express from 'express'
import { User } from '../../domain/user';
import { firebaseUserRepo } from '../../repos';
import { UsersService } from '../../services/usersService'

export async function createUser (req: express.Request, res: express.Response) {
  let body = req.body;

    // Check to see if firstname, lastname, password, email is in the request
    const isFirstNamePresent = body.firstName
    const isLastNamePresent = body.lastName;
    const isEmailPresent = body.email;
    const isPasswordPresent = body.password;

    // If not, end the request
    if (!isFirstNamePresent || !isEmailPresent || !isLastNamePresent || !isPasswordPresent) {
      return res.status(400).json({ 
        message: `Either 'firstName', 'lastName', 'email' or 'password not present`
      })
    }

    // Check to see if already registered
    const existingUser = await firebaseUserRepo.findByEmail(body.email);
        
    // If already registered, return AlreadyRegisteredError
    if (existingUser) {
      return res.status(409).json({
        type: `AlreadyRegisteredError`,
        message: 'User already registered'
      })
    }

    let errorMessage;

    // Validation logic
    if (UsersService.validateFirstName(body.firstName)) {
      errorMessage = 'Invalid firstName';
    }

    if (UsersService.validateLastName(body.lastName)) {
      errorMessage = 'Invalid lastName';
    }

    if (UsersService.validateEmail(body.email)) {
      errorMessage = 'Invalid email';
    }

    if (UsersService.validatePassword(body.password)) {
      errorMessage = 'Invalid password';
    }

    // If invalid props, return InvalidUserDetailsError
    if (errorMessage) {
      return res.status(400).json({
        type: 'InvalidUserDetailsError',
        message: errorMessage
      })
    }

    // Create user
    let user: User = {
      firstName: body.firstName,
      lastName: body.lastName,
      email: body.email,
      password: body.password
    }

    // Save user to database
    try {
      await firebaseUserRepo.save(user);
    } catch (err) {

      // Log this to monitoring or logging plugin but don't return
      // the backend error to the client.

      return res.status(500).json({
        message: 'Unexpected error occurred'
      })
    }

    return res.status(201).json({
      type: 'CreateUserSuccess',
      message: 'Success'
    })
}

What's wrong with this? Well, there's nothing really wrong with this of course. The code works. However, the only way for us to test this would be to perform a black-box test: an End-to-End test. Why? Because in order to set up this test, we need to bring an entire Express.js webserver with us, connections to a real firebase instance, and we can only validate the application logic through response codes.

While I believe E2E testing should be a part of your testing strategy, I don't think it should be your testing strategy.

Let's now discuss the path to fixing this.

Architectural patterns

In Part VII - Architecture Essentials from solidbook.io, we learn about architectural styles and architectural patterns.

Architectural styles are general overarching ways in which you can construct your application to help you solve a particular architectural challenge. The three main ones being structural, message-based, and distributed architectures.

It's kind of like how we have design patterns. Where we typically think of design patterns as these things that help us solve challenges at the class level, architectural styles and their patterns help us solve problems at the architectural level.

Architectural challenge? What's the challenge we're facing here?

The challenge 95% of us who are building non-trivial production applications are tasked with is not only how to encode these business rules within software effectively, but to do so in a way that we can verify that they work, and that we can safely change those rules later on down the road.

In other words, the challenge is that we have strict testing requirements.

Which architectural pattern is specifically meant to help us solve the strict testing requirement problem? Ah yes, the layered architecture.

A layered architecture

The solution to our testing problem is a structural architectural style called the layered architecture.

If you're a regular reader of this blog, you're likely already familiar with it, perhaps by the name of a hexagonal architectural, the onion architecture, or the famous clean architecture.

For those uninitiated, the main idea is that you separate the concerns of your application into layers:

A layer for domain logic
One for application
One for infrastructure
And an adapter layer which merely describes abstractions that make it possible for your infrastructure to hook into your application layer code using dependency injection

Core code corresponds to the domain, application, and adapter² layers while infrastructure corresponds to the infrastructure layer.

Benefit — testing options

The primary benefit of this type of separation is that we give ourselves a lot more testing options.

Before, if we wanted to write our acceptance tests, we'd have to write them in a sort of E2E-test style, bringing all of our databases, services, and real-world APIs along for the ride, maybe even playing with real money or something.

Instead, if we want to write our acceptance tests as unit tests, we have the option for that now.

For a use case test, we can mock and stub out dependencies to infrastructure and just focus on testing the application core — make sure that it fails when it should, that it attempts to save to the database, that it attempts to make a call to an external API, or delete something when it should — but without actually really making that happen.

Then if we want to test that our infrastructure does work, we can test that separately in integration tests. This is just a part of a larger testing strategy. But alas — now we have options.

Let me show you how I'd probably rewrite and acceptance test this feature using the layered architecture pattern.

Demonstration

I'd start with the BDD-style Given-When-Then test.

# modules/users/useCases/createUser/createUser.feature
Feature: Create user

Scenario: Creating a user
  Given I provide valid user details
  When I attempt to create a user
  Then the user should be saved successfully 

Scenario: Invalid password
  Given I provide an invalid password
  When I attempt to create a user
  Then I should get an invalid details error

Then I'd write the test:

// modules/users/useCases/createUser/createUser.spec.ts
import { defineFeature, loadFeature } from 'jest-cucumber';
import * as path from 'path';
import { IUserRepo } from '../../repos/userRepo';
import { CreateUser, CreateUserResult } from './createUser'
import { UserRepoSpy } from '../../testObjects/userRepoSpy'

const feature = loadFeature(path.join(__dirname, './createUser.feature'));

defineFeature(feature, test => {
  let result: CreateUserResult;

  let email: string;
  let password: string;
  let firstName: string;
  let lastName: string;

  let createUser: CreateUser;
  let userRepoSpy: UserRepoSpy;

  beforeEach(() => {
    createUser = undefined;
    userRepoSpy = undefined;
  })

  test('Creating a user', ({ given, when, then }) => {
    
    given('I provide valid user details', () => {
      // Arrange
      email = 'khalil@khalilstemmler.com';
      password = 'hello'
      firstName = 'khalil'
      lastName = 'stemmler';

      userRepoSpy = new UserRepoSpy([]);

      createUser = new CreateUser(userRepoSpy);
    });

    when('I attempt to create a user', async () => {
      // Act
      result = await createUser.execute({ email, password, firstName, lastName });
    });

    then('the user should be saved successfully', () => {
      // Assert
      expect(result.type).toEqual('CreateUserSuccess');
      expect(userRepoSpy.getTimesSaveCalled()).toEqual(1);
    });

  });

  test('Invalid password', ({ given, when, then }) => {
    given('I provide an invalid password', () => {
      email = 'khalil@khalilstemmler.com';
      password = ''
      firstName = 'khalil'
      lastName = 'stemmler';

      userRepoSpy = new UserRepoSpy([]);

      createUser = new CreateUser(userRepoSpy);
    });

    when('I attempt to create a user', async () => {
      result = await createUser.execute({ email, password, firstName, lastName });
    });

    then('I should get an invalid details error', () => {
      // Assert
      expect(result.type).toEqual('InvalidUserDetailsError')
      expect(userRepoSpy.getTimesSaveCalled()).toEqual(0);
    });
  });
});

And then I'd write the implementation of the use case (worrying about the integration tests later):

// modules/users/useCases/createUser/createUser.ts
import { Result } from '../../../../shared/core/result';
import { UseCase } from '../../../../shared/core/useCase';
import { Email } from '../../domain/email';
import { FirstName } from '../../domain/firstName';
import { LastName } from '../../domain/lastName';
import { Password } from '../../domain/password';
import { User } from '../../domain/user';
import { IUserRepo } from '../../repos/userRepo';

type CreateUserInput = {
  email: string;
  password: string;
  firstName: string; 
  lastName: string;
}

type CreateUserSuccess = {
  type: 'CreateUserSuccess'
}

type AlreadyRegisteredError = {
  type: 'AlreadyRegisteredError';
}

type InvalidUserDetailsError = {
  type: 'InvalidUserDetailsError';
  message: string;
}

type UnexpectedError = {
  type: 'UnexpectedError'
}

export type CreateUserResult = CreateUserSuccess 
  | AlreadyRegisteredError 
  | InvalidUserDetailsError
  | UnexpectedError;

export class CreateUser implements UseCase<CreateUserInput, CreateUserResult> {
  private userRepo: IUserRepo;

  constructor (userRepo: IUserRepo) {
    this.userRepo = userRepo;
  }

  public async execute (input: CreateUserInput): Promise<CreateUserResult> {

    // Check to see if already registered
    const existingUser = await this.userRepo.findByEmail(input.email);
        
    // If already registered, return AlreadyRegisteredError
    if (existingUser) {
      return {
        type: 'AlreadyRegisteredError'
      }
    }

    // Validation logic
    let emailOrError = Email.create(input.email);
    let firstNameOrError = FirstName.create(input.firstName);
    let lastNameOrError = LastName.create(input.lastName);
    let passwordOrError = Password.create(input.password);

    let combinedResult = Result.combine([ 
      emailOrError, firstNameOrError, lastNameOrError, passwordOrError 
    ]);

    if (combinedResult.isFailure) {
      return {
        type: 'InvalidUserDetailsError',
        message: combinedResult.errorValue()
      }
    }

    let userOrError = User.create({
      email: emailOrError.getValue() as Email,
      password: passwordOrError.getValue() as Password,
      firstName: firstNameOrError.getValue() as FirstName,
      lastName: lastNameOrError.getValue() as LastName
    });

    if (userOrError.isFailure) {
      return {
        type: 'InvalidUserDetailsError',
        message: userOrError.errorValue()
      }
    }

    let user = userOrError.getValue() as User;

    // Save user to database
    try {
      await this.userRepo.save(user);
    } catch (err) {

      // Log this to monitoring or logging plugin but don't return
      // the backend error to the client.

      return {
        type: 'UnexpectedError'
      }
    }

    return {
      type: 'CreateUserSuccess'
    }
  }
}

There are many nuances in this improved version. Read about Use case Tests or check out the YouTube video for this article for a more detailed breakdown.

Conclusion

To recap, we learned that:

The big problem with testing code that relies on databases, APIs, the file-system, or anything infrastructural is that we often couple our core code — the actual application core, the essential complexity — to our infrastructural code.
Architectural styles and patterns are solutions to common architectural problems.
The reason the layered architecture is so helpful for testing is that that it allows us to use dependency inversion to separate core code from infrastructure code. This gives us testing options. We can then develop a testing strategy that works.
One way to acceptance test your features is to use a [Use ], by using a Use Case test, which is able to test the application core without relying on infrastructure — and that's very handy because it enables us to exhaust all of the success and failure states without needing to bring slow database connections or network requests along for the ride.

FAQ

Question: Do I have to do this?

Do I have to do this to test my backend code? Of course not. If you're building something super simple like a CRUD app, a proof of concept, or just exploring, I don't think you need to get to this level.

You can always just merely black-box test your entire backend, meaning — from the API, you just send it HTTP requests and assume that things work properly if your database get saved with the appropriate records.

This does, however, leave a lot of gaps in our testing capabilities.

For example, how exactly would you be able to verify that a confirmation email was sent when it should have been? You can't exactly programmatically log into your Gmail account to find that out.

I think you normally want a mixture of this black-box testing — testing from the outside, and some white-box testing, testing from the inside. Having both is a part of a healthy testing strategy on any software you intend on maintaining for an extended period of time.

Question: Is this just on back-end or does this apply to front-end too?

You can also decouple the layers of your front-end code as well.

I've written a little bit of philosophy on how this would work in the front-end via Client-Side Architecture Basics, and while I sometimes use these techniques, it depends on the testing requirements.

For example, if you're building what I call a list/detail-view application where all you really have to do is fetch data and present it, then no - I don't think this is entirely necessary. Your front-end testing strategy could just be to perform E2E tests and that's it (though I'd advise writing these tests in as BDD-acceptance-test-style as possible).

However, if you're building something very complex, like a digitial audio workstation in your browser, and there's a lot of logic that cannot be tested by merely perceiving and clicking, then yes, you're going to want to enforce a more rigid set of architectural layers for unit testing application or domain logic as well.

So ultimately, it depends on your testing requirements. How rigid are they? What's your testing strategy?

Question: What about testing React components? How do you test them?

Let's apply the same philosophy.

Understand what it is that we'd like to test
Develop a testing strategy
Separate core code from infrastructure code

It depends on what you'd like to do. If you'd like to test your pure react components with unit tests, then yes — we'll want to enforce some level of separation, especially if it relies on infrastructure like Apollo Client or an HTTP-enabled service class or React hook that fetches data from a RESTful API.

If you don't care so much about testing your React components, and you care more about testing the features of the application from a user level (very common for view/list-detail applications), this calls for E2E tests. And it doesn't call for a strict decoupling of core from infrastructure. Go right ahead and write your E2E tests on top of your React code. Even better, start by writing your E2E tests using the Page Object pattern, and then write the minimum required React code to make your E2E tests pass.

People seem to have different opinions on what a unit test is and is not. I do not believe that testing a component which contains other components disqualifies my test from being a unit test. For example, if I wanted to test a pure React component, like Table and it was decomposed into smaller sub-components for purely cosmetic reasons like TableRow and TableColumn, I still believe testing Table is a valid unit test. Why? Because I believe unit tests are more about if we're testing core code or infrastructure code. If we test code that relies on databases, network requests, the filesystem, or even the system clock, we are no longer testing units - we are testing integrations. Integrations between what? Between code you wrote (core) and code someone else wrote (infrastructure). This is the strategy we use for acceptance testing as use case tests.
↩
Technically, you can say the adapter layer is a core layer because it doesn't contain any infrastructural concerns. It's completely comprised of abstractions. I like to think of it as a bridge between core and infrastructure.
↩

No title

Sun, 08 Aug 2021 04:05:26 GMT

In a layered architecture there are often a number of mutable, shared, external, out-of-process infrastructural dependencies. We can break these down even further into two categories: managed or unmanaged.

Managed dependencies

An external, mutable, out-of-process, shared dependency is considered managed when we:

have full control over it because it is specific to our application AND
interactions with it are hidden from the outside world AND
if an external system wants to interact with it, that has to be done through your application's API

Examples of managed dependencies

Your application database

How to test managed dependencies

The best way to test a managed dependency like an application database is to write a contract test (which is a form of an integration test).

Since the interface for your repository object defines all of the things an implementatation must be able to do, your test exercises each of the methods in the interface, confirming that each works properly.

There is absolutely no mocking in this type of test. It needs to be as real as possible.

Unmanaged dependencies

An external, mutable, out-of-process, shared dependency is considered unmanaged when we:

don't have full control over it because part of it is not fully owned by us OR
it is also used by other applications and changing it could potentially break other applications

Examples of unmanaged dependencies

A message bus (used by other applications in your enterprise)
An email-sending adapter (because it actually sends an email using an email service like Sendgrid or Mailchimp)
A payments API like Stripe or PayPal
Any other external API not owned by you or your team

How to test unmanaged dependencies

In a layered architecture, there are two ways we can test unmanaged dependencies:

Use case test: Test that our application will call the command-like operation at the correct time and with the correct arguments
Integration test: Test that we can communicate with the external, unmanaged dependency

For example, production usage of unmanaged dependencies might look like the following:

// Publishing an event onto an event bus
bus.publishEvent(event);

// Sending an email
emailService.sendEmail(email);

// Charging a credit card
stripe.createCharge(customer, { ... })

Because both tests test different aspects of the application, it's often a good idea to write both use case tests and integration tests.

Use case testing unmanaged dependencies

In use case tests, the way we verify that our application has called these dependencies at the right time and with the correct parameters, is to use a mock.

Rule: As per the CQS principle, only use mocks to assert against commands (outgoing interactions); never use mocks for queries (use stubs for that)

Consider we're writing a makeOffer use case in a vinyl-trading application. Using a use case test to ensure that it works properly, one observable outcome/state change is to test that an email gets sent to the vinyl owner.

Because we don't want to pass in a real email-sending service that actually sends emails, we'll need a way to validate that the use case told the (unmanaged) dependency - the outgoing adapter - to send the email.

We can do this with either a mock or a spy (the difference is that a spy is a hand-coded version of a mock).

We will use both a mock and a spy in the following example:

// modules/notification/notificationService.ts
export interface INotificationService {
  sendEmail (email: Email): Promise<void>;
}

// modules/trading/useCases/makeOffer/makeOffer.ts
export class MakeOffer {
  constructor (
    private vinylRepo: IVinylRepo, 
    private tradesRepo: ITradesRepo,
    private notificationService: INotificationService
  ) {

  }
  async execute (request: any) {
    ... // Implementation here
  }
}

// modules/notification/mocks/notificationsSpy.ts
import { Email, INotificationService } from "./makeOffer";

// Here, we define an `INotificationService` which can be subtituted in.
// We keep track of the emails sent and write specific methods in order
// to help us assert that we'll have called a command-like operation
// (the outcome-related behavior) when we should have.

export class NotificationsSpy implements INotificationService {
  private emailsSent: Email[];

  constructor () {
    this.emailsSent = [];
  }

  // This is the behavior related to the outcome. The command.
  public async sendEmail (email: Email): Promise<void> {
    this.emailsSent.push(email);
  }

  // This is a method we add to make it easier to assert 
  // in our test.
  getEmailsSent () {
    return this.emailsSent;
  }
}

// modules/trading/useCases/makeOffer/makeOffer.spec.ts
import { ITradesRepo, IVinylRepo, MakeOffer } from "./makeOffer";
import { NotificationsSpy } from "./notificationSpy";
import { createMock } from 'ts-auto-mock';

describe('makeOffer', () => {
  describe(`Given a vinyl exists and is available for trade`, () => {
    describe(`When a trader wants to place an offer using money`, () => {
      test(`Then the offer should get created and an email should be sent to the vinyl owner`, async () => {

        // Arrange 
        // Stub: This is used to perform a query - no assertion on this
        let fakeVinylRepo = createMock<IVinylRepo>();
        
        // Mocks: These are used to validate commands (state changes)
        let mockTradesRepo = createMock<ITradesRepo>();        let notificationsSpy = new NotificationsSpy();
        let makeOffer = new MakeOffer(
          fakeVinylRepo, 
          mockTradesRepo,
          notificationsSpy
        );

        // Act 
        let result = await makeOffer.execute({
          vinylId: '123',
          tradeType: 'money',
          amountInCents: 100 * 35
        });

        // Assert 
        expect(mockTradesRepo.save).toHaveBeenCalled();        expect(notificationsSpy.getEmailsSent().length).toEqual(1);      })
    })
  })
})

Some people will say "use mocks for unmanaged dependencies (external APIs, etc). Don't use mocks for managed ones (application database)". This doesn't take into consideration cases when you're using your database as event storage and it also then says that it's illegal for you to confirm that you've performed a save() operation at the end of a use case (which, according to the following test, is part of what we should be testing).

I think a better rule is the following:

Rule: Use mocks for commands, use stubs for queries

A Google search on this principle informs me that Mark Seemann agrees.

Also, because queries can be seen as implementation details:

Rule: Do not perform assertions on queries

If we're unit testing a query (use case) and we want to ensure that it returns the correct data, we should only assert on the result.

Integration testing unmanaged dependencies

Since other types of tests cover the contract and behavior, all we care about is that our application can communicate with the dependency. For example, if we're using Stripe's payment API, we merely want to ensure that our application can connect to Stripe.

This is substantially trickier. As Matthias Noback suggests in “Advanced Web Application Architecture”, you can either:

Write the test against the real service
(Ideal) Write the test against a sandbox environment the third party offers
Write the test against a fake server that you run
Write the test against a fake or mock HTTP client offered by the client library that you use
Write the test against a fake or mock of the HTTP client interface that you use

No title

Sun, 08 Aug 2021 04:05:26 GMT

Originating from the idea of a stunt double (like in action movies), there are two general categories of testing objects that we use in testing: mocks and stubs.

Mocks

Mocks: We use a mock to stand in and assert against command-like operations (outgoing interactions or state changes against dependencies). We pass mocks in to the system under test (SUT) and later check during the assert phase of a test that the correct calls to change the dependencies' state were made.
Spies: These are exactly the same thing as traditional mocks except that we hand-roll spies manually, whereas with traditional mocks, mocking libraries like ts-auto-mock help you create mocks in a single line or two (I highly recommend this package over basic mocking with Jest). You can see an example of a hand-rolled spy here.

Stubs

Dummies: These don't do anything. Nor are they used during a test. These are objects that are just used to fill up parameter lists so that a constructor, function, or method will execute. They can often be null or empty-string.
Stubs: A stub is a dependency that we can configure to return different values in query-like scenarios. This generally takes some effort to do.
Fakes: A fake is practically the same thing as a stub. They only differ in the sense that we create a fake to sub in for a dependency that doesn't exist yet. This can be done very quickly.

When to use each?

The general rule of thumb is that you should:

Use mocks for commands, use stubs for queries

However, there are more specific rules as well depending on the type of test you're writing.

Unit tests (domain layer objects): Don't use mocks. Use stubs when necessary.
Use case tests (unit test): Only use mocks to assert that commands were invoked against dependencies. Use stubs when necessary.
Contract tests (integration): You're testing the contract between a managed dependency like your application database. Don't mock anything. Ensure that the repository (the adapter to your database) can do everything that the interface (the contract) says it needs to be able to do.
Integration tests (incoming adapter tests): You're testing that the incoming communication mechanism (such as an HTTP request from a web server, GraphQL API, webhook or some other incoming port) calls the correct application layer use case. Don't mock the HTTP calls. Use an HTTP testing framework like Supertest to make real requests to your application. You can stub the response for the use case because the behavior is covered in use case tests.
Integration tests (outgoing adapter tests): You're testing that you can connect to an unmanaged dependency like Stripe, PayPal, an email-sending service, or some piece of infrastructure shared between several applications in your enterprise (like AWS SES). While writing a test against the real thing is better, choose one of these options.
End-to-end tests: Treat the entire application like a black box. Don't mock anything. If you're using 3rd party APIs like Stripe or PayPal for example, if provided, use their Sandbox Environments (see Stripe sandbox docs, see Paypal sandbox docs). Use these specifically for your E2E tests instead of mocking them out entirely. Why? Because we want to test against the most production-like environment as possible. You can, however, stubs (see the Builder pattern) to clean up the creation of request data if necessary.

Additional testing principles

Do not perform assertions on queries; assert the result instead

For example, in a use case test, the following type of assertion is a violation of the rule and can lead to brittle tests. This is due to the fact that a query is an implementation detail.

// While testing that a `createOffer` use case in a vinyl-trading application
// can successfully save and email the vinyl owner.

expect(mockTraderRepo.getTraderReputation).toHaveBeenCalledWith('123')
expect(mockVinylRepo.getVinylOwner).toHaveBeenCalledWith('123')

You should be able to create a mock with merely an interface (the contract)

If you have to manually define the implementation for your mock every single time you want to use one, you'll run into the situation where everytime you add or change a method on your interface, you'll have to update all your mock implementations in your test files.

Take a look at "How to Mock without Providing an Implementation in TypeScript". We explore this problem and provide a solution using ts-auto-mock.

How to Mock without Providing an Implementation in TypeScript

Sat, 07 Aug 2021 15:04:10 GMT

🌱 This blog post hasn't fully bloomed. It's very likely to change over the next little while.

I've been spending some time attempting to really understand the philosophy of testing in software design. There is a tremendous amount of varying thought, but my goal is to find some truth and crunch it into something digestible.

A couple of the questions I've been wrestling with are:

Because I use Jest as my test runner and mocking comes with it out-of-the-box, I figured I'd use Jest to create my mocks and that'd be it. Unfortunately, as a diligent blog reader pointed out, I wasn't actually writing mocks. I was inadvertly writing stubs and incurring the negative implications of that slight as well.

Upon further research, I realized that this was an issue many TypeScript developers using Jest are still currently running into.

In this post, I'll explain how many of us are not actually mocking properly using Jest, what some of the implications of that are, and how to fix it.

What is a mock?

First of all, what's a mock?

The term "mocking" is often overloaded (we've purposely done that here) to refer to the concept of a subbing in a dependency for a test double, which is an umbrella term for either a "mock" or a "stub".

Fundamentally, we use a mock to stand in for a dependency that we'll issue command-like operations (outgoing interactions or state changes against dependencies) on. And we use stubs to provide data for query-like operations in tests.

A standard use case test

Here's a problematic example of a use case test written using Jest.

import { 
	INotificationService, 
	ITradesRepo, 
	IVinylRepo, 
	MakeOffer 
} from "./makeOffer";

describe('makeOffer', () => {
  describe(`Given a vinyl exists and is available for trade`, () => {
    describe(`When a trader wants to place an offer using money`, () => {
      test(`Then the offer should get created and an 
      email should be sent to the vinyl owner`, async () => {
        
        // Collaborator #1 - Should be a stub object.
        // We have to provide an implementation otherwise
        // we'll get a compilation error.
        let fakeVinylRepo: IVinylRepo = {
          getVinylOwner: jest.fn(async(vinylId: string) => {
            return { id: '4', name: 'Jim' };
          }),
          isVinylAvailableForTrade: jest.fn(async (vinylId: string) => {
            return true;
          }),
        }

        // Collaborator #2 - should be a mock
        // Unfortunately, we also need to provide an implementation of the
        // interface.
        let mockTradesRepo: ITradesRepo = {
          saveOffer: jest.fn(async (offer) => {
            return;
          })
        }

        // Collaborator #3 - should also be a mock object
        // Again, implementation required.
        let mockNotificationService: INotificationService = {
          sendEmail: jest.fn(async (email) => {
            return
          })
        }

        let makeOffer = new MakeOffer(
          fakeVinylRepo, 
          mockTradesRepo,
          mockNotificationService
        );

        let result = await makeOffer.execute({
          vinylId: '123',
          tradeType: 'money',
          amountInCents: 100 * 35
        });

        // We are confirming that the two command-like operations
        // have been called by looking commands invoked on the mocks.
        expect(mockTradesRepo.save).toHaveBeenCalled();
        expect(mockNotificationService.sendEmail).toHaveBeenCalled();
      })
    })
  })
})

Let's discuss the collaborators here.

The first collaborator is the fakeVinylRepo. Because this is used for queries, it's not going to be a mock of any sort. I want this to be a fake (a type of stub).

The second and third collaborators are intended to be used to verify that an "offer was created" and that an "email was sent" as per the test definition. Both of those things are command-like operations that should be changing state in dependencies. That means that we're looking at these things as if they're mocks.

Problems

What's wrong with this?

My mocks are actually stubs

As was pointed out to me by one blog reader, if you need to provide an implementation to your mock, you're not really creating a mock anymore - you're creating a stub. This makes sense if we really think about the definition of a mock and a stub.

I tried removing the implementation from my design, but I found that with Jest, I couldn't do that and keep my code happy and compiling.

let mockTradesRepo: ITradesRepo = jest.fn(); // Error

I could just any type this, but I don't want to. I'm documenting using an interface to help future test readers understand that what is being passed in here is of type IVinylRepo, not just any object. What I needed was the ability to merely specify the interface of a mock object and let the testing framework create the mock for me.

Unfortunately, I've yet to find a way to do this with Jest. It seems like I have to provide an implementation. This is problematic, because as one StackOverflow user commented,

Although it's technically true that a mock just needs to have the same shape as the interface, that misses the whole point. The whole point is to have a convenient way to generate a mock given an interface, so that developers don't have to manually create mock classes just to, say, stub out a single function out of a dozen methods every time you need to run a test.

Brittle test code

The larger issue here is that if we have to provide an implementation for every test double in our test files, every time we go and add a new method to the interface for an adapter, our tests will break until we go back and update all the mocks and stubs in our tests.

interface ITradesRepo {
  getOfferById: (id: string) => Promise<Offer, None>; // New method breaks tests
  saveOffer: (offer: Offer) => Promise<void>;
}

We obviously can't have that.

Mocking and stubbing with nothing but an interface using ts-auto-mock

I've stumbled upon a wonderful library written by the TypeScript-TDD community called ts-auto-mock.

With ts-auto-mock, we avoid the problem of needing to provide an implementation for each mock and stub. We just give it the interface and it fills that out for us.

// makeOffer.spec.ts

import { ITradesRepo, IVinylRepo, MakeOffer } from "./makeOffer";
import { createMock } from 'ts-auto-mock';
import { NotificationsSpy } from "./notificationSpy";

... 

// Don't care about providing implementations for the stubs
// and the compiler won't yell at us either
let fakeVinylRepo = createMock<IVinylRepo>();

// Here's our first mock object
let mockTradesRepo = createMock<ITradesRepo>();

// And our second mock object.
// We've also written this as a spy instead. You'll see why
// in a moment.
let notificationServiceSpy = new NotificationsSpy();

... 

// This compiles! 
let makeOffer = new MakeOffer(
  fakeVinylRepo, 
  mockTradesRepo,
  notificationServiceSpy
);

...

// Our assertions
expect(mockTradesRepo.saveOffer).toHaveBeenCalled();
expect(notificationsSpy.getEmailsSent().length).toEqual(1);

ts-auto-mock provides trivial implementations of all of methods on the interface at runtime, so if within my MakeOffer use case, I was to call any of the methods on the test doubles (mocks and stubs), it wouldn't result in a runtime failure.

// makeOffer.ts

export class MakeOffer {
  constructor(
    private vinylRepo: IVinylRepo,
    private tradesRepo: ITradesRepo,
    private notificationService: INotificationService,
  ) {}
  async execute(request: any) {

    // This is just to demonstrate that none of these methods exist yet, 
    // but we can still call them and verify that they work
    // in our tests! 
    const owner = await this.vinylRepo.getVinylOwner('');
    const available = await this.vinylRepo.isVinylAvailableForTrade('');

    this.tradesRepo.saveOffer({
      vinylId: '123',
      tradeType: 'money',
      amountInCents: 100 * 35,
    });

    this.notificationService.sendEmail({});

  }
}

You'll also notice in the test file that I've written the notificationService as a spy instead. Generally, you use a spy when you want more control as to how you'll verify that the state-changing command was issued on a dependency.

// modules/notifications/mocks/notificationSpy.ts

import { Email, INotificationService } from "./makeOffer";

export class NotificationSpy implements INotificationService {
  private emailsSent: Email[];
  constructor () {
    super();
    this.emailsSent = [];
  }

  public async sendEmail (email: Email): Promise<void> {
    this.emailsSent.push(email);
  }

  getEmailsSent () {
    return this.emailsSent;
  }
}

Because this is a traditional concrete-class-implementing-an-interface, if I add new methods to the INotificationService, I'll have to update it here, probably with a throw new Error('Not yet implemented') statement until I figure out how it should work in the spy.

This could be better because I can maintain this single spy and use it for various tests, but I'm still working out how we can use ts-auto-mock for other use cases like this.

Summary

In TypeScript, we're forced to provide an implementation for test doubles in Jest.
- By definition of mocks and stubs, this means each test double is a stub.
- It also means our tests and test doubles will be brittle since adding new methods to an interface requires changing the test doubles.
Use ts-auto-mock to create pure mock objects using merely an interface
- For an example, with Jest see jest-ts-auto-mock

Thank you Vittorio Guerriero!

When to Use Mocks: Use Case Tests

Wed, 04 Aug 2021 15:04:10 GMT

If you're not familiar with mocking, it's a pretty controversial topic in testing.

Mocking implies the use of some sort of test double which we sub in for real components. When you're testing, you obviously want to get as much confidence as possible that your code is working correctly, right? So then, why would we bother using mocks if all it's going to do is reduce your confidence?

A reader sent in a question expressing his concern to me recently. He said,

Mocking vs stubbing has always caused conflict with me, because to me it seems that mocking violates the fact that we are only interested in testing the external api/contract of a code module as it tests whether specific internal functions were called. If I am not mistaken, when we desire to test a function/method, we are not interested in what other functions/methods it calls, but only in providing the correct output. Please correct me if I am wrong.

You’re right on the money. Mocking alone isn't the best way to test an application. However, mocking does have benefits when used as a part of a larger testing strategy. Let's explore.

Prerequisites

You're familiar with test doubles and know the the difference between mocks and stubs.

The correct conceptual model of OO

We have to constrain this discussion to the idea of OO, because mocking is a lot less useful in FP. It’s just an entirely different conceptual model.

The conceptual model you should have with OO is one of a “web of objects”, or like as Alan Kay had initially imagined, cell biology. Think of objects as cells each with their own tiny little “mini-computers” and encapsulated state within them. You can even think of it as routers in a computer network.

How you should really think of OO, as a "Web of Objects"

Behaviour is realized through the messages. For example, cell A tells cell B to do something (which may delegate some work to cell C without cell A needing to know that happened).

What do we have to test?

In a layered architecture, we have to test:

Incoming adapters (GraphQL, REST, incoming webhooks)
Managed outgoing adapters (application database)
Unmanaged outgoing adapters (message bus, email/notification service or other public API calls)
The application core (use cases)
Domain objects (entities, value objects, domain services)

Because each of these tests need to verify different things, we have to use different testing strategies for each one.

Let's focus specifically on #4, testing the application core use cases. But first, we have to discuss some of the underlying philosophy for how to use test doubles correctly.

Command-query separation

Use cases are the features of our applications. If you'll recall, features can be either commands or queries, as per the CQS principle.

Commands are unique from queries in the sense that they perform a state change. It could be saving an entity to a database, queuing an email, saving an event to some event storage, and so on.

Ultimately the rule with test doubles with respect to CQS is that we:

Use mocks for commands, use stubs for queries

And when it comes to performing assertions, we:

Do not perform assertions on queries. Assert only results and that commands were invoked using mocks.

These rules are in place to ensure that your tests aren't susceptible to ripple. It's to prevent you from writing the brittle sort of test that asserts against implementation details. And since query-like operations can be seen as implementation details, we create brittle tests when we assert against them.

In testing a use case, our goal is to test 1) that the outgoing interactions - the commands - were issued from our application core, and 2) we get the intended outcome for the scenario.

Writing a use case test

Let's assume you have a use case that you need to implement using TDD. You even have the functional requirements written out using Gherkin in Given-When-Then format.

Feature: Make offer

Scenario: Making a monetary offer
	Given a vinyl exists and is available for trade
	When a trader wants to place an offer using money
	Then the offer should get created 
	And an email should be sent to the vinyl owner

Fantastic. We know, at a high-level, from the user's perspective, one of many scenarios for this use case that we'll need to test.

As you likely know, the TDD process is to start with a failing test, make it pass, and then refactor it. Where a lot of developers get stuck is trying to figure out how exactly to write tests for this, especially because the solution is going to involve databases, external services, and other infrastructural concerns that will slow down tests and cause it to no longer be a unit test.

What's not a unit test?: Michael Feathers says a test is not a unit test if:

It talks to the database
It communicates across the network
It touches the file system
It can’t run at the same time as any of your other unit tests
You have to do special things to your environment (such as editing config files) to run it.

This constraint moves our discussion to the two general flavours of TDD - two ways we can write tests: inside-out and outside-in.

Outside-in vs inside-out

Inside-out — Most of us are familiar with this. It's when we start from the inside layers where code is pure and doesn't involve infrastructure. We unit test outwards. This is typically what we think of when we consider regular unit tests. It's also known as Classic TDD. You can see a demonstration of this here.
Outside-in — This is when we start from the outer boundary, have some understanding of the layers of our application, and code inwards.

If we're to start at the use case boundary, it's impossible for us to write a unit test with the inside-out approach. Since we rely on infrastructure (like databases, web services, and external services), we will fundamentally break a rule of unit tests. Use cases test the core application functionality. They fetch domain objects so that inner layer objects (domain) can execute the essential complexity of the problem at hand. We want these tests to be fast - not painful, show, flaky, and hard to configure.

To write a use case test (that can still technically be seen as a unit test) means we have to use the outside-in approach (the mockist approach).

Starting with the outcomes (commands/state-changes)

With any test, we have to stop and think about the outcomes first. These will be either command-like operations or the output result (success or failure) returned from the use case.

First of all, we expect the use case to be successful in this scenario, so there's that.

But if we look back to our acceptance criteria for this test, we'll notice that are two commands:

the offer should get created
an email should be sent to the vinyl owner

Some TDD-practioners advocate for starting with the assert phase of the "arrange-act-assert" structure of a test. The reason? Often, when we start with creating the objects, it's a lot harder to maintain YAGNI and it forces the design into things that aren't actually necessary for us to accomplish the behavior. Once you get comfortable writing tests using TDD, I also advise this technique.

Here's what the start of our outside-in test could look like.

// modules/trading/useCases/makeOffer/makeOffer.spec.ts

describe('makeOffer', () => {
  describe(`Given a vinyl exists and is available for trade`, () => {
    describe(`When a trader wants to place an offer using money`, () => {
      test(`Then the offer should get created and an email should be sent to the vinyl owner`, async () => {
        // Arrange

        // Act

        // Assert 
        expect(result.isSuccess()).toBeTruthy();                    // result        expect(mockTradesRepo.saveOffer).toHaveBeenCalled();        // outcome         expect(notificationsSpy.getEmailsSent().length).toEqual(1); // outcome        expect(notificationsSpy.emailWasSentFor())          .toEqual(result.getValue().offerId);                      // outcome      });
    });
  });
});

At this point, we are very much in the red of the red-green-refactor loop. I'm referring to objects that don't exist, but that's completely OK, because that's what we're supposed to do. We're not too concerned about how we'll make this work. At this point, we're concerned about how we'll know that it worked. That's the big thing you get when you write test-first.

Here's what we're doing:

I'm assuming that we'll get a result object back from the use case. I'm assuming this result will be successful.
I'm also writing about a collaborator called a mockTradesRepo, and because one of the requirements is that the "offer should get created", I know that the command (the object-oriented message we'll send to a real TradesRepo) will be called saveOffer. We expect that to have been called.
Lastly, I used my imagination to consider how I could ascertain that an email was sent and that it was sent in the correct context. If we return the offerId in the result, we can link that to the email. I need a mock object here, but I know that a hand-rolled mock object, one that we write custom test methods on is actually called a spy, we name the notificationsRepo mock object a notificationsSpy accordingly.

Acting out the behavior

Continuing to work backwards, we'll act out the behavior to make this thing work. I'm specifically focused on the when part of the Given-When-Then statement: "When a trader wants to place an offer using money".

I write something like the following:

// modules/trading/useCases/makeOffer/makeOffer.spec.ts

describe('makeOffer', () => {
  describe(`Given a vinyl exists and is available for trade`, () => {
    describe(`When a trader wants to place an offer using money`, () => {
      test(`Then the offer should get created and an email should be sent to the vinyl owner`, async () => {
        // Arrange

        // Act
        let result = await makeOffer.execute({          vinylId: '123',          tradeType: 'money',          amountInCents: 100 * 35,        });
        // Assert 
        expect(result.isSuccess()).toBeTruthy();                    
        expect(mockTradesRepo.saveOffer).toHaveBeenCalled();       
        expect(notificationsSpy.getEmailsSent().length).toEqual(1); 
        expect(notificationsSpy.emailWasSentFor())
          .toEqual(result.getValue().offerId);                     
      });
    });
  });
});

So far so good.

Create the subject and collaborators in the arrange phase

There are a couple of objects we need to create here:

Subject: The MakeOffer use case which returns a result.
Collaborator #1: The TradesRepo (which we will mock).
Collaborator #2: The NotificationService (which we will mock with a spy)

Here's where we merely create these objects and focus on getting our code to at least compile.

// modules/trading/useCases/makeOffer/makeOffer.spec.ts
import { ITradesRepo, MakeOffer } from './makeOffer';import { NotificationsSpy } from './notificationSpy';import { createMock } from 'ts-auto-mock';
describe('makeOffer', () => {
  describe(`Given a vinyl exists and is available for trade`, () => {
    describe(`When a trader wants to place an offer using money`, () => {
      test(`Then the offer should get created and an email should be sent to the vinyl owner`, async () => {

        // Arrange        let mockTradesRepo = createMock<ITradesRepo>();        let notificationsSpy = new NotificationsSpy();        let makeOffer = new MakeOffer(          mockTradesRepo,          notificationsSpy,        );
        // Act
        let result = await makeOffer.execute({
          vinylId: '123',
          tradeType: 'money',
          amountInCents: 100 * 35,
        });

        // Assert 
        expect(result.isSuccess()).toBeTruthy();                    
        expect(mockTradesRepo.saveOffer).toHaveBeenCalled();       
        expect(notificationsSpy.getEmailsSent().length).toEqual(1); 
        expect(notificationsSpy.emailWasSentFor())
          .toEqual(result.getValue().offerId);                     
      });
    });
  });
});

Here we have the interface for the TradeRepo:

interface ITradesRepo {
  saveOffer(offer: any): Promise<void>;
}

And here we have the spy for the NotificationsService:

export class NotificationsSpy implements INotificationService {
  private emailsSent: Email[];

  constructor() {
    this.emailsSent = [];
  }

  public async sendEmail(email: Email): Promise<void> {
    this.emailsSent.push(email);
  }

  getEmailsSent() {
    return this.emailsSent;
  }

  emailWasSentFor () {
    if (this.getEmailsSent().length === 0) throw new Error('No emails sent');
    if (this.getEmailsSent().length > 1) throw new Error("More than one email sent");
    return this.getEmailsSent()[0].originatingId;
  }

  async hello() {}
}

The NotificationService interface:

export type Email = { originatingId: string };

export interface INotificationService {
  sendEmail(email: Email): Promise<void>;
}

And a minimal, trivial implementation of the MakeOffer use case.

// modules/trading/useCases/makeOffer/makeOffer.ts

export class MakeOffer {
  constructor(
    private tradesRepo: ITradesRepo,
    private notificationService: INotificationService,
  ) {}

  async execute(request: any) {
    ...
  }
}

That should let our code compile.

We're still pretty red but now, we can focus on getting to green.

ts-auto-mock: You may notice that we're using ts-auto-mock instead of Jest to create mocks. I wrote about why we're doing this in How to Mock without Providing an Implementation in TypeScript.

Adding stub objects

As we start to implement the MakeOffer use case, we realize that we can't really do this without the existence of a VinylRepo of some sort to actually fetch the vinyl entity and determine if it's available for trade or not.

// vinylRepo.ts
export interface IVinylRepo {
  getVinylOwner(vinylId: string): Promise<Trader | Nothing>;
  isVinylAvailableForTrade(vinylId: string): Promise<boolean>;
}

// modules/trading/useCases/makeOffer/makeOffer.spec.ts
import { ITradesRepo, IVinylRepo, MakeOffer } from './makeOffer';import { NotificationsSpy } from './notificationSpy';
import { createMock } from 'ts-auto-mock';

describe('makeOffer', () => {
  describe(`Given a vinyl exists and is available for trade`, () => {
    describe(`When a trader wants to place an offer using money`, () => {
      test(`Then the offer should get created and an email should be sent to the vinyl owner`, async () => {

        // Arrange
        let fakeVinylRepo = createMock<IVinylRepo>();        let mockTradesRepo = createMock<ITradesRepo>();
        let notificationsSpy = new NotificationsSpy();
        let makeOffer = new MakeOffer(
          fakeVinylRepo,          mockTradesRepo,
          notificationsSpy,
        );

        // Act
        let result = await makeOffer.execute({
          vinylId: '123',
          tradeType: 'money',
          amountInCents: 100 * 35,
        });

        // Assert 
        expect(result.isSuccess()).toBeTruthy();                    
        expect(mockTradesRepo.saveOffer).toHaveBeenCalled();       
        expect(notificationsSpy.getEmailsSent().length).toEqual(1); 
        expect(notificationsSpy.emailWasSentFor())
          .toEqual(result.getValue().offerId);                     
      });
    });
  });
});

And the use case changes to:

// modules/trading/useCases/makeOffer/makeOffer.ts

export class MakeOffer {
  constructor(
    private vinylRepo: IVinylRepo,
    private tradesRepo: ITradesRepo,
    private notificationService: INotificationService,
  ) {}

  async execute(request: any) {
    ...
  }
}

It's good that we're discovering this later after we've already written out what we're going to assert: the command-like operations and the result for the use case. Sometimes, when we start with creating the objects in the "arrange" phase, we're tempted to assume that the objects which perform queries are somehow important to validating the correctness of our tests.

For example, in an earlier version of this very post, I was doing this:

expect(fakeVinylRepo.getVinylOwner).toHaveBeenCalledWith('123')

This leads to brittle, hard-to-maintain tests because the queries have nothing to do with the outcomes (state changes + result). If the implementation changes, then we experience ripple and up having to change our tests. And that's yucky. Special thank you to Steve in the comments section for pointing this out.

Double loop TDD and turning this test green

Great start.

Is this enough? Absolutely not. This test will not pass until we have gone in, fleshed out the rest of the MakeOffer use case, and probably also end up creating domain entities for Email, Offer, and probably Vinyl too (each of which we should be unit testing with the inside-out approach).

Some call this approach to TDD double loop TDD. You have an outer loop (the one that we've been writing in this post so far) which shapes the way the use case could work and exercises the scenarios (these are often the acceptance tests), and you have the inner loop which ensures that the smaller, pure domain objects are properly tested as well.

This means that the outer TDD loop stays red for much longer than the inner TDD loop. But once both the outer and inner loops pass, you're done that scenario.

This is a powerful tool for your toolbox, but it is important to remember that it is not mutually exclusive to the classic (chigago-Kent Beck) style of TDD where you pass in real objects.

Integration tests and managed/unmanaged dependencies: There is much to test in a clean (hexagonal, onion, ports and adapters, layer) architecture and it can get pretty complicated to explain. There are incoming adapters and outgoing adapters. Incoming adapters are things like GraphQL and REST. Outgoing adapters are things like databases or facades that we're written to external services like Stripe and so on. Each of these out-of-process, external dependencies are said to be either a managed or unmanaged dependency. Depending on the type, we write different kinds of tests for them. For example, repository objects (adapters to our application database), are managed output adapters. The best way to test it is to confirm that it adheres to contract of the interface - that is, to perform a contract test. Read about managed and unmanaged dependencies here and what each type of test in a testing strategy is for, below.

Using stubs to force code paths

We know that most use cases have typically one happy path and numerous sad paths. By adjusting the input data and setting up stubs to return different data in different contexts, we get the ability to force our code through paths to see if our use cases return the correct error result (on failure) or invoke the commands on dependencies in success cases (as we've seen here).

For example, in ddd-forum, a forum application, look at all the different ways a DownvotePost use case can fail.

import { Either, Result } from "../../../../../shared/core/Result";
import { DownvotePostErrors } from "./DownvotePostErrors";
import { AppError } from "../../../../../shared/core/AppError";

export type DownvotePostResponse = Either<
  DownvotePostErrors.MemberNotFoundError |
  DownvotePostErrors.AlreadyDownvotedError |
  DownvotePostErrors.PostNotFoundError |
  AppError.UnexpectedError |
  Result<any>,
  Result<void>
>

We have to exercise these paths with tests. So we better have the ability to be nimble with our tests.

Part of a larger testing strategy

One of the main ideas why we use a layered architecture is because of the testing options it gives us. Ultimately, you can think of your application as core and infrastructure.

If you understand and enforce boundaries between these two types of code, you have a myriad of ways to set up a testing strategy for your application. Here’s one for a full stack GraphQL + React application for example:

Unit test all smaller, pure, core components like entities, domain services, aggregates, value objects, utility classes and so on
Also unit test your use cases with mocks to confirm that we perform commands against infrastructure dependencies; use stubs to force code to go down different code paths — do this for all of your acceptance tests
Integration test all of your incoming and outgoing adapters (GraphQL API, database, cache, event subscribers, etc)
End to End test — from the front-end to the GraphQL API, through the database and the cache — a select few acceptance tests to get that extra bit of confidence

With that said, your testing strategy could look something like this:

You see? In reality, we want to use a mixture of techniques to cover as much ground as possible.

Further reaading

https://enterprisecraftsmanship.com/posts/when-to-mock/

Existential State Machines

Mon, 24 May 2021 15:04:10 GMT

Originally published in volvox vault's issue on metamorphosis.

Someone said I was a writer the other day. When the hell did that happen? Over the last two years, I've been writing blog posts - blog posts that would help my past self figure out how to stop writing code that looks like it was written by a drunken kitten (possibly more like a kitten at step six of a seven-step rehab program).

When exactly did I become a writer, though? Was it when I hit CTRL+S on that first sentence? Those first few blog posts? Maybe it was when I realized I wasn't writing email newsletters for my auntie and uncle anymore. Or perhaps it was when I looked up and realized I had published over 50 blog posts, and people seem to be finding my book somewhat decent?

I don't know. But I know that my focus wasn't on becoming a writer; it was about consistently doing shit that writers tend to do.

Many of us have goals we'd love to see come to fruition, but for whatever reason, it can be hard to stay on track. You know - you want to write a bestseller, get ripped, and play sold-out shows. But when you realize you've been trying to write the same chapter for months, your abs don't look like a cheese grater yet, and Iggy Pop isn't playing your LP for his pet bird, you might feel like you want to throw in the towel.

Over the last 400 days or so, I've read many books on productivity, philosophy (mainly about ethics, absurdism, and existentialism), and software design. Remarkably, I think there's a fascinating parallel to be drawn here. At least, I'm going to try.

In this article, I'd like to do several things. First, I'd like to help elucidate a better way to meet your goals by creating habits that stick. As James Clear writes in Atomic Habits, habit change has more to do with identity than anything else. To convince you just how much capability you have in rewriting your beliefs, I'll rely on some of the main ideas from existentialism. I then present an exercise for fleshing out the identities, values, and principles that'll lead you to your goals. Finally, we'll wrap this up with a computer science-y conceptual model and technique to start chaining habits together tomorrow.

Identity

In James Clear's research on habits in Atomic Habits, he argues that the critical way to build habits is not to try hard but instead to identify who we want to be.

To accomplish our goals, most people compose a set of habits that they'll follow until they meet them. Only after people achieve their goals do they declare that they are the person they want to be.

Outcome-based belief: "If I write this book and it sells 5000 copies, then I'm a writer."

Indeed, this is one way to go about it. To Clear, these are outcome-driven habits. The thing about outcome-driven habits is that we only feel motivated to continue performing them if we feel like we're getting closer to our goal. The downside of this approach is that any small failure - a step backward instead of a step forward - holds a lot of weight in our lives. Minor setbacks have the power to wreck our entire system. Minor setbacks in the outcome-driven approach to habits are the reason why people sometimes quit their dieting and weight loss routines when they notice that they've plateaued or put weight back on.

Instead, what Clear recommends is the identity-based approach to habits.

The identity-based approach to habits is to:

Decide on who we want to be
Prove that we are that person with small wins, over and over

According to Clear's research in behavioral science and psychology, actual behavior change comes from identity change. Therefore, if we want to write a book, work backward to that goal by seeing ourselves as writers. From there, we ask ourselves, "what are the types of values and principles that writers have"? Clear thinking, meeting deadlines, etc. And what types of habits do they have? They probably wake up and journal every morning, right?

Once we do this exercise and we start to believe that we are the person we want to be, the easier it is to stick to the habits.

Identity-based belief: "I am a writer, so I will do the things that writers do"

Existentialism

If you don't believe you can change your identity, your beliefs, and ultimately, your habits — well, my friend — existential philosopher Jean-Paul Sartre and his (less-buggy-eyed) antecedents have a different opinion on that.

Jean-Paul Sartre taught us that "existence precedes essence." That is, we exist, and then at some point, we recognize that we have a consciousness. And perhaps if we're introspective enough, at another point down the road — we also realize that we have free will, and thus the ability to determine our nature.

"Every action you take is a vote for the type of person you wish to become." — Jean Paul Sartre

While Sartre believed that animals have a sort of predefined nature — their own immutable, unchangeable state machines (a fancy term that we're going to dive into momentarily) — he believed that we have a more privileged concept of being.

Sartre and the other existentialists believed that we'd be lying to ourselves (committing an act of bad faith — as they say) by pretending that we don't have the freedom to choose who and what we are — to fall victim to social pressures (being-for-others) and to never take a moment to consider why one does what one does. You know, like going to college straight out of high school, going for the highest paying job, buying a house, getting married, etc. These are all well and good things if we consciously decide on them for ourselves. But to pretend we don't have a choice in the matter? That's a disservice to what it means to be human. We must choose our being (being-for-itself).

How to change your beliefs

If you have a couple of hours, here's an exercise I've personally applied to my life and found incredibly helpful to put all of this reading to practice:

Write down your biggest goals in life right now (write a book, finish my album, buy a home)
Ask yourself "who is the type of person that would accomplish that goal"?
Label that person as an identity you will believe is a part of you (i.e., "musician", "the entrepreneur", "the writer")
For each identity, write down the top five values they likely hold. You can look to your role models or others who live or have lived that possess the qualities you admire.
Lastly, for each value, write down the top five habits (principles) that those who possess each value are most likely to do. For example, "entrepreneurs probably write a list of todos for the next day so that they know exactly what they need to do when they wake up" and "health-minded people probably cook the majority of their meals using real ingredients".

Now you know who you want to be, you understand the virtuous action behind each identity, and you know why you're doing it. The next step is to prove to yourself that these identities are a part of you.

Relax on single-mindedly hitting your goals; instead, make like the stoics, and focus on nothing else but putting in the reps and sticking to your system. Eventually, you'll see results as a side-effect.

“We are our choices.” — Jean Paul Sartre

Optimizing our behaviour

Based on what Clear says about human behavior, cycles, and sequence, and what we know about existentialism, is it crazy to think that we get a chance to create our very own existential state machines?

For example, he says that going to the bathroom leads to washing and drying your hands, which then "reminds you that you need to put the dirty dishes in the laundry, so you add laundry detergent to the shopping list, and so on".

It's almost like there's an encoding built into our brains, a default set of next steps or state changes. Wait, state?

State machines

In computing, we use conceptual model called state machines to model the way that they should behave over time.

Your washing machine is actually a really great example of a state machine. It would be perfectly valid for the state of the washer to transition from OFF to ON and then perhaps to WASH before being PAUSED upon opening the door to sneak an extra sock or two in. Then, from PAUSED, back to WASH, before finally transitioning to DONE.

Based on what Clear says about human behavior, cycles, and sequence, and what we know about existentialism, is it crazy to think that we get a chance to create our very own existential state machines?

Since we get to decide what we do, is there a way to develop new state machine behavior for ourselves? Can we develop better automatic habits? Productive ones? That might be the key to some top-tier productivity if we can hack it.

Habit-stacking

If you want to add a new habit today, one of the best ways to do it is to identify a current habit and stack your new one right after it.

After I [CURRENT HABIT], I will [NEW HABIT]

For example:

Relationships. After I sit down to drink my coffee in the morning, I will send my partner a good morning text.
Planning. After I turn out the lights in the living room before bed, I will write my todo list for tomorrow morning.

Imagine what your state machine could look like. This works because it's slotting in new things around what we currently already do.

The hard part is to stick to it — to train the synapses of our identities — to make it automatic. And to come back to it when life throws us curveballs. Clear says that the only way to make it automatic is to repeat it enough times. All the more reason to stick to the system. It'll fall into place.

"Before enlightenment, chop wood and carry water; after enlightenment, chop wood and carry water". — Zen proverb

Conclusion

If you can rewrite your beliefs (you can), you can build habits that stick and help you meet your goals. It's fun to think of your behavior as a state machine that you can frequently tune and work on. But it's even more fun to watch yourself accomplishing what you've set out to do by merely focusing on putting in the reps and making it as enjoyable as possible.

What Would Albert Camus Think About Software Development?

Mon, 26 Apr 2021 15:04:10 GMT

🌱 This blog post hasn't fully bloomed. It's very likely to change over the next little while.

Albert Camus was a French philosopher most well known for his ideas about absurdism.

Sometimes regarded as an existentialist philosopher among his peers like Jean-Paul Sartre and Simone De Beauvoir, he contemplated existence, the complexity of the world, and eventually declared that any meaning we find in the world is there because we have personally assigned it.

As software developers, the very essence of what we do is to create something productive and meaningful within the inherent entropy of the world.

As Julian Shapiro says, the world is a patchwork of systems. And in the real world, we are constantly clashing against complexities. We maneuver though clumsy incumbent systems like "making friends in a new city", "finding an apartment that suits our needs", and "getting our driver's licenses renewed".

Software cuts through this complexity. We help people get from A to B with less stress, less effort, faster, and easier.

But not all complexity is equal. In our attempts to cut through complexity, we often create complexity.

I imagine that if Camus were a software developer, his famous quote about suicide could be reworked to say:

"There is but one truly serious philosophical software design problem and that is maintainability. Judging whether code is maintainable or not amounts to answering the fundamental questions of software design. All the rest — whether we should use React, a hexagonal architecture, server-less functions — comes afterwards. These are games; one must first answer."

Our mission with software design (as we discuss in the introduction of solidbook) is fundamentally to implement these two high-level goals - one for users and the other for developers. If we're not doing both well, our software design attempts are failing.

1. [Users] Satisfy the needs of the user
1. [Developers] Do it in the most maintainable way possible

The first goal is accomplished by implementing the essential complexity, that is - the features. If it were possible to use a no-code builder to reliably do this every time, then we'd likely be doing that. However, that's not where we are as an industry, so we write code to realize systems that solve problems for people.

The second goal exposes surface area for accidental complexity. We need to use paradigms, languages, and tools to build software. Some of these introduce inherent concepts that are known to cause (like state and sequence, according to Ben Moseley in his influential "Out of the Tar Pit" paper), and some of them introduce other subtle forms of complexity like:

Cognitive load
Ripple
Poor discoverability
Poor understandability
and leaky abstractions

It's for this reason that I'm starting to believe, more and more, that aligning with the essence of the problem should be the school of thought to which we master software design.

I believe that we should seek out techniques that bring us asympotically closer to implementing the bare minimum - the essential complexity of the problem - and mitigates as much accidental complexity as possible.

Why I Recommend a Feature-Driven Approach to Software Design

Fri, 23 Apr 2021 04:05:26 GMT

Introduction

Here are a few pressing questions:

What is the single-most important concept you can tell me about software design?
How do you structure a (frontend or backend) so that it's testable, flexible, and maintainable?
What am I supposed to test against in my tests? Classes? Methods? React components? How can I write tests that actually give me confidence in my code?

At first glance, these questions might appear to be a little bit unrelated. General software design advice, project structure, testing best practices. Hard to see the link, right?

After having studied software design for a few years, the answer to each of these questions — and nearly all questions about how to write testable, flexible, and maintainable software, is to focus on the features.

That is, I'm telling you to be feature-driven.

Features are the only essential complexity

For "Part II — Humans & Code" of solidbook.io, I had a chance to read Out of the Tar Pit by Ben Moseley.

In it, he discusses the ideas of essential complexity and accidental complexity.

Essential complexity

Software, in its simplest form, is a way to tame complexity in the world. We do this by developing use cases (or features, you may call it) that cut through entropy and perform something useful for us.

If my goal is to "make friends in a new city", I can divide the application that helps me do that up into use cases/features. Three features/use-cases may be:

Sign up
Edit profile
Get friend suggestions

The essential complexity involved in solving the problem of "making friends in a new city" is to first identify all of the features - then, to merely implement them.

Once all the features/use cases/vertical slices are implemented, we're done.

Accidental complexity

However, since we live in the real-world, we have to write code to accomplish this.

And in most mainstream programming languages, the concepts of state and sequence exist.

What's wrong with state and sequence?

Well, according to Ben Moseley, state and sequence are the two leading causes of accidental complexity.

State — "Have you tried turning it off and on again?" Has anyone ever said that to you before? This is what happens when a system ends up in an illegal state. And maintaining state is hard. We have to keep track of variables, instances, etc.
Sequence — "First we do this, and then we need to do this, but only if this happens". Most of us write in languages that support procedures and conditionals. When we have to ensure that things happen in a particular order, we're also introducing surface area for complexity.

Accidental complexity is when the complexity we're faced with is not actually related to the complexity of the problem (like the essential ones — the features). Instead, accidental complexity is related to the way we solve problems.

That means that languages or paradigms that contain state and sequence inadvertently introduce accidental complexity.

Okay, can we write code without state and sequence?

It turns out that we can.

Mitigating accidental complexity with functional programming

This is where (purely) functional programming shines. Scott Wlaschin's Domain-Modeling Made Functional book depicts an approach to get as close to the essential complexity as possible by modelling features/use cases (he calls them workflows) using minimal-to-no state and sequence.

For example, here's some pseudo-code for a feature that can be modelled entirely using F#'s algebraic type system.

From "Domain Modeling Made Functional" by Scott Wlaschin - highly recommended reading.

This is a really thorough approach. I think it's amazing. But real pure-ish functional programming isn't for the faint of heart. This is some seriously disciplined programming. By modeling the entire domain from scratch, similar to the way we build up math equations from first principles, this is truly the way that we make illegal states unrepresentable.

However, most of us want a little bit of state and sequence. Modelling purely functional features like this can be challenging. For many, composing software this way is an entirely different way to think about programming. Even RxJs can be challenging.

https://patrickroza.com/blog/result-composition-and-error-handling/

See Patrick Roza's Result Composition and Error Handling techniques in TypeScript. I currently think that pure functional programming in TypeScript sits further on the structural side than it does on the developer experience side.

Structure vs. developer experience — FP is hard

Like most things in design, there's a push-pull between priorities.

In software design, I call it the balance between Structure and Developer Experience.

You want a structured approach to doing things, but you don't want it to hinder your productivity.

Structure vs. Developer Experience - from solidbook.io

Balancing structure vs. developer experience - from solidbook.io

Therefore, the majority of the web development industry codes in paradigms and with languages that deal with state and sequence (TypeScript, JavaScript, Python, etc).

Well, that's certainly some accidental complexity that we're taking on isn't it? Yes... and it's fine.

But that doesn't mean we don't need to keep tabs on it.

Luckily for us, OOP folks from the 90s discovered a great feature-first way to consistently, reliably, and confidently write code containing state and sequence.

It's called TDD.

Use feature-first TDD to tame (state and sequence) complexity

There's more to Test-Driven Development than just unit testing React components and classes.

The best ROI comes through figuring out a way to continue to add and change code, and ensure that the features of our application still work. This is where the majority of our efforts should lie — because the features are the essential complexity, after all.

The "feature-first way" to do TDD is to do Double Loop TDD.

Outer loop: Write the acceptance test (which can be written as an integration or end-to-end test); this is the essential complexity written in English.
Inner loop : Red-Green-Refactor your components, classes, etc until the outer loop passes.

This isn't a new thing, I just think that we're all struggling to agree on the terminology and recognize that we're all talking about the same things (see this blog post from the Dodds).

A collection of feature-first techniques

Some of the best techniques I've found that help us align with the essence of the problem at hand and cut out the noise (tech stack, language, paradigm, etc) are as follows:

Event Storming
Event Modeling
Use-Case Driven Design
Acceptance Test-Driven Development (also called Double Loop TDD)
... way more, to be continued

Benefits of a feature-first approach

We get a lot of benefits when we apply a "feature-first" approach.

Project structure

At the folder-level:

More cohesive feature folders
More discoverable folder structures
Less "cognitive load" flipping around files/folders
Screaming architecture (easy to remember what the system does)

Overall system simplicity

At the system level:

Consistent testing strategy with a high ROI (acceptance test for each feature)
Simpler architecture
Easy to keep tabs on coupling between features/vertical slices (SRP)

Software craftsmanship

At the professional level

Decouple large projects into sets of features
Provide better estimates
Understand where new technologies fit into a vertical slice (feature)

Conclusion

DDDForum (GitHub Repo)

You can work in a feature-driven way on both front-end and back-end architectures. Take a look at the code for DDDForum, the app we build in solidbook.io.

The backend follows a feature-first project structure, organized by use cases.

The frontend application is currently being refactored to follow a feature-first approach for the next iterations of the Client-Side Architecture Basics guide. In the next iterations, you'll learn:

How to craft a feature-first project architecture
How to consistently evolve code using TDD & Unit, Integration, and E2E tests regardless of your stack

Part II — Humans & code from solidbook.io

I'll be writing more content about this phenomenon on the blog, but if you're looking for an in-depth discussion on how to design human-friendly codebases, the Humans & Code part of solidbook.io just went out last week. We learn how to structure repos, organize things, name them, handle errors (and more) — all in a feature-first way.

Check it out here.

Introduction to Test-Driven Development (TDD) with Classic TDD Example

Wed, 31 Mar 2021 15:04:10 GMT

🌱 This article hasn't fully bloomed. It's very likely to change over the next little while.

Watch on YouTube

If you're like me, you see programming as a type of trade ¹.

Over the past year, I've been paying more attention to how traditional tradespeople work. I've been looking for parallels between how they work and how we work as software developers.

When I most recently took my car in to get an oil change, I noticed that the technicians seemed to consistently, reliably, and confidently get the job done, and get it done right. One client after the next. They weren't making messes. They weren't missing their estimates. They weren't discovering that things won't work at the last minute - in fact, I've even seen them come out, 2 minutes into the job to let someone know that there's more work to be done than they initially expected.

It appears, at least to me, as if these are principled workers, and — forgive the pun, but it seems like they're not reinventing the wheel.

Part of what it means to be an Agile software craftsperson is to consistently deliver value to the customer. That's hard. But that's what we're constantly trying to do here.

The goal of this blog is to discover the best techniques for writing testable, flexible, maintainable code, and to teach others how to do it too. Today, the software quality attribute we're most interested in is testability.

And in my experience:

The best way to write testable code ² is to write the test first

Introduction

This is the first post in a series on Test-Driven Development (TDD): a test-first technique for developing software 🧪.

In this introductory post, you'll build a beginner foundation for TDD. We'll learn about what TDD is, what makes it important, and how developers are using it to consistently deliver value on real-life projects. We'll also discuss what makes testing so challenging to get right, and in the end, we'll wrap up with a demonstration of the Classic TDD process, using TDD to construct a palindrome checker.

You'll learn everything you need to know to get started practicing the TDD Red-Green-Refactor process. This is just the starting point. We need to get this foundation down first before we can learn how to apply advanced TDD techniques in the real-world ³.

Let's begin.

Why tests?

There are lots of reasons why tests are helpful, but the two best reasons, in my opinion, are confidence and feedback.

Confidence ⭐ ⭐ ⭐ ⭐

They say that change is a constant in software development.

To find the confidence to safely add, remove, or refactor code without the fear of introducing bugs and regressions, we need tests.

This is especially important later on in a project when the codebase is much larger than it was at the beginning, and there's way more code that one human being can mentally account for anymore.

How projects with good tests, bad tests, and no tests progress over time. Without tests (or with bad tests), it's hard to continue to safely make progress because new changes introduce regressions. Refactoring without tests is a great way to introduce regressions.

I can tell you from experience that this is not a fun situation to be in— thousands of lines of code into a project with no tests and no safety. It's a great way to turn a promising codebase into an unstable mess.

Feedback ⭐ ⭐ ⭐ ⭐ ⭐

In my opinion, the most important reason for tests is feedback.

Tests give us feedback to let us know:

When regressions have been introduced
What our progress towards implementing a feature looks like
That we actually understand the customer requirements
And if our designs are feasible

Depending on how you think about it, confidence may actually come from the feedback we get from tests.

[Principle]: Listen to your tests — Tests that are hard to write typically signal a deficiency in design. Use the immediate feedback you get from feeling out how hard it is to write a particular test to reconsider the design.

Feedback is so important that Kent Beck lists it as one of the primary values of Extreme Programming: the influential Agile software development methodology.

More reasons to write tests

Measure progress ⭐ ⭐
Sculpt out public APIs ⭐
Understand requirements ⭐ ⭐
Keep a feature in scope ⭐ ⭐
Documentation for other developers ⭐ ⭐ ⭐

Why testing is hard

I didn't start writing tests until later in my career. When I first started out, I knew that we should probably have them, but as for writing them? Yeah, right. Like many new developers, I was lost as to how to even get started. After surveying the landscape, here's why I think testing is hard.

Testing isn't regularly taught in schools

This may not be the case for some readers, but the majority of my college/university/bootcamp-going peers weren't exposed to how to properly test code until a mentor sat down and showed them how to do it.

As an industry, we struggle to agree on testing terminology

Ask two developers what they believe an integration test is. As Fowler writes, there are two completely different notions of what this means, and we still haven't converged on a standard.

This is true for much of the other test types as well (unit, acceptance, E2E, contract, etc).

It also appears that your organization style and role (front-end, back-end, full-stack) play a role in how we see certain types of tests as well.

Understanding what to test and how to test it takes practice

Knowing what to test is hard. And this is especially misleading for newer developers since a lot of libraries and frameworks tell you how to test code within their library or framework, but don't actually give you best practices for testing your code within them.

The magic rule here is to test against behavior. And if you understand the requirements, we can pretty much turn those user stories or customer requirements directly into tests.

[Principle]: Prefer tests against behavior, not implementation — Seek to test "behavior" using the language of the domain to write the tests. There will be times when the tests you need to write are actually against more technical concepts (ie: integration tests), but you can always write your tests to test behavior, not implementation.

As for knowing how to test behavior— that's a different story.

Most of the time, the code we write is more complex than plain ol' vanilla JavaScript or TypeScript. Typically, we're writing code that relies on dependencies like web servers, caches, databases, and even front-end library code like React or Vue.js ³.

It takes some up-front planning and foresight to figure out how we're going to test our code in most of these scenarios.

We leave testing until the very end

I also think that testing is hard because developers often write tests after the production code has been written. While this approach sometimes works, I don't think it's the best way to go about things.

Writing all the tests at the end isn't really acting like we value feedback, because we're leaving all the uncertainty of "if we can even test this thing", "if this thing was designed well", and "if this thing even works" to the very end.

Testing is a part of architecture

This has been a massive realization for me. And I hope it will be for you as well. Testing is a part of architecture.

Ralph Johnson, co-author of the famous design patterns book said this of architecture:

"... it is the decisions you wish you could get right early in a project”

Architecture is about the expensive, hard to change stuff like choosing a tech stack (React, Apollo, GraphQL, Mongo), an architectural style (Reactive, Event-Driven, Transaction Script), or in this case — our testing approach.

The trouble with not thinking about how we're going to test early on is that we're leaving a lot of room for uncertainty down the road towards the end of the project.

We're not sure if there are edge cases we're missing, if there are structural problems with the way we've written our code, and if we're even going to be able to test the thing.

Your stress level over time when you realize code is hard to deploy and test towards the end of the project.

[Principle]: Expose uncertainty early - Decide on how you're going to test and deploy your application at the start of the project. Get your test architecture set up in Sprint 0.

As Agile software developers, we should value feedback, exposing bad design and uncertainty as early as possible.

"I'm not a great programmer, I'm just a good programmer with great habits" — Kent Beck, the creator of Extreme Programming

At this point, I hope you're sold on what tests can do for us and why we'd want them.

Now let's talk about the TDD process.

The Test-Driven Development (TDD) Process

TDD (test-driven development), is a technique — or a process for developing software. The goal is to keep code quality high and keep you productive, even as projects grow to be really large and complex.

Red-Green-Refactor

The TDD process works by following the Red-Green-Refactor loop. It goes:

Red — Write a failing test
Green — Write just enough code that will pass the failing test
Refactor — Criticize the design and refactor the code, keeping the tests intact

We should like this process because it keeps tight feedback loops. It gives us the ability to produce cleaner, simpler designs and helps us introduce abstractions only when they are absolutely necessary (see YAGNI — You Aren't Gonna Need It).

Should I always follow the TDD loop?: I know what you're thinking. Khalil, you can't expect me to completely follow this rule. You probably don't even do this. You're right. I don't always follow it. Rules are a great way to get started, but sometimes I break them. When I'm driving, I'll sometimes perform a rolling stop instead of coming to a complete stop. When I'm crossing the street, sometimes I'll take a quick gander to my left and right before jaywalking. I'm of the mindset that if you master this technique, you'll have the skill to decide when to break it, and to do so with confidence.

Types of tests (unit, integration, E2E, acceptance)

There are different types of tests, and you can apply the Red-Green-Refactor process to each of them. As we mentioned earlier, the scopes of these tests are up for interpretation, and they may be slightly different if you're a front-end or back-end developer, but in general, they are:

Unit ⭐ — Test an individual, isolated component
Integration ⭐ ⭐ ⭐ ⭐ — Test that multiple units work together OR "tests that confirm our code works against code we don't own" (like external APIs, databases, caches, etc)
End-to-End ⭐ ⭐ — Tests that act as a user actually using the application; tests the entire stack from top-to-bottom
Acceptance ⭐ ⭐ ⭐ ⭐ ⭐ — Domain-driven tests that verify a user story (also comparable to a use case, customer test, command/query, feature, or vertical slice) works as expected.

The big picture: How TDD works on real-life projects

Allow me to give you the big-picture so you can see where we're going with TDD and why I find it so incredibly powerful.

In the real-world, a good testing architecture typically involves more than one type of test.

In Extreme Programming, tests are a mandatory part of planning and feedback loops.

As written about in the influential "Extreme Programming Explained" and "Growing Object-Oriented Software Guided By Tests" books, the test we start with is the acceptance test: that is — the test that most closely represents the feature we want to build. From here, we build out the internals of the feature using other tests including unit tests.

Starting with the acceptance test

Starting with the acceptance test, we convert the user story into a behavioral test written using the exact same language from the domain. This means our tests should read like plain English, and represents the exact user story that we're about to build.

Acceptance tests read like domain-driven customer requirements. These are the user stories, commands/queries, use cases, and so on.

From here, we identify the objects and their public APIs (properties, methods, etc) for the feature we need to realize while maintaining a single layer of abstraction. When we need to, we step a layer deeper into the objects we come up with and write more specific tests.

This technique is called Double Loop TDD and it's how we sculpt our solution to fit our tests.

Double Loop TDD

The idea of Double Loop TDD is to maintain two TDD loops.

The outer loop:

is the acceptance test loop
and it catches regressions

The inner loop:

is the unit test loop
and it measures progress towards implementing a feature

When we're writing tests for the outside acceptance test loop, we're usually coding from the outside-in. When we're writing tests for the inner unit test loop, we're coding inside-out.

Inside-Out and Outside-In. These are also names for two schools of thought for TDD.

Classic (Inside-Out/Chicago) and Mockist (Outside-In/London) TDD

Classic TDD (also known as Inside Out or Chicago style TDD) is a style of unit testing where we start from the unit tests, and build outwards, fleshing out the internal details of the objects we know we need. This is probably the most straightforward and commonly known form of TDD.

Whereas with Mockist TDD (also known as Outside In or London style TDD), we start at the boundaries of the application (typically the controller) and end up coding inwards, creating mock objects to build out the shape of the system and the APIs of objects we know we're going to need to fulfill the acceptance test.

Both styles of TDD are useful. By using a mixture of both inside-out and outside-in, we can compose an elegant approach to TDD our way through an entire project.

[Principle]: Inside-Out when you know how to build it, and Outside-In when you're working out the pieces.

We've covered about 70% of the essential theory for TDD so far. If your head is spinning, that's OK. One of the goals in this introduction is to show you the big picture of TDD and how it's used to develop software from start to finish.

Next, you'll want to get the technique down.

Let's walk through a demonstration.

Reminder: If you prefer to see the demonstration visually, you can watch me code it here.

Demonstration: Classic TDD — Palindrome Example

We'll build a palindrome checker using the Classic TDD approach.

Requirements

Create a palindrome checker

It should be able to detect that a string is a palindrome: that is, it is the same word or phrase in reverse.

Words like "mom" are palindromes

Words like "bill" aren't palindromes

It should still know that something is a palindrome, even if the casing is off. So that means that "Mom" is still a palindrome.

It should also be able to detect palindromes in phrases like "Was It A Rat I Saw" and "Never Odd or Even" too.

Rules

So we know we're going Red-Green-Refactor. According to Robert. C Martin, the three rules of TDD for this are:

You are not allowed to write any production code unless it is to make a failing unit test pass

You are not allowed to write any more of a unit test than is sufficient to fail; and compilation failures are failures.

You are not allowed to write any more production code than is sufficient to pass the one failing unit test

Getting started

I recommend trying this example as well using the Simple TypeScript Starter repo. To get started, paste this command in your terminal.

git clone https://github.com/stemmlerjs/simple-typescript-starter.git cd simple-typescript-starter npm install && npm run test:dev

1. Write the failing test

The first test I'll write is that the palindrome checker can tell that "mom" is a palindrome. I will:

Write the test name using the requirements

Pretend that something called palindromeChecker exists and that it has an isAPalindrome method on it.

Expect the method to return true for mom.

Save

index.spec.ts

describe('palindrome checker', () => { it('should be able to tell that "mom" is a palindrome', () => { expect(palindromeChecker.isAPalindrome('mom')).toBeTruthy(); // ❌ }); })

At this point, our test should be failing because nothing named palindromeChecker exists and it doesn't have an isAPalindromeMethod.

2. Write the simplest code to make the test pass

Let's move to green. In this next step, we:

Create a PalindromeChecker class

Give it an isAPalindrome method

Return true (the simplest thing that would work)

and import it in our test

Implementing this, the index.ts containing our PalindromeChecker looks like this:

index.ts

export class PalindromeChecker { isAPalindrome (str: string): boolean { return true; // This is the simplest thing! } }

And our test file now looks like this:

index.spec.ts

import { PalindromeChecker } from './index' describe('palindrome checker', () => { it('should be able to tell that "mom" is a palindrome', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome('mom')).toBeTruthy(); // ✅ }); })

Our test passes 🎉.

Let's move to the next step.

3. Refactor

When refactoring, keep a lookout for duplication (at least three times) and code smells.

However... There's none here yet, so let's move to the next failing test.

4. The next failing test

Let's grab the next test. We're going to test that "bill" isn't a palindrome. Our tests should look something like this now.

index.spec.ts

import { PalindromeChecker } from './index' describe('palindrome checker', () => { it('should be able to tell that "mom" is a palindrome', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome('mom')).toBeTruthy(); // ✅ }); it('should be able to tell that "bill" isnt a palindrome', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome('bill')).toBeFalsy(); // ❌ });})

Since our test fails, we're in the red phase.

Let's turn it green.

5. Write the simplest code to make the test pass

Currently, this is what our PalindromeChecker class looks like:

index.ts

export class PalindromeChecker { isAPalindrome (str: string): boolean { return true; // This is the simplest thing! } }

There are several ways to go from Red to Green. If we're really writing the simplest possible thing that would work, then we'd likely find no simpler thing to do than this:

index.ts

export class PalindromeChecker { isAPalindrome (str: string): boolean { if (str === 'mom') { return true; } else { return false; } } }

But we know where this leads — so let's implement the Obvious Implementation of reversing the string and seeing if it matches the originally provided one.

index.ts

export class PalindromeChecker { isAPalindrome (str: string): boolean { const reversed = str.split("").reverse().join(""); return reversed === str; } }

And if we save that, we should notice that our test passes.

index.spec.ts

import { PalindromeChecker } from './index' describe('palindrome checker', () => { it('should be able to tell that "mom" is a palindrome', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome('mom')).toBeTruthy(); // ✅ }); it('should be able to tell that "bill" isnt a palindrome', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome('bill')).toBeFalsy(); // ✅ }); })

Nice — now we're in Green again.

6. Refactor

Refactoring doesn't just mean refactoring production code. It means refactoring our test code too. And you may have noticed that there's some duplication in our test setup.

index.spec.ts

import { PalindromeChecker } from './index' describe('palindrome checker', () => { it('should be able to tell that "mom" is a palindrome', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome('mom')).toBeTruthy(); // ✅ }); it('should be able to tell that "bill" isnt a palindrome', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome('bill')).toBeFalsy(); // ✅ }); })

However, we're going to use the Rule of Three to remove duplication. So we'll leave it for now.

[Principle]: Use the Rule of Three to remove duplication —Introducing abstractions too early is one cause of poor design. Let's wait until we see duplication three times before we make an effort to refactor it.

7. The next failing test

The last test I'll demonstrate is the test case that specifies that a word is still a palindrome, even if it includes capitals and lowercase letters. The business rule here dictates that a word is a palindrome regardless of if it is capitalized or not.

Let's write the failing test.

index.spec.ts

import { PalindromeChecker } from './index' describe('palindrome checker', () => { it('should be able to tell that "mom" is a palindrome', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome('mom')).toBeTruthy(); // ✅ }); it('should be able to tell that "bill" isnt a palindrome', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome('bill')).toBeFalsy(); // ✅ }); it('should still detect a palindrome even if the casing is off', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome("Mom")).toBeTruthy(); // ❌ });})

Aha! Look. We have that duplication we were talking about a moment ago. But we're not in the refactor phase yet. We're about to turn this Red test Green.

We impose a little bit of discipline here and shift our focus to making this test pass.

8. Write the simplest code to make the test pass

At first glance, the simplest thing to do is to merely add .toLowerCase() to both the original and the reversed strings. So let's do that.

index.ts

export class PalindromeChecker { isAPalindrome (str: string): boolean { const reversed = str.split("").reverse().join(""); return reversed.toLowerCase() === str.toLowerCase(); } }

And if we save it and check our tests, we should see it pass as well.

index.spec.ts

import { PalindromeChecker } from './index' describe('palindrome checker', () => { it('should be able to tell that "mom" is a palindrome', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome('mom')).toBeTruthy(); // ✅ }); it('should be able to tell that "bill" isnt a palindrome', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome('bill')).toBeFalsy(); // ✅ }); it('should still detect a palindrome even if the casing is off', () => { const palindromeChecker = new PalindromeChecker(); expect(palindromeChecker.isAPalindrome("Mom")).toBeTruthy(); // ✅ }); })

9. Refactor

And now that we're in the refactor phase, we can look to improving that duplication we saw earlier.

Jest, our test runner, has a way to specify things that we must do before each test. And creating the PalindromeChecker is what we'd like to do here.

We can clean up our code with the following:

index.spec.ts

import { PalindromeChecker } from './index' describe('palindrome checker', () => { let palindromeChecker: PalindromeChecker; beforeEach(() => { palindromeChecker = new PalindromeChecker(); }) it('should be able to tell that "mom" is a palindrome', () => { expect(palindromeChecker.isAPalindrome('mom')).toBeTruthy(); // ✅ }); it('should be able to tell that "bill" isnt a palindrome', () => { expect(palindromeChecker.isAPalindrome('bill')).toBeFalsy(); // ✅ }); it('should still detect a palindrome even if the casing is off', () => { expect(palindromeChecker.isAPalindrome("Mom")).toBeTruthy(); // ✅ }); })

That looks good to me!

Continue

We can continue on like this by writing more and more failing tests, making the code more generic to make them pass, and improving the design.

I'll leave the next two tests for you to do.

index.spec.ts

import { PalindromeChecker } from './index' describe('palindrome checker', () => { let palindromeChecker: PalindromeChecker; beforeEach(() => { palindromeChecker = new PalindromeChecker(); }) it('should be able to tell that "mom" is a palindrome', () => { expect(palindromeChecker.isAPalindrome('mom')).toBeTruthy(); }); it('should be able to tell that "bill" isnt a palindrome', () => { expect(palindromeChecker.isAPalindrome('bill')).toBeFalsy(); }); it('should still detect a palindrome even if the casing is off', () => { expect(palindromeChecker.isAPalindrome("Mom")).toBeTruthy(); }); it('should be able to tell that "Was It A Rat I Saw" is a palindrome', () => { // You do this one! }); it('should be able to tell that "Never Odd or Even" is palindrome', () => { // And this one! }) })

Conclusion

We've covered a lot of ground here.

We discussed why we want to write tests, what TDD is and why it's important, what it looks like in the real world, and we demonstrated the basic technique: implementing the Red-Green-Refactor loop using Classic TDD.

Homework

Master the Classic TDD technique. This should feel like breathing before we jump into Outside-In TDD and combine 'em to do Double Loop TDD. Try the Red-Green-Refactor process out on more coding katas.

Try this out when you're building your domain objects (like value objects and entities).

Next in the series

How to Test Code Coupled to APIs or Databases

More coming

You also likely have questions at this point:

"how do you write acceptance tests on the front-end"?

"what is the difference between an acceptance test and an integration test"?

In that case, leave a question in the comments down below or let me know what you'd specifically like to see.

To see where we're going next with this series, check out this twitter thread and take a look at the new outline for solidbook.io.

[1]: Some developers see programming as an art, math, science, etc. I think there are really good arguments for all of these ways to look at programming, but here's my way of thinking. If you're being paid to write code to make money, save money, or protect revenue, then you should treat the work you do as a craft, because the negative economic consequences for doing it poorly far contrast those of programming for exploratory, experimental, philosophical, or creative reasons.

[2]: There are two broad-stroke categories of test: white box and black box. White box testing is more concerned with what's on the inside. You could say these are tests for the developer. Black box tests are more concerned with the observable output and less about the internals or what's going on within the code. You could call these tests for the customer. When we refer to testable code, we're referring to the mean between these two extremes. If your application adhered to CQS and a query doesn't change the system, then you could technically black box test any badly written code from the outside. It is significantly more difficult (maybe even sometimes impossible) to successfully white box test badly written code. This is why it's a good idea to treat tests as architecture - to start with it before writing any code.

[3]: Architecturally speaking, there is core code and there is infrastructure code. Core code is the code that you use your hands to write. On the backend, with respect to the clean or hexagonal architecture, this is application and domain-layer code. On the front-end, with respect to client-side architectural layers, this is the interaction layer and UI logic. It's the code that you own. Infrastructure code is the stuff that you don't own. Databases, caches, and web servers on the backend and view layer libraries, GraphQL clients, browser APIs and HTTP clients on the front-end. In the real-world, we use both. And this is where many of our testing challenges arise. How do we decouple our core code from the infrastructure code so that we can test our code, and test it fast? These are questions that are better asked and answered earlier than later on in a project.

GraphQL Schemas vs. RESTful DTOs

Fri, 12 Mar 2021 15:04:10 GMT

In the last post, I mentioned the fact that DTOs (Data Transfer Objects) and GraphQL schemas serve a similar purpose; they create a strict abstraction layer that the client (web, mobile, or desktop apps) can rely on. This leaves backend services to merely figure out how to implement the data.

It creates a relationship that looks like this:

If you're a subscriber here, you know that I value the emails I get in response to my bi-weekly newsletters. One reader wasn't so convinced about the similarities between GraphQL schemas and RESTful DTOs in the last post and had some criticisms to share (thank you for doing so).

In this post, I'd like to examine those criticisms for GraphQL as a RESTful replacement and share some principles for managing graphs in production.

Opinion: GraphQL exposes everything

The first piece of criticism was:

"A DTO explicitly limits what is exposed to the client, whereas I'm finding with GraphQL that developers feel encouraged to expose everything but the kitchen sink up front."

Here, the reader is referring to the fact that to create a DTO, we need to explicitly type it out by hand.

For example, a DTO for a vinyl-trading application, say VinylDTO, could be composed to look like this:

type Genre = 'Post-punk' | 'Trip-hop' | 'Rock' | 'Rap' | 'Electronic' | 'Pop'; interface TrackDTO { number: number; name: string; length: string; } type TrackCollectionDTO = TrackDTO[]; // Vinyl view model / DTO, this is the format of the response interface VinylDTO { albumName: string; label: string; country: string; yearReleased: number; genres: Genre[]; artistName: string; trackList: TrackCollectionDTO; }

You may be looking at that and thinking, "man — this sure does look a lot like a GraphQL schema". And I'd opt to agree with you there. The GraphQL schema version of this might look like:

enum Genre { PostPunk, TripHop, Rock, Rap, Electronic, Pop } type Track { number: Int! name: String! length: String! } type Vinyl { albumName:String! label: String! country: String! yearReleased: Int! genres: [Genre!]! artistName: String! trackList: [Track!]! }

So where are we getting the "developers feel encouraged to expose everything but the kitchen sink upfront" sentiment from?

I think it's coming from the fact that there are tools out there to auto-generate GraphQL CRUD APIs for you. In those cases, if you're intending to use your auto-generated GraphQL API as your public GraphQL API, then yes — you're exposing everything to the client.

But this simply just isn't the way we recommend building GraphQL APIs at Apollo.

Have you read Principled GraphQL?

Principled GraphQL

Principled GraphQL is Apollo's best practices guide for creating, maintaining, and operating a data graph. It's what we practice and preach at Apollo.

As someone who maintains a website about software design principles and best practices, I really respect the fact that Principled GraphQL puts a stake in the ground for what it means to do GraphQL properly. It's written by the founders of Apollo based on thousands of conversations with developers implementing GraphQL at companies of all sizes. Conversations with those with practical experience; now that's the way to discover phronesis.

Principle #5 is:

Use An Agile Approach to Schema Development

One of the main ideas about this principle is that "each field should be added only in response to a concrete need by a consumer for additional functionality".

This means that schema design is an iterative process and you should only be adding things to your graph that is actually needed by the consumers of your graph. This means collaboration. It means conversations. This is "Domain-Driven GraphQL Schema Design".

Any field not explicitly and decidedly needed by the client is an implementation detail. It's likely that there are some things in your database that shouldn't be exposed to the client directly. For example, if you were doing CQRS, in a command, there could exist an invariant within an aggregate that operates on a particular piece of state that doesn't have a read model equivalent. Consider an insurance application. You might assign a user a score and save that to their profile. Perhaps it determines what they get approved for. Perhaps this is something that you don't want to expose to the client directly.

Opinion: You can't change already added fields

The other main piece of criticism I noticed was that you can't change fields. They said:

"The server must still maintain all of the fields it initially made available or risk breaking clients", and "now there is no backing out."

I can see why you'd think that, but there is backing out.

GraphQL APIs are tasked with the same set of challenges that any API layer technology is tasked with. Among those challenges, the big question of "how do we make changes to the API and deprecate parts of it over time" arises.

Well, in GraphQL there's no concept of an API version like v1 or v2 in REST. Such a way to version your API is coarse and can become quite an engineering effort.

Instead, GraphQL gives you a declarative and fine-grained way to keep track of, evolve, and deprecate parts of your API.

Use a schema registry

Principle #3 is to:

Track the Schema in a Registry

Similar to how we track source code in Git, the idea is that we should be tracking what our graph looks like overtime in a schema registry.

I highly recommend the talk, "The Do's and Don'ts of GraphQL Schema Design" by Michael Watson — Solutions Engineer at Apollo.

The schema registry keeps track of the history of the graph and it can be used to model graph variants (like example, staging, production, or dev branches).

How to track your schema in a registry: You can get started with a local dev graph in Apollo Studio and then publish variants of your graph to a managed schema registry when you're ready. Read the docs on schema reporting and get started with a local graph here.

Now, if you're familiar with REST — you've probably set up different example, staging, and production environments before. Not much difference there besides a more explicit and declarative way to do this.

The next part is uniquely GraphQL.

Deprecating fields

You can deprecate old fields. Here's what that might look like in your schema:

type ExampleType { oldField: String @deprecated(reason: "Use `newField`.") newField: String }

GraphQL gives you the ability to deprecate fields, signaling to clients that they should probably use the new fields, with GraphQL directives.

That's great and all, but how do we do this safely?

Client awareness

Apollo has a cloud platform with a variety of tools that help you build, explore, and manage your graph in production: it's called Apollo Studio.

One of the many features in Apollo Studio is GraphQL "Client Awareness". Client awareness lets you know ahead of time which clients are using which fields in your Graph and how much traffic those fields get.

This is probably one of the safest ways to know when you can pull a deprecated field from your graph.

With this tool, a workflow to deprecate and update a field might look like this:

Create a new field and deprecate the old one using the @deprecated directive (perhaps optionally recommending the new field to use instead).

Publish the graph changes to a schema registry.

Give teams time to start using the new field.

Check Apollo Studio to confirm that the new field's traffic is ~zero.

Create a PR to remove the old field.

Deploy the change to production.

Conclusion

GraphQL schemas create a strict abstraction layer between the client and the backend services, similar to DTOs in REST. Since they're both API layer constructs, they share a lot of the same problems. What we've found makes the real difference is a principled approach and great tooling.

Recommended

The Do’s and Don’ts for your schema and GraphQL operations by Michael Watson

PrincipledGraphQL by Apollo

How to configure Apollo Server directives

Use DTOs to Enforce a Layer of Indirection | Node.js w/ TypeScript

Wed, 24 Feb 2021 15:04:10 GMT

Have you ever heard the expression that "all problems in computer science can be solved by adding another layer of indirection"?

I'm not sure about all, but if the structural problems in software design can be reduced to being either an issue with coupling or cohesion, then yeah - this makes a lot of sense.

Indirection

To understand the concept of indirection, we need to understand the fact that software doesn't do a whole lot unless we can connect things together. This means that components (whether it be a class, object, package, server, or client application) typically rely on other components to do something meaningful.

Take web applications for example. Most web applications implement the client-server architecture. This usually means that the client relies on the server.

And if we're being realistic, it's a lot more likely that there will be several clients that rely on the server (like a web, mobile, and desktop client). Notion is a great example of this. They have a web, mobile, and desktop app.

In this dependency relationship, the clients rely on the server. A lot. It means that if, for some reason, the server were to change things or break, that change or breakage has the potential to ripple into each client that relies on it.

This is the nature of tight coupling. It puts the clients in an awkward position of needing to constantly keep on top of the latest changes to the server and protect itself against failure. A substantial change could break the entire client. This isn't great. Not great at all.

In fact, just this week, I realized that the browsers introduced a new security feature called strict-origin-when-cross-origin that was stopping solidbook.io readers from getting into the online Wiki. Since my application depends on the browser, it means I'll always have to keep on top of things the browser does that could cause my app to break.

What can we do about this problem?

Introduce a layer of indirection

Let me walk you through a real-world example of how to use DTOs — Data-Transfer Objects to introduce stability and protect clients from change in a RESTful API.

A trivial RESTful API controller

Consider this very stripped down, simple controller that pulls the current user and returns a raw Sequelize ORM object as a result of the GetCurrentUser RESTful API call.

const GetCurrentUserController = (sequelize) => { return async (req: Express.req, res: Express.res) => { const userId = req.decoded.userId; const user = await sequelize.db.User.findById({ where: { user_id: userId } }); return user; } }

For a long time, this is how I used to build out my controllers in my RESTful API layer.

We're not doing anything fancy. We're just using an ORM to get the data we need and return it to the caller. Straightforward, right?

What's the shape of the user object on our RESTful API? Because it's hooked up directly to our ORM, it'll return whatever that, the ORM, returns.

{ firstName: 'Khalil', lastName: 'Stemmler', id: 1, email: `khalil@khalilstemmler.com`, isAdmin: true, isStudent: false, isEmployer: false, }

Cool. That's alright. The design could be better. But let's imagine that this is what we've chosen and it's been like that for quite some time now. Consider that client teams have used our API and built things on top of it already.

Now this happens.

We want to change the API

After months of living with our design, we realize that we don't like it — particularly the isAdmin, isStudent, isEmployer part.

We realize that we're moving towards having more and more roles and that hard-coding the roles as a part of the user entity actually breaks domain encapsulation. The user entity has become somewhat coupled the specifics of other subdomains (students, jobs).

An astute software designer may also recognize that this is a violation of the Open-Closed principle. A module should be open for extension but closed for modification, right? But it seems that every time we want to add a new role (ie: extend this entity's use case), we have to modify it directly (by adding a new column to the database in the form of an is[role] structure). Not good.

We goofed up here. That's OK. The design got us to this point, and that's what matters. We're here today and we want to refactor it.

We decide that it would make more sense to have something like this instead:

{ firstName: 'Khalil', lastName: 'Stemmler', id: 1, email: `khalil@khalilstemmler.com`, roles: ['Admin'] }

That's much better. The concept of roles seems much more appropriate within a users subdomain. Not only that, but we're doing good by the Open-Closed Principle: we can extend this entity by adding a role, not changing the structure of the entity.

Lovely.

So, we go ahead and make the change by writing a Sequelize migration, looking through our code and changing every controller that mentions a user to do the following.

const GetCurrentUserController = (sequelize) => { return async (req: Express.req, res: Express.res) => { const userId = req.decoded.userId; const user = await sequelize.db.User.findById({ where: { user_id: userId }, include: [ { model: sequelize.db.Role, as: 'Roles' }, ] }); return { ...user, // This is how we maintain the structure isAdmin: user.Roles.find((r) => r.name === "Admin"), isStudent: user.Roles.find((r) => r.name === "Student"), isEmployer: user.Roles.find((r) => r.name === "Employer") } }

Hopefully we've found and changed every place in our codebase where we return a user; because if we miss one, we might break the API for our clients.

Disadvantages to this approach

Not only is this messy, but even despite best efforts, it's too easy to break. Clients are practically hooked up directly to any changes we make to our database. There is almost no encapsulation from the implementation details.

The main issues here are that:

Our API is not stable

It forces us to get the design absolutely correct the first time around (pretty unlikely)

We'll have to scour our codebase for other places affected by database changes

It makes implementing API changes incredibly hard

Drives us to tolerate bad design decisions because changing them is just too much of a hassle

And we know where this leads — it's the broken windows theory. Software rot is just down the road.

Enforcing a layer of indirection

Using Data Transfer Objects

DTOs are data transfer objects. They're not much. They're simple interfaces or classes that represent the shape of your API.

The idea is to use DTOs to create a contract for your API.

If we modelled the initial shape of the user entity as a part of our RESTful API, we'd likely have a DTO that looked like this.

interface UserDTO { firstName: String; lastName: String id: Number; email: String; isAdmin: boolean; isStudent: boolean; isEmployer: boolean; }

Then we'd use another pattern, the data mapper pattern, to hold the responsibility of mapping the raw sequelize object to the correct DTO shape

const GetCurrentUserController = (sequelize) => { return async (req: Express.req, res: Express.res): Promise<UserDTO> => { const userId = req.decoded.userId; const user = await sequelize.db.User.findById({ where: { user_id: userId }, include: [ { model: sequelize.db.Role, as: 'Roles' }, ] }); return UserMap.toDTO(user); } }

This can be improved even further by encapsulating the ORM logic in a repository.

const GetCurrentUserController = (userRepo: IUserRepo) => { return async (req: Express.req, res: Express.res): Promise<UserDTO> => { const userId = req.decoded.userId; const user = await userRepo.findUserByUserId(userId); return UserMap.toDTO(user); } }

Advantages to this approach

Now instead of relying on the server (and everything the server relies on), the clients rely only on the DTOs.

The advantages are that:

The DTOs become a strict contract for our API. The clients rely on it. The server implements it.

We've implemented architectural dependency inversion between the client and the server.

The DTOs act as a layer of indirection and shield the clients from internal changes to the way the API is resolved.

Data mappers act as a single location for object transformations. Instead of going through the codebase and changing API code that deals with raw sequelize users all over the place, it's done in one place.

If this smells a lot like the things GraphQL does, it's because it is. GraphQL acts as a strictly-typed abstraction that clients rely on and services implement.

Different objects for different needs

I think a layered architecture is one of the best things you can do for non-trivial web applications of complexity.

You have domain objects (that your domain layer needs), data transfer objects (that your clients need), and the raw sequelize objects (that your database needs). The mapper handles transformations between all of 'em.

Continue reading

Want to learn how to do implement this in your next project?

Read the next post here: Implementing DTOs, Mappers & the Repository Pattern using the Sequelize ORM [with Examples] - DDD w/ TypeScript

Transcribing Video With DDD Discussion | Domain Driven Design w/ TypeScript

Mon, 22 Feb 2021 15:04:10 GMT

I'm always interested in what blog readers are building using Domain-Driven Design and Clean Architecture practices.

In this post, I'd like to share an email exchange I held with a reader who is working on encapsulating the complexity of building an application capable of transcribing videos given their public URL.

Very neat!

If this is something you're looking into doing, awesome. If not, hopefully this discussion is enriching anyway. I often like to be a fly on the wall in technical discussions, and by the law of the internet - I assume someone else must as well!

Hi Khalil! I've been impressed by your articles and I bought your online book, solidbook.io. And also I've been practicing and introducing DDD in real projects.

Recently, I've taken a task to handle streaming media. Download media files and reformat them. And transform the audio of media into a text. And then combine them into one media streaming like HLS with VTT. All of the above processes are conducted with "stream".

At this point I wonder how I can handle "stream" in DDD? Should I have to handle stream as domain object? Or infra? With traditional REST API app, I've almost understood concept of DDD and applied several DDD to several projects. But it is not easy to handle things like stream :( Can you share your experience or inspiration? Thanks in advance :)

"Your question,

"Should I have to handle stream as domain object? Or infra?"

The answer to your question is likely both. Though it depends. I'll explain.

First, let's recall the concept of high-level vs. low-level policy.

High-level policy is what we do

low-level policy is how we do it

The high-level policy is the domain layer and the low-level policy is your infrastructure layer

Domain layer:

entities & aggregates — protects invariants around object behavior & state changes

value objects — protects invariants around validation and object creation logic

Infrastructure

repositories — object persistence and retrieval

Even highly technical domains centered around expensive I/O things can benefit from decoupling the domain from infrastructure. Consider building a new database technology, a virtual synthesizer, or in your case — streaming media.

These are all heavily I/O-heavy kinds of problems to solve. They interact with files, upper-layer protocols, user inputs, and sound. A real-world example that comes to mind is actually what powers wiki.solidbook.io. I wrote a library that converts Notion markdown pages into a website, PDF, and EPUB.

The domain layer has objects that represent these I/O concepts as domain objects.

But the majority of the I/O lives in repositories. For example, if I wanted to create a file, I'd create a file entity (high-level), the pass it to the fileRepo to persist it to the file system (low-level) at the path specified by the getAbsoluteFilePath method.

The whole reason we use abstraction layers and decouple domain from infrastructure is to separate concerns, decompose complexity, and increase cohesion by maintaining a single level of abstraction at a time.

OK, Khalil — I get it. Now, how would you do this? How would you go about solving this?

1. Event Storming or Event Modelling

Just like we walk through with the forum app we build in solidbook, I'd recommend you start with what you should do on any domain-driven design project: use Event Storming or Event Modelling to discover the domain (ie: the high-level policy and what we do).

This is a collaborative activity that you'd normally do with your team and domain experts (the customer or a representative for the customer), but I'm making a ton of assumptions on my own here.

Here's what I've landed on (you might want to download or right-click and open in a new tab to see the image better).

Commands: StartStreamJob (this is the only public one done by the user — the rest are done by the system itself), ExtractTextFromAudio, CombineMedia, CompleteStreamJob.

Events: StreamJobStarted, ExtractedTextFromAudio, CombinedMedia, StreamJobCompleted.

Now, this might not be exactly correct, but this is what I got from reading your email. You'll want to do this yourself with your team (which hopefully includes the customer or a representative for the customer like a product manager).

We now have all the use cases.

Next, we should probably discussion how much you want to handle vs. how much you want to outsource.

Do you want to write the transcoding, transcribing, and reformatting logic yourself or would you prefer to use a service that does it well already?

I'd personally recommend using a service like AWS. They have:

Transcode/reformat — https://aws.amazon.com/elastictranscoder/

Transcribe Audio (get the text from audio) — https://aws.amazon.com/transcribe/. For example, here's the API to start a transcription job using the Transcribe API.

This stuff is going to live in infrastructure. We'll write adapters for this, and if we want, we can switch any of these things out and write our own implementation. I imagine:

Transcoder interface — implemented by an AWSElasticTranscoder

Transcriber interface — implemented by an AWSTranscriber

2. State changes flow through the aggregate as a saga

A "saga" is a series of local transactions.

A streamJob has several of these.

The high-level policy is that our saga goes → reformatMedia, extractAudio, and combineMedia. That's how we complete a streamJob.

The low-level policy is that, in AWS, all of this work is done in S3 buckets. So when our job is done, we'll likely want to do some cleanup. A streamJobCompleted event could be subscribed to in order to clean up those temporary files in our S3 buckets.

3. An example use case

We might realize that the streamJob aggregate needs the following properties: JobState, InitialMedia, TextFromAudio, MediaAudio, TranscribedMedia, etc. These are all domain objects used to get from the beginning to the end of the saga. And they are created as we progress through.

For example, the TranscribeAudioUseCase, which is invoked after a MediaReformmated event, might look something like this:

class TranscribeAudioUseCase { ... execute (streamJobId: string): Promise<void> { const streamJob = await this.streamJobRepo.getJobById(streamJobId); if (!streamJob.shouldTranscribe()) { throw new Error("Not ready to transcribe"); } // Gets the url of the S3 bucket containing the reformmated media then // transcribes it. The streamJob aggregate's `setTranscribedText()` method // is called, updating the state of the aggregate and saving the transcribed text // to the streamJob aggregate as well. A "MediaTranscribed" event is created and // added to the aggregate. await this.awsTranscriber.transcribe(streamJob); // The aggregate is saved and the "MediaTranscribed" event is dispatched. this.streamJobRepo.save(streamJob); } }

And here's a snippet of what that state-changing method might look like."

class StreamJob extends AggregateRoot<StreamJobProps> { ... setTranscribedText(transcription: Transcription): InvalidStateError | void { if (this.state !=== "MediaReformmated") { return new InvalidStateError(); } this.props.transcription = transcription; this.props.state = "MediaTranscribed"; this.addDomainEvent(new MediaTranscribed()); } }

Khalil! Thank you for your comprehensive reply. And I noticed I have to learn about saga in DDD. Your example is assuming that we outsource transcription to 3rd-party service. I need to explain my situation in detail. I need to handle logic - such as transcribing, combining, creating files, removing files, etc. - in a pipeline in stream.

A video source is out there. Video source is generating video chunks continuously. In my service, the system gets a video chunk from a stream and converts it into a proper audio format. And sends it to the transcribing service and gets transcripts. And then the system saves the audio chunk and transcript as files and updates the playlist. Lastly, the system removes old files that are not necessary anymore.

I also posted pseudo code for my current implementation.

class TranscribeAudioUseCase { ... execute(videoSourceUrl: Url) { const videoSourceStream = this.createVideoSourceStream(videoSoureUrl) videoSourceStream .pipe( new Writable({ write: async (data, encoding, next) => { const converted = await this.audioConverter.convert(data) next(converted) } }) ) .pipe( new Writable({ write: async (data, encoding, next) => { const transcript = await this.transcriptionService.transcribe(data) next([data, transcript]) } }) ) .pipe( new Writable({ write: async (data, encoding, next) => { // playlistManager add new audio and transcript files and removes old files await this.playlistManager.update(data[0], data[1]) next() } }) ) } }

At this point, I have several questions.

1 - Do you think it's okay that a "stream" object is exposed to use cases or domain?

2 - In dealing files on domain, are domain objects allowed to have saved paths or not? For example, if we introduce AudioSegment in the domain layer, is it okay that AudioSegment has a path where it is saved? How can you decide a property is for domain or for infra?

3 - Sometimes, talking with domain experts is too abstract. I know we should develop that concept in detail and usually I do. But like this case, sometimes domain experts do not know about the concept of stream. And they just say "From this URL, get a video and extract text from it and combine them and show them to me". I know it is good to use those terms into domain objects. But I think in this case, it depends a lot on what protocol of video source and what protocol we should serve. If we use HLS we should save target duration, sequence number and so on. If we use RTMP we should save values for RTMP packets. I think serving protocols like RTMP, HLS are infra. Thus, they should not corrupt domain or be exposed to domain. But it affects too much what we should put in domain objects.

Do you think it's okay that a "stream" object is exposed to use cases or domain?

"The stream is just a functional way to handle logic. There's probably a way to do this with for or while-loops too. That being said, this approach is elegant and I see no problem with you doing this in the use case. You probably want to be able to promise-ify this and report when it's complete to the caller (controller, most likely).

As for making the concept of a stream a part of the domain, you could if that makes sense. You could model a stream as a domain object.

In dealing files on domain, domain objects are allowed to have saved paths or not? For example, if we introduce AudioSegment in the domain layer, is it okay that AudioSegment has a path where it is saved?

If it makes sense to the integrity of the domain object, then yes. Again, domain objects can know about technical stuff like file paths, permissions, and such so long as it makes sense to the domain. Your domain is inherently technical. For a userProfile, I'd also save the profilePictureURL as a value prop. Similar situation.

How can you decide a property is for domain or for infra?

I don't typically have any entities in infrastructure. My infrastructure usually consists of the building blocks on which the application can operate. We're talking repos, databases, controllers, web servers, adapters to external APIs like Stripe, Google Calendar, etc.

You said, "protocols like RTMP, HLS are infra". Protocols like HTTP are typically abstracted away from the domain layer on most of the things I build, but you are building something where the concept of protocols are inherent to the domain. The concept of HTTP probably belongs as far away from the domain in a pet store application but it also probably belongs a lot closer if your domain is a file-sharing application, or better yet - a media-streaming application.

Sometimes, talking with domain experts is too abstract. I know we should develop that concept in detail and usually, I do. But like this case, sometimes domain experts do not know about the concept of a stream. And they just say "From this URL, get a video and extract text from it and combine them and show them to me". I know it is good to use those terms in domain objects. But I think in this case, it depends a lot on what protocol of video source and what protocol we should serve. If we use HLS we should save target duration, sequence number and so on. If we use RTMP we should save values for RTMP packets.

Yeah, that's complexity. That's why we want a domain layer.

I smell a practical application of the Strategy Pattern here. Consider an abstract StreamProtocol domain object. It defines the general algorithm necessary for what you need to do to deal with the stream in all cases, but leaves out specific details like handling sequence numbers for HLS or what you should do for RTML.

This is also the Open-Closed Principle. Your Protocol object is high-level, and the concrete implementations are low-level. The Protocol domain object contains the law. The concrete implementations follow the law, but in their own unique ways. To add support for a new protocol, you merely need to write the new concrete implementation."

No title

Tue, 09 Feb 2021 04:05:26 GMT

🌱 This wiki entry hasn't fully bloomed. It's very likely to change over the next little while.

Entropy

Entropy is a fancy word for complexity, randomness, uncertainty or a state of disorder.

It was first recognized in the study of thermodynamics but to us, programmers - it's at the heart of everything we're working to do with software.

Entropy is the real world

We're in the business of creating things that solve problems for people in the real world.

And you don't have to be Albert Camus to realize that:

The world is entropy. It is complex, random, uncertain, and disorderly.

The role of software is to cut through that chaos and produce a simplified (or good enough) model that operates top of that complexity.

This is hard. This is where all our efforts go.

Things we do to tame complexity

Learn the domain with Agile planning & discovery practices: If we're going to get anywhere close to building something useful and maintainable, we better start with learning the language of the domain. Event Storming (or Event Modeling), BDD, user stories to build a shared understanding and map out what we want to build.

Use Agile technical practices: DDD, BDD, TDD, acceptance tests, pair programming to model the domain, catch regressions, and ensure quick feedback loops.

Design patterns and principles: SOLID, YAGNI, Simple Design, etc.

Clean code: Style, formatting, conventions, object calisthenics, etc.

Agile/XP techniques and processes help us wrangle entropy into acceptance tested features - see "13. Features (use-cases) are the key" from solidbook.io

There is, however, one truly foundational technique we use to solve the complexity problem. It's one we've relied on ever since the early structured programming days. It's the technique of functional decomposition: decomposing the problem into smaller pieces.

Functional decomposition is great but it can have both positive and negative impacts coupling & cohesion: the best measures of structural software quality there is.

Coupling

Coupling is a measure of how intertwined two components (and by components, we mean: routines, methods, functions, classes, modules, etc) are.

It's said that we should strive for loose coupling.

Loose coupling is good because it works to reduce maintenance costs by making code more testable and flexible.

Loose coupling (ideal): Components are pretty independent and not tied to others. This helps testability and flexibility.

Tight coupling: Components are inseparable. Changes made to one component in the group will likely ripple into the other dependent components. This hurts maintainability, testability, and flexibility.

It's not really possible to have zero coupling because the components we write need to be hooked up to each other in order for our software to do anything meaningful.

That being said, when things are loosely coupled, it paradoxically hurts the other metric - cohesion. That is, we can say that loose coupling generally correlates to low cohesion (which is bad).

What's cohesion then?

Cohesion

Cohesion is a measure of how related components are within a particular module. It's a measure of how much they belong together. Do you have stuff in here mostly focused on the topic at hand? Or is there stuff in here that probably belongs elsewhere?

You want to strive for high cohesion.

Tip: If you have classes that you call helpers that does a lot of random different things for many different areas of your code and isn't quite focused on one particular problem, it likely means that your cohesion is low.

Here's an example:

// Bad cohesion - not focused on a single responsibility class RequestHelpers { ... createUser (details) { ... } getEmailFromRequest (req) { ... } checkForumExists (req) { ... } isAuthenticated (req) { ... } endTransaction (req) { ... } } // Good cohesion class TextUtil { public static strip (text) { ... } public static isPalindrome (text) { ... } public static replaceAll (text, replacement) { ... } public static toCamelCase (text) { ... } }

High cohesion is good because it works to reduce maintenance costs by making code more understandable.

High cohesion (ideal): When a component (like a class, for example) has methods that all appear to be related to an underlying concept, then the class is said to have high cohesion. This helps understandability.

Low cohesion: Many classes with only a single method (that would belong better together) or one class with lots of methods (that aren't related) is an example of low cohesion.

The problem with high cohesion is that it typically correlates to tight coupling.

What do we do?

You see the conundrum we're in?

We optimize for cohesion we lose coupling.

We optimize for coupling we lose cohesion.

The doctrine of the mean: The most virtuous act is not the extreme - one side or the other. Instead, it's the middle path; this is what the best spot is. ━ Me, paraphrasing Aristotle's ethics

It turns out that there's another mental model we can use to balance dependencies.

Connascence

It's called connascence: a generalization of coupling and cohesion into one "holistic approach" according to connascence.io.

What is it?

Also according to connascence.io,

Connascence is a metric, and like all metrics, it is an imperfect measure. However, connascence takes a more holistic approach, where each instance of connascence in a codebase must be considered on three separate axes:

Strength. Stronger connascence is harder to discover or harder to refactor.

Degree. An entity that is connascent with thousands of other entities is likely to be a larger issue than one that is connascent with only a few.

Locality. Connascent elements that are close together in a codebase are better than ones that are far apart.

The three properties of Strength, Degree, and Locality give the programmer all the tools they need in order to make informed decisions about when they will permit certain types of coupling, and when the code ought to be refactored.

Well that's powerful.

The following image lists the different forms of connascence, ordered by strength.

You want to prefer refactorings that push you towards the weaker connascence types (when a component is changed, it's less likely that that change will ripple into another component) over the stronger ones (more likely to introduce ripple).

Apparently this concept goes way back to 1992 when Meilir Page-Jones first introduced it in "Comparing Techniques by Means of Encapsulation and Connascence". It was also written about in Fundamentals of Object-Oriented Design in UML in 1999, and recently expanded on in 2009 in a presentation called "The Grand Unified Theory of Software Development" by Jim Weirich.

I have more to say about this, but I'm still figuring out the best way to present it. So for now, I'll recommend you read the official site on the topic and check out the examples. I'll update this page at some point.

Resources

https://connascence.io/

https://programhappy.net/2020/06/22/connascence-why-is-it-so-important/

https://codesai.com/2017/01/about-connascence#nota5

No title

Tue, 09 Feb 2021 04:05:26 GMT

🌱 This wiki entry hasn't fully bloomed. It's very likely to change over the next little while.

ORMs like Sequelize and Prisma use the Active Record pattern which lets you add, remove, delete, and update data from SQL server without ever needing to write any SQL code. For simple cases, they work amazingly. They even tend to work pretty well for more non-trivial queries as well. The downside is that when we run into the scenario where we need to use our ORMs to write really complex queries, it's not uncommon that they plan and execute inefficient SQL code. In situations like this, the abstraction of the ORM has leaked, and we're forced to either learn how to make the ORM write a more performant query, or to manually write the more efficient SQL query ourselves.

The Law of Leaky Abstractions

Joel Spolsky, co-founder of StackOverflow, famously coined the Law of Leaky Abstractions which states:

All non-trivial abstractions, to some degree, are leaky.

Much of software development is creating abstractions. We write collections to abstract a list of items. We write value objects to make objects appear as if they're primitive types. We use the file system API which makes it seem like it's a structured hierarchy of files and folders. We write RESTful APIs and client libraries to abstract the complexity of interacting with our backend services.

All non-trivial abstractions (the more complex ones), to some degree, leak. As in, they leak implementation details.

Examples in modern tooling

React.js — React is a view layer library that lets you create user interfaces. While the API for React is small and you can get far by following the rules of React (like only put hooks in classes with render functions or other hooks), it's not uncommon to run into scenarios where you need to learn more about how React determines when to re-render, when it calls effects and when components are unmounted.

Apollo Client — Apollo Client automatically handles the GraphQL data fetching logic and caching logic for you. However, to evict or manipulate data in the cache, you need to learn how it normalizes that cached data.

The Django Web Framework — I swear this is 95% magic.

You can probably think of thousands more leaky abstractions. If it's non-trivial (meaning, it does something really substantial), then it's only a matter of time until you run into a leak.

Should we not use tools with leaky abstractions?

No, that's not the point. Tooling ultimately saves us time. Who wants to go and write another React for the sake of... oh wait.

Alright, well. The point is to remember The Law of Leaky Abstractions. Keep in mind that everything non-trivial will eventually leak. And when it does, you may have to roll up your sleeves to understand how the abstraction works behind the scenes.

This can help you make decisions as to what kinds of tools you want to use.

Don't believe that anyone that says their tool can abstract everything away. It's just a matter of time until you're forced to learn the internals.

How do we apply this knowledge to our coding practices?

When you're designing abstractions, you want to make sure that other developers understand how to use them strictly from the public interface. You want to shield others from the complexity behind the scenes of your abstraction.

But if something doesn't work correctly, or they can't figure out how to fit what you wrote for their use case, it's said to be a leaky abstraction.

See here for an example.

Bonus (aside): Mark Phillips (Supreme Dreams) has a really funny video that almost perfectly demonstrates the idea of a leaky abstraction in real life. Check it out.

Related

Maintain a Single Layer of Abstraction at a Time | Object-Oriented Design Principles w/ TypeScript

Coupling, Cohesion & Connascence | Wiki

Also read

React is a Leaky Abstraction

Maintain a Single Layer of Abstraction at a Time | Object-Oriented Design Principles w/ TypeScript

Mon, 08 Feb 2021 15:04:10 GMT

Some people think programming is similar to story-telling. If that's the case, then good code is an incredibly boring and one-dimensional story.

Here are two stories.

Be honest. Which of those two stories did you finish reading? The first one, right?

But why?

Well, the first story is quick and succinct. We get a birds-eye view of what's going on, and if we're curious to know more, we can ask and go a level deeper. The second story gives us everything. And it's pretty fatiguing.

The more complex your system becomes, the more you need to lean on abstractions to keep code readable and maintainable.

Since much of software development is reading code, using encapsulation and information-hiding to abstract away complexity and lower-level details becomes critical.

Used correctly, the plus side of this is that we end up with code that's easier to read, easier to ramp up and learn the domain, and it can — along with tests, even act as the primary form of documentation.

Used incorrectly, or not at all, we end up with leaky abstractions, poor cohesion, and (since "we're always building an API") poor APIs for our fellow teammates and future maintainers to build on top of.

Therefore, the object-oriented design principle in question is:

Maintain a Single Layer of Abstraction at a Time (or in other words, Don't Mix Different Levels of Abstractions)

By maintaining a single layer of abstraction at a time, we write more cohesive and loosely coupled functions, methods, classes, and modules, which in turn improves the readability and maintainability of your code.

Let's take a closer look at this and how it works.

Prerequisite reading

Before reading this article, I highly recommend the following. I've noted the relevant takeaways from each, but I recommend reading them if you find yourself confused or you don't find yourself aggressively nodding your head in agreement.

Coupling, Cohesion & Connascence — Entropy (complexity) is at the heart of what makes it hard to write software. To tame complexity, we decompose the problem into smaller parts. But software is only useful when those parts are connected. Coupling and cohesion are the two best measurements of the quality of our decomposition attempts. We should strive for loose coupling and high cohesion.

(Abstraction) Layers | Wiki — We use layers (domain, application, infrastructure, and adapter) to decompose a web application's complexity into separate concerns. The introduction of layers means more classes (and evidently, the coupling between layers of classes — though we can use Dependency Inversion to mitigate this). Still, it can bring the benefit of higher cohesion within classes.

Leaky abstractions — If we need to know about an object's internals to use it correctly, depending on the layer of abstraction, we may be leaking implementation details. This leads to lower cohesion and tighter coupling between components.

Another real-world example

Let me walk you through a real-world example.

Take my initial attempt at a use case (application service) that syncs my Notion Habits dashboard to my Google Calendar.

useCases/syncHabitsToCalendar.ts

import { UseCase } from "../../../../shared/logic/UseCase"; import { Calendar } from "../../domain/calendar"; import { CalendarService } from "../../domain/calendarService"; import { Habits } from "../../domain/habit"; import { googleCalendar } from "../../services"; import { GoogleCalendar } from "../../services/googleCalendar"; import { HabitsPage } from "../../services/habitsPage"; export class SyncHabitsToCalendar implements UseCase<void, void> { private habitsPage: HabitsPage; private googleCalendar: GoogleCalendar; constructor ( habitsPage: HabitsPage, googleCalendar: GoogleCalendar ) { this.habitsPage = habitsPage; this.googleCalendar = googleCalendar; } public async execute () : Promise<void> { console.log('Loading habits... 5 seconds'); await this.habitsPage.load(5000); const habits: Habits = this.habitsPage.getAllHabits(); const calendar: Calendar = await this.googleCalendar .getCalendarForCurrentMonth(); const { creates, updates, deletes } = new CalendarService() .planSync(habits, calendar); await Promise.all(creates.map((c) => googleCalendar.create(c)); await Promise.all(updates.map((u) => googleCalendar.update(u)); await Promise.all(deletes.map((d) => googleCalendar.delete(d)); await this.habitsPage.cleanup(); return; } }

What's happening here?

The habitsPage is an abstraction over a Puppeteer instance that loads and stores habits from my Notion habits page. We then create a CalendarService (domain service) by passing in both a habits collection and a calendar object.

We then run a series of Promise.all statements, passing the create, update, and delete actions to the correct googleCalendar method.

Finally, we tell the habitsPage to clean up — and by this, we're really telling the Puppeteer instance to destroy itself to free up resources.

Problems?

There are some subtle issues.

Application layer responsibilities

Let's first remember the responsibilities of a use case in the application layer in a layered (clean, hexagonal, etc.) architecture. Regarding the application layer:

It contains the features — the use cases, of our application

It is concerned with the rules that govern the application itself. For example, in a pet grooming application, a domain layer business rule might state that a pet must have an owner. Such invariants could be enforced within entities, value objects, and aggregates within the domain layer. Conversely, an application layer rule might enforce the fact that you can only edit a grooming appointment if you're one of many owners of the pet (authorization logic).

And use cases (one of the main implementation patterns from the application layer) primarily exist to:

Retrieve domain objects from IO (using repositories to databases or adapters to external APIs) so that it orchestrate the execution domain objects' encapsulated business rules and persist any events or state changes they create.

With that in mind, we can call out a few leaking abstractions in the first attempt of this design.

Problem #1: [Leaking abstraction] Calling load on the habits page

Notice that the first thing we do in the execute method of the SyncHabitsToCalendar use case is to call load(waitTimeInMilliseconds: number) on the habitsPage object?

useCases/syncHabitsToCalendar.ts

export class SyncHabitsToCalendar implements UseCase<void, void> { ... public async execute () : Promise<void> { console.log('Loading habits... 5 seconds'); await this.habitsPage.load(5000); const habits: Habits = this.habitsPage.getAllHabits(); const calendar: Calendar = await this.googleCalendar .getCalendarForCurrentMonth(); const { creates, updates, deletes } = new CalendarService() .planSync(habits, calendar); await Promise.all(creates.map((c) => googleCalendar.create(c)); await Promise.all(updates.map((u) => googleCalendar.update(u)); await Promise.all(deletes.map((d) => googleCalendar.delete(d)); await this.habitsPage.cleanup(); return; } }

Why should the use case need to know that we need to load the habitsPage before fetching something from it? If we didn't call load and instead just called getAllHabits, what would happen?

We'd always get an empty list back. Behind that load method is the process of:

starting up a puppeteer browser instance

going to the page to load the habits

using cheerio.js to scrape the habits and parse them into domain objects

Needing to call load before we can call getAllHabits is an example of a leaky abstraction. We appear to abstract away the complex details of getting all the habits from my Notion page. Instead, we've shifted the knowledge required to utilize this object's API onto the user (which, in the real world, could be a coworker or a future maintainer). That's not great.

A leaky abstraction like this can lead to a lot of bugs, and evidently — frustration.

Bonus (aside): Mark Phillips (Supreme Dreams) has a really funny video that almost perfectly demonstrates the idea of a leaky abstraction in real life. Check it out.

To fix this problem, we could relocate the logic to a better place: inside the HabitsPage class. Here, we encapsulate knowing when to call load in HabitsPage with the following:

services/HabitsPage.ts

export class HabitsPage { ... public async getAllHabits (): Promise<Habits> { if (!this.hasLoadedHabits()) { await this.load(5000); } return this.habits; } }

Client usage in the use case looks like this now:

useCases/syncHabitsToCalendar.ts

export class SyncHabitsToCalendar implements UseCase<void, void> { ... public async execute () : Promise<void> { const habits: Habits = await this.habitsPage.getAllHabits(); const calendar: Calendar = await this.googleCalendar .getCalendarForCurrentMonth(); const { creates, updates, deletes } = new CalendarService() .planSync(habits, calendar); await Promise.all(creates.map((c) => googleCalendar.create(c)); await Promise.all(updates.map((u) => googleCalendar.update(u)); await Promise.all(deletes.map((d) => googleCalendar.delete(d)); await this.habitsPage.cleanup(); return; } }

Much better. Moving on.

Problem #2: [Mixed levels of abstraction] There's persistence logic in our use case!

The next thing that needs attention is the fact that we've put persistence logic in our use case.

As a reminder, use cases are only supposed coordinate the interaction between objects.

useCases/syncHabitsToCalendar.ts

export class SyncHabitsToCalendar implements UseCase<void, void> { ... public async execute () : Promise<void> { const habits: Habits = await this.habitsPage.getAllHabits(); const calendar: Calendar = await this.googleCalendar .getCalendarForCurrentMonth(); const { creates, updates, deletes } = new CalendarService() .planSync(habits, calendar); await Promise.all(creates.map((c) => googleCalendar.create(c)); await Promise.all(updates.map((u) => googleCalendar.update(u)); await Promise.all(deletes.map((d) => googleCalendar.delete(d)); await this.habitsPage.cleanup(); return; } }

This isn't the worst thing. Typically when we want to save entities to a repository, we merely pass the new entity off to a save method, which does all the hard work behind the scenes.

However, our current implementation could be a little more cohesive. This is a bit of a detour from what we should be doing here. It also couples our use case to the persistence implementation details (run create first, then update, then delete — perhaps the sequence is important persistence logic). This could make testing this use case more complex.

Instead of decomposing the syncPlan that comes back from the planSync method, let's pass this off to a new object. The new object will contain the knowledge necessary for taking the syncPlan and running the persistence strategy.

useCases/syncHabitsToCalendar.ts

import { UseCase } from "../../../../shared/logic/UseCase"; import { Calendar } from "../../domain/calendar"; import { CalendarService } from "../../domain/calendarService"; import { Habits } from "../../domain/habit"; import { googleCalendar } from "../../services"; import { GoogleCalendar } from "../../services/googleCalendar"; import { HabitsPage } from "../../services/habitsPage"; import { SyncService } from "../../services/syncService"; export class SyncHabitsToCalendar implements UseCase<void, void> { private habitsPage: HabitsPage; private googleCalendar: GoogleCalendar; constructor ( habitsPage: HabitsPage, googleCalendar: GoogleCalendar ) { this.habitsPage = habitsPage; this.googleCalendar = googleCalendar; } public async execute () : Promise<void> { const habits: Habits = await this.habitsPage.getAllHabits(); const calendar: Calendar = await this.googleCalendar .getRecurringEventsCalendar(); const syncPlan = new CalendarService() .planSync(habits, calendar); const syncService = new SyncService(googleCalendar); await syncService.sync(syncPlan); await this.habitsPage.cleanup(); return; } }

The story in our SyncHabitsToCalendar use case is looking a lot more cohesive now.

Not only that, but once I moved the logic to this new object, I realized that there was a performance issue. Google was rate-limiting my API calls. I'd need to add some delay between each operation I run. More complexity started peeking its head out at me. Luckily, we encapsulated this complexity within a new domain service.

services/syncService.ts

import { LogicUtils } from '../../../shared/utils/LogicUtils'; import { SyncPlan } from '../domain/calendarService' import { GoogleCalendar } from './googleCalendar'; export class SyncService { private googleCalendar: GoogleCalendar; constructor ( googleCalendar: GoogleCalendar ) { this.googleCalendar = googleCalendar; } private async executeSyncActions ( milliseconds: number, elements: any[], func: Function ): Promise<void> { let index = 1; try { for (const element of elements) { console.log(` => On action ${index} of ${elements.length}`) console.log(' => Waiting', milliseconds / 1000, 'seconds first...') await LogicUtils.delay(milliseconds); console.log(' => Executing action...') await func(element); console.log(' => Done'); index = index + 1; } } catch (err) { console.log("Hiccup executing action", func) } } public async sync (syncPlan: SyncPlan): Promise<void> { const { creates, updates, deletes } = syncPlan console.log(`Creating ${creates.length} events.`) await this.executeSyncActions(5000, creates, this.googleCalendar.create.bind(this.googleCalendar)) console.log(`Updating ${updates.length} events.`) await this.executeSyncActions(5000, updates, this.googleCalendar.update.bind(this.googleCalendar)) console.log(`Deleting ${deletes.length} events.`) await this.executeSyncActions(5000, deletes, this.googleCalendar.delete.bind(this.googleCalendar)) } }

Problem #3: [Leaking abstraction] Cleaning up the habits page

There was one final thing I considered doing to improve the design. And that's to make it so that the use case doesn't know that it needs to call cleanup on the habitsPage after we've finished the sync.

Why? It could lead to memory issues if someone created a new use case involving this object and forgot to call cleanup at the end.

I haven't come up with a great approach yet — perhaps we could ensure that the encapsulated browser instance is a singleton. That way, we'd never have more than one instance of it running.

In the end, I've decided that I'm OK with this statement living in the use case.

Again, we're always building APIs for other developers, and at the moment, this is going to be one piece of knowledge within the abstraction that will become necessary for others to know about if they're to use the habitsPage API. I'll think about improving that later.

Summary

Here's a summary of the objects involved in this use case, which abstraction layer they belong to, and what their responsibilities are.

The important takeaways here are:

Write more cohesive code by maintaining a single level of abstraction at a time.

Know which layer you're working in and the responsibilities of that layer.

Learn implementation patterns (like value objects, repos, domain services, etc) and which layer they belong to.

Remember that you're always building an API for other developers. Assume that someone else is going to have to know how to use the objects you design. Make the public interfaces foolproof.

Bonus exercise: Refactoring a controller

Given what we've just learned, take a look at this controller.

infra/userController.ts

export class UserController extends BaseController { private db: DbConnection; constructor (db: DbConnection) { this.db = db; } public async createUser ( req: Express.request, res: Express.response ): Promise<CreateUserHTTPResult> { const { body } = this.req; const { username, email, password } = body; // Request validation if (!username | !email | !password) { return this.clientError("Username, email, or password not provided.") } // Entity/value validation const isUsernameValid = username.length > 4 && hasOnlyBasicChars(username); const isPasswordValid = password.length > 4 && validatePassword(password); const isEmailValid = isValidEmail(email); if (!isUsernameValid) return this.clientError("Username not valid") if (!isPasswordValid) return this.clientError("Password not valid") if (!isEmailValid) return this.clientError("Email not valid") // App logic const existingUsernameUser = await this.db.Users.findOne({ where: { username } }); if (existingUsernameUser) return this.conflict("User already has that username") const existingEmailUser = await this.db.Users.findOne({ where: { email } }); if (existingEmailUser) return this.conflict("Account already exists! Sign in."); await this.db.models.Users.create({ username, email, password, emailVerificationState: 'INITIAL' }); return this.ok({ message: 'Successfully created new user' }) } }

How would you improve this? Remember that a controller is a part of the infrastructure layer. What are controller concerns and what are concerns that likely belong elsewhere?

Also related

Separation of Concerns - The introduction of abstraction layers is an exercise of the separation of concerns design principle. It's a way to introduce loose coupling between modules.

Outside-in TDD — Outside-in TDD (Mockist, London-style) is a more exploratory way to perform TDD that involves starting at the top of the system's boundary (like a controller or a use case) and gradually building an understanding of how things can work before committing to building out concrete implementations of classes. I think this topic is related because you can write the public APIs of classes before they exist, designing abstractions in the most human-friendly way possible (while mocking out collaborators for the time being).

5 Ways to Manage Lambda Cold Starts

Sat, 27 Jun 2020 15:04:10 GMT

In this article, you will learn how cold start issues occur, what factors contribute to cold starts, and how to manage Lambda cold starts.

Understanding the Cold Start Problem

A Lambda cold start occurs when a new function instance must be created and initialized. The cold start refers to the delay between invocation and runtime created by the initialization process.

Lambda cold starts occur when there is no available function instance to respond to an invocation. This can happen when instances have expired due to inactivity or when there are more invocations than active instances. Cold starts are an inherent problem with serverless models because providers are unwilling or unable to keep inactive instances alive indefinitely.

To maximize resource use and optimize provider costs, instances are only kept alive for a set amount of idle time. This means that unless your functions are consistently activated, you are going to have to manage latency caused by cold starts.

What Factors Contribute to Cold Starts?

Several factors contribute to cold starts when using Lambda functions. While one of these factors is not controllable (i.e. how long AWS keeps instances alive), most are. Below are some of the factors with the greatest impact that you should be mindful of.

Languages

The length of time it takes to cold start an instance depends significantly on the language you use for your function. For example, JavaScript or Python functions will initialize much faster than C# or Java-based ones.

Additionally, just-in-time (JIT) compilation, like that used by .NET languages, substantially increases start time. This is because the language’s machine agnostic assemblies must be converted to machine-specific assemblies at initialization.

Function chains

In many applications, developers use chains of functions to perform tasks. This chaining is fine when functions are already available but can cause significant overhead if not. When chaining, each function in the chain must wait for the possible initialization and response of any later functions.

This ordered process compounds cold start times and can cause Lambda functions to time out. If timeouts occur, you have to start the invocation process all over, creating even more latency.

Virtual Private Clouds (VPC)

While VPCs can help keep your data more secure, these networks are a barrier to the performance of Lambda functions. This is because most Lambda functions require an external connection for request and response operations. This connectivity requires the creation of an elastic network interface (ENI).

When instances are initialized inside a VPC, the cold start process is extended by the creation of the ENI. During this process, an IP must be allocated and the interface attached to the function. Lambda can take up to 10 seconds to complete this process.

How to Manage Lambda Cold Starts

While you cannot completely eliminate the chance of cold starts when using Lambda you can decrease the impact. When configuring your functions, consider the following tips.

1. Measure current performance

Measuring the current performance of your configurations can help you identify where you need to make changes and can help you anticipate latency. In particular, you should evaluate how frequently your functions are being invoked, how many invocations occur at once, and how long invocations take. You can then use these values to determine how many dedicated instances you need.

Performance can be measured directly using metrics sent to AWS CloudWatch or you can use third-party tools. The former works well for individual functions but can only provide limited information and is not easily searchable. The latter enables you to invoke your functions many times, can provide visualizations of the results, and enables you to more easily evaluate aggregate data.

2. Keep your functions warm

Once your instances are initialized, you can keep them active to avoid the majority of your cold starts. Keeping functions warm can be done by either invoking the function or using warming logic with handler functions. Warmer logic pings functions to check their status, keeping the instance alive even if it isn’t fully invoked.

To accomplish this, you can either manually configure your logic or you can use tooling to automate the process for many instances. Manual setup involves inserting logic into your handler functions and triggering that logic with CloudWatch Events and cron jobs. Or, with tooling, you can reuse the same logic for concurrent instances.

3. Choose the right memory settings

When using Lambda, your memory and CPU resources are directly tied. The more memory you provision, the more processing power your functions can access, and the faster initialization can occur. While you may be tempted to keep memory resources low to save costs, doing so can increase your cold start and runtimes times and end up costing you more regardless.

To avoid higher costs due to insufficient resources, you need to find the balance point between resources and latency. This requires understanding how frequently your functions are invoked, the revenue dependent on those functions, and the latency during cold start and warm invocations.

4. Store dependencies external to your function

With Lambda, you can load dependencies into your function container external to your function. This enables you to prep dependencies for invocations and prevents functions from needing to load dependencies each time the instance is called.

While this process won’t reduce your cold start time, you can use it to reduce your runtimes and offset initial latency. When you use this method in combination with function warming, you can significantly reduce your function runtimes.

5. Keep your function small and single-purpose

Keeping your functions small and single-purpose helps ensure that functions execute quickly and reduces the chances of timeout. It can also reduce the total number of instances you need, thus reducing the chance of cold starts.

When function invocations finish quickly, that instance is sent back to your available pool. The more available instances in your pool at any given point, the lower the chance that you need to create a new instance to answer an invocation request. Ensuring that functions execute efficiently is a way of “increasing” your pool without requiring more instances.

Conclusion

A Lambda cold start occurs when function instances are not available to respond to invocations. There are several factors contributing to this issue. The language you use, for example, impacts the length of time it can take to cold start an instance. If you are using function chains, the order of the functions can cause timeouts. If you are planning on setting up your Lambda on VPC, you need to be aware that the process of initializing Lambda functions on VPCs can take up to 10 seconds.

Unfortunately, you can’t completely eliminate cold starts, but you can significantly reduce the impact this issue may cause. You can do that by measuring current performance and keeping your functions warm, small, and single-purposed. You can also store dependencies externally, and make an effort to choose the right memory settings for your project.

IV. Conclusion

Wed, 24 Jun 2020 15:04:10 GMT

This is page four of a guide on Client-Side Architecture basics. Start at page one.

The work isn't over

I know this article expressed a lot of new ideas. To be completely honest, I spent about three months thinking about this off and on, and I'm likely going to spend some time revising it and trying these ideas out with real-world projects.

Stay tuned for updates on this. If you subscribe to the newsletter, I'll let you know when I publish major updates.

Let's look at what we've covered.

A new client-side development philosophy

Zooming out, we started with this:

And then zooming in, we landed on this:

Let's recap the statements I've made.

Model-View-Presenter isn't good enough for our needs anymore

The problems we're solving on the client-side are much more complex than they were 20 years ago. Because of that, the starting point for an architecture probably can't be MVP.

Design principles have persisted for a long time to help us write better software

I honestly think that reading books and learning from the past is one of the best ways to avoid future mistakes. Design principles are great. You don't need to always follow them, but know the rules before deciding whether you want to break 'em.

There are no silver bullets

There really aren't. With this advanced client-side architecture, what we introduce in structural quality, we lose in developer experience based on the potential learning curve involved. But ask yourself this question: is the complexity related to nature of the problem itself, or is it just related to the way we're solving it?

This is for developers where "use what works for you" is daunting, and would like a good conceptual starting point for an air tight React architecture

I'm really excited about these ideas. I've been milling around with this for a couple of months now but I think it's really important today. If you're a developer that has been told to use "what works for you", that's still incredibly good advice. But if you run into any of the pain-points in your React project like suddenly facing issues adding features, changing code, and feeling like things have turned to mush, this might help.

Next steps

We shouldn't stop here. Here are a few thoughts to chew on. Feel free to ask questions and leave suggestions on ways I can make this guide more helpful.

Choosing a stack: Given this philosophy, what factors into deciding upon a stack? Compatibility? Developer experience? Maintainer(s) activity?

Organizing code: How do you organize code within a React project? What principles are at play?

Dealing with prop drilling: If we have these clear-cut layers, doesn't that mean we'll have to do a lot of prop drilling?

Test-Driven Development: Can we discuss testing in more detail? What does a test-driven approach to client-side application development look like in 2020? Should we write BDD-style acceptance tests with React Testing Library and Jest? Or should invest into testing for functional correctness using E2E tools like Cypress.io?

Real-life project: What does this look like in practice?

I. Architecture

Wed, 24 Jun 2020 15:04:10 GMT

This is page one of a guide on Client-Side Architecture basics.

Why we need a client-side architecture standard

Allow me to paint the picture of why we need a better client-side architecture. To start, let's look at the foundation we're all currently working on top of.

Model-View controller

You've probably heard of Model-View-Controller, the architectural pattern that describes how to design apps involving user interfaces.

MVC says that we should split our application into model, view, and controller layers. This is so each layer can focus on their own respective responsibilities.

The model handles data and logic

The view handles presentation

and the controller turns user events into changes to the model

Model-view-controller architectural pattern. Used to separate the concerns between a client-side web app and backend services.

Most full-stack apps are comprised of a client-side portion, utterly separate from backend services. When users ask to make a change from the UI, it makes things happen by interacting with the backend through some API: in MVC, the API is the controller.

This works! And we like it. At least, we must — it's one of the first architectural patterns we teach to new developers learning how to build full-stack apps.

Model-View presenter

Where Model-View-Controller explained how to separate the concerns of a full-stack web application, Model-View Presenter, a derivation of MVC, told us how to separate the concerns of the client part.

Model-View-Presenter is the architectural pattern typically used within client applications. It's a derivation of the MVC pattern.

In Model-View-Presenter, the view creates user events.

Those user events get turned into updates or changes to the model.

When the model changes, the view is updated with the new data.

It's heartening to realize that every client app uses some form of the model-view presenter pattern.

MVC & MVP are too generic

MVC and MVP are great starters. They give you a good enough understanding of the communication pathways from a 5000-ft view.

Unfortunately, they both suffer from the same problem: being too generic.

In both MVC and MVP, the biggest challenge is that the M is responsible for way too much.

As a result, developers don't know which tools are responsible for which tasks.

In MVC and MVP, the model is ambiguous. This makes matching the correct tool up to the task feel like a puzzle.

Tasks of the model

In the real world, the model portion in most client-side web apps does a lot.

State management — Most apps need a way to fetch state, update state, and configure reactivity to when state changes, the view can re-render.

Networking & data fetching — The actual data-fetching concern is sometimes conflated as part of the model. The data fetching and networking aspect of an app need to know about backend services, formulate requests, handle responses, and marshal data, but it also needs to signal request metadata (ie: isLoading, error). What about features like optimistic updates? Is that a concern of the model? I think it is.

Model behavior (ie: domain, app, or interaction logic) — Deciding what happens next when a user clicks submit, or wants to interact with something on the page is a form of interaction logic. Sometimes there are rules we need to enforce. They can be simple — like validation before sending off an API call. They can be complicated — like deciding if a chess piece can be dropped on the selected square. Some call this app logic, which I believe to be sufficient since it describes how our app responds to user events. Alternatively, I call it interaction logic because it explains what happens in response to user interaction. There's one other kind of logic here, and that's domain logic. Domain logic doesn't normally have anything to do with the application itself. Instead, it originates from an understanding of the domain. For example, while displaying a modal before move a chess piece might be application/interaction logic, enforcing the policy that a knight can only move in an L-shaped fashion is a form of domain logic. That rule originates from understanding the domain of chess where the application-specific logic holds rules about how the user interacts with the app. Usually, domain logic lives behind backend services, and if we break the rules, we can get an error back as a response. Still, sometimes we co-locate it on the client-side, especially for more complex applications.

Authentication & authorization logic (specific type of model behavior) — This is another specific type of model behavior, but it's common enough to mention. Most of the time, authN & authR finds itself being used from within the view layer (show Login screen if not authenticated). Sometimes, it manifests in the application/interaction layer as well, preventing access to specific operations.

Tools used within the model

These are all common challenges to solve. In 2020, the developer toolbox for a React developer looks a little something like this:

React hooks

Redux

Context API

Apollo Client

xState

react-query

and now, recoil

Each of these are capable of addressing a particular piece of the ambiguous model we described, but matching the correct tool to the proper concern can be tricky.

Of course, it's tricky. We don't have a standard language to describe the different concerns. Instead of thinking about the tools right away, I think we need to back up and look at the bigger picture of the problems to be solved.

We need a shared language to talk about client-side architecture

We need a shared language to describe these architectural concepts upon which we either:

Configure a library or a framework to solve

Write the code ourselves for

Most React developers know the implementation-specific terminology like hooks, reducers, context, and props, but architectural concerns are sometimes misunderstood.

Having a shared understanding of what constitutes client-side architecture concerns enables us to:

Have better design conversations

Communicate which concerns are addressed by which tool

Avoid code from concerns creeping too profoundly into another

If we can, as a community, communicate a shared understanding of the concerns that make up the model (and the other parts), I think we can more easily answer questions like this:

Where do we put application logic in a React/Redux app? What about an Apollo Client one? What about a [insert new library/framework here] one?

Should I use container components?

Should I put my GraphQL mutations inside of my component?

Do I need to write tests for Apollo Client? Redux?

What kind of logic should I put in a React hook?

I've got good news for you...

We've already solved this problem

Let's not discredit the software design and architecture research done over the last 30 years.

While the tools and approaches to web development have changed at a miraculous pace, software design principles and patterns have remarkably, remained the same.

Knowledge drain: When (domain, trade, scientific) knowledge is lost or forgotten over time.

Let's look at backend development.

How did we solve the ambiguous model problem when building out backend services?

Initially, with MVC on the server, we thought the model could be services, ORMs, or even the database itself. Each of these are part of the model, but they're not the entire model.

According to wikipedia, the model is supposed to represent the data, logic, and any rules of the application.

When backend developers discovered that MVC doesn't provide enough insight as to how to structure the model portion, we used design principles to create more advanced architectures, like the clean architecture.

The clean architecture sheds more light on how to structure your backend in a testable and flexible way. It also helps accommodate for more advanced applications containing business logic.

The clean architecture (which has many similar variants — see layered, hexagonal architecture, or ports & adapters) provides specifics as to what the M in the model is.

By splitting the model into infrastructure, application, and domain layers, we exercise the separation of concerns design principle, and we're left with a much easier to reason about architecture.

The middle layers (domain and application) are the purest. It's the code that we, the developers, have to write from scratch. And since our app doesn't do much unless we can hook it up to the real-world using things like web servers, databases, APIs, and caches, the adapter layer provides a flexible way to integrate those infrastructural dependencies into our app, while keeping them distanced from our domain and app layer code.

To learn more about the clean architecture read Organizing App Logic with the Clean Architecture [with Examples].

A layered architecture like this comes at the cost of being more complex than a simple single-tiered one, but let's be honest — sometimes we have to solve some damn hard problems.

A layered architecture has a lot of benefits:

It makes it extremely clear which tools are needed at which layer of the stack.

It keeps concerns separate and enables you to keep your app and domain layer code unit testable.

It allows you to mock out expensive to test things, and swap libraries and frameworks (not that you do that too often — but, in case you ever needed to, you can).

That's hella cool.

Quick question.

Where's our client-side version of this?

Client-side needs — testability, flexibility, and maintainability

Let's back up a bit.

Before we discuss an equivalent client-side architecture, let's talk about our needs first.

We don't want to dogmatically copy the clean architecture.

What are we really looking for when we talk about architecture on the client-side? Why does any of this matter? Why don't we just write all of our code in a single file (actually, some of us do write single file components)? Is architecture about file organization, or is it about something more?

It's about writing testable, flexible, and maintainable code.

Testability

I've noticed that an alarming amount of developers don't write tests for their front-end code.

It could be a conscious decision of choosing not to write tests — which is one thing, but it could also be a lack of education for how to write code so that it can be tested.

I've found that depending on your testing strategy, your needs to separate concerns changes. If you're only going to be writing integration tests, then separation of concerns matters less. If you're going to be writing a lot of unit tests, then mocking is going to be your saving grace, and separation of concerns is paramount.

If you understand the app you're building and the complexity of it, you can kind-of gauge this upfront.

When to write unit tests
+

Unit testing is the preferred approach for testing your client app if there's a heavy amount of interaction/app logic, like a metadata layer in a 3D rendered game, chess game logic, a boating application, or a streaming site like Twitch.

If the accuracy of the most important use cases cannot be verified by merely observing the side effects in the view, then unit tests are the way to go.

When to write integration tests
+

Kent C. Dodds recommends integration tests when 90% of your users' primary use cases can be tested against by observing changes to the view in response to user interaction. In this case, we're talking about basic CRUD apps.

The view is an implementation detail, and it's recommended to not test against implementation details. Testing library provides an excellent suite of tools to run integration tests on React apps through the view without focusing on implementation details.

Flexibility

It's not so often that we need to switch from REST to GraphQL or swap out APIs, but there are a select few cases that we should enable flexibility for.

Swapping out view components — Keeping app logic out of your presentational components allows you to swap out how the component looks from how it works, as painlessly as humanly possible.

Changing model behavior — If your app is the interaction-logic heavy kind of app that needs lots of unit tests, using dependency inversion to mock out API and framework code enables you to run fast tests against the behavior of the model.

Maintainability

Maintainability is our ability to constantly provide value. If we struggle to change code or add new features, maintainability is low.

It's worthy to note that if developer experience is low, there's a chance maintainability is low as well.

Here's an argument to challenge everything I've praised about a clean architecture so far. Looking at it from another point of view, too many layers and too many rules traditionally yields low developer experience for newer developers less familiar with the approach.

This might be why so many new developers prefer to use React over Angular. Angular is actually quite opinionated and forces you towards a particular style of architecture. React lets you do whatever.

There's a balance to be struck here. We want the structure of architecture, but we want the developer experience of knowing what to do and having the freedom to do it however we want.

Design is the balance of conflicting priorities

Office furniture = cost vs. quality

Note-taking = context vs. compression
— Tiago Forte

And more relevant to us:

Software design = structure vs. developer experience

I believe that developers who care not only about getting the job done but also getting it done right will push through learning curves.

Client-Side Architecture Basics [Guide]

Wed, 24 Jun 2020 15:04:10 GMT

Welcome

When I first learned React and Redux in 2015, I made an enormous mess of the production codebase I was working on.

Back then, class-based components and Redux were the coolest kids on the block. This was my first time prepping up to work on a real-world React project, so I bought the best courses I could find on the topics and dove in.

After a couple of months, according to the React community, the way I was doing things were outdated. There were newer, cleaner, and recommended best practices for me to follow.

I think it's incredible that we question the way we do things. But my vast gap in knowledge of client-side architecture left me always finding it necessary to play catch up to refactor to the new approaches.

These sentiments were also recently echoed in this painfully accurate Twitter thread from Joel Hooks.

Developers are confused about where to start and what to choose when they start learning to write high-quality React applications. They want a strong foundation of knowledge and the confidence to architect React applications at scale built to current industry standards.

They are frustrated that there are no widely accepted standards for building React applications consistently, coherently, and with minimal risks to their professional reputations and livelihoods.

On top of that, there is a sea of choices and tradeoffs that React developers face every day. From how the project is even started, which framework to use, how to manage state, how does it get bundled, accessibility, and deployment just to get going.

At each f🐬kin' stop, there's a new choice to make: a new chance to be wrong.

Trying to build a solid foundation in React feels like a slot machine. — @jhooks, June 17th, 2020

Some point later in my developer journey, I too realized this was happening. I realized I could never keep up to date with the current trends of the libraries and frameworks I was using. I also decided I didn't want those nuances to dictate my professional reputation or livelihood. It wasn't good enough for me. I needed something better.

Eventually, I sought out the originating principles— the fundamentals, to client-side architecture. I sought out to construct a standard for building client-side apps. A professional one. One based on the software design principles that have helped us design robust software for decades.

This guide teaches you client-side architecture fundamentals.

It's the result of my research using first principles on how to design and develop robust, flexible, testable, and maintainable client-side applications.

During my experience working on client-side apps of varying sizes, I've realized that some serious upfront design on the architecture can have a significant impact on the quality of the code for the duration of its life.

From simple dashboards to multi-layered apps with rendering layers, domain logic, multiple types of users - this guide teaches you the essential design principles front-end developers inadvertently code around within their everyday programming jobs.

This guide proposes architectural standards for client-side web development.

While we're primarily focused on React, because it's the most popular library with the least structure, the principles are transferrable to any configuration of view-layer library or framework.

Programming tends to seem more like a trade than a science. Each tool, be it a state management library, API, or a transport-layer technology, is best suited to solve a particular set of problems. As a developer and a tradesperson, it's good to know how the tools in our toolbox are best used.

... if all you have is a hammer, everything looks like a nail.

At the end of this guide, you'll learn a standard for web development. You'll have a clear understanding of the discrete layers of concerns in a client-side app: from the view layer to various forms of state management, and how to handle interaction (app) logic.

Prerequisites

Familiarity with at least one front-end view layer library (like React) or an opinionated framework like Angular.

Knowledge of at least one state management approach (like React Context, Redux, Angular services, NgRx, etc).

(optional) Knowing When CRUD & MVC Isn't Enough | Enterprise Node.js + TypeScript

(optional) "A Treatise on State" by Jed Watson

II. Principles

Wed, 24 Jun 2020 15:04:10 GMT

This is page two of a guide on Client-Side Architecture basics. Start at page one.

The most influential client-side architecture design principles

While the clean architecture works, we don't need a copy of it on the client-side. However, I do think it's a good idea to look at the same design principles and practices that formed it and apply those to the client.

You'll notice that each principle, in some way, is about enforcing some structural constraints as to what can be done, and how things are organized.

In my opinion, these are the most crucial design principles. They influence 90% of what constitutes good client-side architecture.

Command-Query Separation

Separation of Concerns

Command Query Separation

Separate methods that change state from those that don't

Command Query Separation is a design principle that states that an operation is either a command or a query.

commands change state but return no data, and

queries return data but don't change state.

Operations are the same thing as interactions.

The primary benefit of this pattern is that it makes code easier to reason about. Ultimately, it urges us to carve out two code paths: one for reads, and one for writes.

The simplest way to see it in action is at the method-level.

Commands

Consider the methods createUser and selectTodo. These are both command-like operations.

function createUser (props: UserDetails): Promise<void> { ... } function selectTodo (todoId: number): void { ... }

Notice that neither of these methods return anything. They're both void. That's what a valid command is.

That means that the following methods aren't valid commands.

function createUser (): Promise<User> { ... } function selectTodo (): Todo { ... }

Queries

Queries are operations that return data and perform no side-effects. Like these, for example:

function getCurrentUser (): Promise<User> { ... } function getUserById (userId: UserId): Promise<User> { ... }

Why does it matter?

Simplifies the code paths — this is what React hooks does with the accessor/mutator API of useState, and what GraphQL does with queries and mutations.

Operations are easier to reason about — consider how hard (and disastrous) it would be to test a query was working properly if it always also performed a side-effect that changed the state of the system.

All features can be thought about as operations: commands or queries. If you want to make sure that all your features have integration tests, ensure a good separation of commands and queries that the user performs, and test each one. One other interesting discovery: since most pages/routes in your app invoke one or more features, a potentially maintainable folder structure could be formed by co-locating all the concerns and components by features, and then by page/route. The folks behind React Router seem to be on a similar page (sorry); their new project, Remix, features file system routes and route layout nesting.

Apparently, cache invalidation is one of the hardest problems in computer science. It's easier with this. Using CQS, we can be sure that when if no new commands were executed (against a particular item), we can continue to perform queries for directly from the cache. The moment a command is executed, we invalidate the item in the cache. Consider how this might be useful for a state management library.

Separation of Concerns

Consciously enforcing logical boundaries between each of the architectural concerns of your app

Assume we have a list of todos.

When a user clicks delete on the todo, what happens next?

export const Todo = (props) => ( <div className="todo"> <div class="todo-text">{props.text}</div> <button onClick={props.onDeleteTodo}>Delete</button> </div> )

Well, the view passes off the event to a container. That could connect the user event to a method from a React Hook or a Redux thunk. From there, we might want to run some logic, decide if we should invoke a network request, update the state stored locally, then somehow notify the UI that it should update.

That's a lot. And that's a simple app. And when I said we might want to run some logic a moment ago, I wasn't clear about exactly what kinds of logic it could be. It could be authorization logic, validation logic, interaction/domain logic, etc. Instead of putting five different kinds of logic wherever, we can classify it, carve out a place for it to live, and be more structured and conscious about how we connect features together.

Separation of concerns is one of my favorite design principles. It's about thinking the jobs to be done, delegating them to a particular layer that handles those concerns, and then ensuring those layers do their jobs, and their jobs only.

How separation of concerns and CQS work together

CQS said that every feature is an operation. It also said that every operation is either a command or query.

This means that every feature cuts through several concerns to work.

I like to think of features as vertical slices that cut through the stack.

Features are vertical slices

When we add or change features in an application, we're modifying a part of the vertical slice for that feature.

Need to change the way the login component looks? No problem, you're going to add some styles to the presentational component in the presentation layer from the Login feature.

Need to change what happens a when todo open for longer than 30 days was just completed? Want to throw confetti on the screen and say how proud of the user you are? Gotcha. Add some logic to the xState model from the interaction layer for the Complete Todo feature.

I'm a huge fan of this.

Understanding the responsibilities of each layer enables us to better reason about which tools to use per feature.

Using Apollo Client, React Hooks + xState

Application logic: Hooks + xState

State management: Apollo Client (global state)

Data fetching: Apollo Client

Using Apollo Client and plain JavaScript

Interaction logic: Hooks + pojo-observer

State Management: Apollo Client (global state)

Data fetching: Apollo Client

Using REST, Redux, and React Hooks

Interaction logic: Hooks

State Management: Redux (global state). Connect for observability/reactivity, and Thunks for signaling async states.

Data fetching: Fetch or Axios

I first heard of the term vertical slices from Jimmy Bogard. Thinking of features this way reduces the amount of time it takes for developers to figure out where to add or change code.

This is where developers get stuck, figuring out what the layers of the stack are, and which tools can be used at each layer of the stack.

Vertical slices enables us to keep Single Responsibility high if we "minimize coupling between slices, and maximize coupling in a slice" — via Jimmy Bogard. Also read Kent C. Dodd's article on "Co-location".

Why does it matter?

Better visibility as to which tasks need to be done, which layer they belong to, and which tools can be used to address those concerns.

Helps to decide whether we want to implement a layer ourselves or use a framework/library. For example, most developers won't build their own view-layer library for presentational components — they'll use React or Vue. But lots of users will build their own state management system from scratch using Redux and Connect.

III. Layers

Wed, 24 Jun 2020 15:04:10 GMT

This is page three of a guide on Client-Side Architecture basics. Start at page one.

Layers and concerns

We're finally ready to decompose each part of Model-View-Presenter, especially the model part.

Here's a graphic to illustrate that decomposition into something more concrete.

Can you see both CQS and SoC in here?

Let's examine it from the top.

Presentation components

Render the UI and create user events

If you read the title and feel like closing the tab because of this article by Dan Abramov, hang in there. Just wait until we get to container components to decide if you want to bounce 🏀.

Presentation components live within the boundaries of the View portion of Model-View-Presenter. Their entire purpose is to:

Display data in the UI

Generate user events (from keypresses, button clicks, hover states, etc)

Presentation components are an implementation detail

An implementation detail is a low-level detail that helps us accomplish our main goal. But they're not our main goal. If our main goal is to hook up the Add Todo feature, the buttons, styling, and text in the UI is an implementation detail in realizing the feature.

Presentation components can be volatile

Anything subject to frequent change is said to be volatile. Us constantly changing the look and feel of components is what makes them so.

One way to accommodate this phenomenon is to decide on a stable set of reusable components (that either you wrote or grabbed from a component library), then create your views from those.

Even though we could use reusable components, data requirements change frequently.

Take this simple CardDescription component that uses a GraphQL query to describe a card.

const CARD_DESCRIPTION_QUERY = gql` query CardDescription($cardId: ID!) { card(id: $cardId) { description } } `; const CardDescription = ({ cardId }) => { const { data, loading } = useQuery({ query: CARD_DESCRIPTION_QUERY, variables: { cardId } }); if (loading) { return null; } return <span>{data.card.description}</span> }

How likely is it that we'd need to change the styling? What about displaying something like a lastChanged date beside it? Chances are we pretty likely.

Should we include GraphQL queries in our presentation components?

It's good to have GraphQL queries as close to the presentational component as possible. Queries define the data requirements. And since they'll likely need to be changed together if the requirements change, having them close together reduces unnecessary cognitive load accrued by flipping back and forth between files.

One potential downside to putting your queries in your components is that now, if you ever wanted to switch away from GraphQL, your components aren't pure— they're coupled to GraphQL. If you wanted to switch transport-layer technologies, you'd have to refactor every component.

Another potential downside is that to test these components, you'd need to make sure they're wrapped in a mocked Apollo Client provider.

My recommendation is to couple the queries to the components anyways. What you gain in an incredible developer experience is, in my opinion, worth the risk of going fully in with GraphQL and deciding you want to change later down the road.

Note on query performance: It's ok to have lots of queries for super-specific chunks of data like shown above. Using Apollo Client, Apollo handles that complicated logic of checking whether the data is cached already, and if not — it makes a request to get it.

What to test in presentation components

Unit testing implementation details is typically fruitless — especially for volatile things. It doesn't do us much good testing to see if a button is blue or green. Instead, when testing presentation components, we want to test against UI logic.

To demonstrate what I mean, here's a bland, basic presentation component.

/components/Todo.tsx

export const Todo = (props) => ( <div className="todo"> <div class="todo-text">{props.text}</div> <button onClick={props.onDeleteTodo}>Delete</button> </div> )

There's no UI logic involved here. It merely takes in props, hooks up callbacks, and renders some HTML.

Here's another example of the same component, but this time, as a class-based component with UI logic.

/components/Todo.tsx

import React, { Component } from 'react' import PropTypes from 'prop-types' import classnames from 'classnames' import TodoTextInput from './TodoTextInput' type Props = any; export default class Todo extends Component<Props, Props> { static propTypes = { todo: PropTypes.object.isRequired, editTodo: PropTypes.func.isRequired, deleteTodo: PropTypes.func.isRequired, completeTodo: PropTypes.func.isRequired } state = { editing: false } handleDoubleClick = () => { this.setState({ editing: true }) } handleSave = (id: number, text: string) => { if (text.length === 0) { this.props.deleteTodo(id) } else { this.props.editTodo(id, text) } this.setState({ editing: false }) } render() { const { todo, completeTodo, deleteTodo } = this.props return this.state.editing ? ( <TodoTextInput text={todo.text} editing={this.state.editing} onSave={(text: string) => this.handleSave(todo.id, text)} /> ) : ( <div className="view"> <input className="toggle" type="checkbox" checked={todo.completed} onChange={() => completeTodo(todo.id)} /> <label onDoubleClick={this.handleDoubleClick}> {todo.text} </label> <button className="destroy" onClick={() => deleteTodo(todo.id)} /> </div> ) } }

UI logic

View behavior & local component state

The main difference between the two previously shown Todo components is that the second Todo component contained UI logic where the first did not.

UI logic is view behavior

"If you're logged in, show this — otherwise, show this."

"If you're this type of user, show this — otherwise, show this."

"Depending on which page you're on in the signup process, show the correct form".

A component has UI logic when it exudes behavior. Conditionals that determine what to show, or when certain user events get called over others are a form of view behavior (UI logic).

Here's a conditional example from the previous code sample determining what to show.

return this.state.editing ? ( <TodoTextInput text={todo.text} editing={this.state.editing} onSave={(text: string) => this.handleSave(todo.id, text)} /> ) : ( <div className="view"> <input className="toggle" type="checkbox" checked={todo.completed} onChange={() => completeTodo(todo.id)} /> <label onDoubleClick={this.handleDoubleClick}> {todo.text} </label> <button className="destroy" onClick={() => deleteTodo(todo.id)} /> </div> )

Here's a conditional determining which user event to create.

handleSave = (id: number, text: string) => { if (text.length === 0) { this.props.deleteTodo(id) } else { this.props.editTodo(id, text) } this.setState({ editing: false }) }

Component / local state

This is where the first type of state we might encounter: local (component) state.

In Jed Watson's talk from GraphQL Summit 2019 titled, "A Treatise on State", he describes five different types of state when building web apps: local (component), shared (global), remote (global), meta, and router.

Explanations of the five types of state

local (component) : State that belongs to a single component. Can also be thought about as UI state. UI state can be extracted from a presentation component into a React hook. Note: we're about to do this.

shared (global) : As soon as some state belongs to more than one component, it's shared global state. Components shouldn't need to know about each other (a header shouldn't need to know about a todo).

remote (global) : The state that exists behind APIs in services. When we make queries for remote state, we hold onto a local copy of it accessible from a global scope.

meta : Meta state refers to state about state. The best example of this is the loading async states that tell us the progress of our network requests.

and router state: The current URL of the browser.

This state, local (component) state, belongs to a single component. You can call this UI state. It's meant to hold onto data that helps a single component do its job.

To better see what it looks like, let's extract all UI state from this class-based component and refactor to a functional component and a React hook.

/components/Todo.tsx

import React, { Component } from 'react' import PropTypes from 'prop-types' import classnames from 'classnames' import TodoTextInput from './TodoTextInput' import { useState } from 'react' /** * Decompose the UI logic from the presentational component * and store it in a React hook. * * All data and operations in this hook are UI logic for the * component - we've just separated concerns, that's all. */ function useTodoComponent (actions) { // "editing" is a form of local (component) state const [editing, setEditing] = useState(false); const handleSave = (id: number, text: string) => { if (text.length === 0) { actions.deleteTodo(id) } else { actions.editTodo(id, text) } setEditing(true); } const handleDoubleClick = () => { setEditing(true); } return { models: { editing }, operations: { handleSave, handleDoubleClick } } } /** * This component relies on some local state, but none of * it lives within the component, which is purely * presentational. */ export function Todo (props) { const { todo, actions } = props; // Grab our local (component) state and access to other UI logic const { models, operations } = useTodoComponent(actions); // Conditional UI logic return models.editing ? ( <TodoTextInput text={todo.text} editing={models.editing} onSave={(text: string) => operations.handleSave(todo.id, text)} /> ) : ( <div className="view"> <input className="toggle" type="checkbox" checked={todo.completed} onChange={() => actions.completeTodo(todo.id)} /> <label onDoubleClick={operations.handleDoubleClick}> {todo.text} </label> <button className="destroy" onClick={() => actions.deleteTodo(todo.id)} /> </div> ) }

UI logic is what we actually try to test within components

Since UI logic is behavior, this is actually what we want to test against in our integration tests. The behavior. You could write unit tests as well, but it might be trivial if component logic is straightforward. It could be more worthwhile and give you more confidence that the feature is working correctly to integration test both the component and the UI logic together.

Container/controller

The glue layer (pages)

Traditionally, the responsibilities of a container component were to:

Consume user events & pass them to the model

Subscribe to data changes (reactivity) and keep the view updated

This isn't new. The definition of a controller/presenter, all the way back from the Model-View-Presenter pattern, made this distinction.

Do we really need container components?

In 2019, with the advent of React hooks, Dan said we don't.

The main reason I found [container components] useful was because it let me separate complex stateful logic from other aspects of the component. Hooks let me do the same thing without an arbitrary division.

Here are my thoughts.

I fully agree that complex stateful logic shouldn't live within presentation components. When we do that, we don't get the ability to reuse logic across different components.

Now, as for stateful logic in container components? I don't believe it ever should have been in 'em.

Previously, React developers were advised to put data and behavior in container components and write code that determined "how things work". That breaks the rules of what was said to be the responsibility of a container/presenter.

Just because we know to put stateful data and behavior in React Hooks, it doesn't mean we removed the problems a container component solves.

We still need to configure reactivity, sometimes using Redux, sometimes using Apollo Client or something else, and we still need some construct to act as the glue, knowing which components to load up for the features we enable on a page.

Container components are pages

In the following React Router example, we have three main pages: home, about, and dashboard.

src/App.js

export default function App () { return ( <Router> <Switch> <Route exact path="/"> <Home /> </Route> <Route path="/about"> <About /> </Route> <Route path="/dashboard"> <Dashboard /> </Route> </Switch> </Router> ); }

Each page:

is responsible for enable a variable number of features (remember, a feature is a command or query)

has a variable number of presentational components within it, and

knows about shared reactive state, and sometimes connects it to presentational components that need it

Container components are the top-level modules that turn on all the features for a particular page. In Gatsby.js, we call them Page components. Since all client architectures naturally evolve from this Model-View-Presenter pattern, it's unlikely we'll get rid of the presenter (container) entirely.

To demonstrate my point, here's a container component in a React hooks world. It might not look like much, but notice that it fulfills the two responsibilities of a container component.

/modules/home/Home.container.tsx

import React from 'react' import MainSection from '../components/MainSection' import { useQuery } from '@apollo/client' import { VisiblityFilter } from '../models/VisibilityFilter' import { Todos } from '../models/Todos' import { GET_ALL_TODOS } from '../operations/queries/getAllTodos' import { GET_VISIBILITY_FILTER } from '../operations/queries/getVisibilityFilter' import { todoMutations } from '../operations/mutations' const todosAPI = new TodosAPI(); export default function Home () { // Shared (global) or remote (global) state. const { operations, models } = useTodos(todosAPI); const { completeAllTodos, setVisibilityFilter, clearCompletedTodos } = operations; return ( <Layout> <MainSection // Pass data to components activeVisibilityFilter={visibilityFilter} todosCount={models.todos.length} completedCount={models.todos.filter(t => t.completed).length} // Delegate operations to the model actions={{ completeAllTodos, setVisibilityFilter, clearCompletedTodos }} /> <ReportSection // Pass data to components todos={models.todos} // Delegate operations to the model actions={{ completeAllTodos, setVisibilityFilter, clearCompletedTodos }} /> </Layout> ); };

Something is responsible for knowing how to connect to a reactive model, and knowing what to do with events that come from presentation components. That's a container.

Of course, you could call everything a component, but then the explicit communication and delineation of responsibilities we're fighting for is lost.

Container components contain no functionality

The container component is pretty bare. That's a good thing. They're not supposed to contain any functionality. They're not worthy of unit testing. They're just meant to stitch things together. However, if you want to do an integration test all features of a page, just load up the container component and have at 'er.

Interaction layer

Model behavior

We're finally in the most challenging part of a client-side architecture: the model.

The first layer of the model, which is what gets called from the container component, is the interaction layer.

The interaction layer is the behavior of the model

When you click submit to "add a todo", do you jump straight to the GraphQL mutation right away? Do you perform any validation logic? Are there any rules to enforce?

A lot of times, there aren't any rules. Sometimes we can't be bothered and we leave validation logic as something the server handles. This is particularly common on simple dashboard apps. These apps have pretty much no rules to enforce, so an interaction layer doesn't exist.

It goes controller → network request.

Or as we've been doing for a long time, presentation component → network request.

When there is policy to enforce, it's time to think about carving out an interaction layer.

The interaction layer is the decision-making layer

Application (or interaction) logic is the logic that makes a decision as to what happens next.

Let's say you have a command called createTodoIfNotExists. Whatever construct is responsible for the interaction layer contains the code that helps you decide, "should we follow through with this"?

Here's a Redux Thunk example, where sometimes, we need to reach into some form of global state (maybe cached in a store) to make a decision.

/todos/redux/thunks/createTodoIfNotExists.tsx

// Interaction example export function createTodoIfNotExists (text: string) { return (dispatch, getState) => { const { todos } = getState(); const alreadyExists = todos.find((t) => t === text); if (alreadyExists) { return; } ... // Validate // Request } }

Alternatively, here's a React Hooks & Apollo Client example.

/models/useTodos.tsx

function useTodos (todos) { const createTodoIfNotExists = (text: string) => { const alreadyExists = todos.find((t) => t === text); if (alreadyExists) { return; } ... // Validate // Request } return { createTodoIfNotExists } } // Container function Main () { const { data: todos } = useQuery(GET_ALL_TODOS); const { createTodoIfNotExists } = useTodos(todos); ... }

It contains your application's operations

Some refer to this layer as app logic, which works as well because these are all of the operations of your app. The interaction layer contains the discrete set of commands and queries that your users will carry out. These are the use cases.

Having great visibility into these use cases enables us to get pretty structured with our integration testing as well. We can functionally test every use case with edge cases using Given-When-Then style tests.

For example:

Given no todos exist, when I perform CreateTodo, then I should see one todo.

Given I have 3 completed todos and 1 uncompleted one, when I perform CompleteAllTodos, then I should have 4 completed todos.

If you're familiar with Domain-Driven Design concepts, this is the Application Service equivalent.

Shared behavior

This behavior is written to be used by any component. It contains the rules for how shared state is allowed to change.

At this level, we're often handling concerns like auth, logging, or even more domain-specific things like todos, users, calendar, or even chess.

Consider an interaction-layer React hook that contained all your chess game logic.

/hooks/useChess.tsx

function useChess (todosAPI: ITodosAPI) { ... return { operations: { makeMove, isValidMove, ... }, models: { board, players, currentTurn } } }

Read "Domain-Driven GraphQL Schema Design" for the principles and practices for how to use event Storming to discover the subdomains within your app.

Other ways to implement the model

Though most React developers will be comfortable writing their application/interaction layer logic using something like React Hooks, there's tons of other ways to implement the model.

If you like to think of your model as a state machine, the xState library does this exceptionally well and provides capabilities for you to plug your model instance into a React hook.

For those who want to try to model their interaction layer using plain vanilla JavaScript, the pojo-observer library takes advantage of the fact that every client-app is an implementation of the observer pattern. Separating your model code from React hooks, it also provides a way to notify React that the model changed so a re-render is necessary.

Someone once asked me if it's possible to do DDD in the front-end. Initially, I said no, but after sometime thinking about it, it totally is. While the true high-level policy will always live on the backend, the interaction layer is comparable to the Application and possibly Domain layer in DDD.

There are usually several layers

Most of the time, your app will have several of these application / interaction layers.

Here are some more examples of interaction layers that are commonly built out.

Examples of other interaction layers

Auth layer — Extremely common. Check out the useAuth library which implements Auth0 authentication and authorization as a React hook.

Logging — Sometimes it's important to. Luckily, there are many tools out there that can do this for you, but if you needed to build one yourself, it would exist as an entirely separate layer within your model.

Real-time subscriptions — Let's say you're subscribed to a stream of data. When a chunk comes in, you need to process it, and perhaps act on a switch statement to figure out if you should invoke a command. Keep your code clean by delegating this responsibility to a layer.

Complex rendering logic — I once worked on a project that built out really complex call flows for call centers using Angular and D3. Hundreds of different node types could be dragged and dropped onto a surface. When dropped, the way they connected to each other and how they could be used depended on the rendering and application logic, each decoupled from each other.

Metadata layer — Imagine building a multiplayer video game where new prizes and weapons come out every week. How can we prevent hard-coding weapons and prizes?

If you're curious about what a large-scale version of this looks like, check out Twilio's video-app example built with React hooks and context for global state.

🚡 Networking & data fetching (infrastructure)

Performing API calls and reporting metadata state

The responsibilities of a networking and data fetching layer are to:

Know where the backend service(s) are

Formulate responses

Marshal response data or errors

Report async statuses (isLoading)

Reporting metadata state

Jed Watson describes the async states that tell you about the status of a network request as meta state — state about state.

For example, in Apollo Client, the loading variable we deconstruct from the query response is a form of meta state.

const { data, loading, error } = useQuery(GET_ALL_TODOS);

With Apollo Client, that's handled for us. Though if we were to use a more barebones approach, like Axios and Redux, we'd have to write this signaling code ourselves within a Thunk.

export function createTodoIfNotExists (text: string) { return async (dispatch, getState) => { const { todos } = getState(); const alreadyExists = todos.find((t) => t === text); if (alreadyExists) { return; } // Signaling start dispatch({ type: actions.CREATING_TODO }) try { const result = await todoAPI.create(...) // Signaling success dispatch({ type: actions.CREATING_TODO_SUCCESS, todo: result.data.todo }) } catch (err) { // Signaling Failure dispatch({ type: actions.CREATING_TODO_FAILURE, error: err }) } } }

Note: The code example above is a demonstration of doing a little too much. Recall that a Redux Thunk is an interaction layer concern? That means it should only be responsible for the decision-making logic, and no signalling logic, since request signalling is a concern of the networking & data-fetching layer. It can be hard to establish these concrete boundaries sometimes. Especially if the library or framework wasn't designed with separation of concerns in mind.

🗄️ State management & storage (infrastructure)

Storage, updating data, reactivity

A state management library has three responsibilities:

Storage — Hold onto global state somewhere, usually in a store / client-side cache.

Updating data — Make changes to the data in the cache.

Reactivity — Provide a way for view-layer presentation components to subscribe to data, and then re-render when data changes.

State management and networking are often solved together

State management is complex.

Because it's complex, there are libraries out there to make life a little bit easier. Two of those libraries, Apollo Client and react-query, actually handle the networking part for you.

It can be preferable to choose a library instead of building out the state management machinery and networking layer manually.

Apollo Client handles both the state management and data fetching concerns.

Shared global state

Two types of state exist at this layer. They are:

remote (global) state — The state that exists behind APIs in services. When we make queries for remote state, we hold onto a local copy of it accessible from a global scope.

shared (global) : We said earlier, "as soon as some state belongs to more than one component, it's shared global state". And you'll know you need this when two components that rely on the same state don't need to know about each other. To be clear, this type of state can be live in the interaction layer (via hooks and context, for example). Though sometimes, when working with remote(global) state, it's preferable to have something act as a single source of truth, especially if you need to mix remote and local state.

Mixture of remote and local state

We often cache remote state in a client-side cache or store. Since we do that, it's reasonable to try to use the store as a single source of truth. Often, we'd like to add some client-only local variables or pieces of state to the store as well.

Here's a Redux example of adding an isSelected attribute to each of the todos before merging to the store.

switch (action.type) { ... case actions.GET_TODOS_SUCCESS: return { ...state, // Add some local state to the remote state before merging it // to the store todos: action.todos.map((t) => { ...t, isSelected: false }) } }

And in Apollo Client 3, here's the equivalent with cache policies and reactive variables.

import { InMemoryCache } from "@apollo/client"; export const cache: InMemoryCache = new InMemoryCache({ typePolicies: { Todo: { fields: { isSelected: { read (value, opts) { const todoId = opts.readField('id'); const isSelected = !!currentSelectedTodoIds() .find((id) => id === todoId) return isSelected; } } } } } }); export const currentSelectedTodoIds = cache.makeVar<number[]>([]);

We can configure a way to request remote state and client-only shared local state in the same query.

export const GET_ALL_TODOS = gql` query GetAllTodos { todos { id text completed isSelected @client } } `

Storage facades

Most of the time we don't provide direct access to whats stored within the store. Usually, there's some facade, an API, that sits in-front of the data and provides ways for us to interact with it.

In Redux, this is dispatch (for updates) and connect (for reactivity).

In Apollo Client, this is useMutation (for updates) and useQuery (for reactivity).

Even SQL is a form of a storage facade. It's a powerful pattern.

The Three Responsibilities of a Client-Side State Management Solution

Thu, 14 May 2020 15:04:10 GMT

Historically, when starting on a new React project, we’ve had to design and implement the state management infrastructure from scratch in a bare-bones way.

No matter which approach we take, in every client-side application, the generic role of a state management solution is the same: to handle storage, update state, and enable Reactivity.

Storage

Most apps need to hold onto some data. That data may contain a little slice of local state that we’ve configured client-side, or it could be a subset of remote state from our backend services.

Often, we need to combine these two pieces of data, local and remote, and then call upon them in our app at the same time. This task alone has the potential to get pretty complicated, especially when we need to perform updates to the state of our app.

Update state

The Command-Query Segregation Principle states that there are two generic types of operations we can perform: commands and queries.

In GraphQL, we refer to these as queries and mutations.

In REST, we have several command-like operations like delete, update, post, etc and one query-like operation called get.

Most of the time, after invoking an operation in a client-side web app, we need to update the state stored locally as a side-effect.

Reactivity

When storage changes, we need an effective way to notify pieces of our UI that relied on that particular part of the store, and that they should present the new data.

Each state management approach has a slightly different approach

Just about every library available out there right now can adequately handle all three of these responsibilities! Here are some of the most popular approaches right now in the React realm.

Redux

Storage: Plain JS object

Updating state: actions + reducers

Reactivity: Connect

React Context + Hooks

Storage: Plain JS object

Updating state: useReducer (or not)

Reactivity: useContext

Apollo Client

Storage: Normalized cache

Updating state: Cache APIs

Reactivity: (Auto) Broadcast change notifications to Queries

Accidental and Essential Complexity

Wed, 29 Apr 2020 15:04:10 GMT

🌱 This blog post hasn't fully bloomed. It's very likely to change over the next little while because it's a little bit flawed. In fact, an old version of this post was titled "Absolute and Relative complexity". I then discovered that Ben Moseley wrote a famous paper written about this very phenomenon called, "Accidental and Essential Complexity". I'm re-writing this post to build on top of the existing philosophy using the terminology expressed there. The most up-to-date writing I have on this at the moment is in "Chapter 1 - Complexity & The World of Software Design" from solidbook.io. This will eventually catch up

Ever seen a friend, coworker, or maybe even an accidental stranger- trying to solve a problem using a tool that you feel is completely unfit for the task at hand?

Maybe you've felt like a problem would be easier solved using a different tool, language, platform, or feature for the job.

It's frustrating. It's frustrating seeing someone introduce undue challenge for themselves when you feel like you know there's a better way.

In discussions about architecture, choosing a tech stack, and solving problems in general, I've started to use the terms "accidental complexity" and "essential complexity" more often. Maybe you will too.

Being able to point out accidental and essential complexity will enable you to:

Decide when it's a good idea to use TypeScript over JavaScript

Know when to use Domain-Driven Design

Avoid anti-patterns

Know when you're tackling a problem using well-known best practices or introducing more challenge for yourself

Essential complexity

Essential complexity: When the complexity of the problem is related to the nature of the problem itself.

Let's face it. Some things in life are just inherently complex.

Making a vinyl-trading platform with lots of very specific domain-knowledge.

Architecting and constructing a house from scratch.

Using AI to write catchy pop music.

Coming up with a cure for cancer.

There's no easy answer to any of these things. Some of them don't even have an answer.

Often times though, we have a best practice. And a best practice acts as a close enough or the best that we can possibly do today. Best practices have considered all of the current options, and decided upon what the best approach is. Best practices are usually agreed upon by a larger community.

Here's an easy one.

The best practice for tying your shoes is to use your hands to tie your shoes into a knot. That's probably the easiest and most realistic way to tie our shoes, because well- mini robot butlers aren't really a thing yet. A way to make life harder and introduce accidental complexity is to force yourself to tie your shoes using only a pair of chopsticks, never touching your shoelaces with your hands.

Accidental complexity

Accidental complexity: When the complexity of solving the problem is directly related to the way that we have chosen to solve it, and less related to the nature of the problem itself.

When we take an approach to solving a problem that introduces more work or complexity for us, if not immediately, later on down the road, that's accidental complexity.

We don't often introduce accidental complexity purposefully. In software, sometimes it comes up when we:

Want to use a brand new programming language or tool to solve a project that it is not well-suited for

Are forced to use a particular tech stack

Are unaware of the best practices to solving this particular problem (and in turn, implement an anti-pattern)

Accidental complexity is introduced by failing to use best practices

When I was working on my first startup, I wrote the entire thing in vanilla Node.js. Over time, we started to get more and more business logic complexity added to the app. Eventually, I found myself in a position where it was impossible for me to add new code without breaking old stuff.

It was a huge mess because I had designed it with CRUD & MVC (also known as a Transaction Script) where the problem really needed a more robust solution.

In this scenario, the challenge was to build a large-scale business logic-heavy enterprise application.

The best practices for that task are to use Domain-Driven Design, separate your concerns using the clean/hexagonal architecture, write tests, and implement design patterns when necessary.

It turns out that doing all of these things with JavaScript is hard. Harder than it needs to be. There are several reasons why, and I explain why in detail here, but ultimately, it's the lack of a language concept to express abstractions (interfaces and abstract classes). Without abstractions and a type system, implementing contractual software design is harder in JavaScript.

That was the main reason why I made the move to TypeScript for that project; because the best practices for a large-scale business logic-heavy enterprise application were easier to implement in TypeScript than in JavaScript.

Yes, it was a challenge to learn and migrate to TypeScript. But that challenge was guided towards implementing the best practices that the problem afforded. Not all projects have the same needs as a large-scale enterprise app.

Sometimes best practices are a lot of work

I'm not gonna lie. Writing testable code, using DDD, and clean architecture is a lot of work. But if you're working on a project that needs that, and you're able to identify that, then it's the best path we have to the right way to do things.

Anti-patterns can introduce essential complexity

In the evolution of bad code, it starts with code smells. Code smells are indications that something might be wrong with the way we've designed our code and that we should probably take a better look at it.

That's the first step towards bad code, and it's not purposeful. These things just happen sometimes. If we can catch it, that's great. We can refactor and clean up code before it becomes a problem.

class User { // Wait. Why would we need to pass in a "logger" object to create a user? // A logger is an "infrastructural" cross-cutting concern, and it definitely // doesn't belong in our user factory method. // Also, this parameter list will only increase as we add more things to create // the user. We should use an object instead. public static createUser (userName, password, firstName, lastName, logger) { ... } }

An anti-pattern, however, is a little bit more aggressive. An anti-pattern is deliberate approach to solving a problem in a way that goes against best practices.

Anti-patterns have the potential to introduce essential complexity.

Anti-pattern example: Duck-typing with JavaScript

Problem: To prevent developers from passing in invalid objects.

Let's say that we want to create slugs for blog posts using the following function.

// Returns a slug like: "2020/02/12/accidental-complexity-212312 function createSlug (post) { return createDateString(post.postedDate) + "/" + createSlugFromTitle(post.title) + "-" + createRandomNumber(); }

The function expects that we're going to pass in a Post object that has postedDate and title attributes.

But developers keep on passing in the title of the post as a string like so:

const post = "Accidental and Essential Complexity"; const slug = createSlug(post) // ERROR

This is a silly problem. But on projects that are substantially large, it's pretty common to see. Especially if you're not familiar with the entire codebase.

Essential complexity: Need to signal to clients that use the API that the method requires a post object with postedDate and title, and enforce that.

Accidental complexity: Using JavaScript,

For API clients, you need to know the internals of every method. You can't rely on intention revealing interfaces alone.

For API authors, you need to use duck-typing and throw errors.

For API authors, you need to write more comments so that developers know how to use the API. The function signature is not enough to express valid use.

/** * @desc Creates a slug. You need to pass in a post object with `postedDate` * and `title` to create it. */ function createSlug (post) { if (!!post.postedDate === false || isADate(post.postedDate)) { throw new Error("Need to provide a post object with date as 'postedDate'") } if (!!post.title === false || typeof post.title !== "string") { throw new Error("Need to provide a post object with a string title.") } // Also, we would need to include the post title's validation logic here as well // which may include making sure that the title is a certain length. return createDateString(post.postedDate) + "/" + createSlugFromTitle(post.title) + "-" + createRandomNumber(); }

Best practice solutions:

Use a statically-typed language to indicate and enforce valid API usage.

function createSlug (post: Post) { return createDateString(post.postedDate) + "/" + createSlugFromTitle(post.title) + "-" + createRandomNumber(); }

Breaking best practices and implementing anti-patterns

Implementing best practices and avoiding anti-patterns will get us where we need to go in the long run, but will often require some more upfront work.

I think it's occasionally OK to break best practices and implement anti-patterns in order to go a little faster.

I give it my blessing in one condition...

Know what the rules (best practices) are before you decide to break 'em.

I take shortcuts all the time. But I try to always know what the implications are first.

Resources

No Silver Bullets - Accidental and Essential Complexity

Shortly after this post was released to the interwebs, someone kindly pointed me to the "No Silver Bullet – Essence and Accident in Software Engineering" Wikipedia page. Funny. The fact that I had arrived at the concept of accidental and essential complexity myself, yet was already well-researched and discussed as essential and accidental complexity, reinforces the following statement:

"Not a lot has changed about the fundamentals of software design over the past 20 years, but there's a huge lack of training on it" - me, via solidbook.io

If you're interested in learning more about this subject, I suggest checking out this paper called "Out of the Tar Pit" by Ben Moseley.

It builds on top of the classic paper about software complexity originally discussed by Fred Brooks in 1986. It's very well put together and worth your time. Consider reading the first half at least.

Conclusion

In summary, ask yourself if the problem is hard because it's a hard problem, or if you're making it hard by not following best practices.

Enforcing Coding Conventions with Husky Pre-commit Hooks

Tue, 10 Mar 2020 15:04:10 GMT

This post is a part of the Clean Code Tooling series.
You may want to read the previous posts.
1. How to use ESLint with TypeScript
2. How to use Prettier with ESLint and TypeScript in VSCode

Intro

On most projects I've ever worked collaboratively on, someone takes the role of the code cleanliness champion. It's usually the team lead, and often times, their role involves reviewing PRs and making sure love and care is put into the quality of the code.

Quality includes both the chosen coding conventions in addition to the formatting of the code.

Today, it's good practice in JavaScript projects to utilize ESLint to define the project's coding conventions. For example, how does your team feel about using for loops? What about semicolons- are those required? Etc.

Those are conventions.

The other piece of the puzzle is formatting. That's the visual appearance of the code. When there's more than one developer working on a project, ensuring that code looks consistent is something to be addressed.

Prettier is the correct tool for this.

In the previous article, we learned how to combine both ESLint and Prettier, but we didn't learn how to actually enforce the conventions and formatting on a real life project with multiple developers.

In this article, we'll learn how to use Husky to do so on a Git-based project.

Husky

Husky is an npm package that "makes Git hooks easy".

When you initialize Git (the version control tool that you're probably familar with) on a project, it automatically comes with a feature called hooks.

If you go to the root of a project intialized with Git and type:

ls .git/hooks

You'll see a list of sample hooks like pre-push, pre-rebase, pre-commit, and so on. This is a way for us to write plugin code to execute some logic before we perform the action.

If we wanted to ensure before someone creates a commit using the git commit command, that their code was properly linted and formatted, we could write a pre-commit Git hook.

Writing that manually probably wouldn't be fun. It would also be a challenge to distribute and ensure that hooks were installed on other developers' machines.

These are some of the challenges that Husky aims to address.

With Husky, we can ensure that for a new developer working in our codebase (using at least Node version 10):

Hooks get created locally

Hooks are run when the Git command is called

Policy that defines how someone can contribute to a project is enforced.

Let's get it set up.

Installing Husky

To install Husky, run:

npm install husky --save-dev

Configuring Husky

To configure Husky, in the root of our project's package.json, add the following husky key:

package.json

"husky": { "hooks": { "pre-commit": "", // Command goes here "pre-push": "", // Command goes here "...": "..." } }

When we execute the git commit or git push command, the respective hook will run the script we supply in our package.json.

Example workflow

Following along from the previous articles, if we've configured ESLint and Prettier, I suggest to utilize two scripts:

prettier-format: Format as much code as possible.

lint: Ensure that the coding conventions are being adhered to. Throw an error if important conventions are broken.

package.json

{ "scripts": { "prettier-format": "prettier --config .prettierrc 'src/**/*.ts' --write", "lint": "eslint . --ext .ts", ... }, "husky": { "hooks": { "pre-commit": "npm run prettier-format && npm run lint" } } }

Include these scripts in the scripts object in your package.json. Then, at the very least, run prettier-format and then lint as a pre-commit hook.

This will ensure that you cannot complete a commit without formatted code that passes the conventions.

No-loops example

I like to use the no-loops package as an example. This convention doesn't allow developers to use for loops, and instead suggests that we use Array utility functions like forEach, map, and the like.

Adding the plugin and its rule to the .eslintrc:

.eslintrc

{ "root": true, "parser": "@typescript-eslint/parser", "plugins": [ "@typescript-eslint", "no-loops", "prettier" ], "extends": [ "eslint:recommended", "plugin:@typescript-eslint/eslint-recommended", "plugin:@typescript-eslint/recommended", "prettier" ], "rules": { "no-loops/no-loops": 2, // 2 means throw an ERROR "no-console": 1, "prettier/prettier": 2 } }

And then placing a for loop in the source code...

src/index.ts

console.log('Hello world!'); for (let i = 0; i < 12; i++) { console.log(i); }

And attempting to commit, it should exit with a non-zero exit code, which as we know, means an error occurred.

simple-typescript-starter git:(prettier) ✗ git commit -m "Test commit" husky > pre-commit (node v10.10.0) > typescript-starter@1.0.0 prettier-format simple-typescript-starter > prettier --config .prettierrc 'src/**/*.ts' --write src/index.ts 191ms > typescript-starter@1.0.0 lint /simple-typescript-starter > eslint . --ext .ts /simple-typescript-starter/src/index.ts 1:1 warning Unexpected console statement no-console 3:1 error loops are not allowed no-loops/no-loops 4:3 warning Unexpected console statement no-console ✖ 3 problems (1 error, 2 warnings)

And there it is!

Other considerations

If you notice that linting is taking a long time, check out this package, lint-staged. It runs the linter, but only against files that are staged (files that you're ready to push). This was suggested to me by @glambertmtl. Thank you!

How to use ESLint with TypeScript

Wed, 26 Feb 2020 15:04:10 GMT

Intro

Formatting is one of several concerns in the efforts to write clean code. There's a lot of other stuff we should be concerned about as well, but formatting is one of those things that we can set up right off the bat and establish a standard for our project.

ESLint and TSLint

ESLint is a JavaScript linter that enables you to enforce a set of style, formatting, and coding standards for your codebase. It looks at your code, and tells you when you're not following the standard that you set in place.

You may have also heard of TSLint, the TypeScript equivalent. In 2019, the team behind TSLint decided that they would no longer support it. The reason, primarily, is because ESLint exists, and there was a lot of duplicate code between projects with the same intended purpose.

That being said, there are some really awesome TSLint packages out there that, if you would like to use, you can- but just understand that they're not being supported anymore.

So onwards into 2020 and beyond, we're going to continue to look to ESLint for all of our TypeScript (and JavaScript) linting needs!

Prerequisites

Here's what you need to have in order to follow along:

A code editor installed (psst- VS Code is ideal)

An existing codebase (if you need to get setup, you can follow "How to Setup a TypeScript + Node.js Project first and then come back to this article)

Installation and setup

Run the following commands to setup ESLint in your TypeScript project.

npm install --save-dev eslint @typescript-eslint/parser @typescript-eslint/eslint-plugin

Create an .eslintrc file.

touch .eslintrc

In it, use the following starter config.

{ "root": true, "parser": "@typescript-eslint/parser", "plugins": [ "@typescript-eslint" ], "extends": [ "eslint:recommended", "plugin:@typescript-eslint/eslint-recommended", "plugin:@typescript-eslint/recommended" ] }

Ignoring files we don't want to lint

Create an .eslintignore in order to prevent ESLint from linting stuff we don't want it to.

touch .eslintignore

Then add the things we want to ignore. In the following code sample, we're ignoring the dist/ folder that contains the compiled TypeScript code. If you're compiling your TypeScript code to a different folder, make sure to use that instead of dist. You should be able to find this in your .tsconfig (see the previous guide).

node_modules dist

Adding a lint script

In your project package.json, lets add a lint script in order to lint all TypeScript code.

{ "scripts": { ... "lint": "eslint . --ext .ts", } }

Ready to try it out? Let's run the following command.

npm run lint

For me, since I'm continuing with the previous tutorial, and since my src folder only has a single index.ts in it that prints out some text with console.log(), I don't see anything after I run the command.

src/index.ts

console.log('Hello world!')

What if we wanted to disallow console.log statements in our code?

Rules

There are three modes for a rule in eslint: off, warn, and error.

"off" means 0 (turns the rule off completely)

"warn" means 1 (turns the rule on but won't make the linter fail)

"error" means 2 (turns the rule on and will make the linter fail)

Adding a rule

In .eslintrc, add a new attribute to the json object called "rules".

{ "root": true, "parser": "@typescript-eslint/parser", "plugins": [ "@typescript-eslint" ], "extends": [ "eslint:recommended", "plugin:@typescript-eslint/eslint-recommended", "plugin:@typescript-eslint/recommended" ], "rules": { }}

Rules get added as keys of this rules attribute, and you can normally find the eslint base rules here on their website via the Rules docs.

We want to add the no-console rule, so here is an example of how we can make the linter fail (throw a mean error code) if it encounters a console.log statement with the no-console rule set to error.

We update the .eslintrc

{ "root": true, "parser": "@typescript-eslint/parser", "plugins": [ "@typescript-eslint" ], "extends": [ "eslint:recommended", "plugin:@typescript-eslint/eslint-recommended", "plugin:@typescript-eslint/recommended" ], "rules": { "no-console": 2 // Remember, this means error! } }

And then run the linter again.

npm run lint

And we should get an angry linting result.

/simple-typescript-starter/src/index.ts 2:1 error Unexpected console statement no-console ✖ 1 problem (1 error, 0 warnings)

And if we wanted, we could turn the rule off by setting it to 0 - off.

{ "root": true, "parser": "@typescript-eslint/parser", "plugins": [ "@typescript-eslint" ], "extends": [ "eslint:recommended", "plugin:@typescript-eslint/eslint-recommended", "plugin:@typescript-eslint/recommended" ], "rules": { "no-console": 0 } }

Run the linter.

npm run lint

...and silence.

Using rules in a real-life project

There's a reason that all three of these modes exist. When you set the linter to off for a rule, it's because you and your team probably don't care about that rule in a particular base configuration you're using (we're currently using the ESLint recommended config but you can also go and use Shopify's, Facebook's or several other companies' configs as starting points instead).

When you set the rule to error - 2, it means that you don't want the code that breaks your coding conventions to even make it into the repo at all. I think this is a great act of professionalism and empathy towards the codebase, your fellow teammates, and future maintainers. A popular approach to actually enforce code conventions with this tool is to set up your project with a tool like Husky so that when a teammate tries to commit code or push code, you can tell your linter to check the code first before the operation executes. It's a great habit, though sometimes, if the rules are overly restrictive, it can slow down and annoy your teammates.

To remedy overly restrictive rules, the warn - 1 setting means that yes, you want you and your team to adhere to that rule, but you don't want it to prevent you from moving forward.

Adding a plugin (features)

ESLint also allows you to add one-off features to your config. These are known as plugins.

Here's a fun one. It's called no-loops.

Check out this list of other awesome-eslint plugins and configs.

no-loops is a plugin that will enable you to enforce a convention specifying that for and while loops are illegal and that you should use functions like map and forEach instead.

Install it like this.

npm install --save-dev eslint-plugin-no-loops

And then update your .eslintrc with no-loops in the "plugins" array, and add the rule to the "rules" attribute like so.

{ "root": true, "parser": "@typescript-eslint/parser", "plugins": [ "@typescript-eslint", "no-loops" ], "extends": [ "eslint:recommended", "plugin:@typescript-eslint/eslint-recommended", "plugin:@typescript-eslint/recommended" ], "rules": { "no-console": 1, "no-loops/no-loops": 2 } }

Now if we update our code to include a for loop...

src/index.ts

console.log('Hello world!'); for (let i = 0; i < 10; i++) { console.log(i) }

And run the lint command again...

npm run lint

We'll see the errors that restricts loops appear.

/simple-typescript-starter/src/index.ts 2:1 warning Unexpected console statement no-console 2:1 error 'console' is not defined no-undef 4:1 error loops are not allowed no-loops/no-loops 5:3 warning Unexpected console statement no-console 5:3 error 'console' is not defined no-undef 5:17 error Missing semicolon babel/semi 5:17 error Missing semicolon semi 6:2 error Newline required at end of file but not found eol-last ✖ 8 problems (6 errors, 2 warnings) 3 errors and 0 warnings potentially fixable with the `--fix` option.

Extending a different base config

Let's say that you wanted to start with a different base config- Shopify's, for example.

Here's how to do that.

Looking at the readme, we need to install it by running:

npm install eslint-plugin-shopify --save-dev

Update our .eslintrc

{ "root": true, "parser": "@typescript-eslint/parser", "plugins": [ "@typescript-eslint" ], "extends": [ "plugin:shopify/esnext" ], "rules": { "no-console": 1 } }

You can add several base configs to your project by including them in the array, though you may end up seeeing the same linting rules twice or more.

Now when we run the lint command again with npm run lint, we can see a few errors reported based on the base Shopify config and our no-console rule.

/simple-typescript-starter/src/index.ts 2:1 warning Unexpected console statement no-console 2:1 error 'console' is not defined no-undef 2:28 error Missing semicolon babel/semi 2:28 error Missing semicolon semi ✖ 4 problems (3 errors, 1 warning) 2 errors and 0 warnings potentially fixable with the `--fix` option.

Fixing linted code with ESLint

You might have noticed that at the end of the error message, it says "2 errors and 0 warnings potentially fixable with the --fix option."

You can run ESLint and tell it to fix things that it's able to fix at the same time.

Using the --fix option, let's add a new script to our package.json called lint-and-fix.

{ "scripts": { ... "lint": "eslint . --ext .ts", "lint-and-fix": "eslint . --ext .ts --fix" }, }

Running this command against our code, I expect that it will put a semicolon on the console.log statement that we had.

Let's run it.

npm run lint-and-fix

The result is that less errors are reported. We don't see the error about semi-colons anymore.

/simple-typescript-starter/src/index.ts 2:1 warning Unexpected console statement no-console 2:1 error 'console' is not defined no-undef

Because sure enough, the linter added the semi-colon.

src/index.ts

console.log('Hello world!');

That's really awesome. But what if we don't want to run the linter all the time to fix our code? What if there was a way that we could, while coding, have it automatically format things based on our conventions?

We can! With Prettier. Read the next article, "How to use Prettier with ESLint and TypeScript in VSCode".

Deploying Serverless GraphQL APIs with TypeScript on Netlify

Sun, 23 Feb 2020 15:04:10 GMT

Deploying a Serverless GraphQL API with TypeScript is essentially the same as deploying one with JavaScript.

The only difference is in the build steps.

For example, using vanilla JavaScript on Netlify, the build step in our package.json is usually to copy all of the vanilla JavaScript code in our src folder over to the functions folder.

package.json

{ "scripts": { "start": "nodemon", "bundle": "cpx src/**/* functions/bundle" } }

While building with TypeScript is effectively the same, just through different means. In a TypeScript environment, we configure a tsconfig.json and run tsc as the build step.

package.json

{ "compilerOptions": { "allowJs": true, "module": "commonjs", "moduleResolution": "node", "pretty": true, "sourceMap": true, "target": "es2017", "outDir": "./functions/bundle", "lib": ["es6"], "resolveJsonModule": true, "types": ["node"], "typeRoots" : ["./node_modules/@types", "./src/@types"], "experimentalDecorators": true, "emitDecoratorMetadata": true, "esModuleInterop": true }, "include": [ "src/*.ts", "src/**/*.ts", "src/**/*.js" ], "exclude": [ "node_modules", "src/**/*.spec.ts", "src/**/*.spec.js" ] }

package.json

{ "scripts": { "start": "nodemon", "bundle": "tsc" } }

Either way, the result is the same since Netlify needs all of the source code used in the functions direction to be bundled before it's deployed.

Resources

Check out these starters for deploying your own Serverless GraphQL APIs on Netlify with TypeScript or vanilla JavaScript.

Starters

Serverless GraphQL on Netlify API Starter Project

Serverless GraphQL on Netlify API using TypeScript Starter Project

Tutorial

If you're interested in how the starters were built, read "How to Deploy a Serverless GraphQL API on Netlify [Starters]".

How to Deploy a Serverless GraphQL API on Netlify [Starters]

Sun, 23 Feb 2020 04:05:26 GMT

You know, I'm very proud to be a developer in 2020. The access we have to tools and services is impressive. The barrier to entry towards crafting side-projects, startups, businesses, etc.- it's almost non-existent.

With tools like Netlify, we can deploy a serverless Apollo GraphQL server for free in seconds. Un. Real.

While I like being able to do things quickly, I'm also one who believes that "if we do it, we know it". So I'm going to give you a few useful resources on how to deploy serverless GraphQL APIs and then walk you through the process so that you can better appreciate what's going on.

Huge shoutout to the great Trevor Blades who cut 80% of the time it would have taken me to figure this stuff out on my own. Most of this is based off of his Countries GraphQL API.

Prerequisites

You should have Node and npm installed

You should understand the concept of serverless functions

You have a code editor installed (preferably VS Code, it's the champ, don't @ me)

Goals

By the start of this article (like- right now), you should be aware of two resources to deploy GraphQL APIs on Netlify. Check them out below.

Serverless GraphQL on Netlify API Starter Project

Serverless GraphQL on Netlify API using TypeScript Starter Project

If you just need to get something up and running, you can go to either of those projects, click "Deploy to Netlify", and you're good to go. You can even stop reading right here.

The second goal for the rest of this article is to explain how these starters work, what they can do, and how you can use them for your next serverless project.

We will go through the process of creating a GraphQL API using Netlify from scratch with vanilla JavaScript (though the process is similar with TypeScript).

Initial setup

First we need to create a folder.

mkdir serverless-starter cd serverless-starter

Setup Node.js package.json

Next, we'll create a package.json and say "yes" to everything.

npm init -y

Install dependencies

We don't need a whole lot to get things going. In your console, paste the following command to install the dependencies.

npm install --save apollo-server apollo-server-lambda npm install --save-dev cpx nodemon

apollo-server: When we're developing locally, we're going to want to run our code against a local GraphQL endpoint. Let's install this so that we can craft an excellent local development experience.

apollo-server-lambda: This is a version of Apollo Server optimized to work on serverless lambda functions. It's what we use to deploy to production.

cpx: In Netlify, we need to bundle all of our source code in the functions/ folder, so we need a way to copy the code from our src/ folder in a cross-platform-friendly way to the functions/ folder.

nodemon: We all like cold-reloading. Install this so that we can get live updates when we change any files.

Creating our server implementations

Let's create our server implementations at src/server.js.

touch src/server.js

In server.js, we're going to import both the ApolloServer from apollo-server and apollo-server-lambda, but we're going to call the one imported from the lambda project, ApolloServerLambda instead.

server.js

const ApolloServer = require('apollo-server').ApolloServer const ApolloServerLambda = require('apollo-server-lambda').ApolloServer const { gql } = require('apollo-server-lambda');

Now let's add a simple type definition and resolvers object.

server.js

const typeDefs = gql` type Query { hello: String } `; const resolvers = { Query: { hello: () => "Hi! Love from @khalilstemmler 🤠." } };

The last thing in this file- let's create two factory functions: one for the lambda server, and one for the regular server. We'll also export these from the module as an object.

server.js

function createLambdaServer () { return new ApolloServerLambda({ typeDefs, resolvers, introspection: true, playground: true, }); } function createLocalServer () { return new ApolloServer({ typeDefs, resolvers, introspection: true, playground: true, }); } module.exports = { createLambdaServer, createLocalServer }

Sweet, now let's get our local server going.

Configuring our local server

Create src/index.js, the entry point to our local development efforts.

touch src/index.js

Then import createLocalServer from ./server, create a server instance, and fire it up.

src/index.js

const { createLocalServer } = require("./server") const server = createLocalServer(); server.listen().then(({ url }) => { console.log(`🚀 Server ready at ${url}`); });

Nice! That's how we get a local GraphQL server up and running.

Starting up the local server with Nodemon

Because we like cold-reloading, let's create a nodemon.json file in the root of the project and add configure it to listen to changes to anything in src/ or functions/ (this is where we will configure our GraphQL server).

nodemon.json

{ "watch": ["src", "functions"], "ext": ".js", "ignore": [], "exec": "node ./src/index.js" }

In the package.json, we can add a start script that simply executes nodemon like so.

package.json

{ "scripts": { "start": "nodemon" } }

To start the project, run this command.

npm run start

Then go to http://localhost:4000/ and check out the local server.

Configuring our lambda server

Now that we've got the local server running, it's time to hook up the one that will run on Netlify.

Create the functions directory

At the root of the project, create a directory called functions/- this is where we'll tell Netlify to look for our functions.

touch functions

Create the serverless graphql endpoint

In functions, create a file called graphql.js.

touch functions/graphql.js

Whatever we put in the functions folder, Netlify will make it publicly visible as if it were in a directory.

When we deploy this, we will be able to reach this file directly by going to https://OUR_SITE_NAME/.netlify/functions/graphql.

Let's get access to our createLambdaServer factory function, call it, and then expose a handler using server.createHandler, passing in the most permissive CORS configuration.

functions/graphql.js

const { createLambdaServer } = require('../src/server') const server = createLambdaServer(); exports.handler = server.createHandler({ cors: { origin: '*' } });

This looks good, but there's a problem here. Netlify can't reach outside of the functions folder and into src.

If we were to deploy this, Netlify would yell at us and tell us that it can't find createLambdaServer.

The solution is to bundle the code in src and copy it over to the functions folder.

Bundling our source code

In our package.json, let's add the bundle script.

package.json

{ "scripts": { "bundle": "cpx src/**/* functions/bundle" }, }

This command takes everything in src and copies it to functions/bundle.

Let's give it a run.

npm run bundle

You should see a new folder appear in the functions folder with the source code from src.

Go into functions/graphql.js and update the reference to the createLambdaServer function.

functions/graphql.js

const { createLambdaServer } = require('./bundle/server') const server = createLambdaServer(); exports.handler = server.createHandler({ cors: { origin: '*' } });

We're pretty much all set to go! The last thing to do is configure Netlify.

Netlify configuration

Here's the stuff that's magic to me.

Empty index.html

I'm not entirely sure if this is necessary (someone correct me), but an empty index.html file should live at the root of the project.

touch index.html

Redirects file

By default, our endpoint lives at https://OUR_SITE_NAME/.netlify/functions/graphql. We don't want to have to refer to it like that; we'd prefer to just see to the site like https://OUR_SITE_NAME/.

To remedy this, create a _redirects file at the root of the project.

touch _redirects

Then add the following config.

_redirects

/ /.netlify/functions/graphql 200!

The exclamation mark signals that we will rewrite everything at the root URL to /.netlify/functions/graphql. Read more about Shadowing from the docs.

Netlify.toml file

The last thing we need to do is tell Netlify what the build command is and where our functions live.

We can do set this up with a netlify.toml file.

Let's create one at the root of the project.

touch netlify.toml

And since we need to bundle our code before we deploy, let's use npm run bundle as the build command.

netlify.toml

[build] command = "npm run bundle" functions = "functions"

That's it!

Deploying to Netlify

Two effortless ways to deploy this are:

Click the "Deploy to Netlify" button in either starter. It will clone the repos to your GitHub account and deploy them.

Manually clone the repos, log in to Netlify, and then select a repository to deploy.

Nested GraphQL Resolvers & Separating Concerns

Sat, 22 Feb 2020 15:04:10 GMT

Unfortunately, in GraphQL, you can't nest operations to apply better grouping (namespacing / separation of concerns) to your services.

Here's an example. I was building my own personal data graph with all kinds of cool things on it like my spotify, my Google calendar, and my github activity.

You can check out my personal data graph at stemmlerjs-graph.netlify.com

I wanted to design the schema where each service lived at the top-level. That would enable me to separate concerns and place all operations within the services like so:

schema.graphql

type Spotify { getCurrentSong (): Song getRecentlyListenedToSongs (): [Song]! ... } type Calendar { getCalendarForMonth (month: String!, year: Integer): CalendarResult! ... } type GitHub { getRecentActivity (): GitHubActivityResult! ... } type Query { spotify: Spotify calendar: Calendar github: GitHub }

Looking at the structure of the root Query type, you'd assume that the resolvers object would assume the same shape of the Query type, like so.

resolvers.ts

const resolvers = { Query: { spotify: { // Will not get invoked. spotifyGetCurrentSongPlaying: async () => { const currentSongResult = await getCurrentSong.execute(); return currentSongResult.isRight() ? currentSongResult.value : null } } } }

Unfortunately, this won't work. It's not that it isn't valid GraphQL, but it's just that if we were to do this, none of our nested resolvers will ever get invoked.

With ApolloServer, if we want to refer to a type nested deeper than one level, that type needs to be defined as its own attribute on the resolvers object like the following.

resolvers.ts

const resolvers = { Query: { spotify: () => ({}) // Return nothing }, Spotify: { spotifyGetCurrentSongPlaying: async () => { const currentSongResult = await getCurrentSong.execute(); return currentSongResult.isRight() ? currentSongResult.value : null } } }

This works!

Although on a larger application, we'd like to be able to enforce some sort of namespacing. Some separation of concerns.

Take the following example of a GraphQL schema with the users and movies subdomains. Without namespacing, we end up with large schemas that look like the following.

schema.graphql

type Query { Movie(_id: String, movieId: ID, title: String, year: Int, description: String, first: Int, offset: Int, orderBy: [_MovieOrdering]): [Movie] Actor(actorId: ID, name: String, _id: String, first: Int, offset: Int, orderBy: [_ActorOrdering]): [Actor] User(userId: ID, name: String, _id: String, first: Int, offset: Int, orderBy: [_UserOrdering]): [User] } type Mutation { CreateMovie(movieId: ID, title: String, year: Int, description: String): Movie UpdateMovie(movieId: ID!, title: String, year: Int, description: String): Movie DeleteMovie(movieId: ID!): Movie AddMovieActors(from: _ActorInput!, to: _MovieInput!): _AddMovieActorsPayload RemoveMovieActors(from: _ActorInput!, to: _MovieInput!): _RemoveMovieActorsPayload AddMovieRatings(from: _UserInput!, to: _MovieInput!, data: _RatingInput!): _AddMovieRatingsPayload CreateActor(actorId: ID, name: String): Actor UpdateActor(actorId: ID!, name: String): Actor DeleteActor(actorId: ID!): Actor AddActorMovies(from: _ActorInput!, to: _MovieInput!): _AddActorMoviesPayload RemoveActorMovies(from: _ActorInput!, to: _MovieInput!): _RemoveActorMoviesPayload CreateUser(userId: ID, name: String): User UpdateUser(userId: ID!, name: String): User DeleteUser(userId: ID!): User AddUserRating(from: _UserInput!, to: _MovieInput!, data: _RatingInput!): _AddUserRatingPayload RemoveUserRating(from: _UserInput!, to: _MovieInput!): _RemoveUserRatingPayload }

A poorly namespaced GraphQL schema.

You may be able to see that the grouping and cohesion between related operations are not present here. It's visually challenging to group all operations related to users, actors, and movies into units. Everything is mixed together.

In Domain-Driven Design, each subdomain contains only the operations that are related to that subdomain. Subdomains are well encapsulated.

For example, looking at these operations, I can deduce that we have two subdomains- a users one and a movies one (actor is a concept that would also belong to movies).

Therefore,

All the operations for users go into the users subdomain.

All the operations for movies go into the movies subdomain.

users.schema

type Query { User(userId: ID, name: String, _id: String, first: Int, offset: Int, orderBy: [_UserOrdering]): [User] } type Mutation { CreateUser(userId: ID, name: String): User UpdateUser(userId: ID!, name: String): User DeleteUser(userId: ID!): User AddUserRating(from: _UserInput!, to: _MovieInput!, data: _RatedInput!): _AddUserRatedPayload RemoveUserRating(from: _UserInput!, to: _MovieInput!): _RemoveUserRatedPayload }

movies.schema

type Query { Movie(_id: String, movieId: ID, title: String, year: Int, description: String, first: Int, offset: Int, orderBy: [_MovieOrdering]): [Movie] Actor(actorId: ID, name: String, _id: String, first: Int, offset: Int, orderBy: [_ActorOrdering]): [Actor] } type Mutation { CreateMovie(movieId: ID, title: String, year: Int, description: String): Movie UpdateMovie(movieId: ID!, title: String, year: Int, description: String): Movie DeleteMovie(movieId: ID!): Movie AddMovieActors(from: _ActorInput!, to: _MovieInput!): _AddMovieActorsPayload RemoveMovieActors(from: _ActorInput!, to: _MovieInput!): _RemoveMovieActorsPayload AddMovieRatings(from: _UserInput!, to: _MovieInput!, data: _RatingInput!): _AddMovieRatingsPayload CreateActor(actorId: ID, name: String): Actor UpdateActor(actorId: ID!, name: String): Actor DeleteActor(actorId: ID!): Actor AddActorMovies(from: _ActorInput!, to: _MovieInput!): _AddActorMoviesPayload RemoveActorMovies(from: _ActorInput!, to: _MovieInput!): _RemoveActorMoviesPayload }

There's still references between subdomains, and that's ok, because in software, we eventually need to connect pieces together.

That said, it's a good idea to be explicit about those relationships- this is where tools like Apollo Federation come in handy because we can compose schemas and be precise about where fields are resolved from between services.

Mutations

Unfortunately, the story of nested resolvers for mutations is a lot shorter. See here and here.

Solutions

Let's look at two approaches to remedy our design issue.

1. Enforcing a GraphQL-operation naming pattern for both queries and mutations

Best for modular monolith applications where several subdomains are housed from within the same project (or in fancy DDD-talk, the same bounded context).

With a modular monolith, users might be folder with everything related to users, where movies is its own folder with everything related to movies, as per screaming architecture.

DDDForum, the app that we build at the end of solidbook.io using Domain-Driven Design, is a modular monolith.

Suggested by @dncrews on GitHub, he suggests using the following GraphQL-operation naming pattern.

<primaryResource><Action><SecondaryResource>

Updating the previous schema with this pattern may make it look like this.

graphql.schema

type Mutation { # Movies subdomain movieCreate movieUpdate movieDelete movieAddActor movieAddActors movieRemoveActor movieRemoveActors movieAddRatings actorCreate actorUpdate actorDelete # Users subdomain userCreate userUpdate userDelete userAddRating userRemoveRating }

It's not the perfect solution, but it's good enough for many use cases.

Modularizing your schema: If you're not into having everything in one file, there are several ways you can modularize your GraphQL schema. Two popular tools are merge-graphql-schemas and graphql-modules, though I've also resorted to simple string interpolation with Apollo Server.

2. Federated services

If you're part of a larger organization, and other teams are can take ownership of their own GraphQL endpoints for their respective services, Apollo Federation is a good idea.

With Federation, we can get that separation of concerns at the service level by delegating operations to the appropriate GraphQL endpoint in the organization using an Apollo Gateway. Here's the configuration of an Apollo Gateway.

gateway.ts

const gateway = new ApolloGateway({ serviceList: [ { name: 'accounts', url: 'http://localhost:4001' }, { name: 'products', url: 'http://localhost:4002' }, { name: 'reviews', url: 'http://localhost:4003' } ] }); const server = new ApolloServer({ gateway }); server.listen();

Configuration of an Apollo Gateway to implement a federated GraphQL architecture.

In a Federated architecture, we can use the @provides, @key, and @external directives to compose schemas, be explicit about the relationships between services, and define which service is responsible for resolving a field in question.

reviews.schema

type Review { body: String author: User @provides(fields: "username") product: Product } extend type User @key(fields: "id") { id: ID! @external reviews: [Review] } extend type Product @key(fields: "upc") { upc: String! @external reviews: [Review] }

Clone and try out the Apollo Federation Demo if you're interested in this approach. To learn more about how it works and how to get started, check out the docs.

Conclusion

In order to achieve true separation of concerns at the service level, try out Apollo Federation.

If you're working on a smaller project or working in a monolith, consider enforcing a naming pattern with the namespace (subdomain) at the front of the GraphQL operation.

How to Get the Currently Playing Song using the Spotify Node.js API & TypeScript

Sat, 22 Feb 2020 04:05:26 GMT

Intro

When I first started building khalilstemmler.com, I stubbed out a component to present the current song that I was listening to on Spotify. Y'know, I just wanted to add a neat little feature to make the site a little bit more unique.

Fast forward eight months, and it's still not hooked up. After about four attempts, I've finally gotten the hard part down (integrating with the API). At some point, I might code the React component.

This fun Spotify feature is part of my Personal Data Graph.

The reason I need a Personal Data Graph for this feature is, to fetch authenticated user resources from Spotify, the service that asks for the data needs to be, well- authenticated... as me.

The docs from Spotify's Authorization Guide told me that I needed to use the Authorization Code approach, one of 3, to make this work.

I'm going to plow through this process and show you, very imperatively, the steps involved to authenticate a backend service to act on your behalf to fetch your currently playing song from Spotify.

Prerequisites

You're going to need to be using Node version 10x or higher for this to work. The reason for that is because the code that talks to Spotify relies on the URLSearchParams global.

Installation and Setup

Clone or fork my repo over at https://github.com/stemmlerjs/stemmlerjs-graph to follow along.

Run the following commands to clone, install, and set up the project.

git clone git@github.com:stemmlerjs/stemmlerjs-graph.git cd stemmlerjs-graph npm install mv .env.template .env

Creating a Spotify Developers Account

Go to the Spotify Developers Dashboard and sign up for an account.

When you're all signed up, you'll get to the Spotify Developers Dashboard that shows your applications. Click "Create An App".

Spotify Developers Dashboard. You won't have an application already when you sign up the first time.

Follow the prompts and select Non-Commercial for the type of integration that we're creating (if you are creating a non-commercial app, that is).

Getting your Client ID and Client Secret

When you log in, you should be able to see your Client ID. Underneath it, you can click to see your Client Secret.

First homework on the repo you cloned is to go into the .env file and supply the Client ID and Client Secret into the file for SPOTIFY_CLIENT_ID and SPOTIFY_CLIENT_SECRET.

.env

SPOTIFY_CLIENT_SECRET= # paste here SPOTIFY_CLIENT_SECRET= # paste here as well :-)

Setting up our application

Now we need to configure the application with the website and redirect URI.

On the dashboard page, click "Edit Settings", and a modal should appear.

Configuring the Website

For website, you don't need to have a website up and running yet, so just put in the name of a website that you own.

If you don't own a website, you can use google.com.

Configuring the Redirect URI

This is where Spotify sends us after we've logged in. They send us to the URL that we supply, but also give us back an authorization code. We use that authorization code to get an access and refresh token.

If you have a website, you can put any URL from your domain here, and Spotify will redirect us there after logging in.

If you don't have a website, again- that's OK, we can use http://google.com/callback or something, but do understand that the domain that owns the website that we redirect to is fully capable of reading the authorization code we receive.

When you enter your Redirect URI, don't forget to press the "Add" button beside the URL, and THEN click "Save". That's how attempt #1 of hooking this up failed for me - though you're much smarter so that won't happen to you.

Whatever you've provided as the Redirect URI, make sure to update the .env with this as well.

.env

... SPOTIFY_REDIRECT_URL=https://khalilstemmler.com/callback # For example

Getting the initial Authorization Code

Take this URL and substitute your Client ID and Redirect URI, where I've placed PASTE_HERE, then paste the entire link into your browser.

https://accounts.spotify.com/authorize?client_id=PASTE_HERE& response_type=code&redirect_uri=PASTE_HERE& scope=user-read-private%20user-read-currently-playing

When you paste the link in your browser, you'll be asked to log in using your Facebook or email. Do that.

Successfully logging in will bring you to this page that asks you to confirm the type of data that we're asking for. The authorization scope that we're interested in is being able to see the user's currently playing song. Click "agree" here.

After clicking "agree", we'll get sent to the redirect URI that we passed in earlier, but with our authorization code as a code parameter in the resulting URL (check the URL bar).

We're going to want whatever comes after ?code= in the URL, so copy that.

Using the Authorization Code to get an Access Token and the Refresh Token

In the root of the repository, run the following script with the code you copied from the URL.

npm run get-refresh-token <paste-your-code-here>

This will ask for a permanent refresh token that we can use to make subsequent requests to the API and never have to worry about logging in again (that's dope).

If all goes well, console output should look like this.

Copy the refresh token that was emitted to the console into the .env file.

.env

... SPOTIFY_REFRESH_TOKEN= # Paste the refresh token that we just got here

A couple of things to note if this doesn't work for you.

You have to run this script within roughly a minute of getting the authorization code from Spotify. This is a security by opportunity- they reduce the total amount of time that someone can use that code hanging out in the open. If you get a 400 error, go back to Getting the initial Authorization Code and paste the link again to generate a new authorization code.

It will also fail if you've forgotten to supply the SPOTIFY_CLIENT_ID, SPOTIFY_CLIENT_SECRET, and SPOTIFY_REDIRECT_URL values.

The last reason this might fail is if you're not running Node version 10 or higher.

Getting the currently playing song via GraphQL Playground

In the repo, I've already gone and done all the hard work of setting up a GraphQL server, defining the schema, and mapping the API results to a song type.

The action happens in spotifyService.ts which you can peruse if you like.

If you'd like to play around with the graph via GraphQL playground, run:

npm run start

Then go to http://localhost:4000/, fire up your Spotify, play some music, and then run this query to see the results.

{ spotifyGetCurrentSongPlaying { artist { name } title previewUrl externalUrl currentProgress album { name } } }

Voila!

If you're interested in my terrible music taste, pop over to stemmlerjs-graph.netlify.com and see what I'm listening to currently.

That's it! I hope this helps someone.

How to Build a Compiler?

Mon, 17 Feb 2020 15:04:10 GMT

Today, in the Advanced JavaScript Developers group, someone asked the question:

Musing from a Noob: so if you were to write a translator from one language to another, say JS to Java or something arbitrary, would that translator be written almost entirely in like regular expressions or something? My google-fu is solid, but I'm trying to get some professional opinions first.... What y'all think?

I'm no professional in compiler development (I never got a chance to take Brock University's awesome compiler course), but I once wrote a LISP interpreter in Java (takes in some LISP code and executes it).

For this, you might be able to hack your way through it with regular expressions, but I think the proper way is to build an Abstract Syntax Tree.

In all programming languages, there's the concept of a grammar. The grammar is something that defines all legal expressions in a language (much like English does).

For example, in JavaScript, we're allowed to do:

const x = 12

But we can't do:

const x y z = 12

The grammar that formalizes this might look something like:

"[variable type] [variable name] = [value]"

And we didn't follow that rule in the second example.

If languages each have a set of grammar rules, and those rules are all the legal expressions, then there are primarily two parts to building a compiler.

Establish the language rules (grammar)

Be able to read a file, parse it, then build an validate an Abstract Syntax Tree from that grammar. If you can't build the syntax tree, it's because some grammar is wrong (this is the concept of a syntax error).

There's also the validation of the tree itself.

For example, in Java, for a single tree, where a tree is a file (a program, like myApp.java), you're only allowed to have the public static void main (String[] args) once in the program, because that's the entry point.

So I think in order to PROPERLY do this conversion, you need to:

Build the grammar for JavaScript

Build the grammar for Java

Know what the conversion rules from one grammar to the other are

Build the syntax tree for JS code

For each node (or a group of nodes) in the tree, run it against the conversion rules for Java and build a new syntax tree (a Java one).

Convert the syntax tree to Java source code!

(optional) run it through a prettier thing to make it look clean :p

Definitely leave a comment below if you know a bit about the subject, you've built a compiler before, or you have some useful resources for others.

Resources

I found this page, Introduction to Programming Languages/Grammars. It seems really useful.

There's also Babel, a modern JavaScript compiler that you might have used. I suspect peeking at the source code here would be insightful :)

Other stuff:

[Free] https://www.craftinginterpreters.com/the-lox-language.html

[Paid, but the content is really good apparently!] https://coded-classes.dpdcart.com/product/81271

Ensuring Sequelize Hooks Always Get Run

Sun, 02 Feb 2020 15:04:10 GMT

Intro

In "Decoupling Logic with Domain Events [Guide] - Domain-Driven Design w/ TypeScript", we use Sequelize Hooks to decouple business logic, allowing the system to respond to events in a fashion similar to the Observer Pattern.

Sequelize hooks are places we can write callbacks that get invoked at key points in time like afterCreate, afterDestroy, afterUpdate, and more.

infra/sequelize/hooks/index.ts

import models from "../models" import { DomainEvents } from "../../../core/domain/events/DomainEvents" import { UniqueEntityID } from "../../../core/domain/UniqueEntityID" const dispatchEventsCallback = (model: any, primaryKeyField: string) => { const aggregateId = new UniqueEntityID(model[primaryKeyField]) DomainEvents.dispatchEventsForAggregate(aggregateId) } (async function createHooksForAggregateRoots() { const { User } = models User.addHook("afterCreate", (m: any) => dispatchEventsCallback(m, "user_id")) User.addHook("afterDestroy", (m: any) => dispatchEventsCallback(m, "user_id")) User.addHook("afterUpdate", (m: any) => dispatchEventsCallback(m, "user_id")) User.addHook("afterSave", (m: any) => dispatchEventsCallback(m, "user_id")) User.addHook("afterUpsert", (m: any) => dispatchEventsCallback(m, "user_id")) })()

In Domain-Driven Design, after a transaction completes, we want to execute domain event handlers in order to decide whether we should invoke any follow up commands or not.

forum/subscriptions/afterUserCreated.ts

import { UserCreated } from "../../users/domain/events/userCreated" import { IHandle } from "../../../shared/domain/events/IHandle" import { CreateMember } from "../useCases/members/createMember/CreateMember" import { DomainEvents } from "../../../shared/domain/events/DomainEvents" export class AfterUserCreated implements IHandle<UserCreated> { private createMember: CreateMember constructor(createMember: CreateMember) { this.setupSubscriptions() this.createMember = createMember } setupSubscriptions(): void { // Register to the domain event DomainEvents.register(this.onUserCreated.bind(this), UserCreated.name) } private async onUserCreated(event: UserCreated): Promise<void> { const { user } = event try { await this.createMember.execute({ userId: user.userId.id.toString() }) console.log( `[AfterUserCreated]: Successfully executed CreateMember use case AfterUserCreated` ) } catch (err) { console.log( `[AfterUserCreated]: Failed to execute CreateMember use case AfterUserCreated.` ) } } }

In the Sequelize Repository, where we deal with persistence logic, I have noticed that the hook callbacks do not get called if no new rows were created and no columns were changed.

In the save method of a repository, you'll find code where we rely on a mapper to convert the domain object to the format necessary for Sequelize to save it. You'll also find code that determines if we're doing a create or an update based on the domain object's existence.

forum/repos/implementations/sequelizePostRepo.ts

export class SequelizePostRepo implements PostRepo { ... public async save (post: Post): Promise<void> { const PostModel = this.models.Post; const exists = await this.exists(post.postId); const isNewPost = !exists; const rawSequelizePost = await PostMap.toPersistence(post); if (isNewPost) { try { await PostModel.create(rawSequelizePost); await this.saveComments(post.comments); await this.savePostVotes(post.getVotes()); } catch (err) { await this.delete(post.postId); throw new Error(err.toString()) } } else { await this.saveComments(post.comments); await this.savePostVotes(post.getVotes()); // Persist the post model to the database await PostModel.update(rawSequelizePost, { where: { post_id: post.postId.id.toString() } }); } } }

In the highlighted lines, I expect the afterUpdate hook to get called, though it will not in scenarios where there were no are differences.

To fix this, Sequelize's update method's second parameter configuration object allows you to pass in hooks: true.

forum/repos/implementations/sequelizePostRepo.ts

await PostModel.update(rawSequelizePost, { where: { post_id: post.postId.id.toString() } // Be sure to include this to call hooks regardless // of anything changed or not. hooks: true,});

Doing this will ensure that the hooks get run everytime.

Domain-Driven GraphQL Schema Design using Event Storming

Fri, 31 Jan 2020 22:04:10 GMT

Introduction

Designing your GraphQL schema is a potentially expensive task.

It's potentially expensive because if we respect the principles of GraphQL, the (company-wide) GraphQL schema becomes the singular source of truth for our client applications, and the contract for our sever-side data sources.

The single source of truth isn't exactly the type of thing you want to get terribly wrong the first time around.

This sentiment is shared by expert GraphQL-r and developer from Pinterest Engineering, Francesca Guiducci, who says to "involve others when designing the GraphQL schema" (via Netlify, Jan 21st, 2019).

I'm with Francesca on this one. Your GraphQL API is the language others use to build on top of your services, and if it doesn't come from a shared understanding of the domain, some domain concepts could end up being really off.

Designing a GraphQL schema is not usually something a single developer creates in isolation.

Domain-Driven GraphQL?

Domain-Driven GraphQL means putting an effort into understanding the domain you work in, and using that knowledge to drive development against a model of the business that mimics how it works in the real world.

Code that acts as a software representation of the business is more correct, more tolerant to change, has less inaccurate and undesireable side-effects, and can be understood by everyone.

Often, developers are really good at developing, but don't hold a great understanding of the domain. To fix that, we can spend more time interacting with people who do understand the domain.

You're probably more likely to find a rare unicorn strolling down your company office than you are to find someone who is both a GraphQL Expert AND a Domain Expert - from Marc-André Giroux's talk at GraphQL Summit 2018.

This post was inspired by a tweet I blasted out recently. It describes how we can stack our odds of success for building an accurate GraphQL schema by involving domain experts in a process called Event Storming.

🖼️Domain-Driven GraphQL Schema Design (in 6 tweets)👇#graphQL #eventstorming #dddesign pic.twitter.com/5P2p0XzI2q
— Khalil Stemmler (@stemmlerjs) January 23, 2020

Let's talk about it.

Event Storming

By definition, Event Storming is:

A group or workshop-based modeling technique that brings stakeholders and developers together in order to understand the domain quickly.

It all started when a developer named Alberto Brandolini found himself short on time to organize a traditional UML use case design session with some clients. Thinking quickly, he improvised with some sticky notes, markers, and a whiteboard- inadvertently creating Event Storming.

Today, Event Storming has become something of a staple in the DDD community.

It's a quick (and fun!) interactive design process that engages both developers and business-folk to learn the business and create a shared understanding of the problem domain.

The result is either:

a) A big-picture understanding of the domain (less precise but still very useful).

b) 🔥A design-level understanding (more precise), which yields software artifacts (aggregates, commands, views, domain events) agreed on by both developers and domain experts that can be turned into rich domain layer code.

By running an Event Storming session, we end up answering a lot of questions and have all the pieces needed to construct a rich GraphQL schema.

Using an Event Storm to design a GraphQL Schema

You can use this approach in any context. If you've been asked to build a new app, or you want to start to introduce GraphQL on an existing project, you can take the following approach.

💡Step 1. Domain Events

Plot out all of the events that happen for the main story in your application as past-tense verbs.

For my Hackernews-like app, DDDForum.com, that looks like:

UserCreated => MemberCreated => PostCreated => CommentCreated, CommentUpvoted, etc.

Events drawn out chronologically on a timeline from left to right.

💡Step 2. Commands

For each Domain Event, write the Command that causes it. These are your GraphQL mutations.

They should be in an imperative form. If you know the name of the role/actor that performs it, you can document that as well.

Commands accompanying their respective Domain Event chronologically on a timeline from left to right.

If you know the actor, that can be documented as well. Here, we know that Members are the only ones that are allowed to issue the Create Post command.

💡Step 3. Aggregates (write models)

For each Command/Domain Event pair, identify the Aggregate (or write model) that they belong to.

In Event Storming, we normally place the Aggregate (write model) above the Command/Domain Event pair.

Commands accompanying their respective Domain Event chronologically on a timeline from left to right.

💡Step 4. Enforce boundaries & identify top-level fields

Identify the top-level GraphQL fields by applying Conway's Law. If you're building a Modular Monolith, these will be your subdomains. If you're working with Distributed Microservices, these will be your Federated GraphQL endpoints.

Segregate the Events, Commands, and Aggregate groups from each other based on the relevant self-organizing teams that play a part in the story of our application.

📖 This is probably the hardest and most misunderstood step. I'll create more documentation about this in the future. You may want to read about Conway's Law => (read here)

The story often flows between several subdomains/bounded contexts. For example, when a UserCreated event in the Users subdomain gets fired off, we subscribe and immediately issue a CreateMember command from the Forum subdomain.

💡Step 5. Identify Views / Queries

For each Aggregate, identify all possible Views (GraphQL queries) required in order to give users enough information to perform Commands (GraphQL mutations).

The Post aggregate has several different views.

💡6. Create the GraphQL Schema

Finally, create our GraphQL Schema from the discovered Commands (Mutations) and Queries.

In Modular Monoliths, you can break the schema up into separate files by extend-ing the Query and Mutation or by using graphql-modules.

infra/http/graphql/schemas/forum.ts

enum PostType { text link } type Post { slug: String title: String createdAt: DateTime memberPostedBy: Member numComments: Int points: Int text: String link: String type: PostType } type Member { memberId: String reputation: Int user: User } type PostCollectionResult { cursor: String! hasMore: Boolean! launches: [Launch]! } extend type Query { postById (id: ID!): Post postBySlug (slug: String!): Post popularPosts (pageSize: Int, after: String): PostCollectionResult recentPosts (pageSize: Int, after: String): PostCollectionResult memberById (id: String!): Member } extend type Mutation { createPost (input: CreatePostInput!): CreatePostPayload createMember (input: CreateMemberInput!): CreateMemberPayload } `

For Federated GraphQL architectures, you can compose schemas using the @key directive that comes from @apollo/federation.

Final thoughts

Modeling with events when CRUD doesn't fit

This is a controversial opinion, but one I feel strongly about arguing. I believe that most naturally occurring systems don't naturally fit within the rigid confines of CRUD.

Complex systems are everywhere we look.

By definition, Julian Shapiro says that "a system is anything with multiple parts that depend on each other."

I'm not just talking about software. Consider these real world systems.

Getting started at a new job: Applying to jobs, performing interviews, receiving offers, conducting negotiations, acceptances, paperwork, then (optional) relocation, onboarding/training.

Buying a condo: Finding a place, getting approved for credit (if you don't get approved, you might have a new prerequisite goal - improve credit, earn capital, or ask relatives), making an offer, accepting an offer, signing paperwork, then making payments.

Getting your driver's license: (Optionally) attending driving school, booking the test, passing the test (or failing the test and re-booking), getting your picture taken, getting your temporary license, then getting your license mailed to you.

As human beings, these systems don't seem that complex in the real world. The reason is because we tend to tame the complexity of them and how we progress through them by conceptualizing them as a series of events.

For example, before you accept a job, you need to have applied to jobs, performed an interview, and then gotten an offer.

See how those steps depend on each other? While I wish I could always just jump to the offer step, it's not possible.

Software systems often serve the singular purpose of making life easier for humans by consolidating the number of events that require human intervention.

Examples of consolidated systems
+

These are examples of systems consolidated by software.

LinkedIn consolidates the system of making professional connections.

Facebook consolidates the system of maintaining personal friendships.

Uber consolidates the system of on-demand based transportation like taxis.

How I Write Testable Code | Khalil's Simple Methodology

Mon, 20 Jan 2020 15:04:10 GMT

Understanding how to write testable code is one of the biggest frustrations I had when I finished school and started working at my first real-world job.

Today, while working on a chapter in solidbook.io, breaking down some code and picking apart everything wrong with it, I realized that several principles govern how I write code to be testable.

In this article, I want to present you with a straightforward methodology you can apply to both front-end and back-end code for how to write testable code.

Prerequisite readings

You may want to read the following pieces beforehand. 😇

Dependency Injection & Inversion Explained | Node.js w/ TypeScript

The Dependency Rule

The Stable Dependency Principle - SDP

Dependencies are relationships

You may already know this, but the first thing to understand is that when we import or even mention the name of another class, function, or variable from one class (let's call this the source class), whatever was mentioned becomes a dependency to the source class.

In the dependency inversion & injection article, we looked at an example of a UserController that needed access to a UserRepo to get all users.

controllers/userController.ts

import { UserRepo } from '../repos' // Bad /** * @class UserController * @desc Responsible for handling API requests for the * /user route. **/ class UserController { private userRepo: UserRepo; constructor () { this.userRepo = new UserRepo(); // Also bad. } async handleGetUsers (req, res): Promise<void> { const users = await this.userRepo.getUsers(); return res.status(200).json({ users }); } }

The problem with this approach was that when we do this, we create a hard source-code dependency.

The relationship looks like the following:

UserController relies directly on UserRepo.

This means that if we ever wanted to test UserController, we'd need to bring UserRepo along for the ride as well. The thing about UserRepo, though, is that it also brings a whole damn database connection with it as well. And that's no good.

If we need to spin up a database to run unit tests, that makes all our unit tests slow.

Ultimately, we can fix this by using dependency inversion, putting an abstraction between the two dependencies.

Abstractions that can invert the flow of dependencies are either interfaces or abstract classes.

Using an interface to implement Dependency Inversion.

This works by placing an abstraction (interface or abstract class) in between the dependency you want to import and the source class. The source class imports the abstraction, and remains testable because we can pass in anything that has adhered to the contract of the abstraction, even if it's a mock object.

controllers/userController.ts

import { IUserRepo } from '../repos' // Good! Refering to the abstraction. /** * @class UserController * @desc Responsible for handling API requests for the * /user route. **/ class UserController { private userRepo: IUserRepo; // abstraction here constructor (userRepo: IUserRepo) { // and here this.userRepo = userRepo; } async handleGetUsers (req, res): Promise<void> { const users = await this.userRepo.getUsers(); return res.status(200).json({ users }); } }

In our scenario with UserController, it now refers to an IUserRepo interface (which costs nothing) rather than referring to the potentially heavy UserRepo that carries a db connection with it everywhere it goes.

If we wish to test the controller, we can satisfy the UserController's need for an IUserRepo by substituting our db-backed UserRepo for an in-memory implementation. We can create one like this:

class InMemoryMockUserRepo implements IUserRepo { ... // implement methods and properties }

The methodology

Here's my thought process for keeping code testable. It all starts when you want to create a relationship from one class to another.

Start: You want to import or mention the name of a class from another file.

Question: do you care about being able to write tests against the source class in the future?

If no, go ahead and import whatever it is because it doesn't matter.

If yes, consider the following restrictions. You may depend on the class only if it is at least one of these:

The dependency is an abstraction (interface or abstract class).

The dependency is from the same layer or an inner layer (see The Dependency Rule).

It is a stable dependency.

If at least one of these conditions passes, import the dependency- otherwise, don't.

Importing the dependency introduces the possibility that it will be hard to test that component in the future.

Again, you can fix scenarios where the dependency breaks one of those rules by using Dependency Inversion.

Front-end example (React w/ TypeScript)

What about front-end development?

The same rules apply!

Take this React component (pre-hooks) involving a container component (inner layer concern) that depends on a ProfileService (outer layer - infra).

containers/ProfileContainer.tsx

import * as React from 'react' import { ProfileService } from './services'; // hard source-code dependencyimport { IProfileData } from './models' // stable dependency interface ProfileContainerProps {} interface ProfileContainerState { profileData: IProfileData | {}; } export class ProfileContainer extends React.Component< ProfileContainerProps, ProfileContainerState > { private profileService: ProfileService; constructor (props: ProfileContainerProps) { super(props); this.state = { profileData: {} } this.profileService = new ProfileService(); // Bad. } async componentDidMount () { try { const profileData: IProfileData = await this.profileService.getProfile(); this.setState({ ...this.state, profileData }) } catch (err) { alert("Ooops") } } render () { return ( <div>Im a profile container</div> ) } }

If ProfileService is something that makes network calls to a RESTful API, there's no way for us to test ProfileContainer and prevent it from making real API calls.

We can fix this by doing two things:

1. Putting an interface in between the ProfileService and ProfileContainer

First, we create the abstraction and then ensure that ProfileService implements it.

services/index.tsx

import { IProfileData } from "../models"; // Create an abstraction export interface IProfileService { getProfile: () => Promise<IProfileData>;} // Implement the abstraction export class ProfileService implements IProfileService { async getProfile(): Promise<IProfileData> { ... } }

An abstraction for ProfileService in the form of an interface.

Then we update ProfileContainer to rely on the abstraction instead.

containers/ProfileContainer.tsx

import * as React from 'react' import { ProfileService, IProfileService } from './services'; // import interface import { IProfileData } from './models' interface ProfileContainerProps {} interface ProfileContainerState { profileData: IProfileData | {}; } export class ProfileContainer extends React.Component< ProfileContainerProps, ProfileContainerState > { private profileService: IProfileService; constructor (props: ProfileContainerProps) { super(props); this.state = { profileData: {} } this.profileService = new ProfileService(); // Still bad though } async componentDidMount () { try { const profileData: IProfileData = await this.profileService.getProfile(); this.setState({ ...this.state, profileData }) } catch (err) { alert("Ooops") } } render () { return ( <div>Im a profile container</div> ) } }

2. Compose a ProfileContainer with a HOC that contains a valid IProfileService.

Now we can create HOCs that use whatever kind of IProfileService we wish. It could be the one that connects to an API like what follows:

hocs/withProfileService.tsx

import React from "react"; import { ProfileService } from "../services"; interface withProfileServiceProps {} function withProfileService(WrappedComponent: any) { class HOC extends React.Component<withProfileServiceProps, any> { private profileService: ProfileService; constructor(props: withProfileServiceProps) { super(props); this.profileService = new ProfileService(); } render() { return ( <WrappedComponent profileService={this.profileService} {...this.props} /> ); } } return HOC; } export default withProfileService;

Or it could be a mock one that uses an in-memory profile service as well.

hocs/withMockProfileService.tsx

import * as React from "react"; import { MockProfileService } from "../services"; interface withProfileServiceProps {} function withProfileService(WrappedComponent: any) { class HOC extends React.Component<withProfileServiceProps, any> { private profileService: MockProfileService; constructor(props: withProfileServiceProps) { super(props); this.profileService = new MockProfileService(); } render() { return ( <WrappedComponent profileService={this.profileService} {...this.props} /> ); } } return HOC; } export default withProfileService;

For our ProfileContainer to utilize the IProfileService from an HOC, it has to expect to receive an IProfileService as a prop within ProfileContainer rather than being added to the class as an attribute.

containers/ProfileContainer.tsx

import * as React from "react"; import { IProfileService } from "./services";import { IProfileData } from "./models"; interface ProfileContainerProps { profileService: IProfileService;} interface ProfileContainerState { profileData: IProfileData | {}; } export class ProfileContainer extends React.Component< ProfileContainerProps, ProfileContainerState > { constructor(props: ProfileContainerProps) { super(props); this.state = { profileData: {} }; } async componentDidMount() { try { const profileData: IProfileData = await this.props.profileService.getProfile(); this.setState({ ...this.state, profileData }); } catch (err) { alert("Ooops"); } } render() { return <div>Im a profile container</div> } }

Finally, we can compose our ProfileContainer with whichever HOC we want- the one containing the real service, or the one containing the fake service for testing.

import * as React from "react"; import { render } from "react-dom"; import withProfileService from "./hocs/withProfileService";import withMockProfileService from "./hocs/withMockProfileService";import { ProfileContainer } from "./containers/profileContainer"; // The real service const ProfileContainerWithService = withProfileService(ProfileContainer);// The mock service const ProfileContainerWithMockService = withMockProfileService(ProfileContainer); class App extends React.Component<{}, IState> { public render() { return ( <div> <ProfileContainerWithService /> </div> ); } } render(<App />, document.getElementById("root"));

GraphQL's Greatest Architectural Advantages

Sat, 18 Jan 2020 22:04:10 GMT

Readers that frequent this blog know that I often look for answers to the hard questions in software design and architecture 🧐.

Over the past few years, we've seen companies of all shapes and sizes like Expedia, Nerdwallet, and AirBnb incrementally adopt GraphQL throughout their organizations.

In this article, we'll discuss specific architectural advantages to using GraphQL in your next project or adopting it for an existing one.

We'll talk about:

Where GraphQL belongs in the modern web application architecture.

Why infrastructure code shouldn't be our focus on web development.

The Data Graph: the newest layer in the modern web application stack.

How the Data Graph unblocks frontend developers and removes the need for them to rely on backend developers.

How to scale and separate concerns with Apollo Federation.

How GraphQL and Apollo Federation eliminates the need to do API versioning.

The Hexagonal architecture

Alistair Cockburn's "Hexagonal architecture," says that the inner-most layer of our architecture holds the application and domain layer. Right outside of that layer are the adapters (or ports).

Think of ports as "a way to connect the outside world to the inside world." The outside world is full of technologies that we can use to build our application on top of. Outside, you'll find databases, external APIs, cloud services, and all kinds of other stuff. If we practice dependency inversion, we can safely involve them within our application by defining ports. Ports are abstractions. Contracts. They often appear in the form of interfaces or abstract classes.

Alistair Cockburn's "Hexagonal Architecture".

I believe pretty strongly in this type of architecture because it enables us to:

Delay the decision on exactly which type of web server, database, transactional email provider, or caching technology until it's absolutely necessary to decide. We can always use an in-memory implementation of the port for initial development efforts.

It also prioritizes writing code that can be tested by using Dependency Injection. This helps to minimize concrete dependencies that can make code untestable.

Lastly, it adjusts our focus to the application and domain-specific stuff. The stuff we can't just buy off the shelf or download.

Infrastructure Components

GraphQL servers and HTTP servers are examples of infrastructure components.

Infrastructure components: Fundemental components that comprise the foundation of a web application. Based on the clean (or hexagonal) architecture, databases, web servers, and caches are outer layer infrastructure components.

We call infrastructural components infrastructure because they are foundational to the application we build on top of it. It's within this foundation- this skeleton, that we write our rich, domain-specific applications. The infrastructure is simply the driver.

Another primary characteristic about infrastructure components is that they aren't the focus of our development efforts for a project.

Infrastructure components are tools trusted by the industry, and we only need to configure them to make them work.

Configuration of a GraphQL API involves:

Installing GraphQL

Exposing a server endpoint

Designing a schema

Connecting resolvers to data sources

This is work, but it's not the focus of our work on a project.

If a certain piece of infrastructure technology is trusted by the industry, there's a good chance that we can expedite our development efforts by letting that tool do its job instead of developing our own infrastructure component.

Consider databases. They're infrastructural too.

For example, a Postgres database is one of several databases we could use for a new project. Imagine the pushback you'd get from your team if you tried to convince them it was essential for the project to write your own database from scratch.

The scenario seems silly when we think about how it applies to choosing a persistence technology, but choosing your web application API style (transport/client-server technology) is a similar situation.

It was last year (2019) when I realized that APIs cut deeper into the stack than we think. APIs touch the edges of our front-end frameworks' data stores, and they touch the contracts (see Design by Contract) of our backend services.

If this sounds a bit virtual, that's because it kinda is. At Apollo GraphQL, we call that virtual layer the Data Graph. And Apollo builds tools that improves developer productivity working with it.

The Data Graph

I first heard the idea of a Data Graph from Matt DeBergalis, CTO of Apollo GraphQL, at GraphQL Summit 2019.

I highly recommend Matt's talk at GraphQL Summit 2019 where he introduces the concept of the Data Graph.

The Data Graph is virtual layer that sits in between our client-side application and a GraphQL server. It holds the entirety of an organization's data and provides the language for how to fetch and mutate state within the entire organization.

The Data Graph is a declarative, self-documenting, organization-wide GraphQL API.

To me, the Data Graph is a previously missing layer in the modern application stack.

Basic full stack Apollo Client + Server application stack.

The Data Graph brings the remote state closer to client-side local state

Three challenges all front end frameworks need to solve are data storage, change detection, and data flow.

React developers typically need to patch on Redux or Context, and write lots of boilerplate code to satisfy these requirements.

When Apollo released Apollo Client with apollo-link-state, they enabled React developers to meet all three needs satisfied with a lot less code.

Apollo-link-state (now baked directly into Apollo Client 2 and 3), made it possible for developers to write queries that address both remote state and local state at the exact same time. Remote state (sitting on a server) feels much closer.

Take this query to fetch a particular dog by its breed, for example:

queries/dog.js

const GET_DOG = gql` query GetDogByBreed($breed: String!) { dog(breed: $breed) { images { url id isLiked @client # signal to resolve locally } } } `;

In this query primarily intended to fetch remote resources, we can use the @client directive to refer to properties we want to fetch from our local cache based on a client-side schema. Very cool that this can be done in the same request. Think about the amount of spread and Object.assign()s operations we normally do in a Redux environment.

Depiction of the the simplified data-fetching architecture where the view is any front-end framework - Source nerdwallet.

The Data Graph, with Apollo Server and Client on both ends of the connection, simplifies fetch logic, error logic, retry logic, pagination, caching, optimistic UI, and various other types of boilerplate data-plumbing code.

The Data Graph stretches from client to server and has an answer for the most common infrastructural problems when fetching data and mutating state in modern web applications.

In order communicate with a backend service through GraphQL, Apollo Client exposes several client-side methods that, when invoked, converts the operation into the appropriate API to cross the Data Graph.

On the Apollo Server end, those API calls pass control off to resolvers responsible for fetching data using ORMs, raw SQL, caches, other RESTful APIs or anything else you can think of. For mutations, a resolver can simply pass control to an application layer use case.

By making the use cases the focal point of your application, switching from REST to GraphQL (or supporting both) is a breeze.

GraphQL is self documenting

One bummer in mainting a RESTful API is keeping documentation up to date and adequate.

There are two things that can change for a RESTful API.

The route + method combination

The request shape + parameters

A route + method combination example is that it's quite easy for someone to move the operation that creates a user from POST /users to POST /users/new. An API change like this would likely go unnoticed, break all clients of the API, and be virtually impossible for the API client to detect what the combination was changed to.

A request shape + parameters example is one where the POST request to /users/new used to only require an email and a password, but now, it requires a username property as well. The only way for an API client to understand what to do to remedy the request would be to check the error response (hopefully, error messages are descriptive otherwise this too would be impossible).

If you consider introspection adequate documentation, then GraphQL is self-documenting and impossible for your API documentation to get out of sync.

Using GraphQL Playground, one can browse all of the capabilities of a GraphQL endpoint.

GraphQL Playground's GraphQL explorer can display all of the capabilities of a GraphQL endpoint due to the ability to execute introspection queries.

In REST-land, I've only seen APIs shipped with this amount of metadata built using Swagger. This is a very powerful feature that not only makes the code the single source of truth for documentation, but opens up the surface area of ways we can perform codegen to automate the creation of TypeScript types, client-side libraries, or service-to-service communication.

Because the GraphQL language is ubiquitous and standardized, humans and machines will have an easier time understanding how to integrate and work with it.

Scaling and Separation of Concerns

Principled GraphQL states that,

"Your company should have one unified graph, instead of multiple graphs created by each team."

The emergence of multiple graphs within a company is something that will start to happen as more and more teams catch on to GraphQL.

Using Apollo Federation, each service team can build and manage their own GraphQL service from their bounded context, register it to an Apollo Gateway, distributing GraphQL operations across the entire enterprise.

Apollo Federation lets us grow and expose a single Data Graph composed of several GraphQL endpoints.

In Federation, you can compose schemas and resolve fields from other services/bounded contexts. When a request comes in, those fields get resolved from the appropriate service 👍.

Solving this problem is not uncommon for any organization of substantial size.

A single endpoint

The Open-Closed Principle from the SOLID principles states that:

"A component/system/class should be open for extension, but closed for modification".

Architecturally, because GraphQL only exposes a single endpoint to the client, it satisfies this principle.

All of the complexities of how fields are being resolved are hidden from the client, and all it needs to focus on is how to build on top of the GraphQL server.

An illustration that depicts the evolution of an organization's Data Graph over time.

Empowers frontend developers

The Data Graph reduces front-end developers' reliance on backend developers to develop new endpoints for new use cases.

Small UI changes where we rip out components or realize that we've misjudged our data requirements and actually need a few more fields for components, happen quite often. Because this happens so often, and because REST is so rigid, it creates this dependency on the back-end team to make changes to the REST APIs any time we need adjustments.

Depending on the team structure, each of the following questions signal potential to reduce developer productivity and create a reliance on the backend team.

Is the team fragmented? Are front-end developers strictly front-end, or are they also allowed to work on the other side of the stack?

Are your backend developers remote?

Are your backend developers in the office?

GraphQL removes the need to manage API versions

Principled GraphQL also has a strong opinion on versioning. It states that:

"The schema should be built incrementally based on actual requirements and evolve smoothly over time."

This means that the team should practice agile schema development by iteratively making changes instead of chunking changes in huge releases.

That's great, and all, but I live in the real world just like you. I know that that's not always possible, at least not without the proper tooling.

The Apollo platform has a feature called schema validation that lets you test every change against live production traffic and shows you when breaking changes are proposed, enabling teams to converse on how to continue.

Neat!

Getting started: For more information on how to use Schema Validation, check out the docs, Evans Hauser's talk, and sign up for a free Teams trial.

Conclusion

Within the modern web application architecture, GraphQL and RESTful Web Servers are infrastructure components.

Infrastructure components are fundemental components that comprise the foundation of a domain-specific web application we code within them.

Infrastructure components aren't the focus of most web development projects, so we should aim to spend most of our time on application and domain layer code.

The Data Graph is a declarative, self-documenting, organization-wide GraphQL API, brings remote state closer to the client-side, and can be scaled using Apollo Federation.

Front-end developers are empowered to create their own data-fetching use cases without the reliance on backend developers by using the Data Graph.

GraphQL removes the need to manage API versions and Apollo's Graph Manager can faciliate production schema validation.

There are a lot of scenarios where GraphQL makes sense architecturally.

I'm still in the process of trying everything out for myself. I'll be updating this article periodically as I get my hands dirty, so check back in a little while.

No title

Wed, 08 Jan 2020 04:05:26 GMT

Every frontend framework has their own opinionated way to solve the same three problems.

1. Data storage

Defining the shape, changing, and accessing data from the store

Apollo Client + React Hooks

Using Apollo Client we can create a store to represent the local state as a client-side cache, fundamentally replacing the need to use Redux to maintain a client-side store.

Queries/mutations that need remote state can also be written in combination with remote fields to handle use cases where we add flags or booleans to data for local presentation.

queries/dog.js

const GET_DOG = gql` query GetDogByBreed($breed: String!) { dog(breed: $breed) { images { url id isLiked @client # signal to resolve locally } } } `;

It's pretty nifty.

Anything data not found in the client-side cache is fetched remotely and cached on its way in.

2. Change detection

Signaling to components that data has changed

3. Data flow

The direction that changes flow and responding to change

Decade in Review | 2010-2020

Thu, 02 Jan 2020 15:04:10 GMT

At the time of writing this, I've been on Earth for about ~2.4 decades. That means that ten years ago, I was the very tender age of fourteen. Remember fourteen? Remember all that confusion, boredom, and frustration (and maybe also rebellion)?

At fourteen, everything seems possible. Our dreams of becoming a rockstar, playing sold-out shows, traveling the world, and other big and grandiose things don't seem all that ridiculous. At fourteen, everyone's got their own time blanket to cover their eyes with to stay hidden from the real-world boogeyman: reality. Usually, at fourteen, we don't have to make any of those big decisions just yet.

Flash forward ten years, and I'm twenty-four.

I've made all the most significant decisions of my life so far including what I want to do for a living, who I want to spend my time with, where I want to live, and what I want to achieve in the next decade.

In ten years, I have to say that I'm really proud of how far I've come.

I've gone from high school to working in a factory, to running my own lawn care business, graduating from college and university, learning various instruments, recording 7 full-length albums, playing shows, working tech jobs, building a startup, creating products and blogging, and eventually landing my dream job.

In this post, I'd like to recollect some of the most profound, weird, and memorable moments over the last decade.

2010

I laughed almost every day of high school

Every single day of high school in Brantford, Ontario felt like it was the funniest day on Earth. Surrounded by cartoon characters, I met hilarious people like James, a weed-smoking compulsive that thought he needed to buy the friendship of his peers with chocolate bars stolen from his dad's convenience store. I also met people like Phil, a lanky stoner influenced by Marilyn Manson, who often attended class with a white contact in one eye. Hilarious.

I can barely remember a moment I wasn't laughing (and I probably bought myself several additional years of life with all of that laughter).

Favourite artists

Green Day (I could play all of their songs on drums; my favorite album was Insomniac)

Blink-182 (what? they were good)

Weezer (Pinkerton was the emo jam back then)

AFI (I still bop their first few punk records)

2011

I discovered my love for programming in Grade 11 Computer Science

Formerly a math teacher, Mr. Levac was suddenly tasked with teaching high school students the fundamentals of Computer Science.

Having gone to university for programming later, I thought he did a fantastic job. We learned about searching, sorting, arrays, hashtables, and all the basics of programming with Java. By the end of the class, I had programmed a two-player game of chess.

I was very fortunate to have the opportunity to take a programming class in high school.

This class was the time I ever experienced that empty mind feeling you get when you get into the flow of programming. I always felt the class was short, and that's likely because I was having so much fun with it.

I learned the value of backing up files in multiple places when I lost an album I worked an entire summer on recording

I spent an entire summer in my parents' basement working on a punk album using a crappy little Dell netbook, Audacity, and FL Studio.

After completing the record and rendering it all out to WAV and MP3s, I was devastated when my hard drive with all of the original project files on it- died.

Hard lesson learned that year.

If it doesn't exist in at least two places, it doesn't exist at all.

I still have the final exported audio files, but listening to the way it was mixed back then makes me cringe. If only I could go back in time...

This is what my setup looked like at the start of this decade.

Favourite artists

Nine Inch Nails

Queens of the Stone Age

2012

My parents wanted me to be a lawyer, so I did a co-op to test it out

In Ontario, we have to take a Careers class and complete a test that tells you what job you'd be well suited for. I seriously think we need to re-evaluate how those tests work because mine spit out that I should become a Bereavement Counselor 🙃. Hmm.

Anyway, even though I liked computers and coding, my parents really wanted me to try to get into law. After all, what parent doesn't want their kid to be a lawyer?

I figured why not, let's treat it as an experience, right? I'm so glad I found a co-op placement at the Brantford Superior Court of Justice through my school. Within two days, I realized how much it wasn't for me.

Periodically in between doing really boring filing work, I got time to sit in on custody battles, people in trouble for shoplifting, vandalism, theft, and all kinds of other creatively heinous crimes, but it totally wasn't the kind of thing I wanted to spend my time thinking about.

A young Khalil working at the Superior Court of Justice in Brantford with endless boxes of pages to find and replace in books.

Even though it was 4 months of boredom and the thing I enjoyed the most was helping troubleshoot intranet issues, I appreciated the experience because now I know for sure.

Favourite artists

Not going to tell you because it's embarrassing.

2013

I lost a $20,000 scholarship and couldn't find a part-time job, so I worked in a factory

I think the Schulich Scholarship is actually Canada's biggest scholarship (today, it's $100k CAD for STEM students). I was nominated for it!

It came down to two other students and me. Unfortunately, I didn't win it. Unfortunately for the person who won it, they ended up dropping out of school and giving back all the money after the first year to go into trades instead. Nonetheless, I figured things out and was able to fund my way into Brock University/Sheridan College's Computing and Internet Telecommunications Program.

Because Brantford's part-time job economy for high school students was pretty poor, the only job I was able to find was working as a temp in a soap factory.

That sucked. It felt like I barely saw daylight for months. Though I did get some pretty hilarious stories from it.

Started school at Brock University and met some of my best friends to this day

The main difference I've noticed between college and university is that in college, you end up doing a lot of busy work, but in university, the assignments that you get can take anywhere from an hour to 2 weeks to complete.

For example, in Mathematical Reasoning, we'd often get these single page assignments with one or two formulas to solve. I noticed that we ended up spending more time just trying to understand the problem than we did actually solving the problem.

And when I got to college, it was just a lot of work- not as theoretically challenging, but man I always had something keeping me busy.

That first week at Brock, getting my ass kicked in the Mathematical Reasoning class, I went to the math lab, rallied up a group of first years also struggling, and worked through those problems together. Several of those first years have become really good friends to me that I still keep in contact with to this day.

Working on Axion EP in my dorm.

Stuff I made

Before I left for university, me and two buddies recorded The Sandpaper Disco II, an electronic/dance album with a few good songs :p

In the Winter, I recorded Axion EP in my dorm room, a downtempo trip-hop/electronic record under the pseudonym of Cyanide Canaries.

Favourite artists

Massive Attack

Dead Kennedys

2014

I wrote my first piece of distributable software: a glitch art program

After taking COSC 1P02 and learning about basic data structures like Queues and Stacks, I was feeling like I learned some really cool stuff but wasn't sure how to apply it in the real world just yet.

I had this idea for a really long time that I wanted to figure out how to build my own glitch art program.

Glitch art is exactly what it sounds like. You take an image, mess it up by changing some of its digital representation, save it back, and then look at it.

It wasn't until second-year' sCOSC 2P03 - Advanced Data Structures that things finally clicked.

I remember staying up all night at my friend Kirk's house, getting the code on Glitchie to work.

The first program I ever built (that I was proud of).

You can download the jar and try it out on Github. I haven't tested it on Mac yet.

Ran a lawn-care business in order to raise more funds to pay for my next years of university

In the summer, I needed to raise some more money to head back for the second year of school.

I was really fed up with factory work, so I came up with the idea of running my own lawn-care business with my stepdad. We called it "Steps Lawn Care."

Honestly, we made a killing.

Since my stepdad had his own full-time job, he would only help me out in the evenings, so throughout the day, I would drive the truck with the trailer across Brantford, Paris, Hamilton cutting and rolling lawns. I was able to pull in $300-600 a day on average if it wasn't too hot outside.

Tip: You want to get up really early and get the big lawns out of the way when you've still got a little bit of that early morning chill goin' on. Pushing a lawnmower in 25-degree heat is straight-up death 💀.

Stuff I made

Really influenced by the trip-hop genre and artists like Massive Attack, Tricky, and Portishead, I recorded this warm trip-hop + surf-rock album called Separation Stories in between classes. My process was to record an entire surf-rock song, cut the guitar riffs up into little samples, slow 'em down, and then organize them into a downtempo track using a Novation Launchpad. I would record vocals, more guitars, and place other embellishments over the track in FL Studio.

Favourite artists

The Clash

Beastie Boys

2015

Moved to Ottawa and worked at IBM

My first co-op job through school was at IBM in Ottawa.

It was one of the longest and coldest Winters I've ever experienced. It felt like I was living in an extended version of The Girl With The Dragon Tattoo (the Daniel Craig one).

I was a Technical Analyst at IBM and accompanied by two of my brock buds, Imran and Sulman. Lots of fun times being thrown into the deep end, resolving customer service requests on technologies that we practically knew nothing about. To this day, I still can't tell you what a star schema is.

Quip of the decade: Once, I forgot to pack a lunch and decided that I would just spend my lunch break reading my copy of Kim Gordon's autobiography that I brought from home. I entered the elevevator with Sulman, Imran, and our team lead, Cam. Cam asked me where my lunch was, and I told him I forgot it. A couple seconds of silence passed. Before the doors to the elevator opened, Sulman murmured the words, "Readers digest". Two more seconds passed. Then, we all bursted out with uncontrollable laughter.

Freak punk shows

Ottawa and Montreal have thriving art scenes. One particular night after work, I walked across town in the blistering cold looking The Gabba Hey, an underground punk venue located in an industrial district of Ottawa.

Upon walking in, noticing punks decked out in studs, leather, and spandex, I was sure that I was going to be the only one attending this show in a full suit and tie getup. While that was technically correct, it was the virtual presence of Mr. Node and his cut-up nine-to-five suit and tie uniform that also occupied the space.

Pretention aside, before any bands played that night, we were shown a documentary about a Montreal freak punk band called Thee Nodes.

Mr. Node was a mental mummy businessman that performed with toilet paper wrapped around his head and had about as much to say as Jello Biafra, although in a less witty and cutting way.

Mr. Node himself alongside Marc The Narc performing at a house show. You can feel the heat and congestion from just looking at this photograph.

I don't know. Something happened here witnessing this. I've never seen music performed so passionately. It was probably more performance art than music. Art had been so important to me for such a long time, and on that day, something shifted in my head. I'm not sure how to explain it. Weird that I'd even mention it on a decade retrospective, but it's unexplainably important to me for some reason.

Further reading: I find these guys fascinating. If you're interested, check out this Vice article titled "Thee Nodes Went to the Future and Came Back Insane". If you're still intrigued, I just learned that the original documentary "Who Are You Mr. Node?" is finally on YouTube.

Started learning computer networking at Sheridan College in Oakville

The university program I chose was half computer science and programming, and half computer networking.

These first few weeks were kind of rough because I was joining a second-semester class that had already learned all the basics and were getting ready to apply real-world networking protocols like OSPF, RIP, MLPS, and IS-IS.

I got a couple of books off someone, busted my ass that first month, and somehow caught up. Because I'm more interested in programming and software design, I almost never need to utilize any of the core internet routing and networking knowledge I acquired at school, but I at least have an appreciation for just how complex the internet really is.

Did my first software development co-op and learned the hard way how important it is to test your code

My first actual software development job was as a Java Developer at a company in Cambridge that builds ERP Software.

The codebase was very bad.

My first assignment was to "take some SQL code, put it in this block of Java code, and run it." I was also told that I'd "know if it works if it doesn't return any rows."

Weird stuff.

The hardest lesson I ever learned was that I should never skip out on running tests and making sure my code works because once, I neglected to spend time doing any testing, and the code went out to the client completely busted. I was called in to speak to the manager that comes up with the requirements and given an incredibly stern and upsetting talking to.

Though there might have been a better way to go about doing that, I saw it from his point of view. Test your code, kids.

Stuff I made

At this time, I was mostly into noise rock and recorded an album called "Death to the Police Force." One of my better-sounding releases.

Favourite artists

Sonic Youth

Black Flag

2016

Started working on Univjobs

Back at Sheridan, I was introduced to a fellow named Charles, who was as interested in programming and entrepreneurship as I was. Charles met with the mayor of Mississauga and asked her what the biggest problems in Mississauga were. She said that the single biggest problem was brain leak, meaning that all of the smart STEM students were leaving Mississauga to find tech jobs in Toronto while there are plenty of opportunities in Mississauga that no one knew about.

Charles asked me to help him with his database schema and later asked me to help him develop the platform. Seeing how passionate he was about solving the problem, I decided to invest my time to work with him on what would become Univjobs.

The application was built using React.js, Redux, Node, and eventually refactored to TypeScript and deployed on AWS.

Regularly had nightmares about my computer networking professor

Bill Farkas is well-known for being one of the hardest professors you'll ever have. He created the Internet Telecommunications program in the 90s, took a bet on TCP/IP, and will most certainly make you learn. You will not graduate if he doesn't feel like you put in the effort.

The way I feel about it now is kind of the way that you feel about parents that were hard on you growing up- you know they were hard on you for your own good, but man, it really sucked.

Bill expected students to put in 60-70 hours per week and come prepared for lectures and seminars. I used to spend 15-20 hour days in the computer networking lab and often slept there in order to attend the class the next day. Unfortunately, that was the norm at the time. Any social life or balance I had before attending Bill's classes were completely killed by the time I started.

For me, the most positive outcome I got from taking Bill's classes was my improved ability to present technical information in a concise and approachable manner.

Bill would often draw out a network topology, cut a few lines, and then ask someone to come up and explain to the class how the network will converge.

The challenge is: if you provide any superfluous information, Bill will roast you in front of the class.

If you don't provide enough information, Bill will roast you in front of the class.

If you provide incorrect information, Bill will roast you in front of the class.

In fact, you have to work really hard to not get roasted in front of the class.

Either that or just accept the fact that you will get roasted in front of the class. Talk about facing your fears of public speaking.

A very valuable experience that did a lot more good than bad, though some couldn't take it, which made the graduating class pretty small.

A goofy-looking Khalil in 2016.

Stuff I made

My tastes had shifted more towards post-punk and popular music by this time. I recorded "Dig It Up," probably my most accessible music to date.

Favourite artists

The Birthday Party

Kleenex

Gang of Four

The Drums

2017

Finally freed of 18 hour days, began to build a social life again

Free from Bill, I felt like I could actually live life again. I started getting back into exercising, meeting new people and just having fun again.

This was the best summer I've ever had. I was able to turn a really sleepy commuter school community into everyday feeling like a party.

My minimal, brutalist workstation in Oakville, Ontario. Not by choice.

Univjobs makes it onto Blog Toronto & Toronto Guardian, and the traffic demolishes our tiny AWS t2.micro

I was sitting in class when all of a sudden, 600 new people signed up for Univjobs within about 3 minutes.

Looking at the traffic, we discovered that Univjobs had made it to the front page of Blog Toronto with an article titled "There's a new way for students to look for jobs in Toronto".

For sure, the tiny little EC2 instance died, and in came hundreds of emails from people not getting their VerifyAccount email.

This was the main event that turned me onto serverless architectures.

Guerilla marketing. We printed off all kinds of Univjobs leaflets and put them everywhere. It didn't take long before we got a cease and desist from the school though.

Stuff I made

While I was at Brock taking Robotics and Discrete Event Simulation, I wrote and recorded Debutante, a gutteral post-punk album.

Towards that end of the year, I did Candy Body. This has been the last bit of music I've formally released.

Favourite artists

The Wolfgang Press

The Fall

2018

Graduated from college/university

After 4 and half years of school, 3 co-op jobs, and 8 different apartments, I had finally finished college and university obtaining both my Bachelor's degree in Computer Science and a Diploma in Internet Communications Technology.

Focused on Univjobs and side-projects to pay rent

After school, I had to figure out if I wanted to pursue Univjobs or get a real job. For a while, I was able to pay my rent by working as an independent freelancer, but I really hated the aspect of trying to find work and land contracts rather than just focusing on doing good work, so I got a real job.

Univjobs television interview.

Started work at Dev6

Dev6 is a small front-end consultancy in Mississauga. I worked here for about 8 months and coded mostly in TypeScript on Angular projects. It was also here that I met Sean Hopen, a senior consultant who would introduce me to Uncle Bob, TDD, DDD, and clean coding.

Favourite artists

Cypress Hill

Sleaford Mods

Mac Demarco

Wu-Tang Clan

2019

Focused on software architecture and domain-driven design

2-ish years into working on Univjobs, I realized that we were having a lot of troubles pushing out new features because the codebase had just gotten so massive and disorganized.

I ended up spending over $500 in software design and architecture books, took a couple of months to read 'em, then refactored Univjobs to TypeScript with Domain-Driven Design.

A growing collection of software design books.

Started blogging @ khalilstemmler.com

During the time that I was reading these books and improving the terrible code that I had written, I realized that there are a lot of common pitfalls in building large-scale applications that the JavaScript community has neglected.

Half in order to solidify my learning, and half to give back and raise awareness to software craftsmanship in the JS community, I started writing about DDD, Enterprise Node.js / TypeScript, and software design at khalilstemmler.com.

Over the course of the year, traffic has grown to approximately 700 visitors per day, which I'm really proud of. I'm planning on keeping that momentum going in 2020 with topics about GraphQL, DDD, TypeScript, and Event Sourcing.

I was finally able to afford moving into an apartment with natural lighting. About time.

Univjobs gets government funding

While on paid vacation, my co-founder called me to tell me that Univjobs secured OCE funding from the Government of Canada.

I quit my work at Dev6 and focused on Univjobs full-time.

Unfortunately, it didn't go so well, particularly because we were building a two-sided marketplace. This left us in a chicken-and-egg scenario where we were left to figure out how to get students as well as how to get employers and jobs at the same time.

We came up with a good plan to build a Campus Reps referral program where every referral a campus rep got, they could make $1.

Testing it out with a few reps, we noticed that it was incredibly effective. In about 5 minutes of work, reps were able to get 80-100 signups and make around $60 bucks, easy.

Unfortunately, due to budget cuts, thanks to the Ford government, we never actually saw the OCE money that we were waiting on.

Started writing solidbook.io

On occasion, I would work out of an entrepreneurship hub in Mississauga. It was there that I met Eric Rafat, founder of FoundersBeta. Eric saw my website traffic and convinced me to take my writing more seriously and even attempt to build a product to sell as well.

That's when I got the idea for solidbook.io - The Software Design and Architecture Handbook. I wanted to consolidate everything that I thought every developer should know about software design and architecture into one resource instead of having it split across the 15 books I had to buy to learn it.

To my surprise, people started pre-ordering the book. Eventually, I started pulling in around $2/3K per month on book sales and was able to pay my rent without Univjobs.

Today, finishing solidbook.io is one of my primary goals of 2020.

Co-founder leaves Univjobs

We got so close to securing investment for Univjobs. We were able to convince a UK investor to fund our $500K seed round by explaining that we would use the money to grow the heck out of the student side with the Campus Rep software.

Unfortunately for the rest of the team, our CEO ghosted the entire team for weeks and neglected to answer phone calls.

About a week into the ghosting, the site went down entirely because of non-payment. Charles, our CEO, was the only one who had access to pay the AWS bill, but no one could reach him.

A week later, he called a team meeting to make an announcement. The announcement was that he was considering taking a job at Lyft late-2020 and didn't feel like doing Univjobs anymore.

It was quite a shame because we were right at the gates of funding. Although I feel like it was for the best because remote communication was poor, a lot of trust was broken, and morale on the team was very low.

Everyone went their separate ways, and by the end of it, our first startup company Univjobs grew to 5000 students and a couple hundred employers in about 2 and a half years.

Started at Apollo GraphQL

Around the same time that I was trying to decide whether to continue working on Univjobs, Peggy Rayzis discovered my blog through Tania Rascia's "Developer Blogs to Follow (2019)" list and pitched the idea of working as a Developer Advocate for Apollo GraphQL.

Dropping Univjobs and choosing Apollo was a no-brainer. In October, I flew to San Francisco on my own for the first to meet the team and attend GraphQL Summit 2019.

First time ever being to California. First time ever traveling to another country by myself.

Great times, lots of learning, lots of GraphQL, and lots of karaoke.

I'm really excited about 2020 and the opportunity to work with Apollo, work on cool projects, meet developers around the world, learn from others, and help people be successful with Apollo, GraphQL, TypeScript, and DDD.

It's realzies. Look out, 2020 :)

Favourite artists

Jonathan Fire Eater

Ghostface Killah

MF Doom

Part-Time

John Maus

People we've lost this decade

Mark E. Smith

Stewart Lupton

Rowland S. Howard

James Gandolfini

Stan Lee

Stephen Hillenburg

Why Event-Based Systems? | Enterprise Node.js + TypeScript

Fri, 27 Dec 2019 15:04:10 GMT

There are several ways to design systems. When I first starting building applications, I would start out with the database. Eventually, I realized that the view dictates the data, so I moved to starting out with the UI. Then I realized that I was failing to represent a lot of important business rules, so I began to search for something else.

Eventually I found Domain-Driven Design and starting building applications by first understanding the domain, then working upwards from the domain layer.

I then realized the utility of domain events and the fact that you can illustrate an entire business's operations on a timeline with domain events.

Designing systems from domain events instead of columns in a database, endpoints on an API, or components in a view, has so many benefits.

Here are just a few.

Communication

Thinking in terms of domain events is the closest we can get to expressing what happens in the real world. Features can be communicated by non-technical folk as a series of domain events.

UserRegistered -> EmailVerificationSent -> EmailVerified

Auditing & Reasoning

When we represent a system using events, we can reason about the way that the system changed over time since the current state is computed by applying all of the domain events that ever occurred. In some domains, this is mandatory for audits.

Estimates

Software can be a very creatively unbounded process. Find and count all the views (read models) that present data and commands (write models) that change the system and cause domain events, then measure their complexity.

The views and commands that a system needs in order to be complete are called "Use Cases" and adhere to the Command-Query Segregation design principle. Read "Better Software Design with Application Layer Use Cases | Enterprise Node.js + TypeScript" to learn more about how to design with use cases.

Scalability

Sometimes, in large applications- a single transaction might need to pass through or notify several other systems. In high-traffic scenarios, instead of blocking an async call, we introduce "Eventual Consistency" as a way to enable systems to "catch up" eventually.

Complexity

Business rules are often missed in CRUD-first design. In order to avoid unmaintainable if and else statements and anemic domain models, think in terms of events, the commands they sometimes invoke as side effects, and the preconditions that govern how and when they can be invoked.

Conclusion

Event-based architecture seems to be the common place to go when the problem we're solving with software gets really large.

Unfortunately, there's a bit of a learning curve.

If you're interested in learning event-based systems, here's what I think the general learning path looks like:

Get really good at one programming language to the point where it feels like breathing for you.

Understand what an anemic domain model looks like and if you have one.

Learn about Event Storming and how it can help you plan a project using events.

Learn Domain-Driven Design concepts like entities, value objects, aggregates, and domain events.

Learn about CQRS and why you would want to use it in your DDD project.

Learn about Event-Sourcing

Build an application using DDD, CQRS, and Event-Sourcing.

How Apollo REST Data Source Deduplicates and Caches API calls [Deep Dive]

Sat, 14 Dec 2019 15:04:10 GMT

Quite honestly, I don't know if I really enjoy writing optimization code. I'm not saying that I don't like optimized code- I'm saying that optimization code tends to be the last thing that I need to do, has the potential to degrade readability a little bit, and can be pretty challenging to implement. Not only that, I just don't get as excited at the opportunity to optimize something as I do when I get the opportunity to codify things relevant to the business through domain layer code and use cases.

But alas, it's something that needs to be done.

And that's what I think Apollo has done really well with their tooling. They've listened to the community and figured out the common tasks in hooking up a GraphQL server in production that aren't so fun (but totally necessary), and done a lot of that hard stuff for us so that we can start out with an optimized GraphQL server.

Here's what we'll learn by the end of this article:

Potential ways to resolve data for a GraphQL endpoint using RESTful APIs.

Why Apollo's REST Data Source library is a better choice than a wrapped HTTP client library.

How Request Deduplication works in REST Data Source.

How Resource Caching works in REST Data Source.

How you can use the apollo-server-caching library to implement your own caching logic.

Resolving data from a RESTful API

If you're thinking about running a Apollo GraphQL server, a common setup is to resolve data from an external REST API.

Going that route, you might have heard of the Apollo REST Data Source library.

Using it, you can write custom data-fetching methods in order to resolve data from your external RESTful services. The image below depicts a potential architecture.

Example architecture of a client application connected to a GraphQL server resolving data from 4 RESTful API services.

Going through the Apollo full-stack tutorial, the Apollo Docs recommend that we use the RESTDataSource library when trying to fetch data from an external REST API.

We can get this to work in the following 3 steps:

Subclassing the RESTDataSource.

Setting the BaseURL.

Writing data fetching methods that utilize the parent class's get, post, put, patch (and more) in order to fetch data from the external REST API.

A very simple example of a RESTDataSource subclass might look like this:

datasources/vinyl.ts

import { RESTDataSource } from 'apollo-datasource-rest' import { Vinyl } from '../models/vinyl' type VinylCollection = Vinyl[]; export class VinylAPI extends RESTDataSource { constructor() { super(); this.baseURL = 'https://api.whitelabel.com/v2/'; } async getAllVinyl (): Promise<VinylCollection> { // Utilize the `get` convenience method from the parent RESTDataSource class. const response = await this.get('vinyl'); return Array.isArray(response) // Map the vinyl objects from the external API response into the format // that we need for our schema. ? response.map((vinyl: any) => this.vinylReducer(vinyl)) : []; } // More data fetching code ... }

A common question becomes:

"What is the advantage of using the RESTDataSource library over using an abstracted HTTP client?"

Moreover, why do we need to use RESTDataSource? Couldn't we just use Axios and write our own get, post, patch, etc convenience methods if and when we feel like it?

datasources/vinyl.ts

// Using axios instead import axios from 'axios' import { Vinyl } from '../models/vinyl' type VinylCollection = Vinyl[]; export class VinylAPI extends RESTDataSource { constructor() { super(); this.baseURL = 'https://api.whitelabel.com/v2/'; } async getAllVinyl (): Promise<VinylCollection> { const response = await axios({ method: 'GET', url: `${this.baseURL}vinyl` }) return Array.isArray(response.data) ? response.data.map((vinyl: any) => this.vinylReducer(vinyl)) : []; } ... }

You could, but you'd be missing out on some stuff.

Today I learned that RESTDataSource comes with two awesome performance optimizations out of the box:

Request Deduplication and Resource Caching.

Request Deduplication

Request deduplication prevents double-invoking HTTP calls in rapid succession.

That's a common enough problem if your some types in your resolvers rely on the same underlying API call in order to resolve fields.

This chatty, rapid-fire behaviour is the type of thing that Facebook's DataLoader does really well with loading data from an ORM or an in-memory datastore, but isn't as effective when dealing with RESTful API calls that go out onto the network and stress your service endpoints.

Deduplicating the API requests themselves fixes that.

How Request Deduplication works in REST Data Source

The REST Data Source library uses a caching technique called memoization in order to cache the entire Promise invoked when we make a request to a resource. We hold onto the Promise for any request to an external RESTful resource in a Map<string, Promise<any>>.

Memoization: A specific form of caching where for an indeterministic (and potentially expensive) operation, we save the results and return it from memory in subsequent invocations. Check out this article on "How to Memoize Async Functions in JavaScript".

src/RESTDataSource.ts

export abstract class RESTDataSource<TContext = any> extends DataSource { ... memoizedResults = new Map<string, Promise<any>>(); }

When a GET request comes in, we construct a cache key for that request by looking at the URL:

export abstract class RESTDataSource<TContext = any> extends DataSource { ... protected cacheKeyFor(request: Request): string { return request.url; } ... }

That url is what we're going to use as the string key for the Map that holds our memoized requests.

So if we make a request to https://api.kittens/v2/all?sort_by=cuteness, that becomes our key.

Now here's the cool part.

Depending on if the request was a query-like HTTP verb like GET, we'll check to see if there's a memoized Promise<any> by looking it up using the cache key. If there's already a Promise<Request> in progress for that cache key, just return that Promise that we know will resolve to the same thing we're already asking for from a previous request. Otherwise, we'll perform the request and memoize it so that it can be deduplicated for subsequent requests.

However, if we issued a command-like request like DELETE or POST, let's remove that item from the memoization cache because the next time we retrieve it, we can expect it's value to be different.

src/RESTDataSource.ts

export abstract class RESTDataSource<TContext = any> extends DataSource { ... private async fetch<TResult>( init: RequestInit & { path: string; params?: URLSearchParamsInit; }, ): Promise<TResult> { ... // If the request is a QUERY, see if we can get the memoized // result to prevent invoking another API call for a resource that hasn't // yet changed. if (request.method === 'GET') { let promise = this.memoizedResults.get(cacheKey); if (promise) return promise; // Didn't find it, just invoke the request normally and then save // the request to our memoization cache. promise = performRequest(); this.memoizedResults.set(cacheKey, promise); return promise; } else { // If the request is a COMMAND, delete the item from our memoization // cache (because we're about to issue a side-effect) and then perform the request. this.memoizedResults.delete(cacheKey); return performRequest(); } }

To illustrate that this works corrrectly, here's a test case straight from the Apollo REST Data Source library.

src/__tests__/RESTDataSource.test.ts

describe('memoization', () => { it('deduplicates requests with the same cache key', async () => { const dataSource = new class extends RESTDataSource { baseURL = 'https://api.example.com'; getFoo(a: number) { return this.get('foo', { a }); } }(); dataSource.httpCache = httpCache; fetch.mockJSONResponseOnce(); await Promise.all([dataSource.getFoo(1), dataSource.getFoo(1)]); expect(fetch.mock.calls.length).toEqual(1); expect(fetch.mock.calls[0][0].url).toEqual( 'https://api.example.com/foo?a=1', ); }); });

If you fired the same request of one that was already in flight but not resolved yet you’d only generate one Promise (and one fetch)!

Resource Caching

As for the actual responsibility of caching in REST Data Source, that falls into the lap of an Apollo-implemented Apollo Server Cache called HTTPClient.

Resource cache: The more traditional understanding of caching. We save a fetched resource in a storage medium more efficient at retrievals than the initial mechanism we used to retrieve it. For example, fetching an object from memory is faster than reaching into a database (disk) to fetch it. Also, fetching an object from memory is also often faster than making a network call to an external resource.

How it works

When RESTDataSource invokes performRequest, it uses a class called HTTPCache which is composed with a KeyValueCache.

KeyValueCache is a powerful interface that Apollo Server uses internally to provide a caching store for the Data Sources.

In the following code (that powers RESTDataSource), Apollo uses an InMemoryLRUCache, which is a valid subtype of the KeyValueCache.

src/HTTPCache.ts

export class HTTPCache { private keyValueCache: KeyValueCache; private httpFetch: typeof fetch; constructor( // You can pass in your own cache, but Apollo will use an // InMemoryLRUCache by default keyValueCache: KeyValueCache = new InMemoryLRUCache(), httpFetch: typeof fetch = fetch, ) { this.keyValueCache = new PrefixingKeyValueCache( keyValueCache, 'httpcache:', ); this.httpFetch = httpFetch; } ... }

When a request comes in, we run similar logic to that of which we ran in our memoization cache, but this time, we're in control of if the request goes out onto the wire or not.

src/HTTPCache.ts

export class HTTPCache { ... async fetch( request: Request, options: { cacheKey?: string; cacheOptions?: | CacheOptions | ((response: Response, request: Request) => CacheOptions | undefined); } = {}, ): Promise<Response> { /** * 1. Create the cache key. You can either supply a cache key or leave it blank * and Apollo will use the URL of the request as the key. */ const cacheKey = options.cacheKey ? options.cacheKey : request.url; /** * 2. Using that key, see if the cache has the value already. */ const entry = await this.keyValueCache.get(cacheKey); /** * 3. If it doesn't already have the response, we'll need to * get it, store the response in the cache, and return the * response. */ if (!entry) { const response = await this.httpFetch(request); const policy = new CachePolicy( policyRequestFrom(request), policyResponseFrom(response), ); return this.storeResponseAndReturnClone( response, request, policy, cacheKey, options.cacheOptions, ); } /** * 4. Returns the object from the cache (respecting any * cache invalidation policies). */ ... } }

You can read this file and peruse around the rest of the codebase for the Apollo REST Data Source library here on GitHub.

Out of the box, this thing does a LOT. But if you want to roll your own cache, or if you need to hook several GraphQL endpoints up to Redis or Memcached for a single source of truth, that's totally possible as well.

Making your own cache

For REST Data Source, Apollo ships with it's own in-memory caching strategy called InMemoryLRUCache. While this is what HTTPCache that REST Data Source relies on, there's nothing stopping you from writing your own cache strategy.

In order for Apollo to utilize your cache, all you need to do is write a class that implements the KeyValueCache interface and you're good to go.

export interface KeyValueCache { get(key: string): Promise<string | undefined>; set(key: string, value: string, options?: { ttl?: number }): Promise<void>; }

You can get started implementing your own cache using the apollo-server-caching npm library.

Summary

Apollo REST Data Source comes equipped with Request Deduplication and a Resource Cache

Request Deduplication prevents multiple requests to the same resource from getting invoked using memoization.

The Resource Cache prevents multiple requests to the same resource from getting sent out onto the wire.

You can implement your own Resource Cache using the apollo-server-caching library.

Thanks to Trevor Sheer for helping me understand the responsibility division between RESTDataSource and HTTPCache.

No title

Mon, 02 Dec 2019 04:05:26 GMT

If what we ship to our server is a single executable file or script that runs our entire application in one process, we have a monolith on our hands.

A monolithic application containing modules from A to F.

If for some reason, our application relies on another service that contains critical features or business logic for our core domain, it's no longer a monolith. It may be a distributed monolith.

An application that relies on a business-critical module running in a separate process is no longer a monolith.

When to migrate to microservices

A monolith isn't always a bad thing. In fact, I would advocate for starting out with a monolith on most projects.

Factors that can influence needing to break a monolith into microservices are:

Growing team size.

Clear understanding of the subdomains within the code and how they can be assigned to teams.

Horizonal scaling required for select high-traffic or expensive to operate features in the application.

Difference between a monorepo and a monolith

A monorepo is a code repository that we use to store several related repositories for a project.

A monorepo can become a monolith if when we run an application contained from within a monorepo, it has access to all the components needed in order for it to operate, and all those components are self-contained at runtime.

If we needed to run several processes from within the monorepo in order for the application to work, it's no longer a monolith.

Benefits

Ease of local development.

Ease of deployment.

Disadvantages

Single point of failure.

Checking in code can turn into a nightmare if the team size is large. Code conflicts might occur more frequently.

For some applications, it doesn't scale well.

Books that Developers Should Read in 2020 [List]

Fri, 22 Nov 2019 15:04:10 GMT

Whether you’re a new developer or you’re fairly experienced as a programmer, you’ll come to realize that the amount of time you’ve worked at a job isn’t the best way to determine your skill and knowledge as a programmer (I know, tell that to the recruiters 🤫).

What you do in your spare time and how you choose to take learning into your own hands is what’s going to ultimately determine your success in this industry. That’s why it’s so important for us as developers to adopt a growth mindset.

There are some excellent ways to learn and improve as a developer. Some of those ways are pair-programming, online courses, meetups, work experience, building projects and finding a mentor.

One of my personal favorite ways to learn is to crack open a well-written book and try to absorb something from those who have distilled years of knowledge and insight into a permanent artifact.

Here are my personal recommendations that I think all developers (especially junior ones) should read at some point. These books are all highly regarded by professionals in our industry and have the potential to leave a profound impact on the quality of your work and your speed of development & learning.

Some of them stray from the technical details and focus more on giving you practical rules about what it means to be a good developer on the interpersonal and professional level.

1. Clean Code

by Robert C. Martin (Uncle Bob)

After you overcome the basic challenges of development and get comfortable figuring out how to write code to solve problems, it’d be a good idea to take a look at this book. It turns out that making the code work the first time is actually the easy part. The hard part is making your code read well so that others are able to understand it and change it in the future. Remember the last time you had to read code like this?

function calculateIt (a, b) { if (a.delta < b.element.x) { var x = b.element.x; return x - b.delta.x } else { var y = b.next.y; var h = b.element.y * 2; return y - h } }

Who knows what it really does. Code like this might work, but the moment we need to change it, we have to hope the author of the code is still in the company and pray that they are somehow able to decipher what they wrote potentially years ago. When careful attention isn’t taken to write code that’s readable and maintainable, we end up with pockets of code like this that everyone is afraid to touch, and if it ever breaks, we’re in trouble.

Uncle Bob’s “Clean Code” teaches you how to identify what clean code looks like compared to bad code, and it teaches you how to transform it into good code. A task like this sounds trivial to most, but you’d be surprised at how turning a just a few clean software design principles into habits can help you write much more professional and scalable code.

We’re software craftspeople, you know. Building a house is not much different than building an application in principle. We need to pay attention to the details or else it can all become very expensive to fix in the future if not done right first time.

2. The Clean Coder

by Robert C. Martin (Uncle Bob)

This book is not necessarily a technical book as it is a book for teaching you how to be a professional in this industry. Professionals are those who, when faced with challenges, uncertainty and pressure, will continue to treat creating software as a craft and will be determined to adhere to their professional values.

The Clean Coder is full of practical advice on estimation, refactoring, testing, dealing with conflict, schedules, avoiding burnout, and much more. Trusted advice from someone who has spent decades doing this stuff.

One of the best things it teaches, is how to have integrity as a developer, when to say “No” and how to say it.

A book about professionalism.

3. Refactoring

by Martin Fowler

Martin Fowler is one of my favorite authors. The first reason is that he’s hilarious. His approach to writing software books is unmistakably “Fowler”. The other reason is that he’s incredibly good at explaining complex topics, and doing so very simply, in a way that doesn’t fatigue you as a reader.

Refactoring is a book that the creator of Ruby on Rails once said that you should “read before you write another line of code”. Fowler guides you through refactoring a simple application, introducing you to a number of techniques that he’s accumulated and cataloged over his years of consulting.

Fowler shows you how to flip between coding and refactoring, how often you should be committing your code and when you should be writing your tests. Highly recommended. The latest version of this book was updated to present the examples in JavaScript, which was an added plus for me since it’s my favorite language.

Half of this book serves as a reference guide, which you may or may not like- but I think it's important to be aware of how one piece of code can always be refactored to better serve the needs of its users tomorrow.

4. Design Patterns: Elements of Reusable Object-Oriented Software

by Erich Gamma, Richard Helm, Ralph Johnson, & John Vlissides

This is the seminal book on Design Patterns. What are design patterns, you ask? Design Patterns are well-known solutions to commonly occurring problems in software development. If you’re familiar with the patterns, you’ll find that you’ll be able to greatly reduce the amount of time it takes you to put forth the solutions to those problems.

Having a good awareness of design patterns also allows you to communicate your solutions effectively with other developers.

“Yeah, I just used a Facade overtop of whichever database Adapter gets loaded from the Strategy.” “Ahh! Gotcha.”

Yeah, it’s an older book. But it’s still one of the best for reference. If you’d like something on this topic that’s a bit more recent and friendly, I’d also recommend the good “Head First Design Patterns: A Brain-Friendly Guide” by Eric Freeman and friends.

5. Domain-Driven Design: Tackling Complexity in the Heart of Software

by Eric Evans

In order for large code bases to continue to scale, we need to logically split up code into different parts. The idea is to partition your code in a way such that it would be possible for separate teams to work on those parts of your system without affecting anyone else.

The underlying concept that enables moving your code base in this direction is Domain-Driven Design (DDD). It’s an approach to software development where we model the problems that exist in the “problem domain” (the real world) to a number of solution domains.

DDD is incredibly important when a code base gets sufficiently large. Large enterprise companies employ DDD in order to assign teams to parts of the company’s code base.

Diagram showing when an anemic model becomes cumbersome.

Eric Evan’s coined the term “Ubiquitous Language”, which is the term for building a common, all-encompassing language between the developers, the domain experts and any other users or actors in the domain. By using this Ubiquitous Language, it ensures that the most important domain concepts are well understood and get modeled in the software.

The book is a little more technical and challenging than the others, but if you get familiar with these concepts, you’ll be very well off in understanding how today’s largest companies keep their code bases manageable and scalable.

The ideas presented are essential readings before wandering into microservices and all the challenges it will present to you.

Alternative recommendation

I’ve thought about this recommendation a little bit. Introduction to the DDD world is, in my opinion, extremely beneficial for Junior Developers. I believe this to be true because DDD introduces builds familiarity with software architecture, design principles and design patterns. It’s a great way to practically introduce yourself to a higher level of programming.

That said, DDD is a large and challenging topic. For some readers, this book (the seminal “blue book”) by Eric Evans might be better treated as a reference. It was Eric Evans who wrote the first book on DDD.

However, I also read "DDD Distilled" by Vaughn Vernon. It's good.

It’s a really short and sweet intro to DDD, written in order to address the fact that most books on DDD are huge in size.

Definitely try "DDD Distilled" instead. I think this book would be a lot better for most developers in order to get ramped up on the DDD essentials first. For more practical details on how to implement the concepts, refer back to “the blue book” and “the red book”.

7. Clean Architecture

by Robert C. Martin (Uncle Bob)

What? Uncle Bob writes good books, ok?

In school, there’s a lot of focus on algorithms and less focus on software design principles. I think it’s kind of unfortunate because in reality, you don’t encounter that many algorithm challenges too often. Instead, it’s more common that you’ll be faced with the challenge of structuring your code in a way that’s modular, flexible, readable and will allow you to add new features quickly when requirements change. Clean Architecture is about the essential software design principles and patterns that you’ll be able to use in order to face these challenges.

Some of the best takeaways from this book are the cost of dependencies, stable vs. non-stable code and the SOLID principles: a way to write code so that it’s more understandable, flexible and maintainable.

Other aspects of this book that were incredibly useful were concepts of “screaming architecture” and “packaging by component” which are opinions on how to organize your modules so that it practically screams to the reader exactly what the project is all about.

This book goes well hand-in-hand with Domain-Driven Design, which is enabled through the use of a “Layered Architecture” or as Uncle Bob calls it, “The Clean Architecture” (also known as Ports and Adapters). A great book for anyone who wants to up their architecture chops and learn how to effectively design a system at a high level, and do the dance of dependencies at the detail level.

8. The Effective Engineer

by Edmond Lau

Time is our single most valuable asset in life, and we should aim to be more efficient with it. It’s easy to get bogged down and spend a lot of time fixing bugs and wasting effort. Doing repeated work. Bleh. The Effective Engineer is all about getting more done in less time and removing repeated work.

We can mitigate wasted time and effort on repetitive tasks through a framework called “leverage”.

Leverage helps you identify the activities that you can do that produce the most disproportionate results- per unit of time invested. It’s a framework that can apply to anything, whether that be how you learn, how you code, how you debug… anything!

9. The Pragmatic Programmer

by Andrew Hunt & David Thomas

Praised for being easy to follow and understand, The Pragmatic Programmer is a book that should be on the desktop of developers of all levels. Andrew and David are programmers that not only spent many years doing what they do, but paying attention to what they were doing as they were doing it, and then trying to determine if they could do that better.

What came out of their years of introspection was this book, which introduces a number of essential programmer philosophies to follow throughout your career, like “programmers should have a “do it once, or automate” philosophy”.

It includes simple yet detailed advice that you should carry with you in the back of your mind before you write another line of code or start a new project.

Final Words

Books really are some of the best tools to improve your knowledge and skills as a new programmer or Junior Developer. Books tend to have a really high return on investment; did you know you can make a lot of money programming? 😉

These are just a few of the best books out there right now in 2019! None of them are really new, but that’s because programming has maintained the same general philosophies and best practices for years. As a professor I once had to say, “you can make a lot of money in this industry, you just have to read the damn manual”.

Have you read any of these books? What did you think? Any books not on this list that you think newer developers would really benefit from reading? Let us know in the comments!

Additional Resources

Here’s a list of some really excellent articles that cover some of the topics from these books. If you don’t quite have the time to invest in fully blown books right now, familiarizing yourself with these concepts might assist you in your journey to become a better developer!

Architecture & software design

Refactoring.guru https://refactoring.guru/

SOLID Design Principles https://stackify.com/solid-design-principles/

NodeJS and Good Practices https://blog.codeminer42.com/nodejs-and-good-practices-354e7d76362

Implementing the SOLID and the onion architecture in Node.js https://dev.to/remojansen/implementing-the-onion-architecture-in-nodejs-with-typescript-and-inversifyjs-10ad

Better Software Design with Clean Architecture https://fullstackmark.com/post/11/better-software-design-with-clean-architecture

The Clean Architecture http://blog.cleancoder.com/uncle-bob/2012/08/13/the-clean-architecture.html

Web Development

CSS Visual Dictionary

Make Illegal States Unrepresentable! - Domain-Driven Design w/ TypeScript

Thu, 21 Nov 2019 23:04:10 GMT

When we first learn how to code, we're typically most focused on making things work. We visualize the finished product as a coming together of several technologies. A common MVC-mental model for a full-stack web app is: database + React app + Express API.

This works really well for a long time in CRUD and MVC-based applications without lots of business logic because we can (usually) think about them imperatively.

"This string maps to this column in the database. I'll just do some validation in the service to make sure it's not longer than 30 characters".

This approach to structuring business logic is called a "Transaction Script" and it's the most basic one. It's defined by the absense of a domain model and all business rules being represented with simple control statements like if or switch.

Transaction Scripts

Here's an example of creating a user using the Transaction Script approach.

UserService.ts

export class UserService { createUser (email: string, firstName: string, lastName: string): Promise<void> { const UserModel = this.models.User; const isEmailValid = TextUtils.validateEmail(email); const isFirstNameValid = TextUtils.greaterThan(1, firstName) && TextUtils.lessThan(31, firstName); const isLastNameValid = TextUtils.greaterThan(1, lastName) && TextUtils.lessThan(71, lastName); if (!isEmailValid) { throw new Error("Email isn't valid"); } if (!isFirstNameValid) { throw new Error("First name isn't valid"); } if (!isLastNameValid) { throw new Error("Last name isn't valid"); } const user: User = new User({ email, firstName, lastName }); await UserModel.create(user); return; } }

What's wrong with this code?

You know, it's not that bad. It works. But there are a couple of important things here being ignored.

Errors aren't given the love that they deserve. We're making it hard for the calling code to be able to handle the errors possibly thrown by this block.

We're not really solving this problem, we're just putting it off for someone else to figure out how to decipher the errors later. It would be a lot easier for the next person if the errors were treated as types just like everything else in our domain.

There are important business rules here (constraints) that can easily be circumvented.

In order to create a User, we need a valid email, a firstName from 2-30 characters, and a lastName from 2-70 characters.

What's stopping a new developer joining our team from just doing:

const user: User = new User({ email: 'bob', firstName: 'b', lastName: 'c' })

...completely circumenting our efforts defining business logic (validation logic is a form of business logic).

Nothing 😨.

There's another problem too.

Duplicating business rules

It's nice that we're at least trying to prevent against violations to the upper and lower bound lengths for the strings, but what happens when we create the updateUser method in our UserService? How much of that constraint logic do we have to duplicate?

All of it.

And what if the business rules changed a little bit?

What if, in order to create a user, you need to have either an email + firstName OR firstName and lastName?

That would make the User creation logic a little bit more complex, and we'd have to repeat it anywhere that we create users in our code.

That's a textbook anemic domain model. Not usually fun for web development (though, see Entity Component Systems for times it is fun).

This sucks. What should we do?

Make illegal states unrepresentable

Earlier this year, functional programming fanatic John A De Goes blasted out a tweet:

Making illegal states unrepresentable is all about statically proving that all runtime values (without exception) correspond to valid objects in the business domain. The effect of this technique on eliminating meaningless runtime states is astounding and cannot be overstated.

— John A De Goes (@jdegoes) January 28, 2019

This is an awesome quote. In programming (and science), it's often impossible to prove that something is correct, but it's reasonable to prove that something is correct enough.

With types, we can achieve that correct enough, and that makes our code much easier to reason about. And that gives us confidence to know that our business rules are being respected.

This builds trust. This means potentially less unit tests against the business logic, and more tests with various inputs to the system. This makes it safer to refactor code. This enables us to continue to push forward the design. This means that the types serve as a form of documentation to the system. This reduces the amount of supplementary documentation necessary. This enables new developers to onboard faster. This makes us happier.

This also happens to be the core of what Scott Wlaschin teaches with his book, blog, and talks on functional domain modeling.

Let's recap some approaches that we've already covered in detail on the blog, and look at a few alternative approaches as well.

Wrapping primitives with domain specific types to express a common language and prevent misuse

The essence of DDD is to build a shared understanding of the domain with stakeholders, non-technical folk, and technical folk, and then codify that.

Using types, non-techncial stakeholders can understand our code.

Like this.

forum/domain/types.ts

// UserName is from a separate subdomain, `Users` import { UserName } from '../../users/domain/types' type MemberId = string; type Member = { id: MemberId, name: UserName } type Text = string; type Link = string; type PostTitle = string; type PostId = string; type Post = { postId: PostId postedBy: Member, title: PostTitle, content: Text | Link, } type Upvote = { postId: PostId, memberId: MemberId } type Downvote = { postId: PostId, memberId: MemberId }

You probably have a good idea about the type of domain that we're dealing with simply by looking at the names of the types and their relationships.

Using classes (because of its nominal typing) yields a benefit (that I haven't yet been able to get working with purely functional types in TypeScript):

Forcing strictly nominal type checks (based on the name)

For example, even though PostId and PostTitle might just be wrappers to a string value, like this:

interface PostIdProps { id: string; } export class PostId { // Value is protected private props: PostIdProps; // There is only a getter, because it should never be able to change get value (): string { return this.props.id } } interface PostTitleProps { title: string; } export class PostTitle { // Also protected private props: PostTitleProps; // Also only needs a getter get value (): string { return this.props.title; } }

We can't just substitute them in for each other.

export class PostSlug { // We need a PostTitle to create a PostSlug public static create (postTitle: PostTitle): Result<PostSlug> { ... } } let postId: PostId = PostId.create(); let postSlug: PostSlug = PostSlug.create(postId); // Error

We can also think of this as a way to encode business requirements. If the only way to create a PostSlug is to first create a PostTitle, then we're forced to first locate the factory that creates PostTitles.

Think of the amount of illegal states we prevented by wrapping primitives in domain-specific types.

Using public setters sparingly

It's easy to add getters and setters for all of the properties of a class, but should we?

Consider modeling a User entity with a public UserId.

export class User extends Entity<UserProps> { get userId () : UserId { return this.props.userId; } ... }

Should we also add a setter?

That depends on if doing so is valid to the domain.

From a Users, Identity & Access Management context, there's no reason why the UserId should ever change to a new value. Doing so would break the relationships between User and any other subdomains that have a 1-to-1 relationship with User like Member from the Forum subdomain in DDDForum.com.

So lets make it entirely impossible.

export class User extends Entity<UserProps> { get userId () : UserId { return this.props.userId; } // set userId (userId: UserId) { // this.props.userId = userId; // } }

I might even discourage the use of public setters in general because of their rigid method signature. They clash with the way that we handle updates in aggregates using DDD using Results.

Strictly typing "null"s

What should you do in scenarios when you "try to create an object and it fails" or you "try to retrieve an object and it's not found"? How do you represent that?

Those are pretty realistic scenarios to run into, and passing null is a common practice, though not a very good one.

Nulls aren't nice because to the client, they masquerade 🎭 as if they're actually the object(s) you're looking for, but as soon as you call a method or try to access a property on them, they throw errors.

let votes: Votes[] = [vote1, vote2, vote3, null]; for (let vote of votes) { // throws error on the last element console.log(`This vote was cast by ${vote.member.memberId}`) }

Undefined and null-y values are things that we should be prepared to deal with.

In Scott Wlaschin's talk, "Domain Modeling Made Functional", he suggests representing null or undefined with its own type, like Nothing.

type Nothing = null | undefined | '';

Using this type, we can create optional types using unions:

type SomeString = string; type OptionalString = SomeString | Nothing; type SomeNumber = number; type OptionalNumber = SomeNumber | Nothing;

Or we can make it generic in order to strictly represent any type that may or may not have a value.

type Option<T> = T | Nothing; type OptionalString = Option<string>; type OptionalNumber = Option<number>;

Building on that, the type for a Factory Function could adequately

type Email = string; function createEmail (email: string): Option<Email> { return email.contains('@') ? '' as Nothing : email as Email; }

This particular functional design pattern used to represent types that may or may not exist is called a monad.

When we model nothingness as a type in our domain instead of just passing null, it forces any client code that relies on it to accomodate the possibility of nothingness.

I wrote an article describing how to use the Result monad to model nothingness and wrap failure states with error messages. Check it out, it's called "Flexible Error Handling w/ the Result Class | Enterprise Node.js + TypeScript".

Enforcing object creation constraints with factories

Object constraints (in OO we refer to these as class invariants) are the rules that dictate size, shape, length, or the state that a particular domain object is allowed to be in, and at which point in time.

Here are some constraint (invariant) examples:

Length constraints: PostText cannot be longer than 50 characters (overflow), and must be at least 3 characters (underflow)

Pattern constraints: Email must be a valid email address based on a regex formula that we constructed.

Complex constraints: PaidJobDetails cannot be less than 15 dollars an hour if the Job's Location is based in Ontario.

It's important that the language we choose is able to reduce all the possible ways to create a valid domain object to one way.

We can use the factory pattern for this.

Here's a purely functional approach:

domain/postTitle.ts

type Nothing = undefined | null; type EmailAddress = string; type Option<T> = T | Nothing; let createEmailAddress: Option<EmailAddress> = (email: string) => ( !!email === true && email.includes('@') ? email as Email : null as Nothing )

This is great an all, but there's nothing stopping me from doing:

let email: EmailAddress = 'diddle.com'

This is one of the things that I've been able to enforce with classes using an Object-Oriented approach. Using private constructors and static factory methods, we can ensure that the only way domain objects is through the static factory.

interface EmailAddressProps { value: string; } export class EmailAddress extends ValueObject<EmailAddressProps> { get value () : string { return this.props.value; } // Can't invoke with new from outside of class. private constructor (props: EmailAddressProps) { super(props); } // Only way to create an EmailAddress is through the static factory method public static create (email: string): Result<EmailAddress> { const guardResult = Guard.againstNullOrUndefined(email, 'email'); if (guardResult.isFailure() || !TextUtils.validateEmail(email)) { return Result.fail<EmailAddress>() } else { return Result.ok<EmailAddress>(new EmailAddress({ value: email })) } } }

Handling error states

There are errors that we expect and know how to deal with, and errors that we don't expect and don't really know how to deal with.

It's also true that for every single use case, there are at least one or more possible ways that it can fail.

So why is it that we often let error states fall through the cracks?

We should strictly type them and represent them as domain concepts.

For example, if we were building DDDForum.com, and we wanted to code out the UpvotePost use case, here's a namespace containing objects for all the possible ways it can fail.

useCases/UpvotePost/UpvotePostErrors.ts

import { UseCaseError } from "../../../../../shared/core/UseCaseError"; import { Result } from "../../../../../shared/core/Result"; export namespace UpvotePostErrors { export class MemberNotFoundError extends Result<UseCaseError> { constructor () { super(false, { message: `Couldn't find a member to upvote the post.` } as UseCaseError) } } export class PostNotFoundError extends Result<UseCaseError> { constructor (slug: string) { super(false, { message: `Couldn't find a post by slug {${slug}}.` } as UseCaseError) } } export class AlreadyUpvotedError extends Result<UseCaseError> { constructor (postId: string, memberId: string) { super(false, { message: `This post was already upvoted postId {${postId}}, memberId {${memberId}}.` } as UseCaseError) } } }

For that single use case, there are at least three ways that it can fail.

Using the Either<T, U> monad, we can take a step up from the basic Result<T> class that we learned about here, and we can segregate all the possible failure states from the single success state.

Here's what the response type for this use case might look like:

useCases/UpvotePost/UpvotePostResponse.ts

import { Either, Result } from "../../../../../shared/core/Result"; import { UpvotePostErrors } from "./UpvotePostErrors"; import { AppError } from "../../../../../shared/core/AppError"; export type UpvotePostResponse = Either< // Failures UpvotePostErrors.MemberNotFoundError | UpvotePostErrors.AlreadyUpvotedError | UpvotePostErrors.PostNotFoundError | AppError.UnexpectedError | Result<any>, // Success Result<void> >

Note: If you haven't already read it, check out "Functional Error Handling with Express.js and DDD | Enterprise Node.js + TypeScript" to learn how to stop throwing errors and prefer a functional approach to expressing error states.

Conclusion

The main takeaways here are to:

Use Factories and Factory Methods to prevent from repeating validation logic. Encapsulate it in one place. When the creation rules change, you only need to update code in one place. See this article on Value Objects for how to implement this.

Make it impossible for you to create domain objects without going through the factory. Use private constructors to prevent the use of the new keyword.

Wrap primitives with their own domain specific classes in order to make the language more expressive, encode business requirements, and protect their internal values from change operations that don't make sense to the domain.

Prefer update methods over public setters to encapsulate change rules within the entity itself. Make it impossible to change things when they shouldn't be changed.

Strictly type errors, nulls, and nothingness in order to remove the guesswork from client side code. They should always be able to accomodate for when something failed or was not found through static typing.

Types are law

Types are the laws that dictate what's allowed to happen in our domain. Next time you write some code, think not only about what it would take to make the code work, but also think about what could be done using types (and any language features) to prevent anyone else from putting the system in a state invalid to the domain. If you're using vanilla JavaScript, consider the challenges of preventing team members from doing invalid things, and consider if TypeScript would improve the bylaws of your codebase.

References & resources

For a purely functional introduction to how to implement your domain layer code functionally, watch Scott's "Domain Modeling Made Functional" talk. I recommend you check it out. It's well worth your time.

Static Factory Methods | Object-Oriented Programming w/ TypeScript

Thu, 21 Nov 2019 15:04:10 GMT

Go read "When to Use a Private Constructor | Object-Oriented Programming w/ TypeScript".

You'll use static factory methods for the same reason you'll want to use a private constructor.

When to Use a Private Constructor | TypeScript OOP

Thu, 21 Nov 2019 15:04:10 GMT

One of the first things we learn when we start out is how to create instances of objects. Typically, we do this with the new keyword.

class User { public name: string; constructor (name: string) { this.name = name; } } const user: User = new User('Khalil Stemmler');

The actual thing that does the creating is the constructor- and by default, it's public.

Have you ever seen a private constructor? Changing the scope of a constructor to private removes our ability to use the new keyword.

class User { public name: string; private constructor (name: string) { this.name = name; } } const user: User = new User('Khalil Stemmler'); // Error

Why on Earth would you want to do that? We can't create instances anymore. How are we supposed to get Users out of this class now?

All is not lost, and it turns out that there is a very good reason for why you'd want to do this kind of thing.

In essence, it's to enforce object creation rules.

Using the new keyword

When we use the new keyword, there's not really an elegant way to prevent a User from being created if certain validation rules don't pass.

We can throw errors.

class User { public name: string; constructor (name: string) { if (!!name === false) { throw new Error ("Ya need to include a name") } this.name = name; } } let user; try { user = new User(); } catch (err) { } console.log(user); // undefined

But honestly, who wants to live in a world where we can't trust that a simple new statement won't throw errors. I don't feel like adopting trust issues with my codebase.

We've talked about why throwing errors is not great in the "Functional Error Handling" article, so let's think of a better way to prevent bad objects from being created.

Static factory methods

The best way to enforce validation logic against a domain object is to keep the constructor private and use a static factory method to enforce the constraints.

Using the Result<T> class from "Flexible Error Handling w/ the Result Class | Enterprise Node.js + TypeScript", we can statically represent a success or a failure.

interface UserProps { name: string; } class User { private props: UserProps; get name (): string { return this.props.name; } private constructor (props: UserProps) { this.props = props; } public static create (props: UserProps): Result<User> { const guardResult = Guard.againstNullOrUndefined(props.name, 'name'); const isAppropriateLength = TextUtils.isAtLeast(2, props.name) && TextUtils.isAtMost(31, props.name); // Fail with reason if (!guardResult.success) { return Result.fail<User>(guardResult.message) } // Fail with reason if (!isAppropriateLength) { return Result.fail<User>("Must be between 2 and 31 characters") } // Static method can access the constructor return Result.ok<User>(new User(props)); } }

Now, object creation looks like this:

let user: User; let userOrError: Result<User> = User.create({ name: 'Khalil Stemmler' }); if (userOrError.isSuccess) { user = userOrError.getValue(); } else { console.log(userOrError.error) }

Feel free to get even more functional with these different types of errors as well. We can statically type a NullValue error and an InvalidLength error.

Using the Either<T, U> monad from "Functional Error Handling with Express.js and DDD | Enterprise Node.js + TypeScript", we can build return types like:

type UserResult = Either< // Failure types UserErrors.NullValuesError | UserErrors.InvalidFieldLengthError, // Success type User >

I'm Learning Everything GraphQL in 2020

Tue, 19 Nov 2019 22:04:10 GMT

If you've been writing code for at least 15 minutes, you've probably realized that the ability to learn is one of the most important skills you can possess as a developer.

Things change super quickly in our industry.

For example, I heard about this thing called GraphQL a couple of years ago when I built my first GatsbyJS site, I blinked (maybe two or three times), realized it was everywhere, and witnessed my friends building front-end applications faster than me (and apparently having more fun doing it as well).

This year, I joined Apollo, the company that builds the best tools for GraphQL developers, as a Developer Advocate.

I'm really excited about it because I've already been having a lot of fun creating helpful content on advanced TypeScript and Node.js for y'all over the past year.

To me, helping you learn and improve your skills is the most fulfilling and meaningful work I've ever done.

As an Apollo Developer Advocate, I'll be able to spend a lot more time interacting with community by creating apps, contributing to open source Apollo projects, making videos, blog posts, meeting with you in person and online in addition to rallying for you and being a direct line to our Apollo team.

That being said, I have a lot to learn.

I'm pretty much an Apollo noob. So this should be a TON of fun.

Defining success

The first thing I always do whenever I start learning something new is to define what success looks like. After that, I work backwards.

What would I like to be able to comfortably do? What do I want to know?

Minimize the total number of unknown unknowns about GraphQL and the developer ecosystem.

Understand the timeline of major events related to GraphQL, tooling, trends, and adoption.

Identify the major players in the GraphQL universe.

Learn the Apollo platform and know it like the back of my hand.

Understand the biggest pain points for developers using GraphQL today.

Understand the current trends and where GraphQL is going in the future.

Learn best practices in the following categories: Federation, Running GraphQL in Production (Apollo Server, Caching, Architecture), GraphQL Development Tools (Graph Manager), Apollo Client, Schema Design.

Develop my own opinions on GraphQL project architectures and when is the most and least appropriate times to develop products with it.

What would I like to build?

When learning new programming concepts, I like to dive in by getting my hands dirty building real world applications with it.

I really like this approach because it forces me to run into errors and problems that you don't get simply watching videos or reading books, and debugging is where I have some of my biggest mental breakthroughs.

Several smaller React (with Hooks) + Apollo applications in order to try out tooling (something fun and music-related).

Incrementally migrate a large RESTful project to Apollo GraphQL. Perhaps DDDForum.com.

Build an ambitious Domain-Driven Design-based GraphQL project with React, TypeScript, MySQL, and Sequelize using Apollo Federation.

Deploy large and small GraphQL applications on various platforms like Heroku, AWS Lambda, Zeit Now, and Netlify.

Try out different architectures, like Serverless GraphQL.

What would I like to teach/write about?

The last part of my learning is usually to teach it to someone else. If I can't comfortably explain it to a friend or write about it in a blog post, I probably don't know it well enough yet. That's how I like to force myself to retain concepts- by testing that I can teach it.

I'd love to write/speak/train about the following running list of topics:

Best practices towards Enterprise GraphQL Architectures (Apollo GraphQL, Federation, Caching, and CQRS)

Serverless GraphQL

Comparisons between DDD and GraphQL

Choosing Learning Resources

In order to learn, I just pick a couple of resources that will be enough to get me started, and then piece things together by Googling when I get stuck. Here's where I'm thinking of starting.

Learning GraphQL: Declarative Data Fetching for Modern Web Apps

This book by Eve Porcello and Alex Banks was recently recommended to me by my peers. Looks like a good starting point.

Apollo Fullstack Tutorial

Apollo has a pretty cool full stack tutorial that I plan on walking through to get familiar with how devs are building apps on the Apollo platform.

I'm also planning on helping migrate this to TypeScript :)

Principled GraphQL

You know me. I like a set build a set of principles early on to decide if I'm going the direction that I believe I should be with my code.

Principled GraphQL is a neat little resource of principles on GraphQL usage.

Shopify's GraphQL Design Guide

I'm also planning on checking this one out. It seems to be a bit more low-level than Principled GraphQL, and it was recommended to me by a community member at GraphQL Summit 2019.

Read Shopify's GraphQL API Design Guide.

More free and paid stuff?

I recently asked Twitter to help me identify the best free and paid resources.

If I need anything else, I'll go to this thread :)

What are the best free and paid resources to learn GraphQL?

Link courses, books, guides + tag authors, instructors & devs.
— Khalil Stemmler (@khalilstemmler) November 2, 2019

Should be fun!

Stay tuned for more content and subscribe to the newsletter if you want to get notifed when new stuff drops 🎤.

How to Handle Updates on Aggregates - Domain-Driven Design w/ TypeScript

Mon, 18 Nov 2019 15:04:10 GMT

We cover this topic in The Software Essentialist online course. Check it out if you liked this post.

Also from the Domain-Driven Design with TypeScript series.

Introduction

A blog reader recently asked,

"Please suggest to me the best practices to deal with situations where a single field or only a few fields are present for an update against an aggregate".

Ah yes. Updates.

It's inevitable you'll want to perform update commands in a CRUD + MVC or DDD-based application. Everyone has their own way to handle updates in basic MVC applications, but it's not abundantly clear how to design updates against aggregates using DDD.

In the code sample, they had a User aggregate, which looked more or less like this:

Note: I've left comments for improvements that I recommend based on patterns we've explored on this site in previous articles.

domain/user.ts

class User extends AggregateRoot { // The user props. // I would recommend extracting these to props like in the // "Understanding Domain Entities" guide. // The way it is currently, getters and setters are exposed // for everything, and that's not very "Intention Revealing". // Also makes it hard to restrict invalid operations (like // manually changing the id) and makes it hard to enforce // model invariants like ensuring the Phone is a valid // value object instance (can currently assign to null). // Let's take full advantage of the language capabilities. id: number; phone: Phone; email: Email; address: Address; private constructor (id, phone, email, address){ // setting values here, I think } // Very good, there's a public factory method. Should type these // props as their own interface UserProps {} though. public static create (props) { // There should be a Guard class here to ensure all props are valid. return new User({ ...props }); } // This is what's new. We'll talk about this. public static update (props) { return new User({ ...props }); }}

There's definitely ways we could improve User aggregate, but the real pain points are felt from within the UsersService's updatePhone method (which is something that I would normally advocate for representing as an application layer use case) instead.

usersService.ts

export class UserService { ... public updatePhone (updatePhoneDto) { const userEntity = userRepository.getUser(updatePhNoDto.userId); const userModel = User.update({ id: updatePhNoDto.userId, phone: Phone.create(userEntity.phone), email: Email.create(userEntity.email), address: Address.create(userEntity.address) }); userRepository.updateUser(userModel) } }

The primary drawback to address in this code is the fact that the updatePhoneDto probably has a shape like:

interface UpdatePhoneDto { userId: number; phone?: string; email?: string; address?: string; }

And with all of those optional fields, we need to be able to "deal with situations where a single field or only a few fields are present for an update".

If any of either phone, email, or address aren't present, we'll break each value object's create() factory method.

That's not good.

What about non 1-to-1 relationships?

We also have to consider situations where we're not just updating 1-to-1 relationships like User to Phone or User to Address.

How do we handle updates against 1-to-many or many-to-many relationships?

For example, in White Label, an Album can have many different Genres assigned to it.

We need some way that we can keep track or mark which Genres were updated or removed in an update so that we can perform the correct persistence commands (insert? delete? update?).

It can get pretty complex. And that's a necessary complexity when we use domain models to encapsulate business rules and language within code.

The flow is straightforward though.

Plan of action for performing updates

Since in DDD, we usually implement the Data Mapper pattern, the object we retrieve from persistence before we update it will be a plain 'ol TypeScript object.

Our plan for performing an update against and aggregate will look like this:

Fetch the aggregate (simple TypeScript object) we want to change.

Change it.

Pass it off to a repo to save() (or perhaps delete()).

The challenges at step 2 are:

Protecting model integrity (class invariants)

Performing validation logic

Representing errors as domain concepts

Keeping "update code" DRY

Choosing the best ways to represent updates

The challenges at step 3 are:

Performing an atomic update as a single transaction

Knowing whether to perform an update, an insert or a delete based on the changes from the domain model.

Scaffolding across foreign-key tables within the aggregate boundary.

Lets start with the basics.

In this article, we'll cover:

How to handle updates (update, insert, delete) within aggregates with 1-to-1 relationships

How mutations to your domain model can contain class invariants that need to be represented as domain concepts

We'll go into more advanced stuff like 1-to-many relationships in another article.

Prerequisites

In order to get the most out of this article, there are a few things that you might want to read first.

The primary domain-driven design building blocks: domain entities, value objects, aggregate roots and aggregate boundaries.

How to use DTOs, Repositories, and Mappers to persist and transform domain objects to other representations.

The Command-Query Segregation Principle.

How every feature of an application is a use case, which is either a command or a query.

A basic example (1-to-1)

Let's take the example we looked at before: the User aggregate and it's relationship to phone, email and address.

Creating the domain model

I'm going to model the User aggregate a little differently than the example provided based on things we've covered in the Domain-Driven Design w/ TypeScript series already.

domain/user.ts

import { AggregateRoot } from "../../../core/domain/AggregateRoot"; import { UniqueEntityID } from "../../../core/domain/UniqueEntityID"; import { UserCreated } from '../events/userCreated' import { Result } from "../../../core/logic/Result"; import { UserId } from "./userId"; import { Email } from "./email"; import { Phone } from "./phone"; import { Address } from "./address"; import { Guard } from "../../../core/logic/Guard"; interface UserProps { phone: Phone; email: Email; address: Address; } export class User extends AggregateRoot<UserProps> { // Only getters so far, no way to perform changes to // the model after it's been created. get userId (): UserId { return UserId.create(this._id) } get phone (): Phone { return this.props.phone; } get email (): Email { return this.props.email; } get address (): Address { return this.props.address; } private constructor (props: UserProps, id?: UniqueEntityID) { super(props, id); } // The only way to create or reconstitute a User is to use the static factory // method here. public static create (props: UserProps, id?: UniqueEntityID): Result<User> { const guardResult = Guard.againstNullOrUndefinedBulk([ { argument: props.phone, argumentName: 'phone' }, { argument: props.email, argumentName: 'email' }, { argument: props.address, argumentName: 'address' } ]); if (!guardResult.success) { return Result.fail<User>(guardResult.message); } const isNewUser = !!id === false; const user = new User(props, id); // Dispatch a domain event if it's new. Otherwise, it's just an // entity being reconstituted from persistence. if (isNewUser) { user.addDomainEvent(new UserCreated(user.userId)) } return Result.ok<User>(user); } }

Cool, now let's talk about the use case.

Use Case setup

We want to provide a way to update the User aggregate, so let's start by creating a new UpdateUser use case in our use cases folder for the User subdomain.

modules/ users/ # `users` subdomain domain/ ... user.ts ... useCases/ updateUser/ UpdateUser.ts # Use case!

Folder structure following the Screaming Architecture / Package by Component principle.

useCases/updateUser/UpdateUser.ts

export class UpdateUser implements UseCase<any, Promise<any>> { constructor () { // import dependencies } public async execute (): Promise<any> { // execute application layer logic } }

If you've read the Clean Architecture vs. Domain-Driven Design concepts article, you'll remember that the responsibility of use cases at this layer are to simply fetch the domain objects we'll need to complete this operation, allow them to interact with each other (at the domain layer), and then save the transaction (by passing the affected aggregate root to it's repository).

That's just what we'll do.

Let's create a DTO in order to specify the inputs to this use case.

... updateUser/ UpdateUser.ts # Use Case UpdateUserDTO.ts # DTO

useCases/updateUser/UpdateUserDTO.ts

export interface UpdateUserDto { userId: number; phone?: string; email?: string; address?: string; }

And we'll update our use case to use that as the input.

useCases/updateUser/UpdateUser.ts

import { UpdateUserDTO } from './UpdateUserDTO' export class UpdateUser implements UseCase<UpdateUserDTO, Promise<any>> { constructor () { // import dependencies } public async execute (request: UpdateUserDTO): Promise<any> { // execute application layer logic } }

Cool. Now let's dependency inject a UserRepo so that we can get access to the user aggregate that we want to change in this transaction, then let's get it.

useCases/updateUser/UpdateUser.ts

import { IUserRepo } from '../repos/interfaces/userRepo' import { UpdateUserDTO } from './UpdateUserDTO' export class UpdateUser implements UseCase<UpdateUserDTO, Promise<any>> { private userRepo: IUserRepo; constructor (userRepo: IUserRepo) { this.userRepo = userRepo; } public async execute (request: UpdateUserDTO): Promise<any> { let user: User; try { user = await this.userRepo.findUserById(request.userId); } catch (err) { // Handle not found error. } ... // Continue } }

I'm going to hook up some expressive error handling because I don't want a "Not Found" error to go unswallowed by the consumer of this use case.

Note: This is how we can strictly type and express any other application-level (use case) or domain layer errors that might get returned. The primary benefit of not throwing errors but instead representing them as first-class citizens of our app is that we force the client using our use case to handle the error states (that we know for sure will occur at some point).

Two new files.

... updateUser/ UpdateUser.ts # Use Case UpdateUserDTO.ts # DTO UpdateUserErrors.ts # Holds all the unique types of application errors UpdateUserResult.ts # Express the result as a functional error type

Lets create the namespace to represent errors for this use case.

updateUser/UpdateUserErrors.ts

import { UseCaseError } from "../../../../../shared/core/UseCaseError"; import { Result } from "../../../../../shared/core/Result"; export namespace UpdateUserErrors { export class UserNotFoundError extends Result<UseCaseError> { constructor (userId: string) { super(false, { message: `Couldn't find user id {${userId}} to update.` } as UseCaseError) } } }

And then we'll create our strict return type so that clients can know what success and failure states to expect.

updateUser/UpdateUserResult.ts

import { Either, Result } from "../../../../../shared/core/Result"; import { UpvotePostErrors } from "./UpvotePostErrors"; import { AppError } from "../../../../../shared/core/AppError"; export type UpdateUserResult = Either< UpdateUserErrors.UserNotFoundError | // Specific use case error AppError.UnexpectedError | // Global app error Result<any>, // Misc errors (value objects) Result<void> // Success! >

Error handling articles: No idea what I'm doing? First read "Flexible Error Handling w/ the Result Class" and then read "Functional Error Handling with Express.js and DDD".

Then let's update the use case with the return type, represent the not found error, and represent the void success state

useCases/updateUser/UpdateUser.ts

import { IUserRepo } from '../repos/interfaces/userRepo' import { UpdateUserDTO } from './UpdateUserDTO' import { UpdateUserResult } from './UpdateUserResult' import { UpdateUserErrors } from './UpdateUserErrors' export class UpdateUser implements UseCase<UpdateUserDTO, Promise<UpdateUserResult>> { private userRepo: IUserRepo; constructor (userRepo: IUserRepo) { this.userRepo = userRepo; } public async execute (request: UpdateUserDTO): Promise<UpdateUserResult> { let user: User; try { user = await this.userRepo.findUserById(request.userId); } catch (err) { return left(new UpdateUserErrors.UserNotFoundError(request.userId)) } // Update logic goes here return right(Result.ok<void>()) } }

Fantastic. We're all set up with to write some update logic now.

Do you know how the Repository pattern works? Do you know how to convert an ORM model (like Sequelize or TypeORM) to a domain object (like an Aggregate Root, Entity or Value Object)? If not, check out "Implementing DTOs, Mappers & the Repository Pattern using the Sequelize ORM [with Examples] - DDD w/ TypeScript".

Handling update logic

In this particular scenario, we have a DTO with several keys that we'd like to update: phone, email, address.

Depending on the domain, each of these could possibly be modelled as simple value objects because they're more complex than simple string types and have validation rules to dictate what a valid instance of them looks like.

We can start by using each value object's factory method (which encapsulates its respective validation logic).

useCases/updateUser/UpdateUser.ts

import { IUserRepo } from '../repos/interfaces/userRepo' import { UpdateUserDTO } from './UpdateUserDTO' import { UpdateUserResult } from './UpdateUserResult' import { UpdateUserErrors } from './UpdateUserErrors' import { Phone } from '../domain/phone' import { Email } from '../domain/email' import { Address } from '../domain/addres' export class UpdateUser implements UseCase<UpdateUserDTO, Promise<UpdateUserResult>> { private userRepo: IUserRepo; constructor (userRepo: IUserRepo) { this.userRepo = userRepo; } public async execute (request: UpdateUserDTO): Promise<UpdateUserResult> { let user: User; try { user = await this.userRepo.findUserById(request.userId); } catch (err) { return left(new UpdateUserErrors.UserNotFoundError(request.userId)) } // Create value object instances const phoneOrError: Result<Phone> = Phone.create(request.phone); const emailOrError: Result<Email> = Email.create(request.email); const addressOrError: Result<Address> = Address.create(request.address); ... return right(Result.ok<void>()) } }

The problem is that a request to this use case might not contain each of the keys on the dto interface. It's likely that we only want to update phone but not also email and address in a single request.

We can write our own null checks or use a utility like lodash's has function. It will return true if a property exists on an object.

useCases/updateUser/UpdateUser.ts

import { IUserRepo } from '../repos/interfaces/userRepo' import { UpdateUserDTO } from './UpdateUserDTO' import { UpdateUserResult } from './UpdateUserResult' import { UpdateUserErrors } from './UpdateUserErrors' import { Phone } from '../domain/phone' import { Email } from '../domain/email' import { Address } from '../domain/addres' import { has } from 'lodash' export class UpdateUser implements UseCase<UpdateUserDTO, Promise<UpdateUserResult>> { private userRepo: IUserRepo; constructor (userRepo: IUserRepo) { this.userRepo = userRepo; } public async execute (request: UpdateUserDTO): Promise<UpdateUserResult> { let user: User; try { user = await this.userRepo.findUserById(request.userId); } catch (err) { return left(new UpdateUserErrors.UserNotFoundError(request.userId)) } if (has(request, 'phone')) { const phoneOrError: Result<Phone> = Phone.create(request.phone); // update } if (has(request, 'email')) { const emailOrError: Result<Email> = Email.create(request.email); // update } if (has(request, 'address')) { const addressOrError: Result<Address> = Address.create(request.address); // update } ... return right(Result.ok<void>()) } }

Awesome. Now we won't attempt to update something that's not even included in the request.

Focusing in on one of the keys (like phone for example) if we were able to successfully create the value object, we can go ahead and update user with it.

And if we weren't able to create it, yet we included some value for that key in the request, that probably means that there was a validation error that didn't pass and we should let the client know about that (matches the Result<any> error type).

useCases/updateUser/UpdateUser.ts

... export class UpdateUser implements UseCase<UpdateUserDTO, Promise<UpdateUserResult>> { ... public async execute (request: UpdateUserDTO): Promise<UpdateUserResult> { ... if (has(request, 'phone')) { const phoneOrError: Result<Phone> = Phone.create(request.phone); if (phoneOrError.isSuccess) { user.updatePhone(phoneOrError.getValue()) // This will be of Phone type. } else { return left(phoneOrError) // This will be a Result<any> return type. } } ... return right(Result.ok<void>()) } }

We have a new method on the user aggregate called updatePhone(phone: Phone).

domain/user.ts

... export class User extends AggregateRoot<UserProps> { ... get phone (): Phone { return this.props.phone; } public updatePhone (phone: Phone): void { this.props.phone = phone; } ... }

Simple enough.

Here's where it gets interesting...

Atomic Transactions

Lets say we wanted to update phone AND address in single transaction.

If phone was valid but address was not, should we let the transaction partially update the user aggregate or should the entire transaction fail?

It should fail, correct?

Things have the potential to get messy, code can get hard to debug, and things can be challenging to reason about if we were to allow partial transactions. A request to UPDATE the user has to be fully correct for it to succeed.

How do we do this?

Using our trusty Result<T> class and its combine(results: Result<T>[]) method of course!~

We want to keep track of all the changes in a transaction, so let's add a changes: Result<T>[] property and a couple of methods to manage it as well.

useCases/updateUser/UpdateUser.ts

... export class UpdateUser implements UseCase<UpdateUserDTO, Promise<UpdateUserResult>> { private userRepo: IUserRepo; private changes: Result<T>[]; constructor (userRepo: IUserRepo) { this.userRepo = userRepo; this.changes = []; } public addChange (result: Result<any>) : void { this.changes.push(result); } public getCombinedChangesResult (): Result<any> { return Result.combine(this.changes); } ... }

The requirement now is that every time we mutate an aggregate, that needs to issue a Result<T>.

Going back to our User aggregate, we can change the method to accomodate our design decision.

domain/user.ts

... export class User extends AggregateRoot<UserProps> { ... get phone (): Phone { return this.props.phone; } public updatePhone (phone: Phone): Result<void> { this.props.phone = phone; return Result.ok<void>(); } ... }

This might not seem like a great design decision right away. In our updatePhone method, the cyclomatic complexity hasn't really changed at all - there are the same number of code paths. So this additional complexity doesn't make a whole lot of sense upfront.

Note: After we finish this up, I'll show you an example of a real life aggregate that enforces class invariants that dictate when and how it's allowed to change. It makes much better use of the Result type.

Using this pattern, we can add each mutation against the user aggregate to the changes array.

useCases/updateUser/UpdateUser.ts

import { IUserRepo } from '../repos/interfaces/userRepo' import { UpdateUserDTO } from './UpdateUserDTO' import { UpdateUserResult } from './UpdateUserResult' import { UpdateUserErrors } from './UpdateUserErrors' import { Phone } from '../domain/phone' import { Email } from '../domain/email' import { Address } from '../domain/addres' import { has } from 'lodash' export class UpdateUser implements UseCase<UpdateUserDTO, Promise<UpdateUserResult>> { private userRepo: IUserRepo; private changes: Result<T>[]; constructor (userRepo: IUserRepo) { this.userRepo = userRepo; this.changes = []; } public addChange (result: Result<any>) : void { this.changes.push(result); } public getCombinedChangesResult (): Result<any> { return Result.combine(this.changes); } public async execute (request: UpdateUserDTO): Promise<UpdateUserResult> { let user: User; try { user = await this.userRepo.findUserById(request.userId); } catch (err) { return left(new UpdateUserErrors.UserNotFoundError(request.userId)) } if (has(request, 'phone')) { const phoneOrError: Result<Phone> = Phone.create(request.phone); if (phoneOrError.isSuccess) { this.addChange( user.updatePhone(phoneOrError.getValue()) ) } else { return left(phoneOrError) } } if (has(request, 'email')) { const emailOrError: Result<Email> = Email.create(request.email); if (emailOrError.isSuccess) { this.addChange( user.updateEmail(emailOrError.getValue()) ) } else { return left(emailOrError) } } if (has(request, 'address')) { const addressOrError: Result<Address> = Address.create(request.address); if (addressOrError.isSuccess) { this.addChange( user.updateEmail(addressOrError.getValue()) ) } else { return left(addressOrError) } } ... return right(Result.ok<void>()) } }

And then at the end, if all of the changes were successful, we can pass it to the repository to be saved.

useCases/updateUser/UpdateUser.ts

... export class UpdateUser implements UseCase<UpdateUserDTO, Promise<UpdateUserResult>> { private userRepo: IUserRepo; private changes: Result<T>[]; constructor (userRepo: IUserRepo) { this.userRepo = userRepo; this.changes = []; } public addChange (result: Result<any>) : void { this.changes.push(result); } public getCombinedChangesResult (): Result<any> { return Result.combine(this.changes); } public async execute (request: UpdateUserDTO): Promise<UpdateUserResult> { ... if (this.getCombinedChangesResult().isSuccess) { try { // Save! await this.userRepo.save(user); } catch (err) { return left (AppError.create(err)) } } return right(Result.ok<void>()) } }

And the UserRepo knows whether to perform an update or a create because it will first check to see if the entity exists or not.

repos/implementations/sequelizeUserRepo.ts

export class SequelizeUserRepo implements UserRepo { ... async save (user: User): Promise<void> { const UserModel = this.models.BaseUser; const exists = await this.exists(user.email); if (!exists) { const rawSequelizeUser = await UserMap.toPersistence(user); await UserModel.create(rawSequelizeUser); } else { // Update! } return; } }

For more on using Repositories to persist domain entities, read "Implementing DTOs, Mappers & the Repository Pattern using the Sequelize ORM [with Examples]".

Extracting "change" functionality to a interface

Pretty much all update use cases need to use this kind of change functionality so I'd recommend extracting it into it's own separate class and then using composition to add it to use cases that need it.

Design Principle: "Prefer composition over inheritance". - via solidbook.io: The Software Architecture & Design Handbook

import { Result } from "./Result"; // Use cases that need changes can implement this export interface WithChanges { changes: Changes; } // Extracted into its own class export class Changes { private changes: Result<any>[]; constructor () { this.changes = []; } public addChange (result: Result<any>) : void { this.changes.push(result); } public getCombinedChangesResult (): Result<any> { return Result.combine(this.changes); } }

This changes the usage in our use case slightly:

useCases/updateUser/UpdateUser.ts

import { IUserRepo } from '../repos/interfaces/userRepo' import { UpdateUserDTO } from './UpdateUserDTO' import { UpdateUserResult } from './UpdateUserResult' import { UpdateUserErrors } from './UpdateUserErrors' import { Phone } from '../domain/phone' import { Email } from '../domain/email' import { Address } from '../domain/addres' import { has } from 'lodash' export class UpdateUser implements UseCase<UpdateUserDTO, Promise<UpdateUserResult>>, WithChanges { private userRepo: IUserRepo; public changes: Changes; constructor (userRepo: IUserRepo) { this.userRepo = userRepo; this.changes = new Changes(); } public async execute (request: UpdateUserDTO): Promise<UpdateUserResult> { let user: User; try { user = await this.userRepo.findUserById(request.userId); } catch (err) { return left(new UpdateUserErrors.UserNotFoundError(request.userId)) } if (has(request, 'phone')) { const phoneOrError: Result<Phone> = Phone.create(request.phone); if (phoneOrError.isSuccess) { this.changes.addChange( user.updatePhone(phoneOrError.getValue()) ) } else { return left(phoneOrError) } } ... if (this.changes.getCombinedChangesResult().isSuccess) { try { // Save! await this.userRepo.save(user); } catch (err) { return left (AppError.create(err)) } } return right(Result.ok<void>()) } }

Representing invalid updates that break class invariants

I said I'd give you a better example of why using Result<T> from within a domain entity is a good idea when we're performing mutations against it in the context of an update use case.

Here's one from DDDForum.com, the Hackernews-inspired forum app built with TypeScript & DDD from solidbook.io.

A Post can have either a link or text, but not both.

And if you wish to change your link or text, you can only do so if the Post doesn't yet have any comments.

See that? In this scenario, there are business rules that dictate when it's OK for something to change. That's the power of DDD and model-driven design. The shift that should happen in your thinking is to prefer trying to represent those invalid error states as domain concepts, rather than than throwing untyped errors.

Here's a snippet from the Post aggregate.

domain/post.ts

/** Represents the different types of things that can happen when we try * to update the post text or link. */ export type UpdatePostTextOrLinkResult = Either< /** * This error means that we're trying to update a text post when the * post is actually a link post, or vice versa. */ EditPostErrors.InvalidPostTypeOperationError | /** * This error means that the post is sealed due to there already being * comments */ EditPostErrors.PostSealedError | Result<any>, Result<void> > class Post extends AggregateRoot<PostProps> { ... public updateText (postText: PostText): UpdatePostTextOrLinkResult { if (!this.isTextPost()) { return left(new EditPostErrors.InvalidPostTypeOperationError()) } if (this.hasComments()) { return left(new EditPostErrors.PostSealedError()) } const guardResult = Guard.againstNullOrUndefined(postText, 'postText'); if (!guardResult.succeeded) { return left(Result.fail<any>(guardResult.message)) } this.props.text = postText; return right(Result.ok<void>()); } public updateLink (postLink: PostLink): UpdatePostTextOrLinkResult { if (!this.isLinkPost()) { return left(new EditPostErrors.InvalidPostTypeOperationError()) } if (this.hasComments()) { return left(new EditPostErrors.PostSealedError()) } const guardResult = Guard.againstNullOrUndefined(postLink, 'postLink'); if (!guardResult.succeeded) { return left(Result.fail<any>(guardResult.message)) } this.props.link = postLink; return right(Result.ok<void>()); } }