Building Countdoku, a multiplayer Sudoku Web app

tldr: I’ve built a multiplayer Sudoku game which you can play at Countdoku.app. Read on if you are interested in the technical details, (they’re cool, honest!)

Screenshot of Countdoku on a mobile device

Many years ago I became interested in how to generate Sudoku puzzles efficiently, as a thought exercise. Having solved this fairly well writing a program in Java, I continued to build a small multiplayer Java game where people could play the same Sudoku puzzle, taking turns to see who would win. This was long before social media or even Web2, and it ended there.

A few weeks ago I picked up the idea and decided to port it to modern technologies, namely the modern (as of 2022) Web. I decided to build a Progressive Web App (PWA) with NextJS, React, running on Vercel.

Priorities

My priorities with the project were

  • Keep it simple. It doesn’t need to be fancy, just a game you can pop into when you feel like some diversion.
  • Make it fast. It should work well on slow networks and cheap phones.
  • Easy to share. Players should be able to get others to play with them with minimal effort.
  • Kid friendly. I have two young daughters and wanted to make something they can enjoy playing (they love it!)

Technical choices

I am a huge fan of NextJS and React. They make building a web application far easier than it has ever been in the past (I’ve been doing this since 2003 or so, trust me, I’ve seen things you people wouldn’t believe…..). The combination of React’s relatively simple API & easy componentisation, NextJS’ developer environment, and Typescript means I can knock out code quickly and reliably.

Deploying NextJS apps works best on its parent company Vercel’s platform. Of course you can deploy NextJS apps anywhere, but Vercel really is the perfect match for it. Just push to a Github repo and very soon it’ll be live on the web. I also have other Vercel deployed projects, so adding one more was a no-brainer.

My initial plan was to use a persistent connection to receive live updates when another player made a move, so I decided to use Firebase & Firestore, which have SDKs that support this. I’ve had a terrible time in the past using Firestore for complex applications, as it has so many gotchas, however this project was going to be simple, and they even seem to have fixed some gotchas in the past few years – you can now run a query with a ‘not equals’ condition. So fancy!

Need for speed

My initial version of the web app used the standard NextJS and React deployment, with JS running both on the server and on the client. However I found that this was very slow to download on a phone, with over 250kb of JavaScript to download, parse and execute before any application code was used. This was split between React and the large bundle of JS that NextJS sends down the wire to enable fast page transitions, as well as the FirestoreLite SDK.

I realised that I didn’t need any of that. After rendering the page on the server side, the only visual changes that happened during play were:

  • Changing CSS classes on the Sudoku board to highlight some cells.
  • Showing and hiding dialogs.

Knowing this, I disabled all client side NextJS and React JavaScript by exporting the following from the page JS file.

export const config = {
  unstable_runtimeJS: false,
};

I then wrote a single Javascript file called browser/index.js in which I put most of the application code. I separated some other code out into their own small files, e.g. everything to do with accessing the API, and wrote a simple script for copying/transpiling any utility code I shared with the server side, which you can find on Github.

Then, it was a fairly simple thing to use Browserify and Tinify to build the resulting JS file and put it in the public/js folder, to be loaded. While developing, I used node-watch to listen for code changes and rebuild the main.js file.

All of this was duplicating what NextJS gives you for free, but the result was that I loaded 19KB of JavaScript instead of hundreds of KB, and every byte was application code that provided functionality to the user.

Goodbye Client side Firebase

As for the Firebase/Firestore code, I removed the client side SDK, and moved all that functionality to a few simple APIs, which used the Firebase Node SDK instead. Then I simply called these using fetch() on the client. The downside of this is that I no longer had access to the real time updates that Firebase can provide, but those are only available through its full SDK, not the Lite version, and that is simply huge. Replacing it with a regular polling of an API endpoint gives the user basically the same functionality, and I store timestamps on every user change, so the API only returns the changes since your last request, keeping the payload small.

Finally, I added a simple service worker to make this a Progressive Web App. I basically copy/pasted the great example from the Chrome Dev team, and modified it slightly to suit my needs. This not only makes some requests on repeat visits quicker, it also allows the app to run in full screen mode, hiding the browser chrome, which suits this game style of app.

Result

The final result is that Lighthouse on Google Chrome gives a 100 performance score. This is largely because, once the HTML is sent down by NextJS on the server, only 16KB of JavaScript is loaded, and none of it renders anything on the screen at page load time, so there are no layout shifts.

Conclusion

This method of building apps will not suit every use case of course. There are applications where the UI needs to change significantly as the user interacts with it, or new data needs to be loaded and displayed with a UI or template that has not yet been rendered.

However, conversely there are many applications where this is a perfectly valid method of building fast, responsive user experiences. I’m currently building a far more complex application in a similar fashion (hopefully launching in the coming months), and this is working very well. When large UI changes are required, link to a new page and load it. Otherwise have small scripts that slightly modify the DOM as needed, as well as some reusable code for showing/hiding areas of the screen (dialogs, drawers etc), and loading new content rendered server side when necessary.

NextJS and React really do provide a fantastic developer experience, but for anything I want to run on mobile, they’re just too heavy to run client side. This approach keeps the great developer experience, keeps most of the code running on the server, and also gives a great experience to your users no matter how cheap their phone or shoddy their network connection is. Everyone wins.

Finally, I’d love any feedback you have on Countdoku as a game! Feel free to tweet at me, or email me. Try out the easiest mode with young kids, I’ve found they get a kick out of puzzles they can actually solve, before they graduate onto the much harder puzzles later.

Epilogue

One alternative I considered was to use Web Components on the client side. They’re definitely interesting, but since I had a good server side rendering story already, and didn’t need to technically render any new UI on the client side, they would have been overkill. Plain old JS was more than sufficient to make calls to the API, show/hide some pre-rendered dialogs and fiddle with CSS class names on existing DOM nodes. I might give them a go for a future side project however.

Forwarding an email with attachments using SendGrid and Formidable

SendGrid.com is a great service for sending emails programmatically, but it also has an InboundParse feature that will call your webhook for any emails sent to your domain. It can be useful to forward these emails elsewhere, e.g. sending support@mydomain.com to your own personal email.

There’s not really a good example of how to do this, while including attachments, that I could find, so here you go.

import fs from "fs";
// You can use "formidable-serverless" if in a serverless environment like
// AWS Lamba or Google Cloud Functions
import formidable from "formidable";
import sendgridMail from "@sendgrid/mail";
import { AttachmentData } from "@sendgrid/helpers/classes/attachment";
sendgridMail.setApiKey(process.env.SENDGRID_API_KEY);
// See https://www.npmjs.com/package/formidable
interface FormidableFile {
// The size of the uploaded file in bytes.
// If the file is still being uploaded (see `'fileBegin'` event),
// this property says how many bytes of the file have been written to disk yet.
size: number;
// The path this file is being written to. You can modify this in the `'fileBegin'` event in
// case you are unhappy with the way formidable generates a temporary path for your files.
path: string;
// The name this file had according to the uploading client.
name: string | null;
// The mime type of this file, according to the uploading client.
type: string | null;
// A Date object (or `null`) containing the time this file was last written to.
// Mostly here for compatibility with the [W3C File API Draft](http://dev.w3.org/2006/webapi/FileAPI/).
lastModifiedDate: Date | null;
// If `options.hash` calculation was set, you can read the hex digest out of this var.
hash: string | "sha1" | "md5" | "sha256" | null;
}
// Hook this up to your inbound API however you like, e.g.
// using Express
function handleRequest(req, res) {
const form = new formidable.IncomingForm();
form.uploadDir = "/tmp/";
form.keepExtensions = true;
form.type = "multipart";
form.multiples = false;
form.parse(req, async (_err: any, fields, files) => {
handleFormidableResult(fields, files).then(() => {
// Send whatever you want
res.status(200);
res.json({ success: true });
res.end();
});
});
}
async function handleFormidableResult(fields, files) {
const { to, subject, from, html } = fields;
const fileKeys = Object.keys(files);
let attachments: Array<AttachmentData> = null;
let cleanupPromises = null;
if (fileKeys.length > 0) {
const filesInfo = fileKeys.map((key) => files[key]) as Array<
FormidableFile
>;
const attachmentPromises = filesInfo.map((fileInfo) => {
return new Promise((resolve, reject) => {
fs.readFile(fileInfo.path, (err, data) => {
if (err) {
reject(err);
return;
}
const attachment: AttachmentData = {
// Encode the buffer as a base64 encoded string
content: data.toString("base64"),
filename: fileInfo.name,
type: fileInfo.type,
disposition: "attachment",
contentId: fileInfo.hash,
};
resolve(attachment);
});
});
});
// Feel free to do better error handling, where if one file fails to
// read then you still attach others. Keeping it simple here.
attachments = (await Promise.all(attachmentPromises)) as Array<
AttachmentData
>;
// Delete all temp files.
cleanupPromises = filesInfo.map((fileInfo) => {
return new Promise((resolve, reject) => {
fs.unlink(fileInfo.path, () => {
resolve(null);
});
});
});
}
const emailBody = html || fields.text;
const message = {
from: "no-reply@example.com",
to,
subject,
html: emailBody,
envelope: {
from: "no-reply@example.com",
to,
},
attachments,
};
try {
await sendgridMail.send(message);
} catch (err) {
console.error("Sending email failed with error", err, " message ", message);
} finally {
if (cleanupPromises) {
await Promise.all(cleanupPromises);
}
}
}

Uploading a signed video to Cloudinary: a code example

Cloudinary is a fantastic cloud service for storing, serving and transforming images and videos. However the documentation for uploading an image or video from the browser, in a secure fashion, are pretty poor. The various examples are scattered around the place, and none of them shows, in one place how to

  • Sign a request on the server
  • Use that signed request to upload a video
  • Track the progress of the video upload

I had to patch it together for myself, and thought it’d be useful for you all.

// Run in the browser
// This function takes "someId" as a parameter, as an example that you
// may want to link the video upload to some object in your database.
// This is of course totally optional.
function uploadVideo(
someId: number,
file: File,
listeners: {
onProgress: (perc: number) => void;
onComplete: (url: string) => void;
onError: (str: string) => void;
}
): () => void {
let cancelableXhr = null;
fetch("/api/signCloudinaryUpload", {
method: "POST",
cache: "no-cache",
credentials: "include",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
someId
}),
})
.then((res) => {
if (!res.ok) {
listeners.onError("Permission to upload denied");
} else {
return res.json();
}
})
.then((signatureInfo) => {
cancelableXhr = runUpload(
signatureInfo.cloud_name,
signatureInfo.api_key,
signatureInfo.signature,
signatureInfo.public_id,
signatureInfo.timestamp
);
});
function runUpload(cloudName, apiKey, signature, publicId, timestamp) {
const url = `https://api.cloudinary.com/v1_1/${cloudName}/upload`;
const xhr = new XMLHttpRequest();
const fd = new FormData();
xhr.open("POST", url, true);
xhr.setRequestHeader("X-Requested-With", "XMLHttpRequest");
listeners.onProgress(0);
// Update progress (can be used to show progress indicator)
xhr.upload.addEventListener("progress", function(e) {
const progress = Math.round((e.loaded * 100.0) / e.total);
listeners.onProgress(progress);
console.log(
`fileuploadprogress data.loaded: ${e.loaded}, data.total: ${e.total}`
);
});
xhr.onreadystatechange = function(e) {
if (xhr.readyState == 4 && xhr.status == 200) {
// File uploaded successfully
const response = JSON.parse(xhr.responseText);
console.log("response", response);
// Create a thumbnail of the uploaded image, with 150px width
listeners.onComplete(response.secure_url);
}
};
fd.append("api_key", apiKey);
fd.append("public_id", publicId);
fd.append("timestamp", timestamp);
fd.append("signature", signature);
fd.append("file", file);
xhr.send(fd);
}
return () => {
cancelableXhr && cancelableXhr.abort();
};
}
// Run on the server
import { v2 as cloudinary } from "cloudinary";
cloudinary.config({
cloud_name: "", // Your cloud name
api_key: "", // your api key
api_secret: "", // your api secret
});
function signCloudinaryRequest(publicId: string) {
const timestamp = Math.round(new Date().getTime() / 1000);
const apiSecret = (cloudinary.config("api_secret") as any) as string;
const signature = cloudinary.utils.api_sign_request(
{
timestamp,
public_id: publicId,
},
apiSecret
);
return {
api_key: (cloudinary.config("api_key") as any) as string,
signature,
cloud_name: (cloudinary.config("cloud_name") as any) as string,
timestamp,
};
}
function apiHandler(request, response) {
// This assumes that you have a bodyParser set up
const someId = request.body.someId;
const publicId = `${someId}/video`; // use whatever path you like
const signatureInfo = signCloudinaryRequest(publicId);
response.status(200);
response.json({
api_key: signatureInfo.api_key,
cloud_name: signatureInfo.cloud_name,
public_id: publicId,
signature: signatureInfo.signature,
timestamp: signatureInfo.timestamp,
});
response.end();
}

The story of how Google could have killed Facebook with the flick of a switch

As we near the end of this decade, and more importantly the end of the hell that was 2020, I realised two things. First, I survived catching Covid-19, and secondly that I had a good story about the history of Silicon Valley that I’d never written down. I’m still functional, so here, for perpetuity, is my tale.

Back in 2013 I was working in the Ads Interfaces organisation at Facebook, building mostly front end products (other people did the AI, database etc). We had an application called Power Editor which was the kitchen sink of products, and 25% of all Facebook revenue depended on it working. Every single thing you could do with ads on Facebook was supported in Power Editor (we called it P.E. for short). This made it huge, hugely complex and pretty user hostile. However our big spending customers were forced to use it as it was the only way they could efficiently scale – PE had a lot of cool tools for duplication, permuting your ads and working in batches of thousands of changes.

By 2013, PE was creaking under its own weight (about 150k lines of front end JavaScript), and no one wanted to work on it. We had half of one (awesome) engineer supporting it, and she could just about keep it working. I was looking for an opportunity to become a manager, and my manager Brian and I decided that building a team to properly support PE would be a good idea.

I looked into the code base, and was shocked to find that the entire application depended on a technology called WebSQL. It only ran on Chrome, and Google had deprecated WebSQL over a year earlier. I kind of flopped back in my chair, dragged my manager to a room, and told him that Google could shut off 25% of Facebook’s revenue, and lose us all our large accounts, by turning off WebSQL in Chrome, and it could happen any time.

This became a closely held secret in Facebook Ads leadership. We didn’t want to take any chance that word our this vulnerability could get back to Google. They had already deprecated WebSQL, and other browsers had removed it. They would have been well within their rights to just flip a feature flag in Chrome and do the same.

There was a whispered joke among those who knew about it that Google never had to bother building Google +, they could shut us down by changing one boolean in a database.

We quickly put in place a team of 5 engineers and one PM to work on it, with me managing it and coding probably 75% of the time. The plan was fairly complex. There was no easy way to get us off of WebSQL without a full rewrite of 150k lines of JavaScript. We couldn’t just build a new application from the ground up, that would take years to support all the features, and Google could turn off WebSQL at any time. Also, PE was falling apart – it was blocking the entire company from shipping any new ads products at all.

So, we decided to first make it not fall apart with a lot of performance and reliability work. Next, we had a long running project called PE Live, where we took each subset of code and made it read from the live API rather than locally from the WebSQL database. To unblock the other teams we would rewrite the whole thing in ReactJS, whereas it was then built using two frameworks that we had sunset called UkiJS and BoltJS.

This whole process took over three years. By the time it was complete in 2016, we had improved Power Editor so much, with better features, more stability, speed and ease of development for partner teams, that over 50% of all Facebook’s revenue was spent through it. The team grew to 13 engineers, with lots of help from dozens more across the Ads organisation building new APIs and infrastructure to support our work.

Google could have killed it at any time, and there were no complete alternatives for our customers – some third party applications existed, but they were even buggier than PE, were always late with new features (we didn’t have to wait for a new API to be public, they did), and often each specialised in a subset of the features. 50% of our revenue disappearing over night could have happened. That it didn’t and that we moved mountains of code doing the horrible, inglorious work of rewriting hundreds of thousands of lines of spaghetti code in production while people used the product, while also building an infinitely better product, is the most satisfying period of professional work I’ve ever been fortunate enough to experience.

A huge thanks to all the amazing people I worked with on that crazy project, you’re the best team I ever worked with, and I’d hop into the trenches with you again any time!

Power Editor as I left it in 2016, after 6.5 years at Facebook.

[Edit: So this ended up on the front page of Hacker News, and there’s much more conversation about it over at https://news.ycombinator.com/item?id=26086056 ]

Specifying a default form submit button that works with Safari

When you have a HTML form with multiple submit buttons, if the user hits their Enter button, it’ll submit the form and pretend that the first submit button in the form was clicked. Given that the button can have name and value attributes, the server can use this information. For example:

<button name="command" value="save">Save</button>
<button name="command" value="delete">Delete</button>

Unfortunately it’s not always possible to put the button you want to be default first in the DOM. A common hack [1] is to use CSS to either float or absolutely position the default button to appear where you want it. In this case you can put a hidden button at the top of the form that is a duplicate of the one visible to the user, e.g.

<form>
  <div style="display:none">
    <button name="command" value="save">This is a duplicate save button</button>
  </div>

  <button name="command" value="delete">Delete</button>
  <button name="command" value="save">Save</button>
</form>

This works well, except in mobile Safari. It seems that because the duplicate button is hidden, the browser ignores it when the user hits Enter. To fix this, instead of using style=”display: none”, absolutely position it off the screen, e.g.

<form>
  <div style="position: absolute; top: -10000px; left: -10000px;">
    <button name="command" value="save">This is a duplicate save button</button>
  </div>

  <button name="command" value="delete">Delete</button>
  <button name="command" value="save">Save</button>
</form>

And there you have it, a nasty hack to work around the fact there is no way to explicitly specify the default button.

Firebase strips all request cookies

A painfully wasteful discovery today was that, when you are using Firebase Functions to serve your code, it strips all cookies from the request object.  The only hack around this is to use a specially named cookie, “__session” and fit any data you need into that.

For example, in the browser

document.cookie = “__session=some_value”;

and on the server

import {parse} from “cookie-parse”;

exports.myFunction = functions.https.onRequest((req, res) => {
const cookie = req.headers.cookie;
const sessionValue = parse(cookie)[“__session”];
console.log(“Should be some_value”, sessionValue);
});

sms-splitter – Intelligently split SMS messages

At Promise Engineering, we’ve open sourced our code that does intelligent SMS message splitting.  It can do useful things like ensuring messages are split on a space character, and that template tokens are never divided across messages (e.g. an address).

Get it on NPM at https://www.npmjs.com/package/sms-splitter

See the source code at https://github.com/PromiseNetwork/sms-splitter

Read more about it at https://medium.com/promise-eng/intelligently-split-sms-messages-with-sms-splitter-4f3c4d0cc4ec

 

Lightweight authentication and upload to Google Cloud

Authentication to the Google Cloud, as well as uploading to it, is quite simple when using Google’s Cloud services SDK.  A big downside of using these with NodeJS is that they bring 80MB of dependencies, which if you are packaging your application for distribution is a little bit crazy.

Authentication to Google Cloud’s HTTP JSON API (link) can be a bit tricky in Node, as you need to construct your own authentication token, cryptographically sign it, and make sure you’re using all the right data, serialized just so.  I’ve written a small tool, linked above, to do this authentication for you.

Uploading to Google Cloud’s Storage uses the authentication token, and takes a few other parameters that are easy to get wrong.  The tool also does the upload of a simple file to a Cloud Storage bucket that you have the private key for a service account.

Using the code is simple (see the README)

Screen Shot 2018-08-25 at 8.19.40 AM

Please let me know if you’d like any additions or find bugs in the issues tracker.