Uploading a signed video to Cloudinary: a code example

Cloudinary is a fantastic cloud service for storing, serving and transforming images and videos. However the documentation for uploading an image or video from the browser, in a secure fashion, are pretty poor. The various examples are scattered around the place, and none of them shows, in one place how to

  • Sign a request on the server
  • Use that signed request to upload a video
  • Track the progress of the video upload

I had to patch it together for myself, and thought it’d be useful for you all.

// Run in the browser
// This function takes "someId" as a parameter, as an example that you
// may want to link the video upload to some object in your database.
// This is of course totally optional.
function uploadVideo(
someId: number,
file: File,
listeners: {
onProgress: (perc: number) => void;
onComplete: (url: string) => void;
onError: (str: string) => void;
}
): () => void {
let cancelableXhr = null;
fetch("/api/signCloudinaryUpload", {
method: "POST",
cache: "no-cache",
credentials: "include",
headers: {
"Content-Type": "application/json",
},
body: JSON.stringify({
someId
}),
})
.then((res) => {
if (!res.ok) {
listeners.onError("Permission to upload denied");
} else {
return res.json();
}
})
.then((signatureInfo) => {
cancelableXhr = runUpload(
signatureInfo.cloud_name,
signatureInfo.api_key,
signatureInfo.signature,
signatureInfo.public_id,
signatureInfo.timestamp
);
});
function runUpload(cloudName, apiKey, signature, publicId, timestamp) {
const url = `https://api.cloudinary.com/v1_1/${cloudName}/upload`;
const xhr = new XMLHttpRequest();
const fd = new FormData();
xhr.open("POST", url, true);
xhr.setRequestHeader("X-Requested-With", "XMLHttpRequest");
listeners.onProgress(0);
// Update progress (can be used to show progress indicator)
xhr.upload.addEventListener("progress", function(e) {
const progress = Math.round((e.loaded * 100.0) / e.total);
listeners.onProgress(progress);
console.log(
`fileuploadprogress data.loaded: ${e.loaded}, data.total: ${e.total}`
);
});
xhr.onreadystatechange = function(e) {
if (xhr.readyState == 4 && xhr.status == 200) {
// File uploaded successfully
const response = JSON.parse(xhr.responseText);
console.log("response", response);
// Create a thumbnail of the uploaded image, with 150px width
listeners.onComplete(response.secure_url);
}
};
fd.append("api_key", apiKey);
fd.append("public_id", publicId);
fd.append("timestamp", timestamp);
fd.append("signature", signature);
fd.append("file", file);
xhr.send(fd);
}
return () => {
cancelableXhr && cancelableXhr.abort();
};
}
// Run on the server
import { v2 as cloudinary } from "cloudinary";
cloudinary.config({
cloud_name: "", // Your cloud name
api_key: "", // your api key
api_secret: "", // your api secret
});
function signCloudinaryRequest(publicId: string) {
const timestamp = Math.round(new Date().getTime() / 1000);
const apiSecret = (cloudinary.config("api_secret") as any) as string;
const signature = cloudinary.utils.api_sign_request(
{
timestamp,
public_id: publicId,
},
apiSecret
);
return {
api_key: (cloudinary.config("api_key") as any) as string,
signature,
cloud_name: (cloudinary.config("cloud_name") as any) as string,
timestamp,
};
}
function apiHandler(request, response) {
// This assumes that you have a bodyParser set up
const someId = request.body.someId;
const publicId = `${someId}/video`; // use whatever path you like
const signatureInfo = signCloudinaryRequest(publicId);
response.status(200);
response.json({
api_key: signatureInfo.api_key,
cloud_name: signatureInfo.cloud_name,
public_id: publicId,
signature: signatureInfo.signature,
timestamp: signatureInfo.timestamp,
});
response.end();
}

The story of how Google could have killed Facebook with the flick of a switch

As we near the end of this decade, and more importantly the end of the hell that was 2020, I realised two things. First, I survived catching Covid-19, and secondly that I had a good story about the history of Silicon Valley that I’d never written down. I’m still functional, so here, for perpetuity, is my tale.

Back in 2013 I was working in the Ads Interfaces organisation at Facebook, building mostly front end products (other people did the AI, database etc). We had an application called Power Editor which was the kitchen sink of products, and 25% of all Facebook revenue depended on it working. Every single thing you could do with ads on Facebook was supported in Power Editor (we called it P.E. for short). This made it huge, hugely complex and pretty user hostile. However our big spending customers were forced to use it as it was the only way they could efficiently scale – PE had a lot of cool tools for duplication, permuting your ads and working in batches of thousands of changes.

By 2013, PE was creaking under its own weight (about 150k lines of front end JavaScript), and no one wanted to work on it. We had half of one (awesome) engineer supporting it, and she could just about keep it working. I was looking for an opportunity to become a manager, and my manager Brian and I decided that building a team to properly support PE would be a good idea.

I looked into the code base, and was shocked to find that the entire application depended on a technology called WebSQL. It only ran on Chrome, and Google had deprecated WebSQL over a year earlier. I kind of flopped back in my chair, dragged my manager to a room, and told him that Google could shut off 25% of Facebook’s revenue, and lose us all our large accounts, by turning off WebSQL in Chrome, and it could happen any time.

This became a closely held secret in Facebook Ads leadership. We didn’t want to take any chance that word our this vulnerability could get back to Google. They had already deprecated WebSQL, and other browsers had removed it. They would have been well within their rights to just flip a feature flag in Chrome and do the same.

There was a whispered joke among those who knew about it that Google never had to bother building Google +, they could shut us down by changing one boolean in a database.

We quickly put in place a team of 5 engineers and one PM to work on it, with me managing it and coding probably 75% of the time. The plan was fairly complex. There was no easy way to get us off of WebSQL without a full rewrite of 150k lines of JavaScript. We couldn’t just build a new application from the ground up, that would take years to support all the features, and Google could turn off WebSQL at any time. Also, PE was falling apart – it was blocking the entire company from shipping any new ads products at all.

So, we decided to first make it not fall apart with a lot of performance and reliability work. Next, we had a long running project called PE Live, where we took each subset of code and made it read from the live API rather than locally from the WebSQL database. To unblock the other teams we would rewrite the whole thing in ReactJS, whereas it was then built using two frameworks that we had sunset called UkiJS and BoltJS.

This whole process took over three years. By the time it was complete in 2016, we had improved Power Editor so much, with better features, more stability, speed and ease of development for partner teams, that over 50% of all Facebook’s revenue was spent through it. The team grew to 13 engineers, with lots of help from dozens more across the Ads organisation building new APIs and infrastructure to support our work.

Google could have killed it at any time, and there were no complete alternatives for our customers – some third party applications existed, but they were even buggier than PE, were always late with new features (we didn’t have to wait for a new API to be public, they did), and often each specialised in a subset of the features. 50% of our revenue disappearing over night could have happened. That it didn’t and that we moved mountains of code doing the horrible, inglorious work of rewriting hundreds of thousands of lines of spaghetti code in production while people used the product, while also building an infinitely better product, is the most satisfying period of professional work I’ve ever been fortunate enough to experience.

A huge thanks to all the amazing people I worked with on that crazy project, you’re the best team I ever worked with, and I’d hop into the trenches with you again any time!

Power Editor as I left it in 2016, after 6.5 years at Facebook.

[Edit: So this ended up on the front page of Hacker News, and there’s much more conversation about it over at https://news.ycombinator.com/item?id=26086056 ]

Specifying a default form submit button that works with Safari

When you have a HTML form with multiple submit buttons, if the user hits their Enter button, it’ll submit the form and pretend that the first submit button in the form was clicked. Given that the button can have name and value attributes, the server can use this information. For example:

<button name="command" value="save">Save</button>
<button name="command" value="delete">Delete</button>

Unfortunately it’s not always possible to put the button you want to be default first in the DOM. A common hack [1] is to use CSS to either float or absolutely position the default button to appear where you want it. In this case you can put a hidden button at the top of the form that is a duplicate of the one visible to the user, e.g.

<form>
  <div style="display:none">
    <button name="command" value="save">This is a duplicate save button</button>
  </div>

  <button name="command" value="delete">Delete</button>
  <button name="command" value="save">Save</button>
</form>

This works well, except in mobile Safari. It seems that because the duplicate button is hidden, the browser ignores it when the user hits Enter. To fix this, instead of using style=”display: none”, absolutely position it off the screen, e.g.

<form>
  <div style="position: absolute; top: -10000px; left: -10000px;">
    <button name="command" value="save">This is a duplicate save button</button>
  </div>

  <button name="command" value="delete">Delete</button>
  <button name="command" value="save">Save</button>
</form>

And there you have it, a nasty hack to work around the fact there is no way to explicitly specify the default button.

Firebase strips all request cookies

A painfully wasteful discovery today was that, when you are using Firebase Functions to serve your code, it strips all cookies from the request object.  The only hack around this is to use a specially named cookie, “__session” and fit any data you need into that.

For example, in the browser

document.cookie = “__session=some_value”;

and on the server

import {parse} from “cookie-parse”;

exports.myFunction = functions.https.onRequest((req, res) => {
const cookie = req.headers.cookie;
const sessionValue = parse(cookie)[“__session”];
console.log(“Should be some_value”, sessionValue);
});

sms-splitter – Intelligently split SMS messages

At Promise Engineering, we’ve open sourced our code that does intelligent SMS message splitting.  It can do useful things like ensuring messages are split on a space character, and that template tokens are never divided across messages (e.g. an address).

Get it on NPM at https://www.npmjs.com/package/sms-splitter

See the source code at https://github.com/PromiseNetwork/sms-splitter

Read more about it at https://medium.com/promise-eng/intelligently-split-sms-messages-with-sms-splitter-4f3c4d0cc4ec

 

Lightweight authentication and upload to Google Cloud

Authentication to the Google Cloud, as well as uploading to it, is quite simple when using Google’s Cloud services SDK.  A big downside of using these with NodeJS is that they bring 80MB of dependencies, which if you are packaging your application for distribution is a little bit crazy.

Authentication to Google Cloud’s HTTP JSON API (link) can be a bit tricky in Node, as you need to construct your own authentication token, cryptographically sign it, and make sure you’re using all the right data, serialized just so.  I’ve written a small tool, linked above, to do this authentication for you.

Uploading to Google Cloud’s Storage uses the authentication token, and takes a few other parameters that are easy to get wrong.  The tool also does the upload of a simple file to a Cloud Storage bucket that you have the private key for a service account.

Using the code is simple (see the README)

Screen Shot 2018-08-25 at 8.19.40 AM

Please let me know if you’d like any additions or find bugs in the issues tracker.

Useful git shortcuts

Over the years I’ve found myself following a near identical workflow with Git, and I finally got around to making shortcuts for these common tasks.

You can find these all in a Github repo at https://github.com/shaneosullivan/git-shortcuts, where it makes it quick and easy to work your way through many modified files, `git diff` them, `git add` them, `git add` many files with similar names (like images) and more.

Pull requests welcome, what do you find useful and why?

Using React types with Flow

A little tip for using React types, such as React.Node or React.Element with Flow.  If you import React into a component like

import React from ‘react’;

then Flow will give you the error

Cannot get React.Node because property Node is missing in object type

The fix is to import React as follows

import * as React from ‘react’;