I create videos that I don't want to upload to Youtube or Vimeo. There's nothing sketchy about my videos; I just don't want to share my family videos with the world, nor do I need Google's machine learning dissecting my kids into advertising data points. So, I want to self host.

Also, I want to easily share these videos with my extended family. And that needs to be easy for them. Not everyone in my family gets excited by technical challenges. So, I need some simple authentication layer. Finally, I really want my video hosting site to render differently for authenticated and unauthenticated users. This is because I want Open Graph meta headers to give information about a page on the site, even if the page does not render restricted content to unauthenticated users. In other words, I want links to my content that are posted in social media and messaging apps to be descriptive and interesting. Like so:

Finally, I wanted to ditch the traditional server approach that I have used in the past and try to leverage AWS services to build my video hosting site serverless.

The Plan

• Use Gatsby to generate static pages for the site. I like React. I sorta like GraphQl. Gatsby seemed like a useful thing to learn.
• Use AWS Cognito to manage users and to allow users to login with Facebook.
• Somehow manage a list of allowed users. Anyone could attempt to login with Facebook, but only those users on the list would actually succeed.
• Use client-side security to show unauthenticated users a restricted version of the site. This is obviously not secure, so...
• Restrict all access to the actual videos to authenticated users. Even if a malicious user circumvented the javascript, they still wouldn't be able to access the stuff that I actually want to keep private.

The Plan

This proved to be possible with AWS and also quite a bit more complicated than I imagined. As I worked through the problem I discovered that AWS is both over-documented and under-documented. AWS services are always completely documented with respect to available API endpoints, parameters, allowed values, etc. But it is often the purpose of a service, or how it is intended to be incorporated into larger projects that is missing. I think AWS documentation is teleologically under described. ;)

But I eventually figured it out. I figured out the difference between AWS Cognito User Pools and Identity Pools. I figured out why one should secure content with signed cookies rather than with IAM permissions. I figured out how to use Cognito's various tokens. I figured out the separation of concerns between API Gateway, Lambda, Cognito, and CloudFront. And I figured out how to manage it all with CloudFormation - a good step for learning how to operate with infrastructure as code. You can see the whole project in GitHub.

I configured my Cognito User Pool to allow sign ups and logins from a 3rd party identity provider: Facebook. Additionally, because you can configure your user pool to connect to a Lambda function on certain triggers, I was able to use a Lambda function to ensure that the user is on my allow list.

My Gatsby content is stored in S3 and distributed through a CloudFront distribution. This distribution is public. I created a second CloudFront distribution to serve the video content. This distribution requires that requests include signed cookies. The user obtains those signed cookies by making a request to an API Gateway endpoint. That request must include an Authorization header containing the access token from Cognito. The API Gateway endpoint integrates with a Lambda function that creates the signed cookies.

And the whole thing is described in a CloudFormation template.  The template includes 31 AWS resources! Many of these are access-related. Most resources need to be explicitly given access to the other resources with which they communicate. Access is granted by creating more resources.

I also used a handful of scripts to orchestrate creating the CloudFormation stack from the template and tearing it down. One particularly important function of the setup script is creating a key pair with OpenSSL, passing the private key to the signed cookie lambda function and the public key to the video CloudFront distribution.

Here are a couple diagrams:

Resource Diagram

UML Sequence Diagram

Summary

This was definitely not the simplest way to rig up a private video hosting site. But it worked and I learned a lot about Gatsby, AWS, and infrastructure as code.

The following CloudFormation template creates resources that automate the encoding of video. It creates a stack that provides the following workflow:

1. User uploads video file to the /originals directory of a S3 bucket.
2. A lambda function is notified of the newly uploaded file and starts an AWS MediaConvert job to encode the video in both Dash and HLS.
3. The resulting encoded files are stored in the /assets directory of the same S3 bucket.

More detailed explanation following the template.

I was calling this setup "transcodekit" as I worked on it. Feel free to ignore that label wherever you see it. :)

AWSTemplateFormatVersion: 2010-09-09

Parameters:
  BucketName:
    Type: String
    Default: 'transcodekit'

Resources:
  TranscodekitLambdaPolicy:
    Type: 'AWS::IAM::ManagedPolicy'
    Properties:
      Description: Provides necessary access to MediaConvert and CloudWatch logs
      ManagedPolicyName: !Join
        - '-'
        - - !Ref 'AWS::Region'
          - TranscodekitLambdaExecutor
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - mediaconvert:CreateJob
              - mediaconvert:DescribeEndpoints
            Resource:
              - '*'
          - Effect: Allow
            Action:
              - 'logs:CreateLogGroup'
              - 'logs:CreateLogStream'
              - 'logs:PutLogEvents'
            Resource:
              - '*'
          - Effect: Allow
            Action:
              - 'iam:PassRole'
            Resource:
              - !GetAtt MediaConvertRole.Arn

  LambdaExecutionRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: LambdaExecution
      Description: Allows Transcodekit lambda function to start MediaConvert job
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      ManagedPolicyArns:
        - !Ref TranscodekitLambdaPolicy

  TranscodeVideoFunction:
    Type: AWS::Lambda::Function
    Properties:
      FunctionName: TranscodeVideo
      Description: Sends uploaded S3 object to MediaConvert for transcoding
      Handler: index.handler
      Role: !GetAtt LambdaExecutionRole.Arn
      Environment:
        Variables:
          ROLE: !GetAtt MediaConvertRole.Arn
      Code:
        ZipFile: |
          const path = require('path')
          const MediaConvert = require('aws-sdk/clients/mediaconvert')

          exports.handler = async function(event, context, cb) {
            const mediaConvert = new MediaConvert({
              apiVersion: '2017-08-29'
            })
            const s3Record = event.Records[0].s3
            const { base: fileName, name: title } = path.parse(s3Record.object.key)

            try {
              const { Endpoints: [{ Url: endpoint }]} = await mediaConvert.describeEndpoints().promise()
              mediaConvert.endpoint = endpoint

              const hlsResponse = await mediaConvert.createJob({
                Role: process.env.ROLE,
                JobTemplate: 'System-Ott_Hls_Ts_Avc_Aac',
                Settings: {
                  Inputs: [{
                    FileInput: `s3://${s3Record.bucket.name}/${s3Record.object.key}`,
                    AudioSelectors: {
                      'Audio Selector 1': {
                        Offset: 0
                      }
                    },
                  }],
                  OutputGroups: [{
                    OutputGroupSettings: {
                      Type: 'HLS_GROUP_SETTINGS',
                      HlsGroupSettings: {
                        Destination: `s3://${s3Record.bucket.name}/assets/${title}/hls/`
                      }
                    }
                  }]
                },
              }).promise()

              const dashResponse = await mediaConvert.createJob({
                Role: process.env.ROLE,
                JobTemplate: 'System-Ott_Dash_Mp4_Avc_Aac',
                Settings: {
                  Inputs: [{
                    FileInput: `s3://${s3Record.bucket.name}/${s3Record.object.key}`,
                    AudioSelectors: {
                      'Audio Selector 1': {
                        Offset: 0
                      }
                    },
                  }],
                  OutputGroups: [{
                    OutputGroupSettings: {
                      Type: 'DASH_ISO_GROUP_SETTINGS',
                      DashIsoGroupSettings: {
                        Destination: `s3://${s3Record.bucket.name}/assets/${title}/dash/`
                      }
                    }
                  }]
                },
              }).promise()

              cb(null, [hlsResponse, dashResponse])
            } catch (e) {
              cb(e.message)
            }
          }
      Runtime: nodejs12.x

  S3Bucket:
    Type: 'AWS::S3::Bucket'
    DependsOn:
      - S3ExecutionPermission
    Properties:
      BucketName: !Ref BucketName
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: 's3:ObjectCreated:*'
            Function: !GetAtt TranscodeVideoFunction.Arn
            Filter:
              S3Key:
                Rules:
                  - Name: prefix
                    Value: 'originals/'

  S3ExecutionPermission:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !GetAtt TranscodeVideoFunction.Arn
      Action: lambda:InvokeFunction
      Principal: s3.amazonaws.com
      SourceAccount: !Ref 'AWS::AccountId'
      SourceArn: !Sub 'arn:aws:s3:::${BucketName}'

  TranscodekitMediaConvertPolicy:
    Type: 'AWS::IAM::ManagedPolicy'
    Properties:
      ManagedPolicyName: !Join
        - '-'
        - - !Ref 'AWS::Region'
          - TranscodekitMediaConverter
      Description: Provides access to S3 for MediaConvert transcode jobs
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Action:
              - 's3:PutObject'
            Resource:
              - !Sub 'arn:aws:s3:::${BucketName}/assets/*'
          - Effect: Allow
            Action:
              - 's3:GetObject'
            Resource:
              - !Sub 'arn:aws:s3:::${BucketName}/originals/*'

  MediaConvertRole:
    Type: 'AWS::IAM::Role'
    Properties:
      RoleName: MediaConvertExecution
      Description: Allows MediaConvert to gain access to S3
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - mediaconvert.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      ManagedPolicyArns:
        - !Ref TranscodekitMediaConvertPolicy

Getting the video from S3 to MediaConvert

At the time of writing, S3 does not have a notification that will send an uploaded file directly to MediaConvert. So, we'll have to use a Lambda function. S3 will notify our Lambda function of the file and Lambda will kick off the MediaConvert encoding jobs.

This general setup is relatively simple. The tricky part was the required IAM roles and managed policies required to give the different resources permission to interact. The Lambda function has its own role, but must pass a different role to the MediaConvert jobs it initiates. This passed role gives the MediaConvert jobs permission to do what they need to do: read from and write to S3.

Encoding the Video

We use the System-Ott_Hls_Ts_Avc_Aac and System-Ott_Dash_Mp4_Avc_Aac job templates to simplify configuring the encoding settings. These templates produce HLS and Dash encoded videos respectively in a range of resolutions and bitrates.

The jobs themselves are instructed where to read the original files in S3 and where to write the encoded assets when they are done.

Cost

For a 5 minute video, encoding in both HLS and Dash in the variety of resolutions and bitrates provided by the aforementioned MediaConvert templates costs $2-$3.

Happy encoding!

Testing for React Native components is broken

This week, I generated a react-native app with react-native v2.0.1. It went something like this:

The resulting Jest-based testing setup does not function.

TL;DR

You can get a nice Mocha testing setup working for React Native using a custom Mocha config and react-native-mock-render. See below for the modules you'll need to add and the configuration.

The problem

Using react-native-cli v2.0.1, I ran react-native init testproj to generate a clean React Native project. It spit out a project directory with React Native v0.57.3, Jest v23.6.0, and metro-react-native-babel-preset v0.48.1. I then added a simple test called App.test.js like so:

describe('a test test', () => {
  it('is true', () => {
    expect(true).toBe(true);
  });
});

However, running yarn test results in:

$ jest
 FAIL  ./App.test.js
  ● Test suite failed to run

    Couldn't find preset "module:metro-react-native-babel-preset" relative to directory "/Users/bgladwell/repos/testproj"

      at node_modules/babel-core/lib/transformation/file/options/option-manager.js:293:19
          at Array.map (<anonymous>)
      at OptionManager.resolvePresets (node_modules/babel-core/lib/transformation/file/options/option-manager.js:275:20)
      at OptionManager.mergePresets (node_modules/babel-core/lib/transformation/file/options/option-manager.js:264:10)
      at OptionManager.mergeOptions (node_modules/babel-core/lib/transformation/file/options/option-manager.js:249:14)
      at OptionManager.init (node_modules/babel-core/lib/transformation/file/options/option-manager.js:368:12)
      at File.initOptions (node_modules/babel-core/lib/transformation/file/index.js:212:65)
      at new File (node_modules/babel-core/lib/transformation/file/index.js:135:24)
      at Pipeline.transform (node_modules/babel-core/lib/transformation/pipeline.js:46:16)

Test Suites: 1 failed, 1 total
Tests:       0 total
Snapshots:   0 total
Time:        0.259s, estimated 1s
Ran all test suites.
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

There is some problem with how Jest and Babel are (not) working together. In this case, Jest's out-of-the-box configuration is working against us. Dependencies for Jest and Metro are apparently causing some conflict between Babel 6 and Babel 7, but it's quite difficult to isolate which packages are causing the conflict.

There are open issues for React Native and Metro, but no definitive solutions.

A proposal in both issue threads is to change the babel preset used in .babelrc from metro-react-native-babel-preset to react-native. It doesn't surprise me that this works for some people in some situations, but I don't think it's a good idea. Here's why:

The react-native preset is short for babel-preset-react-native. As you might guess, this is a Babel preset package for working with react native code. What is less obvious is that babel-preset-react-native depends on Babel 6 and has been superseded by metro-react-native-babel-preset, which depends on Babel 7. So to regress back to babel-preset-react-native means forcing our project to continue to depend on Babel 6, something that the react-native maintainers have decided against. Also, we have to worry about more than Jest here; Metro (the React Native bundler) depends on Babel 7. It seems unwise to have our tests compiled with Babel 6 while our development and production code is compiled by Babel 7.

Not really sure how to fix Jest

There may be some way to resolve this issue using Jest. I'm just not sure how. And neither does anyone else, judging by the React Native and Metro issues.

But, I am able to get tests working using Mocha. Here's how:

Mocha

Let's add Mocha and Enzyme to our project so that we can test React components. We also add Chai for assertions. Of course, we'll have to add the Enzyme React 16 adapter. We will configure Enzyme to use the adapter below.

yarn add -D mocha chai enzyme enzyme enzyme-adapter-react-16

Next, update App.test.js so that it does something sorta like a real component test:

import React from 'react';
import Enzyme, { shallow } from 'enzyme';
import Adapter from 'enzyme-adapter-react-16';
import { Text } from 'react-native';
import { expect } from 'chai';

import App from './App';

Enzyme.configure({ adapter: new Adapter() });

describe('<App>', () => {
  let wrapper;

  beforeEach(() => {
    wrapper = shallow(<App />);
  });
  it('renders default Text', () => {
    expect(wrapper.find(Text)).to.have.lengthOf(3);
  });
});

Our test assumes that App.js renders a simple component with three <Text> component children, which is how react-native init generated it.

Ok, let's try to run our test:

node_modules/.bin/mocha App.test.js

No dice. We get this error:

...
App.test.js:1     
(function (exports, require, module, __filename, __dirname) { import { shallow } from 'enzyme';      
                                                              ^^^^^^                                 

SyntaxError: Unexpected token import
...

That makes sense. We need Babel to compile those import statements, but Babel is not yet involved. Let's change that:

If you check Babel's documentation for how to use Babel with Mocha, it says to do this:

mocha --require babel-register

The --require option loads a npm module at runtime - in this case it is telling Mocha to compile everything with Babel. But this is incorrect in our case! These instruction are out of date. babel-register is the Babel 6 version of what we want to do. We actually need to do:

mocha --require @babel/register

@babel/register is the Babel 7 version. Let's try it:

node_modules/.bin/mocha --require @babel/register  App.test.js

Oh no what's this? Another error:

module.js:491                                     
    throw err;                                    
    ^                                             

Error: Cannot find module 'Platform'
...

Mocking native modules

This error is telling us that that Node is not able to load Platform. This is because Platform is a native module provided by React Native - i.e. a non-JavaScript module written in Objective-C or Java. Happily, this problem is also solvable by mocking the native modules. If you Google around, most solutions will direct you to a package called react-native-mock. I'm sure this package worked at some point, but it is no longer maintained, and it does not work with React 16.

However, the good people at Root Insurance have forked the project and improved it. Lets use their module: react-native-mock-render.

yarn add -D react-native-mock-render

Now what happens if we also load react-native-mock-render with our tests?

node_modules/.bin/mocha --require @babel/register --require react-native-mock-render/mock App.test.js

OMG! IT WORKED!

Compiling node_modules

If the above is working for you, you're good to go. However, it wasn't enough for my project.

Let's add a fairly large dependency to our project: Native Base. This will bring in a host of other modules.

yarn add native-base

Now add a Native Base component to the App.js file that was generated by react-native init. It should look something like this:

import React, {Component} from 'react';           
import {Platform, StyleSheet, Text, View} from 'react-native';                                       
import { Spinner } from 'native-base';            

const instructions = Platform.select({            
  ios: 'Press Cmd+R to reload,\n' + 'Cmd+D or shake for dev menu',                                   
  android:                                        
    'Double tap R on your keyboard to reload,\n' +                                                   
    'Shake or press menu button for dev menu',    
});                                               

type Props = {};                                  
export default class App extends Component<Props> {                                                  
  render() {                                      
    return (                                      
      <View style={styles.container}>             
        <Spinner />                               
        <Text style={styles.welcome}>Welcome to React Native!</Text>                                 
        <Text style={styles.instructions}>To get started, edit App.js</Text>                         
        <Text style={styles.instructions}>{instructions}</Text>                                      
      </View>                                     
    );                                            
  }                                               
}                                                 

const styles = StyleSheet.create({                
  container: {                                    
    flex: 1,                                      
    justifyContent: 'center',                     
    alignItems: 'center',                         
    backgroundColor: '#F5FCFF',                   
  },                                              
  welcome: {                                      
    fontSize: 20,                                 
    textAlign: 'center',                          
    margin: 10,                                   
  },                                              
  instructions: {                                 
    textAlign: 'center',                          
    color: '#333333',                             
    marginBottom: 5,                              
  },                                              
});

All we added was
import { Spinner } from 'native-base';

and
<Spinner />

Now what happens when we run our test?

node_modules/.bin/mocha --require @babel/register --require react-native-mock-render/mock App.test.js

New errors!

...
node_modules/native-base-shoutem-theme/index.js:1                    
(function (exports, require, module, __filename, __dirname) { import connectStyle from "./src/connectStyle";                                                                                               
                                                              ^^^^^^                                 
                                                                                                     
SyntaxError: Unexpected token import
...

Wait, what? This looks like Babel 7 is no longer doing its job. Why is "import" an unexpected token?

The answer is that Babel 7 no longer compiles node_modules by default. This isn't necessarily bad. If you don't need to do it, compiling everything in node_modules can add a LOT of overhead to your tests.

But we apparently DO need Babel 7 to compile the modules under node_modules that ship with ES2015 assets. To make that happen, we're going to create a new Mocha configuration that we load at runtime. As a bonus, we can put some of our general test config stuff (like Enzyme's adapter config) in there too.

Create config/mocha.js

const Enzyme = require('enzyme');
const Adapter = require('enzyme-adapter-react-16');

Enzyme.configure({ adapter: new Adapter() });

require('react-native-mock-render/mock');

require('@babel/register')({
  presets: [require('metro-react-native-babel-preset')],
  ignore: [
    function (filepath) {
      const packages = [
        'native-base-shoutem-theme',
      ];
      if (packages.some(p => filepath.startsWith(`${__dirname}/node_modules/${p}`))) {
        return false;
      }
      return filepath.startsWith(`${__dirname}/node_modules`);
    },
  ],
});

As you can see, at the top of the file we configure Enzyme with the React 16 adapter, so we can remove that stuff from our test file. We also pull in react-native-mock-render, so we can now omit that from the command line when running our tests.

Then, we require @babel/register (the same module that we were --requireing on the command line` and pass it a configuration object.

presets: [require('metro-react-native-babel-preset')]

This specifies the same Babel preset that is listed in the .babelrc file generated by native-react init. As you may recall, it is the new version of babel-preset-react-native that uses Babel 7.

Next comes the ignore array. You can find documentation for how this works here. In our case, we are providing a function that returns false for anything we do want Babel to compile, true for anything it should skip.

The function says to compile anything in the packages array, skip anything else under node_modules, compile everything else. In our case we want to compile native-base-shoutem-theme, the module with the error from above.

We could simply write the ignore array as an empty array, but this would mean Babel should compile everything, including node_modules. That seems unnecessarily slow. Using the ignore function above, we can simply add the name of any module that ships code with ES2015 import semantics.

Run the new config like this:

node_modules/.bin/mocha --require config/mocha.js App.test.js

And there you go! Working React Native component tests with Mocha and Babel 7.

Summary

We changed the files and commands we used throughout this post. Here are the final versions:

The example test:

import React from 'react';
import { shallow } from 'enzyme';
import { Text } from 'react-native';
import { expect } from 'chai';

import App from './App';

describe('<App>', () => {
  let wrapper;

  beforeEach(() => {
    wrapper = shallow(<App />);
  });

  it('renders default Text', () => {
    expect(wrapper.find(Text)).to.have.lengthOf(3);
  });
});

The Mocha configuration file:

const Enzyme = require('enzyme');
const Adapter = require('enzyme-adapter-react-16');

Enzyme.configure({ adapter: new Adapter() });

require('react-native-mock-render/mock');
require('@babel/register')({
  presets: [require('metro-react-native-babel-preset')],
  ignore: [
    function (filepath) {
      const packages = [
        'native-base-shoutem-theme',
      ];
      if (packages.some(p => filepath.startsWith(`${__dirname}/node_modules/${p}`))) {
        return false;
      }
      return filepath.startsWith(`${__dirname}/node_modules`);
    },
  ],
});

We added the following modules to make all this work:

yarn add -D mocha chai enzyme enzyme enzyme-adapter-react-16 react-native-mock-render

To run a test:

node_modules/.bin/mocha --require config/mocha.js App.test.js

You can/should of course put that in a npm script and omit the node\_modules/.bin/ as well as change App.test.js to be something like **/*.test.js.

P.S.

Snapshot testing with Mocha seems to work with with snap-shot-it and enzyme-to-json. So far, I'm quite happy with Mocha as my React Native test framework.

import { lessons } from 'rails'; // part 3

Bringing techniques used in Rails to Node

Better late than never. I'm happy, if not surprised, to be learning so much from Rails in 2018.

This is the third post in a series about techniques I found in Rails that were so good, they needed to be ported to my Node projects.

Technique 3 - In tests, focus on state

This is another technique gleaned from RSpec, a BDD test framework commonly used as an alternative to Rails' default testing framework, Minitest. RSpec's let expression allows you to abstract setup code in a convenient way. I'll try to illustrate by way of example, but first...

A word of caution: this technique may not agree with your testing sensibilities

In my experience, smart people often disagree on test readability and structure. If you like your tests to be extremely readable, even at the expense of repeating yourself, this technique is probably not for you. If, on the other hand, you don't mind a bit of abstraction in your tests to make things more concise, keep reading. For me, readability is important, but I don't mind that my test files feel more like programs and less like stories. I like the balance that RSpec strikes and let is a big part of that.

A quick explanation of RSpec's let

RSpec's let is, simply put, another way of binding the result of an expression to a variable. You might see let expressions like this:

let(:avg_chicken_weight) { 1.2 }
let(:num_chickens) { 50 }
let(:total_chicken_weight) { avg_chicken_weight * num_chickens }

You can see that let expressions can reference other let expressions.

let expressions often serve similar purposes to before blocks; they contain setup code necessary to for the initial conditions of your tests. Some of the advantages of let are explained nicely in this stackoverflow answer.

The real value of let

But, in my opinion, the real value of let is that allows you to focus on the state that describes test conditions, rather than steps required to create that state.

For example, let's say we're setting up an integration test for some classes in a blog system. Specifically, we're going to test a user's ability to edit someone else's blog post when the user is a member of different groups. Users that are a member of the editors group should be able to edit other users' posts. Users that are a member of the default group should not.

We need to initialize a few different entities for the test. Using RSpec and before block semantics, we might do it like so.

describe 'User permission to edit' do
  subject { @post.edit(@user) }

  before do
    @post = create :post
    group = create :group, is_admin: false
    @user = create :user, group: group
  end

  it 'cannot edit another user's post' do
    expect{ subject }.to raise_error
  end

  context 'when user is a member of an admin group' do
    before do
      group = create :group, is_admin: true
      @user = create :user, group: group
    end

    it 'can edit another user's post' do
      expect{ subject }.not_to raise_error
    end
  end
end

Now let's refactor using let. Notice that the nested context is much cleaner and the setup code has been reduced to a single expression that communicates purpose.

describe 'User permission to edit' do
  subject { post.edit(user) }

  let(:post)   { create :post }
  let(:is_group_admin) { false }
  let(:group)  { create :group, is_admin: is_group_admin }
  let(:user) { create :user, group: group }

  it 'cannot edit another user's post' do
    expect{ subject }.to raise_error
  end

  context 'when user is a member of an admin group' do
    let(:is_group_admin) { true }

    it 'can edit another user's post' do
      expect{ subject }.not_to raise_error
    end
  end
end

With the before blocks approach, we were forced to rehearse the steps necessary to initialize this new test. But this isn't really important when trying to understand the point of the test. The point is the state: the user is now part of an admin group.

Can we do it in Javascript?

Sorta, yeah!

describe('User permission to edit', () => {
  let s;
  beforeEach(() => {
    s = new StateDefinition({
      post: create('post'),
      isGroupAdmin: false,
      group: (state) => create('group', { isAdmin: state.isGroupAdmin }),
      user: (state) => create('user', { group: state.group }),
    });
  });

  const subject = () => { s.post.edit(s.user) }

  it('cannot edit another user's post', () => {
    expect(subject).toThrow();
  });

  describe('when user is a member of an admin group', () => {
    beforeEach(() => {
      s.define({ isGroupAdmin: true });
    });

    it('can edit another user's post', () => {
      expect(subject).not.toThrow();
    });
  });
});

Clearly, we snuck in an new construct, StateDefinition. This class was designed by a colleague and I as we were attempting to reduce the cognitive overhead required to switch back and forth between Rails RSpec tests and our frontend React tests.

StateDefinition

The StateDefinition class is used in the Javascript example as a stand in for the let expressions from the Ruby example. While not as terse, it has some of the same useful properties.

• Properties defined can reference other properties in the state definition.
• Properties can be defined as functions, but are accessed as properties.
• Properties are evaluated as-needed. This means that if you redefine a state property like we do with isGroupAdmin, its value will bubble up to other properties when they are referenced in tests.
• Property values are memoized. Once evaluated, functions that define properties will not be called again. This is useful when state definitions involve stateful operations like creating a database entry or rendering a React component with Enzyme.

The StateDefinition API is pretty simple. You can pass in an object of property definitions at instantiation - definitions that can either be statements or functions. Later, you can update those definitions with the .define() method.

All properties are accessed as properties, even if they are defined as functions. This allows you to not worry about how properties were defined and helps with readability of the tests.

StateDefinition is available as a npm package. Give it a try!

Wrap it up

RSpec's let expressions are fantastic for creating more readable tests and for reducing some repeated setup code. Using StateDefinition, we can enjoy some of the same benefits in Javascript.

There is more work to do on StateDefinition. I think it could be integrated with Mocha to make usage clearer and more seamless.

import { lessons } from 'rails'; // part 1

Bringing techniques used in Rails to Node

Better late than never. I'm happy, if not surprised, to be learning so much from Rails in 2018.

This is the second post in a series about techniques I found in Rails that were so good, they needed to be ported to my Node projects.

Technique 2 - Generate test data with factories

In the RSpec Better Specs documentation, they say, "Do not use fixtures because they are difficult to control, use factories instead. Use them to reduce the verbosity on creating new data."

This is a simple argument, so here it is.

Setup code in your tests like the following isn't pleasant. Nor is it maintainable.

const user = new User({
  name: 'Joe User',
  email: 'joe@fake.com',
  birthdate: new Date('1999-03-30'),
});
const user2 = new User({
  name: 'Bill Jones',
  email: 'bill@fake.com',
  birthdate: new Date('1973-12-08'),
});
const message1 = new Message({
  content: 'This is the content of message1',
  user: user1,
});
const message2 = new Message({
  content: 'Message2 content!',
  user: user2,
});

Why manually create the boilerplate data needed to instantiate these test entities every time we need them? And each time the API changes for these entities, we'll need to update dozens of references. Thankfully, there are libraries that can help us with this.

How about code like this instead?

const user1 = casual.user;
const user2 = casual.user;
const message1 = casual.message({user: user1});
const message2 = casual.message({user: user2});

This makes for much cleaner, much more maintainable tests.

Casual

In the examples, we used casual, a fake data generator. There are other options in the Javascript ecosystem - Chance looks popular - but I like casual because it has a simple, extensible API.

For the above example, you could create casual definitions like so:

const casual = require('casual'),
  moment = require('moment');

casual.define('user', function () {
  return {
    name: casual.name,
    email: casual.email,
    birthdate: casual.date,
  };
});

casual.define('message', function (opts) {
  return Object.assign({
    content: casual.sentence,
    user: casual.user,
  }, opts);
});

You just need to make sure to put those definitions in a file that is loaded before your tests run. You will then have access to casual.user and casual.message whenever casual is required in your tests.

Wrap it up

So that's it. A pattern that is an accepted norm in the Rails/RSpec community works pretty well in JavaScript. Take the 10 minutes required to set up test data factories and enjoy cleaner, easier to maintain tests!

import { lessons } from 'rails'; // part 1

Bringing techniques used in Rails to Node

I'm about a decade late.

After unknowingly circling it for years, I am now working in Rails. When I started at CoverMyMeds in the beginning of 2017, I wasn't sure what I would think of working with a framework that had clearly crossed peak hype years ago. I expected to be somewhat bored with a system that I assumed wouldn't have much to teach me. But working in Rails has been nothing if not educational. I had no idea how many concepts and practices from Rails permeate thttps://giphy.com/search/hackhe libraries and frameworks I have used throughout my career. The Rails community continually discovers and enshrines best practices in the framework and supporting libraries. From what I can tell, this community generally prefers to converge. The JavaScript community, for all of its strengths, generally does not.

I still love working in Node though. So, as I have learned Rails and RSpec, some of the techniques I discovered there proved so useful that I decided to find a way to bring them into my JavaScript projects.

Technique 1 - In tests, use DB transactions

Rails has always supported running your database operations in tests as transactions that are automatically rolled back after each test. By using this feature, you prevent your test database from being inundated by old or even misleading data.

TBH I'm not sure if a transaction with a rollback is faster than a DB write plus a delete, but it's much, much simpler to manage than manual test teardown statements that restore the state of your database. I highly recommend this practice.

Setting this up in Node is going to depend on your DB abstraction layer. I prefer Objection.js and the example presented here uses that library.

For the testing framework, I typically use Mocha, but I think the semantics used here would work with Jasmine and Jest as well.

Setup

Create a helper file that you will use in your test files. I put mine at test/support/transactional_tests.js. We're going to import this file in each test file in which we want transactional tests.

'use strict';

const dbConfig = require('../../knexfile'),
  knex = require('knex')(dbConfig[process.env.NODE_ENV]),
  Model = require('objection').Model;

let afterDone;

beforeEach('initialize transaction', function (done) {
  knex.transaction(function (newtrx) {
    Model.knex(newtrx);
    done();
  }).catch(function () {
    // call afterEach's done
    afterDone();
  });
});

afterEach('rollback transaction', function (done) {
  afterDone = done;
  Model.knex().rollback();
});

Explanation

It's a fun little hack. Let's break it down.

const dbConfig = require('../../knexfile'),  
  knex = require('knex')(dbConfig[process.env.NODE_ENV]),
  Model = require('objection').Model;

Here we are initializing knex and objection. My knexfile is keyed by environment, hence I pass in process.env.NODE_ENV to the knex initialization function.

Now for the main trick. Rather than analyze the code line by line, it's easier to think about how the tests will interact with individual functional sections.

First:

beforeEach('initialize transaction', function (done) {  
  knex.transaction(function (newtrx) {
    Model.knex(newtrx);
    done();
  })
  ...

In the beforeEach section, which will run before every test, we initialize a transaction and inject it into the base Objection model. All Objection models in the test will now use this transaction. Immediately after injecting the transaction, we call the beforeEach function's done callback, so the current test is now free to run.

After the test, the afterEach executes:

afterEach('rollback transaction', function (done) {
  afterDone = done;
  Model.knex().rollback();
});

The afterDone = done; line is like a bookmark; we're holding on to a reference to afterEach's done function so that we can call it later.

Calling .rollback() rolls the transaction back (obviously), but it also results in an exception being thrown that is caught by the original transaction's catch function (which you will see in the beforeEach callback):

...
  .catch(function () {
    // call afterEach's done
    afterDone();
  });

Here, we call afterEach's done function, which we refer to with afterDone. The teardown code is now complete and the test will not hang.

So, the steps can be summarized as follows:

1. Initialize a transaction and inject it into Objection's base Model class.
2. Run test
3. Rollback transaction
4. Catch resulting exception
5. Call afterEach done function

Use it

In any test file in which you want automatic transactions, simply require your helper file.

require('../../support/transactional_tests');

describe('Class under test', function () {
...
});

The end.

If you're using Objection and want cleaner tests, give transactions a try!

I recently returned home from a two-week trip to Ethiopia. My family spent some time in the capital, Addis Ababa, as well Awassa in the south of the country and Bahir Dar in the north.

I had many experiences that will probably influence my view of the world, and many others that were interesting or beautiful or difficult.

But that's not what this post is about. If you are looking for a fun and provocative take on Ethiopia, look no further than Anthony Bourdain's Parts Unknown episode; it has the emotional depth and helpful reflection you are looking for. Or, if you know me personally, please ask me about my experiences there. I'm happy to share.

This post is about some of the practical, nerdy lessons that I learned while traveling around a developing country.

Client-side webapps can be better than server-generated content in high-latency environments

Here in the U.S., I am accustomed to high-speed, low-latency networks. Occassionaly, I have to deal with low-speed, low-latency. In that scenario, everything is pretty much the same, you just wait longer.

But throughout this trip, I was on high-latency networks the whole time. Regardless of the network infrastructure, just trying to reach servers that were probably hosted on a different continent meant long round trips.

So I got to experience first hand what many have been saying for a while: well-implemented, client-side webapps may handle high latency connections better than traditional server-generated pages. Usually this point is made in the context of mobile networks. But it is probably even more applicable for the kind of remote networks that I was dealing with.

Using Gmail illustrated this point perfectly. As you may recall, Gmail provides a link to "Load basic HTML (for slower connections)". If you have a Gmail account, you can try it at mail.google.com/mail/h/. This seemed like the right thing to do over there. After all, my connection felt really slow.

But using the basic HTML, server-generated version didn't feel faster at all. And after I thought about it for a moment, the reason was obvious. In my case, the high-latency network requests weren't necessarily slow, they just sort of behaved erratically; the timing felt off. Sometimes I would make a request and wait 30 seconds before there was any response at all. Sometimes I would get a response almost right away. And this erratic response pattern fits much better with the async architecture of a client-side webapp. The user is already accustomed to initiating an action and then moving on as the action is dealt with in the background. Very often, the speed with which the action is resolved is either not very important or is masked because the user is immediately shown a notification.

So yay for webapps.

Immediate UI feedback is crucial

In addition to Gmail, I also use Yahoo mail. And though Yahoo mail is a webapp with UX similar to Gmail, the experience of using it on a high-latency network was terrible, simply because it does not give immediate UI feedback. When you click to delete a message or to open a different folder, the app waits for the server response to update. This is ok on low-latency, moderately fast networks (except when the response takes longer than usual, which happens not infrequently), but unbearable when dealing with high-latency.

Without immediate UI feedback, you just don't know if anything happened after you clicked. Should you click that button again? Was there an error? What is happening!? Just a simple spinner or some kind of status indicator would alleviate all of that frustration.

I admit that I have built UIs that commit this exact error. I justified it because I thought that the particular response would always be fast. Well, high latency can definitely mess that up. Better to always let the user know when something is happening.

Cheap Chromebooks make for awesome travel dev systems

I needed a development machine to take with me in case something went wrong at work. My normal dev system is a Macbook, but I really didn't want to take it. It seemed too risky and even though its a newer model, it seemed too bulky to carry around in backpack everywhere I went.

So I bought a 11.6" Acer CB3-111 Chromebook. This felt risky at the time. I had never used a Chromebook before. I knew it was possible to run Ubuntu in chroot environments on Chromebooks, but it couldn't possibly work as well as advertised, right? Could I really get my dev environment working on this underpowered little computer? Also, it only has 16GB of SSD space, and some of that is devoted to the Chromebook OS. Would whatever is leftover really be enough to run a Linux distribution?

Well, it all worked out; even better than advertised. First of all, the hardware is great. Yeah, it feels sort of plasticy, but the thing costs less than $200! And Chromebook OS is very smart. Boot time is incredibly fast and the UI is fantastic.

But the Linux chroot setup - well, that was a revelation. Chromebooks are built on Linux, so running Ubuntu in a chroot is nothing like running a virtual machine. There is barely any performance penalty. And the lack of disk space was no problem for me. I really wasn't interested in running a full Gnome or Unity or KDE Plasma UI. I just wanted a shell environment to install my development tools. Thanks to years of sysadmining, I'm happy fixing bugs with just bash and vim.

The Chromebook was probably the best $175 I have ever spent on a computer. I imagine it will be my travel computer for years.

iPhone ups and downs

I took my trusty iPhone 4S. No, that's not a typo. If it ain't broke...

More than I expected from the battery

I was surprised how well the battery did. Battery life is one the main problems with an aging phone. In my normal life, I sometimes barely make it home before I need to recharge. I worried that I would be unable to take pictures at times because I wouldn't be able to recharge often enough.

But in Ethiopia, because I couldn't connect to the local GSM networks, the phone was on airplane mode when we were away from hotels and wifi. Apparently even an old phone can go a long time on airplane mode. And it was probably just my imagination, but recharging on 220V seemed even faster than 110V.

Less than I expected from iCloud and Photos

I did have a problem with not being able to take pictures at all times, but it was more with iCloud's syncing protocol. I use iCloud Photo Library, which means that theoretically all the pictures I take are synced to iCloud whenever the phone is online. And, the Photos app is smart enough to ditch the fullsize images in favor of smaller thumbnails when you are running out of disk space. Since this is an old phone and I have a lot of pictures, I am pretty much always running out of disk space.

What I didn't know is that iCloud will not sync if it can tell you have a poor connection. So there were a few days when I had to frantically delete apps or podcasts or other stuff to make more room for photos even though I did have an Internet connection, albeit a bad one. Once we got to a hotel with a better connection, the Photos app was able to upload, sync, and purge.

New Balance Minimus 10v3s are the perfect shoes for exploring a developing country

These New Balance Minimus trail shoes are one in a series of minimal running shoes I have tried. I got lucky and bought these shortly before the trip. They were perfect. Even in the biggest cities in Ethiopia many of the roads are unpaved or have no sidewalks. Almost everywhere I was walking on dirt and rocks. A shoe designed for trails was ideal both in the cities and in the country side.

This is a great shoe. The Vibram outsole is awesome. Probably the best minimalist shoe I have tried so far.