Custom build in dojo 1.7.

Introduction

In this blog-post I will explain how to do a simple custom build with dojo’s build system. Dojo is a great JavaScript framework and you can read more about it at dojotoolkit.org.

I write this from the experience gained from upgrading our application from dojo 1.6 to dojo 1.7. Dojo’s upgrade to version 1.7 is the biggest change in the dojo core we have seen for a long time, now introducing AMD (asynchronous module loading). This upgrade also affected the build system, which is new for dojo 1.7. We found that we where unable to reuse our current build-set-up after upgrading. I wrote about this because dojo still lacks some easy and simple examples on how to do custom build.

The official documentation for the build system is found at livedocs.dojotoolkit.org/build/buildSystem

If you only are interested in the solution, please head straight to the GitHub repository: https://github.com/ivarconr/dojo-custom-build. Here you will find all the source files used in this tutorial.

Why do we need custom build?

Dojo 1.7 introduced AMD which heavily increases the speed modules, classes and dependencies are loaded. Now your application only need to load dependencies required to satisfy the specific request and can wait with other dependencies until they actually are needed. Modules can even be downloaded to the browser in parallel (huge speed gain). This change in dojo will make many dojo applications start up much faster, especially if they use many modules.

Still, when the application grows, and you get hundreds or even thousands of JavaScript files it will be many HTTP GET request required to download all these small JavaScript files which builds up a typical application. We know by experience that many small files is more costly to download, than if we concatenate them into larger files which are downloaded. This gets even worse if we first have to download module A to find out that it requires module B which again requires module C. We have no way of knowing that we need to download module C before we have downloaded both module A and module B. This is fine in development, but not in production.

Layers to the rescue

What we do in our application is to combine related JavaScript files into layers. We try to see the application from the user-perspective and combines related scripts. We combine our modules in such a way that all files needed to perform a specific action is built into a layer. We also use a core-layer where we put core functionality shared by many different modules (the overlap).

• Core-layer – Contains files/modules used many places in our application. Definitions in this layer is excluded from the other layers. 
• Function-layer-x – Contains files/modules required to perform action x in the application. We have many “x”  layers.

The Dojo Build System – A tutorial (1.7.1)

Requirements

• Dojo source 1.7.1: http://download.dojotoolkit.org/release-1.7.1/dojo-release-1.7.1-src.zip
• To execute the build: node (http://nodejs.org) or Java (1.6)

Project structure

The file structure used in the tutorial project is: ( alsoavailable at GitHub):

├── index-dev.html
├── index-release.html
├── scripts
│   ├── dojo-release-1.7.1-src
│   │   ├── dijit
│   │   ├── dojo
│   │   ├── dojox
│   │   └── util
│   ├── mycompany
│   │   ├── mycompany.profile.js
│   │   ├── package.json
│   │   └── widgets
│   ├── release
│   │   ├── build-report.txt
│   │   ├── dijit
│   │   ├── dojo
│   │   └── mycompany
│   └── release.profile.js
└── styles

I have placed dojo-src under scripts and my own company package inside the “mycompany” folder. It is important to not mix the dojo-source with my own source files. This makes it much easier to upgrade dojo later.

I use index-dev.html during development. In release I switch to index-release.html. I will explain the difference between these files later.

The built files are placed in the relese directory with the build script.

The package profile

The mycompany folder is the folder containing all the files for the “mycompany” package. To help the build system to build this layer I have a package profile, telling the build system important information about this package, such as which files are valid AMD modules, which files are test files and configuration files to only
copy during the build phase.

var profile = (function(){
  copyOnly = function(filename, mid){
	var list = {
	"mycompany/dojo.profile":1,
	"mycompany/package.json":1
    };
	return (mid in list) || /(css|png|jpg|jpeg|gif|tiff)$/.test(filename);
  };

  return {
	resourceTags:{
		test: function(filename, mid){
			return false;
		},

		copyOnly: function(filename, mid){
			return copyOnly(filename, mid);
		},

		amd: function(filename, mid){
			return !copyOnly(filename, mid) && /\.js$/.test(filename);
		}
	},

	trees:[
		[".", ".", /(\/\.)|(~$)/]
	]
  };
})();

This profile is heavly based on dojo’s profile for the dojo package, found under dojo/dojo.profile.js

The layer build profile

To perform the actual build I use a profile on a higher layer (application layer) which tell the build system about the different packages used in the application, and what layers to build. This can be packages we have built our self, such as the mycompany package, our third party packages, such as the dijit package provided by dojo.

release.profile.js Looks like:

var profile = {
    basePath: "./",
    layerOptimize: "shrinksafe.keepLines",
    releaseDir: "./release",
    hasReport: true,

    packages:[
        {
            name: "dojo",
            location: "./dojo-release-1.7.1-src/dojo"
        },
        {
            name: "dijit",
            location: "./dojo-release-1.7.1-src/dijit"
        },
        {
            name: "mycompany",
            location: "./mycompany"
        }
    ],

    layers: {
        "mycompany/layers/core": {
            include: [
                "dijit/_Widget",
                "dijit/_Templated"
            ],
        },
        "mycompany/layers/example": {
            include: [
                "mycompany/widgets/Example"
            ],
            exclude: [
                "mycompany/layers/core"
            ]
        }
    }
};

In the release profile I have defined which packages I have used and the location of them. I have also included the package “mycompany”. This package holds all of the company specific code. The example also shows that I have included two layers:

• mycompany/layers/core – This layer contain the core functionality
• mycompany/layers/example – Our example widget. This includes all modules required for ExampleWidget (all transitive dependencies), but excludes dependencies found in the core layer.

More details on about layer configuration can be found in dojo’s live documentation: http://livedocs.dojotoolkit.org/build/transforms/writeAmd

Executing the custom build

I execute the build from the “mywebapp/scripts/dojo-release-1.7.1-src/util/buildscripts” folder by using the build.sh or build.bat script:

./build.sh profile=../../../release.profile.js -r

The build system will place the built files into the release directory inside scripts because I have defined the relaseDir inside our relase.profile.js. The build-script will decide whether to use node or Java. Node is preferred because it can run the analysis and build process in parallel with multiple threads, while Rhino only supports one thread, which makes it much, much, much slower….

Development mode vs. release mode

Development mode

To enable us to develop without having to build the layers all the time I use a special html-file for starting the application in dev mode named index-dev.html. In this mode we just load required modules asynchronously as they are needed. This is supported by dojo out of the box and we basicly requieres stuff just when we need them. It us much easier to debug if we load all modules individually during development.

I have measured some stats in dev mode (no layers):

• 57 individual HTTP GET requests
• 840ms before onLoad is fired

As seen from the numbers I actually end up with 57 individual request to fetch all resources required for my little application. This is fine in development because it is easy to change code, and we get better debugging in the browser.

Release mode

After I have performed the build I will have a release catalog containing our built javascript, including my layers. This folder contains the built layers which I want to use in my application. In production I will use index-release.html where I have changed the code a bit to also handle the built layers. The important part is to notice the extra require part in the start-app script:

require(["mycompany/layers/core", "mycompany/layers/example"], function() {
  require(["mycompany/widgets/Example","dojo/domReady!"], function(Example) {
    new Example({}, "app");
  });
});

As you can see I require both layer-modules first, which includes all the required JavaScript files in my application. This way I reduces the number of HTTP GET requests to a minimum. I still keep all my files in the release folder. This enables easy fallback for dojo to load dependencies asynchronously if I for some reason should forget (or exclude) a reuired JavaScript in the layer.

Stats in release mode (with layers):

• 5 individual HTTP GET requests
• 355ms before onLoad is fired

This is significantly better performance and better suited for production.

Can different layers include the same modules?

Can I have the same JavaScript bundled into different layers? Yes you can. Dojo will actually handle this very well. You should try though to avoid this when possible, and always evaluate whether these modules can be part of a “core” layer because the layers will be larger if multiple layer modules include the same scripts. This basically comes down to a trade off, we want to limit the size off the core layer and at the same time limit the size of a function layer.

One solution can of course be to have multiple “core layer” for different function groups. It all comes down to you having to find which layering that gives the best performance for your application.

Don’t put everything in one layer

Even though fewer HTTP GET’s are good we must make sure to not put everything inside ONE layer. Putting it in one layer can give us one HUGE file, which basically gives our application an undesired long startup time. We can also gain something from loading a few modules in parallel. We should evaluate the nature of the application when we decide how to define our layers. We should try to group stuff we need together inside modules. It is also important to discover the overlaps between layers and try to create “overlap” layers.

Summary

This ends my blog about how to build custom layers in dojo 1.7.1. I have placed all my files on github: https://github.com/ivarconr/dojo-custom-build. Please post any comments or questions you may have. In the next blogpost I will explain how we are using custom build as part of our maven build process to ensure that our layers are built each time we build our Java Webapplication with Maven.