Know JavaScript: Hoisting

JavaScript is a beautiful language with a lot of hidden secrets. Today I will explain a concept called hoisting. This is a very important feature to be aware of, and is one of those wierd things that suddenly happens in your code which is hard to understand and you end up debugging for hours.

Variable declaration hoisting

1. We start with something simple:

var a = "Hello World";
function doSome() {
   alert(a);
}
doSome();

From the code it is easy to see that “Hello World” will be alerted. a is global variable, also available inside the doSome function. No magic here.

2. We add a new local variable a inside the function, after the alert:

var a = "Hello World";
function doSome() {
   alert(a);
   var a = "hi there";
}
doSome();

You might expect that “Hello World” is alerted again, because the local variable a is not defined at the time of the alert. This is wrong. What actually gets alerted is “undefined”.

This happens because variable declarations in JavaScript are hoisted to the top of the function body. The initialization though is not hoisted. The code is thus executed as:

var a = "Hello World";
function doSome() {
   var a;
   alert(a);
   a = "hi there";
}
doSome();

When we write it like this it is easy to read and understand that “undefined” will be alerted in this code.

Function hoisting

Also function definitions are hoisted. This is actually pretty neat because it means we can use functions defined later in the code.

function doSome() {
   doOther();
   //A lot of code
   function doOther() {
      alert("hello world");
   }
}
doSome();

If we execute the code above, “hello world” is alerted. the doOther function is hoisted to the top by JavaScript. This means that JavaScript hoisted the whole function definition for us, making it available “before” it was defined in the code.

But functions can also be assigned to a variable in JavaScript:

function doSome() {
   doOther();
   //A lot of code
   var doOther = function() {
      alert("hello world");
   }
}
doSome();

In the example above we have changed the code to assign an anonymous function to a variable with name doOther. In this example we experience variable declaration hoisting, and we get an undefined exception:

TypeError: undefined is not a function

Summary

  1. Variable declarations are hoisted to the top of your scope in JavaScript
  2. Whole functions are hoisted to the top of your scope in JavaScript

Always declare variable on the top of your functions.

Want to explore more JavaScript?

I have created a workshop, with examples and tasks. It focus on some of the basic features in the JavaScript language, which all developers should know. It is available on GitHub:

Custom build in dojo 1.7.

Introduction

In this blog-post I will explain how to do a simple custom build with dojo’s build system. Dojo is a great JavaScript framework and you can read more about it at dojotoolkit.org.

I write this from the experience gained from upgrading our application from dojo 1.6 to dojo 1.7. Dojo’s upgrade to version 1.7 is the biggest change in the dojo core we have seen for a long time, now introducing AMD (asynchronous module loading). This upgrade also affected the build system, which is new for dojo 1.7. We found that we where unable to reuse our current build-set-up after upgrading. I wrote about this because dojo still lacks some easy and simple examples on how to do custom build.

The official documentation for the build system is found at livedocs.dojotoolkit.org/build/buildSystem

If you only are interested in the solution, please head straight to the GitHub repository: https://github.com/ivarconr/dojo-custom-build. Here you will find all the source files used in this tutorial.

Why do we need custom build?

Dojo 1.7 introduced AMD which heavily increases the speed modules, classes and dependencies are loaded. Now your application only need to load dependencies required to satisfy the specific request and can wait with other dependencies until they actually are needed. Modules can even be downloaded to the browser in parallel (huge speed gain). This change in dojo will make many dojo applications start up much faster, especially if they use many modules.

Still, when the application grows, and you get hundreds or even thousands of JavaScript files it will be many HTTP GET request required to download all these small JavaScript files which builds up a typical application. We know by experience that many small files is more costly to download, than if we concatenate them into larger files which are downloaded. This gets even worse if we first have to download module A to find out that it requires module B which again requires module C. We have no way of knowing that we need to download module C before we have downloaded both module A and module B. This is fine in development, but not in production.

Layers to the rescue

What we do in our application is to combine related JavaScript files into layers. We try to see the application from the user-perspective and combines related scripts. We combine our modules in such a way that all files needed to perform a specific action is built into a layer. We also use a core-layer where we put core functionality shared by many different modules (the overlap).

  • Core-layer – Contains files/modules used many places in our application. Definitions in this layer is excluded from the other layers. 
  • Function-layer-x – Contains files/modules required to perform action x in the application. We have many “x”  layers.

The Dojo Build System – A tutorial (1.7.1)

Requirements

Project structure

The file structure used in the tutorial project is: ( alsoavailable at GitHub):

├── index-dev.html
├── index-release.html
├── scripts
│   ├── dojo-release-1.7.1-src
│   │   ├── dijit
│   │   ├── dojo
│   │   ├── dojox
│   │   └── util
│   ├── mycompany
│   │   ├── mycompany.profile.js
│   │   ├── package.json
│   │   └── widgets
│   ├── release
│   │   ├── build-report.txt
│   │   ├── dijit
│   │   ├── dojo
│   │   └── mycompany
│   └── release.profile.js
└── styles

I have placed dojo-src under scripts and my own company package inside the “mycompany” folder. It is important to not mix the dojo-source with my own source files. This makes it much easier to upgrade dojo later.

I use index-dev.html during development. In release I switch to index-release.html. I will explain the difference between these files later.

The built files are placed in the relese directory with the build script.

The package profile

The mycompany folder is the folder containing all the files for the “mycompany” package. To help the build system to build this layer I have a package profile, telling the build system important information about this package, such as which files are valid AMD modules, which files are test files and configuration files to only
copy during the build phase.

var profile = (function(){
  copyOnly = function(filename, mid){
	var list = {
	"mycompany/dojo.profile":1,
	"mycompany/package.json":1
    };
	return (mid in list) || /(css|png|jpg|jpeg|gif|tiff)$/.test(filename);
  };

  return {
	resourceTags:{
		test: function(filename, mid){
			return false;
		},

		copyOnly: function(filename, mid){
			return copyOnly(filename, mid);
		},

		amd: function(filename, mid){
			return !copyOnly(filename, mid) && /\.js$/.test(filename);
		}
	},

	trees:[
		[".", ".", /(\/\.)|(~$)/]
	]
  };
})();

This profile is heavly based on dojo’s profile for the dojo package, found under dojo/dojo.profile.js

The layer build profile

To perform the actual build I use a profile on a higher layer (application layer) which tell the build system about the different packages used in the application, and what layers to build. This can be packages we have built our self, such as the mycompany package, our third party packages, such as the dijit package provided by dojo.

release.profile.js Looks like:

var profile = {
    basePath: "./",
    layerOptimize: "shrinksafe.keepLines",
    releaseDir: "./release",
    hasReport: true,

    packages:[
        {
            name: "dojo",
            location: "./dojo-release-1.7.1-src/dojo"
        },
        {
            name: "dijit",
            location: "./dojo-release-1.7.1-src/dijit"
        },
        {
            name: "mycompany",
            location: "./mycompany"
        }
    ],

    layers: {
        "mycompany/layers/core": {
            include: [
                "dijit/_Widget",
                "dijit/_Templated"
            ],
        },
        "mycompany/layers/example": {
            include: [
                "mycompany/widgets/Example"
            ],
            exclude: [
                "mycompany/layers/core"
            ]
        }
    }
};

In the release profile I have defined which packages I have used and the location of them. I have also included the package “mycompany”. This package holds all of the company specific code. The example also shows that I have included two layers:

  • mycompany/layers/core – This layer contain the core functionality
  • mycompany/layers/example – Our example widget. This includes all modules required for ExampleWidget (all transitive dependencies), but excludes dependencies found in the core layer.
More details on about layer configuration can be found in dojo’s live documentation: http://livedocs.dojotoolkit.org/build/transforms/writeAmd

Executing the custom build

I execute the build from the “mywebapp/scripts/dojo-release-1.7.1-src/util/buildscripts” folder by using the build.sh or build.bat script:

./build.sh profile=../../../release.profile.js -r

The build system will place the built files into the release directory inside scripts because I have defined the relaseDir inside our relase.profile.js. The build-script will decide whether to use node or Java. Node is preferred because it can run the analysis and build process in parallel with multiple threads, while Rhino only supports one thread, which makes it much, much, much slower….

Development mode vs. release mode

Development mode

To enable us to develop without having to build the layers all the time I use a special html-file for starting the application in dev mode named index-dev.html. In this mode we just load required modules asynchronously as they are needed. This is supported by dojo out of the box and we basicly requieres stuff just when we need them. It us much easier to debug if we load all modules individually during development.

I have measured some stats in dev mode (no layers):

  • 57 individual HTTP GET requests
  • 840ms before onLoad is fired
As seen from the numbers I actually end up with 57 individual request to fetch all resources required for my little application. This is fine in development because it is easy to change code, and we get better debugging in the browser.

Release mode

After I have performed the build I will have a release catalog containing our built javascript, including my layers. This folder contains the built layers which I want to use in my application. In production I will use index-release.html where I have changed the code a bit to also handle the built layers. The important part is to notice the extra require part in the start-app script:

require(["mycompany/layers/core", "mycompany/layers/example"], function() {
  require(["mycompany/widgets/Example","dojo/domReady!"], function(Example) {
    new Example({}, "app");
  });
});

As you can see I require both layer-modules first, which includes all the required JavaScript files in my application. This way I reduces the number of HTTP GET requests to a minimum. I still keep all my files in the release folder. This enables easy fallback for dojo to load dependencies asynchronously if I for some reason should forget (or exclude) a reuired JavaScript in the layer.

Stats in release mode (with layers):

  • 5 individual HTTP GET requests
  • 355ms before onLoad is fired
This is significantly better performance and better suited for production.

Can different layers include the same modules?

Can I have the same JavaScript bundled into different layers? Yes you can. Dojo will actually handle this very well. You should try though to avoid this when possible, and always evaluate whether these modules can be part of a “core” layer because the layers will be larger if multiple layer modules include the same scripts. This basically comes down to a trade off, we want to limit the size off the core layer and at the same time limit the size of a function layer.

One solution can of course be to have multiple “core layer” for different function groups. It all comes down to you having to find which layering that gives the best performance for your application.

Don’t put everything in one layer

Even though fewer HTTP GET’s are good we must make sure to not put everything inside ONE layer. Putting it in one layer can give us one HUGE file, which basically gives our application an undesired long startup time. We can also gain something from loading a few modules in parallel. We should evaluate the nature of the application when we decide how to define our layers. We should try to group stuff we need together inside modules. It is also important to discover the overlaps between layers and try to create “overlap” layers.

Summary

This ends my blog about how to build custom layers in dojo 1.7.1. I have placed all my files on github: https://github.com/ivarconr/dojo-custom-build. Please post any comments or questions you may have. In the next blogpost I will explain how we are using custom build as part of our maven build process to ensure that our layers are built each time we build our Java Webapplication with Maven.

Fetch twitter feed with JavaScript

Who said you can’t fetch data across domains?

In this blog I show you how to include a twitter feed on you web-page which fetches data from twitter in plain JavaScript, directly from the client’s browser to twitter.com.

Same origin you said?

We all know that “ajax” call’s in JavaScript must follow the same origin policy. This usually leads to using a proxy one the same domain to proxy ajax-calls to the external services. This leads to more work for your server and more code for you to write. Who invented this same origin stuff anyway?

Of course it is supposed to be more secure? for the end user, but is it actually? If we end up proxy all the calls? Is it then more secure? In the end the user has to trust the scripts delivered from our server anyway.

Jsonp to the rescue

But fortunately there exists another trick, namely jsonp. This is a neat trick where you add a “script” tag to your page which reference an external script. The external service (supporting jsonp) will then return a valid JavaScript which call’s the specified callback function with the response-data as input.

<script type="text/javascript"
src="http://somesite.no/someResource?jsonp=parseResponse"></script>


The Twitter feed

jQuery makes jsonp pretty simple and does all the hard work for us through the standard XHR-get call. You simply tell jQuery to fetch the data as jsonp with the “dataType” attribute. And of course, twitter supports jsonp.

In the example below I show how to fetch data from twitter using jsonp, and jQuery.

$.ajax({
    url: "http://search.twitter.com/search.json?q=%23test",
    dataType: "jsonp" }).success(function(data) {
    $.each(data.results, function(idx, result) {
      var time = "<span class='time'>"+ result.created_at + "</span>";
      var user = "<span class='user'>@" + result.from_user + "</span>: ";
      var tweet = $("<li>", {html: user + result.text + "<br />" + time});
      tweet.appendTo("#twitter");
    });
  });

As you can see it is pretty simple. One line to fetch the code and few lines of formatting. The final result is added to the “ul” container with id=”twitter”.

The benefits are:

  • less code
  • less server logic
  • less resources need on your part (the client fetches the data from twitter)
  • easier to implement
  • easier to maintain

Know JavaScript: parseInt()

The last week I have been writing some JavaScript to improve the user-experience on a web-page I am working on. One of the task I had to do was parsing some String values into integer values for numeric comparison. I googled and found JavaScript’s “parseInt()” function. I added this function, and tested it a few times, happy to see that the function transformed my String values into comparable integer values.

var strValue = "8";
var i = parseInt(strValue) ;
/* i has value 8 */

Later I did some modifications in the service delivering values to my webpage. It was convenient for the layout if all values were two digits, so all values below 10 was zero padded. “1” became “01”, “2” became “02” and so on. After a few days strange bugs were reported. apparently some of the parsed values became 0, an illegal value for my web-application.
After some testing I discovered the following:

  • “01” becomes 1
  • “02” becomes 2
  • “03” becomes 3
  • “04” becomes 4
  • “05” becomes 5
  • “06” becomes 6
  • “07” becomes 7
  • “08” becomes 0
  • “09” becomes 0
  • “10” becomes 10

What is going on? Is there a bug in JavaScript’s parseInt() function? After all I headed to the parseInt() documentation. The documentation clearly states:

If the radix parameter is omitted, JavaScript assumes the following:

  • If the string begins with “0x”, the radix is 16 (hexadecimal)
  • If the string begins with “0”, the radix is 8 (octal). This feature is deprecated
  • If the string begins with any other value, the radix is 10 (decimal)

So this tells us that the parseInt() function also takes a radix parameter.  The above results makes sence because 8 and 9 does not exists in the octal number system.  I changed my code to the following:

var strValue = "08";
var i = parseInt(strValue, 10) ;
/* i has value 8 */

Afte this change I was  happy to see that “08” becomes “8” and “09” becomes “9”. The radix parameter is optional, but I will never leave it out again! Just add the required radix as a second parameter to the function, and you will never will get in trouble parsing String values to integer values!