Friday, December 30, 2011

Learning From 2011

It's time to summarize at least few mistakes or epic fails of the year, hoping the next one will try to follow a better direction.
I won't mention anything related to war, politic, Berlusconi, or fashion related stuff, other more prepared than me will do via pictures or posts, all I gonna talk about is the field I am concerned about: web and mobile web oriented technologies and facts.
The order is completely random, so grab a mug of coffee, take few minutes, and read 'till the end, forgetting for once tl;dr philosophy ;)

The Partially (Multi)? Touch Device Case

This has been the most annoying fact of the year: vendors going out with freaking powerful mobile devices with touch or multi touch capable screen/hardware not exposed through the browser and only available for native apps.
We are not talking about potentially "dangerous" technologies as WebGL could be, we are talking about the most basic thing able to break 140% the user experience.
If a user plays a native game or use a native map, as classic Map application could be, she will naturally use at least a finger, or more, to interact with the screen. Interaction does not mean scroll the page and nothing else, interaction means full control of what the user is doing with the screen.
As soon as the user surf the web, the browser pretends to know what the user would like to do on the screen without giving web developers any control over those actions.
The inconsistent behavior comes when the viewport meta tag is used to block, as example, the zooming in and out gesture because the whole layout has been developed for a static viewport that should not change.
In this case, only few vendors ( Apple ) or browsers ( Opera Mobile ) got it right, any other vendor with a WebKit based browser got it wrong. Here a few examples:

IE9 Mobile Epic Fail

IE9 is a great browser, compared with all other previous version, and it's freaking fast. This is true for both desktop and mobile phones but in latter case, IE9 is an epic fail when it comes to User Interaction.
There is no bloody way to intercept a user moving a finger on the screen, and standard W3C Touch Events are not supported.
You may think: oh well, I gonna use mouse events then ... and you are wrong, these are not fired until the user release her finger so ... forget about it, no "scroll" hacks will work as expected there ... also why on earth when everybody else is disabling JS actions during scrolling, IE9 Mobile fires it all the time?


Sad part is that Microsoft is investing a lot into HTML5, but if the most basic thing as Touch Events are, is not there, no reason to have the fastest mobile Canvas implementation because no game will be interactive, and no web app can be created as desired.
Hopefully IE10 will simply follow W3C exposing both Touch and TouchList in a meaningful way ... please don't screw this or few developers will even consider to write software for this browser, thanks.

webOS 2 Sad Fail

I have upgraded few minutes ago my Pre 2 to webOS 2.4.4 ... and I was expecting more from this update. My Palm Pre 2 exposes correctly multi touches through the system, but the web browser does not expose them and this is freaking annoying. Same IE9 behavior and at least at the end of 2011 I was expecting such simple browser update able to bring my Palm to the next level: User Interaction Through The Web ... FAIL.
The good old native Google Maps application for webOS 2.0.X is gone. This app was working like a charm and it was exposing multi touches through the multi touches capable screen ... now I have a slower and flicking version of Bing maps that is not even able to render pinch/spread without loosing the position on the screen since everything gets stuck truly easily.
Now, I have no idea why this happened and why somebody wasted time to do this ... it's just worse than before and the browser still does not expose multi touches ... come on guys, this phone did not deserve to die like this.

Ultra Powerful ... "broken" Androids

It does not matter if we have 16 cores in our expensive smart phone when developers life is made harder for no reason ... with most recent Android phones we could create any sort of web application but here again, no-f*cking-body exposes TouchList via browser except few tablets out there.
OK, still better than nothing, as it is for IE9 Mobile, FF Mobile ( I will come back later to this one ), or webOS browser, but how comes 3rd parties browsers such Opera Mobile are able to bring TouchList and expose multi touches through the multi touches capable screen and native browsers are all lacking of this basic feature?
You don't even want to know how we managed to bring multiple touches in android devices, all you need to know is that is basically a hack and it requires a native application wrapper and this is, in my opinion, ridiculous!
Nowadays, all users with a friend that uses an iPhone, will consider the multi touch web app broken in its android: come on, I use two fingers in other things why I cannot do the same in this web site/application?

FireFox Mobile Fail

I don't know where to start with this browser ... I mean, guys, this is not Desktop, this is a touch screen. If you decide that the user cannot even scroll a paragraph horizontally because your freaking cool settings or bookmark view has to show up instead, you should create your own device and make the phone 3 times larger: 1/3 for the actual screen, 2/3 for your things nobody expect to appear during navigation.
About touches, at least most recent version implements touch events but performances are still too bad compared with native browsers or Opera Mobile, and the fact settings or bookmarks show up when we touch close to the edge of the screen, plus the fact AFAIK there is no way to avoid this, make this browser not suitable for full screen web games/apps based on canvas: slow and unusable ... please fix/drop this!

UpdateIt must be said that latest FF mobile brings better canvas performances and apparently it exposes a not so fast WebGL. I dd not know this the moment I wrote this post so FF Mobile is definitively already going to the right direction.

iOS Gesture Events VS Others

These are not standard and nobody should care. The moment we can rely in TouchList and multi touches exposes through this interface, the problem is solved 'cause we can implement by our self any gesture we want.
No need to wait 'till on spread/pinch are implemted and documented, just please give us TouchList and take your time after to do things properly. TouchList has been already defined ... that's all we need so far, thanks!

The Curious Case of Opera Mobile

Opera did weird things this year ... it switched from partial ES3 support to full ES 5.1 support ... what the hell guys, congratulations! Still there is no way to understand numbers of this browser (again: Opera Mobile, NOT Mini) .
Today, I can say Opera Mobile is one of the fastest and most capable browsers I have been dealing with but the lack of many CSS3 features have been the arrow in the knee together with the fact Opera could have been the Phone Gap Like software of the year while they focused on ... I guess something else.
Don't give up guys, Opera Mobile is amazing and people may chose to download it as soon as most common HTML5/CSS3 things are working as expected and also you may expose freaking cool stuff without necessarily follow W3C, giving web developers the possibility to distribute applications directly through your browser as native wrapper ( and fallback to Webkit for iOS ) .
Opera has always been pioneer, and it's time to be pioneer on mobile too because on desktop I am afraid to tell you it's hard to compete. Firefox, Chrome, and Webkit Nightly, are really advanced and I believe bigger teams, what we are missing is a not so fragmented browser that works as expected in as many devices as possible because what I have seen with all webkit versions I have worked with this year was painful, hard, inconsistent, and WTF development oriented.
Last, but not least, If there is not SQL Database, don't put a native useless Database function on the global scope or shims are screwed and if you don't expose SQL Database, at least expose Indexed DB or shims are screwed.

Android Whatever

While many developers think that Android is an Operating System, I am every day more convinced that Android is a Super Kernel that has almost nothing to do with the Operating System itself.
There are too many damn versions of Android, and even same major versions do not guarantee anything.
Every device must be tested a part because every device, with every vendor specific gotcha, software, or extra cool stuff, may behave differently. Funny enough, while I was thinking this is true for the webkit based browser only, there are many exceptions device specific that do not work as expected even natively ... oh, come on, don't call it OS!
If we think about Windows, almost every piece of hardware is supported. Then we have the classic crapware every single vendor puts by default in its own distributed piece of hardware ... fair enough, at least most basic things gonna work as expected in all of them ... right?
With Android, the most basic free version has almost nothing. Cyanogenmod is the basic example, once you install it, you gonna scream for apps you expect to be there and it comes up you need the specific Gapp package ( Google Applications ).
Same thing with Kindle Fire, based on the very basic version of Android so common apps available for other tablets may not work or may not be there.
This means we can trust only a truly basic set of functionalities that are hopefully working in all distributions out there but you know, this is IT, expectations are rarely matching reality.
The performances tuning per each device is another problem, as problematic are performances of the same app through the browser compared with those through the native layer.
If you wrap your single touch full page canvas game into phone Gap, as example, you will realize how freaking fast it is compared with the same exact code through the browser ... why that? Security layer is off plus bound stuff is limited compared with the whole browser application ... fair enough? No it's not. The moment I realize it is possible to force HW accelerated rendering via native wrapper but it's not possible to do the same via browser I start screaming ... let us develop through the web or stop telling us you support standards because if these are limited intentionally then those not supported are developers themselves.

Mobile Web Apps Too Limited

I really hope I won't need to write a similar post at the end of 2012 but I am not sure there will be many progresses here ...
W3C is slow, plus it may waste their/our time. Web SQL Database is an example, something I loved screwed by dependencies free purist.
I have talked about Web SQL many times and right now I don't care about this specific topic, all I care is about moving forward.
Time to agree and promote such interface to drop it after 2 years is not the way to go.
Time to replace such interface with another one limited, slower, and not widely adopted as IndexedDB is, still not the way to go.
We need a database, please put it there one ASAP and drop 50Mb per domain/tab limit.
No fools guys, if the logic behind the Web is that user must explicitly agree give the user full control.
If a web application requires more than 50Mb give the user the possibility to increase the database.
All native apps do not care about how much space is used, these simply informs the user through system settings how much stored data there is. Google Earth, as example, even maps, these are heavy memory applications that do not ask every 5 megabytes if more space can be allowed ... this was another epic fail.
Once the user accepts the fact this web application would like to use the storage, create a way to simplify the task able to verify how much data is being stored and stop asking!
From UX perspective is like accepting an explicit file upload/manipulation and while the file is being loaded into ram asking every 200Kb if the user would like to increase the memory allocation for that file ...


What If Every Web Page Asks For Space

Nothing, same is for "what if every native app takes 4 gigs of space" ... either space is available or it's not possible ... is that easy ... but again, with an easy to access indicator and a limit that goes far beyond 50Mb the problem should not exist. User can clean the cache, or enable more space but it must be easy, as easy should be for web apps developers to understand the available space because if the device is full already, the 50 Mb limits is pointless in any case plus there are ways to trigger memory consumption behind the scene and fill up gigs of disk space/SSD ... once again, this ask on demand every time is an epic fail, imho.
FYI, the behavior I am talking about is in iOS, where Android asks only once if the disk access is allowed, then it silently fails after we reach the 50Mb limit.

Native VS Web App

Every browser is based on a native web view or similar concept, which means every browser could expose anything. I would like to be able to record user voice via web, to decode it via audio tag, to create real time video chat, to replace all native applications and to forget Android, iOS, webOS, blackberry, Symbian, MeeGo, Windows, whatever SDK ... I want to leave the SDK power to much more complicated things and I would like to be able to do most basic things via browser, all those things that the user could do with native apps, performances a part.
The gap between native and web app should be performances and nothing else indeed ... right now is: all possible nasty access to everything on native side, things that few users are rarely aware of, and almost no access to any functionality through the browser.
No idea if Chrome OS solved this but if it did, I am pretty sure it did with in-house solution, same Windows 8 will do, and this is annoying at least until a jDevice library will come out able to make all these private API consistent across all platforms ... as summary, why Mobile Web App development has to be so hard is a complete mystery to me: is it about your own markets? Fine, then stop selling us your browser is promoting Standard Mobile Web development because this sounds, so far, like a big lie.

Patent Oriented Development

When I heard that Apple patented a slide to unlock the screen I did not feel like "they are so advanced" ... I rather felt like "is anyone trying to patent my finger" ?
I move horizontally my thumb since ever and nowadays I know there is a patent for that and this is bad.
A natural movement should never be patented ... the graphic used to guide the movement could eventually be a patent but the way I do things not.
I don't want a patent to hold a fork, I don't want a patent for the position I use to sleep in a mattress, whatever weird sofa, or a chair, I don't want to do unnatural movements to unlock a touch screen.
What the hell is going on here ... if I use my nose to unlock my old android device I don't have to pay the patent indirectly, right?
Is this what is patent about? Yes, it is.
Patents are about intellectual property of whatever but the problem is that patents are expensive, are not necessarily predictable, and all these patents are doing is to block future development on top of it, unless is not the same company.
Patents are a sort of creativity barrier because if a company invents a generic something, nobody will even try to improve that invention due basic patent behind the original.
Example, I use my right hand to hold the phone and when I am talking with someone I put my right thumb on the right side of the phone.
If there is a button there, it's annoying, while if I have to swipe from left to right ... well, it's still annoying.
Will I ever implement a swiping gesture from right to left to unlock the screen because usually the way I hold the phone is with my right finger on the right side and it's more convenient to me to swipe the other way round as it is convenient for all left hand users? Nope, I don't even want to remember that thing exist, I have to create something completely new being careful that what I am doing does not touch the other patent ... cool, uh?

Hardware Patents

Same concept could be applied for hardware ... was that thing perfect? Yes, I pay the patent. Was that thing not perfect? No, I have to create something new ... even if based on laser(disc), the reason we all waited so much for new standards after the CD.
This happened with Compact Discs, and it brought us DVD first and Blu Ray after. In the meanwhile, another standard better than CDs but a bit less capable than Blu Ray died ... fair enough, we have new hardware for all these technologies but how comes in the hardware field the patent problem is not so strong as it is for the Software one?
3D TV, all vendors are making one, as well as CD players, DVD players, and Blu Ray players ... nobody cares about patents here, at least not the way we all do when it comes to Software.
I don't have any answer for this but I believe patents should be re-evaluated in their meaning because in the fastest field ever, as technology/software is, something that last for 10 years is too much and price to pay is too much as well as too much is the time between the patent pending and the patent applied status.

ES6 Focus

I have talked about this as well ... it's about what's truly needed today, at least in the Mobile Web Development, and the JavaScript.Next focus, often completely abstract over not-so-needed topics as syntax sugar and boring classes.
iOS5 brought massive performances boost to all Apple devices but even having the fastest and complete browser out there, Safari Mobile cannot do miracles when it comes to performances.
The side effect of not having performances oriented techniques in JavaScript and DOM world is that average web developers are not able to create cool stuff without draining the battery or being slightly fast only with most recent mobile devices.
The need for performances in this Mobile era should be priority number one for W3C and a possible ES 5.2 milestone, after that take even 5 years to improve syntax with all sugar you want, if necessary, at least we can push Mobile Web Development to its limit in the meanwhile while many new proposals won't bring anything interesting, as far as I can read, performances speaking.
We can deal with the fact protected and private variables are absent or different in JS, what we cannot deal with anymore is that a proper and fast 3D API is not available, that Parallels are not available, that WebWorkers are useless when it comes to move data back and forward, that postMessage requires a window.open which require a click which is too slow but cannot be faked ontouchend or window.open won't work anymore because not explicit ... that IndexedDB does not provide native way to join data and speed up queries, that DOM manipulation is slow, that we don't control repaints and/or reflows, that binary Arrays are still slow, that Structs like classes/objects are missing, that CSS3 is bringing logic in the view with lack of control ... etc etc ...
I don't care if Proxy are in the browser, if these are 2X slower than good old JS on mobile, you know what I mean? We need more performances now and more APIs finalized to have access to hardware layers only SDK can expose without granting any better usage than an experienced web developer.

WebGL Partial Fail

First of all this is not yet a standard, regardless the effort of the kronos group, which means browsers like Internet Explorer will never adopt this ... fair enough if we can shim through Silverlight but here comes the second point: WebGL, as well as Silverlight, are not available for mobile.
Apple exposed it for Adds only, where adds are most likely those responsible for bad Web performances, generally speaking, due massive usage of Flash or arguable usage of automations to bring canvas Adds able to slow down all netbooks and tables out there .... congratulations, I'll never get this choices ... WebGL not for developers but for designers of adds only ... can I say brilliant?
Too few mobile devices are exposing WebGL if not none of them while basically every mobile device has a GPU compatible with Open GL 2.0 ES
Still experimental, true, and already available mainly for Chrome browser where Webkit, as well as Firefox, could do the same or truly similar. The direction WebGL is taking is not good at all and I can't wait to see cool demonstrations able to run with all WebGL capable browsers ... so far, not many of them ... and this is bad.
Sony enabled WebGL in its latest version of Android, still about fragmentation, so I am expecting at least every other vendor to bring WebGL in our current devices already capable ... it was a massive fail to drop it for the whole 2011, cool things may have been experimented already on devices plus typed Arrays could have been used more even for non WebGL related tasks.

CSS3 Partial Fail

As mentioned before, CSS3 are becoming the most messed part of the whole Web stack, mobile and/or desktop. Examples with counters to change dynamically text via :after selector ... is it just me? I mean ... dafuq is that?
Internet Explorer dropped the most misunderstood proprietary feature as expression was and webkit is introducing any sort of unmanageable logic inside CSS3? ... counters? webkitTransitionEnd withou webkitTransitionStart? No way to trigger properly transitions if not without asynchronous rendering? Lots of hacks to make them work cross platform with control through JS for something written in CSS?
CSS3 is bringing MVC in Web world without the C and if we would like to create robust behaviors/interactions/applications that make sense, programming speaking, we should forget the case where the user has no JavaScript because CSS3 is becoming indirectly strongly dependent on JavaScript for many reasons ... why not bring standard DOM method to control states via JS, rather than control partially what is supposed to happen via CSS3 only? I don't get it, I have seen cool stuff, but I have not seen cool stuff adopted cross browser and I am not seeing cool stuff easy to handle via JS and/or vice-versa.
CSS3 needs notification into the JS world or we gonna have Photoshop like power without tools to control what we are doing.

As Summary

I have more things to say, and more I forgot but I thought a rant on 2011 was needed at least to check, in one year, what happened, what improved, and what did not change at all.
I hope all people that are doing hard work on daily basis to promote a better web won't take anything I wrote personal and will simply consider the point of view of somebody that works with all these things on daily basis so ... it's my personal feedback, on what I think should change ASAP and specially for mobile web development.

Happy New Year Everybody

Wednesday, December 21, 2011

Coercion Performances

There are cases where JS coercion may be wanted/needed/necessary, at least logically speaking.

A classic case is a list of primitives, e.g. strings, or numbers, and a check we would like to perform without creating a new function each time.

// recycled function
function alreadyThere(value) {
// wanted coercion, no cast needed
return value == this;
// shitty code, logically speaking
// one cast per iteration
return value === "" + this;
}

var listOfNames = ["Me", "You", "Others"];

// the pointless check
listOfNames.some(
// the recycled callback
alreadyThere,
// the *passed as object* this
"You"
); // will be true

Now, for above specific case anyone would use an indexOf but this is not the point.
The point is that in some case we may want to do more complicated stuff and compare the result with this

// know if word was already in the dictionary
function alreadyThere(value) {
return value.toLowerCase() == this;
}

// convert the check once per iteration
listOfNames.some(
alreadyThere,
"yoU".toLowerCase()
); // still true


It's About Optimizations

The whole point is to try to perform as less runtime computations as possible.
Following this logic, we may decide to lower-case all items in the list once, and never again, but unfortunately this will require duplicated amount of RAM used per each collection.
The String#toLowerCase() is fast enough for a not so frequent check so why bother the RAM?
The optimization is also to avoid this.toLowerCase() per each entry of the dictionary.
The concept here is again simple, avoid duplicated entries in a generic list of chars, but things may be similar with numbers.

Performances

I have created a specific test able to understand performances gap between coercion and cast per iteration.
Surprising Google Chrome seems to be able to optimize in core once per iteration the cast, resulting almost twice as fast as competitors.
Specially on mobile, the in core coercion seems to be faster, sometimes almost twice as fast as the cast per iteration is.

JIT Oriented Development ?

I believe we should code logically, rather than trust JIT optimization. What I mean is that performances test are always welcome but we all know that better logic and algorithms are always winning in therms of performances and maintainability. In this specific case, specially where mobiles suffer the most, I would never suggest to do the cast per iteration: first of all because only one engine seems to be able to optimize such cast, but we don't know if a more complex object/scenario would perform that well, secondly because if I see a cast per each loop iteration I start smelling laziness all over the place ... the boxing/unboxing has always been a well known problem performances speaking, so how can a developer approve a logic similar to (String)alwaysSameObject per each iteration?

As Summary

Tools such JSLint should just do their business in these cases ... coercion is wanted/needed/logical, and unless every browser will show such gap in common tasks against coercion, I will rarely promote a cast per iteration ... and you know what? I believe that gap in Chrome, is a missed optimization in Chrome itself.

Tuesday, December 13, 2011

Please, Give Us Back __noSuchMethod__ !

For those who don't know what __noSuchMethod__ is here the quick summary: it was a bloody handy non-standard method able to provide a fallback whenever we invoked an object method that did not exist.

var o = {};
o.__noSuchMethod__(function (name, args) {
alert(name); // "iDoNotExist"
alert([].slice.call(args)); // 1,2,3
});
o.iDoNotExist(1, 2, 3); // will produce above alerts


A Bit Of Background

Well, if you are patient enough, you may consider to read this never-ending post in Mozilla mailing list.
The reason that post is called Proxies: get+fn vs. invoke is because Proxy supposes to be the new way to go able to bring us much more power than we probably ever need ... but hey, this is welcome, while what is not welcome, is that Proxy may not be implemented first, which means browsers vendors should have waited to remove __noSuchMethod__ 'cause right now we may not have a pseudo equivalent, and second, but surely not less important, , Proxy does not provide the same functionality.

The Minified Theory Against The Practice

The main argument from @BrendanEich is that JavaScript has properties only so that o.fn() is the equivalent of o.fn.apply(o).
While this is true with any normal object, this is totally different with __noSuchMethod__.
The equivalent of __noSuchMethod__ for that operation, and behind the scene, is:
  • is there a property in o or its __proto__ called fn ?
    • yes, proceed as usual as if it was o.fn.call(o) and throw error if that property was not callable
    • no, is there a __noSuchMethod__ callback to burn instead of throwing an error due undefined property?
      • yes, perform the current operation: nsm_callback.apply(o, arguments)
      • no, throw an error since property was undefined and obviously not callable
Got it? The equivalent of o.fn() in an environment where __noSuchMethod__ was supported is potentially different from o.fn.apply(o) ... I am 100% sure Brendan knows this before and better than me and this is the reason I don't really get his strongest point.
Once again, o.fn() may be the equivalent of nsm_callback.apply(o, arguments) and not o.fn.apply(o).

The Inexistent Theory Against The Practice

If above "reason" was not enough, I have read even worst in the same thread. I am sorry guys, but sometimes you must be realistic and understand that if a developer does, as example, this:

// de-context fn from o and invoke it
(o.fn)();

// exact equivalent of
var fn = o.fn; // GETTER, no invokation
// and after ...
fn()
// this is a problem? we have same with missing bind then ...

rather than this:

// invoke fn through o as default context
o.fn();
// can we see the difference?

it means that developer has much bigger problems than __noSuchMethod__ inexistent ambiguity, that developer does not even know that __noSuchMethod__ exist ... come on!
Going on and on, another point is that get should be all we need to simulate the __noSuchMethod__ behavior through proxies ... but this is completely misleading!

A Getter IS A Getter

Is that trivial ... if we access a generic object property we are doing nothing different from invoking a getter with such object as property context.

o.whatever;
// look for "whatever" property name in o
// if found returns the "whatever" associated value


o.whatever.call
// nothing change, THIS IS NOT AN INVOKE
// look for "whatever" property name in o
// if found returns the "whatever" associated value
// since the value was a function, the call method is usable
// if whatever was not defined, the call method won't exist

Nobody should ever even consider to use property accessor and expect a __noSuchMethod__ behavior ... that property did not exist, what kind of method would you expect to look for?
call is a property of the Function.prototype so following the accessor/getter logic, nothing is ambiguous here.
Accordingly, lattest example is simply an inexistent mistake that hopefully no developer would ever do ... but you know, shit happens, then we learn, then hopefully we don't repeat same shit.

Other Programming Languages

When it comes to PHP, they perfectly managed to make the behavior not ambiguous through the __call magic keyword in classes definition but no, we decided that in JavaScript we cannot even think to put an invoke to make the life easier and completely NOT ambiguous for all of us ... do we?
I still cannot understand where and what is the ambiguous part if we have an explicit invoke declaration ... maybe something a bit harder to solve behind the scene for these poor JS engines? It could be ... should we all limit JS because of this? I don't think so.

Think About Libraries APIs Migrations

I give you the most basic example, the most used JS library with a fake getter and setter behavior: jQuery.

// jQuery simulation of getters and setters behaviors

// the getter
$("body").html(); // return string with content

// the setter
$("body").html("
whatever
");
// set the string with content

If you want, specially for chainability reasons through the simulated setter, the fact html is a method is convenient for the library but this library is stuck forever behind these two methods.
jQuery, at current ECMAScript status, will never be able to switch gradually to real getters and setters ... why that? A simple example:

$("body").html; // returns the string in jQuery 3000

$("body").html(); // shows a "deprecated warning"
// ... and returns the "html" getter

// implementation example
function setInnerHTML(node) {
node.innerHTML = this;
}
Object.defineProperty($.fn, "html", {
get: function () {
return this[0].innerHTML;
},
set: function (html) {
this.each(setInnerHTML, html);
}
});

// the deprecation warning
$.fn.__noSuchMethod__(function (property, args) {
if (property in this) {
console.log("Warning: " + property + " is not a method anymore");
if (args.length) {
// invoke the setter
this[property] = args[0];
// preserve behavior
return this;
} else {
// invoke the getter
return this[property];
}
} else {
throw "Y U NO READ DOCUMENTATION";
}
});

That's it, we can migrate from two different APIs implementing getters and setters whenever we had a similar behavior and bringing gracefully users to the new usage ... no wa can't!

It Is Not About jQuery

I don't even use jQuery so don't get me wrong, this is not my battle here ... the point is that for another private project I am working on I would like to educate developers to use properties correctly but I understand developers may already got use to invoke methods as if it is normal, even when they are simply looking for a getter behavior.

var o = {
whatever:"cool bro",
__noSuchMethod__: function (property, args) {
console.log("we got a bro-blem here, " +
"don't invoke if you want a getter");
return this[property];
}
};

// so that
o.whatever === o.whatever();

Secially last line of code is apparently impossible to reproduce with Proxies, those that suppose to be the new and best way to go, those that give us control on things rarely needed until now, those that made JS.Next group decide that __noSuchMethod__ was evil and it had to be abandoned.
I really hope that JS.Next will not be non-developer expectations behaviors driven because guys, somebody tries to do cool things with this cool language, and if the reason you drop something is because we are all morons, as example misunderstanding the difference of a referenced property through parenthesis ... oh well ... good luck kkfuture JavaScript ...

How To Solve This

Please put an invoke or even better an invokeProperty, preserving invoke for when the object itself is used as if it was callable, in the current Proxy specifications so that who knows what is doing, can keep doing it and who never even bothered with this stuff, won't be affected at all.
Thank you for listening.

Monday, December 12, 2011

Create a JS builder with node.js

A very good and common practice with JS projects bigger than 100 lines of code is to split code in different files.
Benefits are clear:
  • smaller pieces of code to maintain
  • swappable portions for experiments and/or improvements or new features, as example including for a build magic2.js and get it, rather than change drastically magic.js and follow the repository logs
  • better organization of the code, and I'll come back on this in this post
  • possibility to distribute bigger closures, as example the jQuery approach
  • create ad hoc builds including or excluding portion of the library, specially suitable for specific version of the code that must be compatible with IE only


Solutions All Over The Place

There are really tons of solutions able to make the described build process easy to use and easy to go. As example, I have created my own one and I am using it with basically every project I am working with: the JavaScript Builder.
However, this builder requires a couple of extra technologies such Python and Java ... but aren't we using simply JavaScript?
So why not an easy to create guide on how to build your code via JS only?
This is what this post is about, and I hope you'll find useful.

How To Structure Your Project

If all files are in the same directory is not easy to find the right file immediately since these could be many. A good solution I came up with is folder related structure with both namespaces and private keywords paths.
Here an example on how I would structure this library ( and please ignore the library itself )

var myLib = (function (global, undefined) {"use strict";

// private scope function
function query(selector) {
return document.querySelectorAll(selector);
}

function Wrapper(nodeList) {
this.length = nodeList.length;
this._list = nodeList;
}

// a prototype method of the Wrapper "class"
Wrapper.prototype.item = function item(i) {
return this._list[i];
};

// public static query method
query.asWrapper = function (selector) {
return new Wrapper(query(selector));
};

var // private scope variables
document = global.document,
slice = [].slice
;

// the actual object/namespace
return {
query: query,
internals: {
Wrapper: Wrapper
}
};

}(this));

The code should be easy enough to understand. The object used as namespace for myLib has a couple of methods, few private variables and functions and something exposed through the internals namespace.
It does not matter what the library does or how good/badly is structured, what matters is that our folder structure should be smart enough to be able to scale with any sort of allowed JS pattern ... OK?

The Folder

Well, to start with, let's say our source code should be inside an src folder so we can add other folders for tests or builds beside in the same hierarchy.

dist
src
tests
builder.js

We'll see the builder.js later, in the meanwhile, let's have a look into the src folder:

dist
src
intro.js
outro.js
var.js
function
Wrapper.js
query.js
Wrapper
prototype
item.js
query
asWrapper.js
tests
builder.js

The distinction will be much cleaner once you read above list through your editor or even your shell ... query and files are well distributed but bear in mind this is only the first example.
Let's see what we are going to write into each file ?

src/intro.js

var myLib = (function (global, undefined) {"use strict";


src/function/query.js

// private scope function
function query(selector) {
return document.querySelectorAll(selector);
}


src/function/Wrapper.js

function Wrapper(nodeList) {
this.length = nodeList.length;
this._list = nodeList;
}


src/function/Wrapper/prototype/item.js

// a prototype method of the Wrapper "class"
Wrapper.prototype.item = function item(i) {
return this._list[i];
};


src/function/query/asWrapper.js

// public static query method
query.asWrapper = function (selector) {
return new Wrapper(query(selector));
};


src/var.js

var // private scope variables
document = global.document,
slice = [].slice
;


src/outro.js

// the actual object/namespace
return {
query: query,
internals: {
Wrapper: Wrapper
}
};

}(this));

Got it?

Structure Rules

  • every part of the scope can be distributed
  • each file can or cannot be compatible as stand alone with a parser because to test the library we need to build it first ( eventually with automations )
  • function declarations should be included in a dedicated folder called function accordingly with the nested level
  • var declaration per scope could be included in a folder var accordingly with the nested level. Do not create a var folder per each function where you define variables 'cause if you need it it means the function is too complex. Split it in sub task and do not define 100 variables per a single function: closures are the only exception.
  • nested closure must be named in order to be able to define nested closure structure following previous rules. Every minifier will be able to remove function expression names, included named closures, while not every developer would like to deeply understand the whole code to recognize why the nested closure was useful. A classic example is the inclusion inside our own closure of an external library that uses its own closure. in this case name that closure so you know were to look for the library inside your folder structure.
  • function prototypes should be placed inside a prototype folder, inside the function folder.
    We don't need to reassign an object when we want to pollute the function prototype so please stop this awkward common practice ASAP: MyFunction.prototype = { /* THIS IS WRONG */ } and use the already available prototype object defined by default in every ECMAScript standard and per each function declaration or expression.
    If your argument is that the code will be bigger, use the outer scoped variables definition to address the prototype once and reuse this reference within the prototype folder. This approach will make your life easier once you get use to work with structured and distributed JavaScript files.

Specially about last example, we could have set a shortcut to the Wrapper.prototype object in the var.js file and reuse the reference inside Wrapper.
The structured folders will always help you to find references in the library thanks to the lookup that you, as well as the code, have to do.

// in the var.js file
WrapperPrototype = Wrapper.prototype,

// in the Wrapper/prototype/item.js file
WrapperPrototype.item = function item(i) { ... };


The Order Partially Matters

In ECMAScript 3rd or higher edition function declarations are always available at the very beginning of the scope. I really don't know why these are so much underrated in daily basis code ... the fact these are always available means we can reference their prototype at any moment in our code:

var internalProto = (function () {

// address any declaration made in this scope
var WhateverPrototype = Whatever.prototype;
return WhateverPrototype;

// even if defined after a return!!!
function Whatever() {}
}());

alert(internalProto); // [object Object]

Now, the above code is simply a demonstration about how function declarations work ... I am not suggesting a return in the middle, and declarations after, all I am saying is that the order of things in JavaScript may not be relevant, and function declarations are a perfect example.
Another example is the usage of variables ... if a function, as declaration or as expression, reference a variable defined in the outer scope nothing will break unless we are invoking that function before the referenced variable has been defined.

This are really ABC concepts we all should know about JS before even claiming that we know JavaScript ... OK?
Is really important to get these points because to simplify ASAP the builder file we need to rely in these assumptions.

The builder.js File

It's time to create the magic file that will do the job for us in possibly a smart way so that we can cover all edge cases we could think of.
This is the content of builder.js file, in the root of our project

// @name builder.js
// @author Andrea Giammarchi
// @license Mit Style License

// list of files to include
var
scriptName = "myLib", // the namespace/object.project name
fileList = [
"intro.js", // beginning of the closure
"var.js", // all declared variables
"function/*", // all declared functions
"function/Wrapper/prototype/*", // all methods
"function/query/*", // all public statics
"outro.js" // end of the library
],
fs = require("fs"), // file system module
out = [], // output
alreadyParsed = [] // parsed files for visual feedback
;

// per each file in the list ...
fileList.forEach(function addFile(file) {
// if the file contains a wild char ...
if (file.charAt(file.length - 1) == "*") {
// read the directory and per each file found there ..
fs.readdirSync(
__dirname + "/src/" + file.slice(0, -2)
).forEach(function (file) {
// if the file type is js
// and the file has not been defined explicitly
// in the original list
if (
file.slice(-3) == ".js" &&
fileList.indexOf(file) < 0
) {
// call this same function providing the whole path
addFile(this + file);
}
// the path is passed as context to simplify the logic
}, file.slice(0, -1));
// if the file has not been included yet
} else if (alreadyParsed.indexOf(file) < 0){
// put it into the list of already included files
alreadyParsed.push(file);
// add the file content to the output
out.push(fs.readFileSync(__dirname + "/src/" + file));
} else {
// if here, we are messing up with inclusion order
// or files ... it's a nice to know in console
try {
console.log("duplicated entry: " + file);
} catch(e) {
// shenanigans
}
}
});

// put all ordered content into the destination file inside the dist folder
fs.writeFileSync(__dirname + "/dist/" + scriptName + ".js", out.join("\n"));

// that's it

The reason there are so many checks if a wild char is encountered is quite simple ... the order may not matter but in some case the order matters.
If as example a prototype property is used runtime to define other prototype methods or properties, this cannot be pushed in the output randomly but at the very beginning, example

// src/function/Wrapper/prototype/behavior.js
WrapperPrototype.behavior = "forEach" in [];

// src/function/Wrapper/prototype/forEach.js
WrapperPrototype.forEach = WrapperPrototype.behavior ?
function (callback) {[].forEach.call(this._list, callback, this)} :
function (callback) { /* a shim for non ES5 compatible browsers */ }
;

Being file 2 strongly dependent on file 1, the list of files could be written as this:

fileList = [
"intro.js", // beginning of the closure
"var.js", // all declared variables
"function/*", // all declared functions
"function/Wrapper/prototype/behavior.js", // precedence
"function/Wrapper/prototype/*", // all methods
"function/query/*", // all public statics
"outro.js" // end of the library
],

When the wild char will be encountered and the behavior passed to the forEach, this will be simply ignored since it has been pushed already in the previous call.
Same concept could happen if a specific file must be parsed runtime at the end:

fileList = [
"function/Wrapper/prototype/behavior.js", // precedence
"function/Wrapper/prototype/*", // all methods
"function/Wrapper/prototype/doStuff.js" // after all
],

I believe these are already edge cases most of the time but at least now we can better understand what the builder will do.

How To Use The Builder

In console, inside the project folder where the builder.js is:

node builder.js

That's pretty much it ... if you try to open dist/myLib.js after above call you will find your beautiful library all in one piece and ready to be minified, debugged, and tested.
If the process does not take long time you may bind the builder to the Constrol+S action with a potential sentinel able to inform you if any problem occurred, as example checking if the output has been polluted with some redundant file logged through the process.

As Summary

All these techniques may be handy for many reasons. First of all it's always good to maintain a structure, rather than a single file with thousands of lines of code, and secondly once we understand how the process work, nothing can stop us to improve, change, make it ad-hoc for anything we may need such regular expressions to strip out some code before the output push or whatever else could come up for some reason at some point.
The minification can be done the way you prefer, as example adding this single line of code at the end of the process assuming you have a jar folder with, as example, google closure compiler.

require('child_process').exec(
['java -jar "',
__dirname + "/jar/compiler.jar",
'" --compilation_level=SIMPLE_OPTIMIZATIONS --language_in ECMASCRIPT5_STRICT --js "',
__dirname + "/dist/" + scriptName + ".js",
'" --js_output_file "',
__dirname + "/dist/" + scriptName + ".min.js",
'"'].join(""),
function (error, stdout, stderr) {
if (error) console.log(stderr);
}
);

Enjoy your new builder :)

Tuesday, December 06, 2011

On JSON Comments

Another active exchange with @getify about JSON comments and here my take because tweets are cool but sometimes is hard to tell everything you think in 140 bytes ...

Kyle Facts

JSON is used on daily basis for billion of things and configuration files are one, surely common, way to use JSON ( just think about npm packages for node.js ).
His frustration about the fact JSON does not allow comments is comprehensible, and he even created an online petition about allowing comments in JSON specs ... but are comments really what we need?

Just A Side Effect

JSON is extremely attractive as standard, first of all because it's available and widely adopted by basically any programming language in this world, even those that never had to deal with a single JavaScript interlocutor, secondly because it's both simple to parse, and easy to read for humans.
After all, what can be so wrong about comments inside such common serialization standard?
Aren't comments just as easy to parse as white spaces?
The problem is, in my opinion, that we are mixing up an easy process to serialize data, much easier compared to what PHP serialize and unserialize functions do, with the possibility to describe it.
I have always seen JSON as a protocol, rather than a YAML substitute, and as a protocol I expect to be as compact as possible and as cross platform as possible.
Specially about the latter point:
that's annoying to port to every single language
Precisely, what we should understand is that if JSON became so popular without needing comments, maybe the fact today we would like to use it as a "descriptive markup" does not reflect anymore the success, the adoption, and possibilities this standard brought to all these languages?

Improve The Standard

Thanks gosh software is not always stuck behind immutable standards or patents ... and neither is JSON.
If the need for comments is such big topic, define another standard able to combine the good old one with a new one.
If this new standard is truly what developers need, every JSON implementor will spend few hours to test and optimize the well defined standard in order to accept comments, isn't it?
I mean ... RFC 4627 was not meant to be the final solution, that was a Crockford proposal universally adopted.
To create a new standard able to extend RFC 4627 should not be a big problem ... or maybe ...

JSON Is Not A JavaScript Thing

The JSON serialization looks just like JavaScript ... but not JavaScript only.
Other programming languages use curly brackets and squared brackets to define lists and objects ( Python and others ) ... the fact JSON has been accepted so well is probably because the design of the format was indeed already widely adopted, it was not a JS thing, and never should be.
What's the deal here, is define a standard for JSON comments.
Let me better explain this point ...

I am a JS developer and I edit my JSON files via my JS editor ... fair enough ... I want to communicate my data to a server side service, let's say Python.
Python would like to be able to parse my data and produce a file compatible with ... Python, of course.
Does it mean that Python at that point should keep comments in a JavaScript standard? And why that, since the format used to exchange data was already somehow evaluable via Python and right now not anymore due some double slash in the file?

# this is PYTHOOOOOOOOOON
o = {"test": "data"}
o.get("test") # data

Will the renewed JSON work as well?


# still python
o = {"test": "data"} // and some WTF
o.get("test")

>> SyntaxError: invalid syntax

Well done ... making JSON with comments a JS thing python, as other languages, need extra effort to parse data.
What will happen once Python uses the json library?

import json

jsonh.loads(theReceivedData);

Should it print data with python compatible comments or with JS compatible comments so that data is not directly usable anymore for any python application, storing it as example in a file.py or a database ?
And once the renewed JSON has been transformed into Python valid syntax, wouldn't this file not be usable anymore from all others programming languages due possible syntax errors ?

Not That Easy

I like JSON as it is, except few broken implementations quite common even on browsers, because it's about creating a bloody piece of text many other languages can understand basically on the fly.
No need to remember which comment style has been saved with that version of JSON, no need to parse back and finally, no need to ask every single programming language that is using JSON as protocol to update their legacy ... that just worked, and it will aways work for what it was meant: transfer data, not transfer JS like code without functions and/or functions calls only ...
What we are doing, we as JavaScript developer, is to abuse JSON as if it's a piece of our JS code, polluting today with comments, and who knows what else tomorrow.
The right way to go, still in my opinion, would be, once again, to enrich, propose, create, a new standard that allows comments and why not, other features.
As example, what I always found annoying is that in PHP we can unserialize preserving the class, so that we can serialize objects states ... where is this in JSON?
Nowhere, indeed I have spent in my past few hours trying to enrich this protocol ... did it work? Was it interesting ? Probably not, except for my last attempt that is 100% based on current JSON and it's about optimizing bandwidth and performances ... that worked better, accordingly with JSONH github status, still I was not expecting every other that never had this problem to adopt that approach ... you know what I mean?

Still Valid JSON

If it's about writing comments, next snippet is perfectly valid JSON string. All we need to do is to use the replacer in a proper way:

{
"@description": "something meaningful",
"property": "value",

"@description": "something else",
"other": "other value"
}

// parse above text via JSON
console.log(JSON.parse(aboveText, function (key, value){
if (key.charAt(0) != "@")
return value;
}));

Here we are with our object, comments manually written on the original JSON, and every language able to parse them