Complex Value Array in Stylus

Stylus is an awesome CSS pre-processor, which provides much more concise syntax and more powerful feature than its competitors, such as LESS or SCSS.

But now, with more and more features added into Stylus, it seems its syntax become over-weighted. Pitfall come up.

I wish to declare an array of values for box-shadow property. And I can reference them with index:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
drop-shadows = [
0 2px 10px 0 rgba(0, 0, 0, 0.16),
0 6px 20px 0 rgba(0, 0, 0, 0.19),
0 17px 50px 0 rgba(0, 0, 0, 0.19),
0 25px 55px 0 rgba(0, 0, 0, 0.21),
0 40px 77px 0 rgba(0, 0, 0, 0.22)
]
drop-shadow(n)
box-shadow shadows[n]
for i in (1..5)
.drop-shadow-{i}
drop-shadow(i)

And expect it generates

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
.drop-shadow-1 {
box-shadow: 0 2px 10px 0 rgba(0, 0, 0, 0.16);
}
.drop-shadow-2 {
box-shadow: 0 6px 20px 0 rgba(0, 0, 0, 0.19);
}
.drop-shadow-3 {
box-shadow: 0 17px 50px 0 rgba(0, 0, 0, 0.19);
}
.drop-shadow-4 {
box-shadow: 0 25px 55px 0 rgba(0, 0, 0, 0.21);
}
.drop-shadow-5 {
box-shadow: 0 40px 77px 0 rgba(0, 0, 0, 0.22;
}

But I found there is not such thing called Array in Stylus!!!!
There is only Hash, and Hash doesn’t accept number as key!
So finally, I come up something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
drop-shadows = {
'1': 0 2px 10px 0 rgba(0, 0, 0, 0.16),
'2': 0 6px 20px 0 rgba(0, 0, 0, 0.19),
'3': 0 17px 50px 0 rgba(0, 0, 0, 0.19),
'4': 0 25px 55px 0 rgba(0, 0, 0, 0.21),
'5': 0 40px 77px 0 rgba(0, 0, 0, 0.22)
}
drop-shadow(n)
box-shadow shadows[n+'']
for i in (1..5)
.drop-shadow-{i}
drop-shadow(i)

In this piece of code, there are a bunch of pitfalls:

  1. Hash doesn’t accept number as key. So 1: 0 2px 10px 0 rgba(0, 0, 0, 0.16) cause compile error.
  2. '1' != 1, so drop-shadows[1] returns null
  3. There is no type conversion function in Stylus, use the same trick as JavaScript. ''+n convert n into string.

Just found Stylus provides something called List, which is pretty much similar to what array in other languages, except the syntax.

1
2
3
4
5
6
7
8
9
10
11
12
13
drop-shadows = 0 2px 10px 0 rgba(0, 0, 0, 0.16),
0 6px 20px 0 rgba(0, 0, 0, 0.19),
0 17px 50px 0 rgba(0, 0, 0, 0.19),
0 25px 55px 0 rgba(0, 0, 0, 0.21),
0 40px 77px 0 rgba(0, 0, 0, 0.22)
drop-shadow(n)
box-shadow shadows[n]
for i in (1..5)
.drop-shadow-{i}
drop-shadow(i)

So no brackets or parentesis needed.

Page renders improperly in IE before developer tool has opened

Today I found a super annoying issue about IE. Our website works perfectly in any browser except IE. The page isn’t rendered properly in IE 9. Well, this is common, this is the nature of IE. The mysterious issue I found is that once you opened or ever opened the developer tool, open the page or refresh the page, the problem is gone magically!!!!

As a conclusion, opening the developer tool changes the browser behavior!!!!! What a hell! So you know there is something wrong, but once you try to figure out the error message, you have to open developer tool. Once you open the developer tool, the bug is gone! DEAD END!!!

Because I cannot open developer tool, so I have to debug with alert. It is really a horrible experience to me, and feels like inspecting nuclear reaction with a plain optical magnifier or fixing a high-tech spacecraft with stones and clubs.

Since it is client-rich page, a lot of javascript is introduced. So I cannot go through the scripts line by line, instead I have to make an assumption to explain the phenomena spotted, then validate it with experiments, finally correct or extend the assumption according the validation result.

During the process I invalidated a couple of assumptions, some of them are seems very close to the “right answer”, such as “some script is loaded and executed before its dependencies, and developer tool load all the scripts first because it displays all the scripts”.

After spending a couple of hours on it, I put on eye on a line of code that is really out of my expectation: console.warn.

Code breaks the page rendering
1
2
3
console.warn('__proto__ is not supported by current browser, fallback to hard-copy approach');

I displays a warning message to console when a workaround is applied. But a tricky fact about IE 9 is that console isn’t available until developer tool is opened (MSDN reference here))!!!

The fact that console is not available until developer tools is opened really blows my mind away! (Maybe it is because I have little experience work with IE). As a chrome user, I take console as the universal log system for javascript. But in IE, according to the document), the code should check the existence of console every time print a log.

There is another pitfall here, and I saw someone really post it as answer on StackOverflow:

Bad polyfill implemntation
1
2
3
4
5
if (typeof console == "undefined") {
this.console = { log: function (msg) { alert(msg); } };
}

We usually access console as console.log, feels like console is a global instance to access. But actually console is an member of window, its full name should be window.console. When console exists, we can definitely reference to it via console. But if it doesn’t exist, the statement console cause script error! So the following code doesn’t work:

Pitfalls in console existence check
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
if (typeof(console) === 'undefined'){ // Break the script execution
console.log('Never got executed');
}
if (console != null) { // Break the script execution
console.log('Never got executed');
}
if (console) { // Break the script execution
console.log('Never got executed');
}
if (window.console) {
console.log('this works!');
}

To avoid console issue, a ployfill could be very useful. Here is a great implementation available as bower package: console-polyfill

Mac OS X case-insensitive file system pitfall

I was working on the YouTube video playback feature for LiveHall last night, and have it works successfully on my local devbox, which is running Mac OS X. Then I deployed the code to Heroku, without any regression.

But today morning, when I have the demonstrate the new features, I met server error! It says 1 of the 4 javascripts are missing, so the Jade template failed to render.

This is a very wield issue, then I try the same data on my local dev box once again, and it works perfect! But it does yield error on the production! Then I tried to use heroku toolbelt to run ls command on the production, and I found the there are 4 coffee scripts there.
Then I tried to enforce heroku to redeploy the app by using git push --force, but the issue remains!
Then I even tried to dive into the dependency pacakges connect-assets and snockets, but still found nothing useful.

Same code, same data, but different result! Very odd issue!

After half an hour fighting against the code, I suddenly noticed I the file name is RevealJSPresenter.coffee, whose “S” is capital S! And I reference the file with name #= require ./presenter/RevealJsPresenter, whose ‘s’ is a lowercase ‘s’!

And snockets depends on the OS feature to locate the file. On my local dev environment, I’m using Mac. Although OS X allow user to explicitly format the HFS+ into file name case sensitive mode, but it is case insensitive by default. So snockets can locate the file even the case is wrong.
But once I have deployed to heroku, which, I assume, runs Linux, whose file system is absolutely filename case sensitive. So the snockets won’t be able to locate the file due to the case issue.

To resolve the bug, I renamed my file in RubyMine, then try to commit in terminal.
But when I commit, I met another very interesting issue, that git says there is no file changed, so it refused to commit.
It is still the same issue, due to FS is case insensitive, git cannot detect the renaming.

This problem is more common when coding on Windows, but CI or production runs on Linux. And the very common solution is to pull the code in case sensitive environment, then rename the file and commit it.

But I found another more easier way to do it:

Using git mv in terminal to rename the file, which will enforce git to track the file renaming action.

Or

Most of Git GUIs are able to track file name case changing, so you can try to commit the change with the tool, such as RubyMine or SourceTree.

Pitfall in matching line head and line end in regexp

I usually uses \^\ and \$\ to verify user input, e.g:
I uses following regexp to verify whether a user input is valid gmail email address:

Matching Gmail
1
^[a-zA-Z_\.]+@gmail.com$

But in fact it is potentially vulnerable!
According to the RegExp document, ^ and $ is matching to line head and line end!
So I might rush into pitfall when user try to fool me with following input:

bad input
1
"hacker@gmail.com\n<script>alert('bang!')</script>"

Since there is a \n in the string, so $ won’t really match to the end of the string but actually matched to the \n, then the whole string become a valid input, but actually it isn’t!

To avoid such issue, we should stick to \A and \z, which is literally means the the beginning of the string and end of the string!

Pitfall in node crypto and base64 encoding

Today, we found there is a huge pitfall in node.js crypto module! Decipher has potential problem when processing Base64 encoding.

We’re building RESTful web service based on Node.js, which talks to some other services implemented with Ruby.

Ruby

In ruby, we use the default Base64 class to handle Base64 encoding.

Base64#encode64 has a very interesting feature:
It add line break (\n) to output every 60 characters. This format make the output look pretty and be friendly for human reading:

Ruby Base64 Block
1
2
3
4
5
6
7
MSwyLDMsNCw1LDYsNyw4LDksMTAsMTEsMTIsMTMsMTQsMTUsMTYsMTcsMTgs
MTksMjAsMjEsMjIsMjMsMjQsMjUsMjYsMjcsMjgsMjksMzAsMzEsMzIsMzMs
MzQsMzUsMzYsMzcsMzgsMzksNDAsNDEsNDIsNDMsNDQsNDUsNDYsNDcsNDgs
NDksNTAsNTEsNTIsNTMsNTQsNTUsNTYsNTcsNTgsNTksNjAsNjEsNjIsNjMs
NjQsNjUsNjYsNjcsNjgsNjksNzAsNzEsNzIsNzMsNzQsNzUsNzYsNzcsNzgs
NzksODAsODEsODIsODMsODQsODUsODYsODcsODgsODksOTAsOTEsOTIsOTMs
OTQsOTUsOTYsOTcsOTgsOTksMTAw

The Base64#decode64 class ignores the line break (\n) when parsing the base64 encoded data, so the line break won’t pollute the data.

Node.js

Node.js take Base64 as one of the 5 standard encodings (ascii, utf8, base64, binary, hex). Ideally the data or string can be transcoded between these 4 encodings without data loss.

The Buffer class is the simplest way to transcode the data:

Base64 Encoder in Node.js
1
2
3
4
5
6
7
8
Base64 =
encode64: (text) ->
new Buffer(text, 'utf8').toString('base64')
decode64: (base64) ->
new Buffer(base64. 'base64').toString('utf8')

Although encode64 function in node.js won’t add line break to the output, but the decode64 function does ignore the line break when parsing the data. It keeps the consistent behavior with ruby Base64 class, so we can use this decode64 function to decode the data from ruby.

Since base64 is one of the standard encodings, and some of the node.js API does allow set encoding for input and output. So ideally, we can complete the base64 encoding and decoding during processing the data.
It seems Node.js is more convenient comparing to Ruby when dealing with Base64.

e.g. We can combine reading file and base64 encoding the content into one operation by setting the encoding to readFileSync API.

Write and Read string as Base64
1
2
3
4
5
6
fs = require('fs')
fileName = './binary.dat' # this file contains binary data
base64 = fs.readFileSync(fileName, 'base64') # file content has been base64 encoded

It looks like we can always use this trick to avoid manually base64 encoding and decoding when the API has encoding parameter! But actually it is not true! There is a BIG pitfall here!

In our real case, we uses crypto module to decrypt the the JSON document that encrypted and base64 encoded by Ruby:

Base64 Deocde and Decrypt
1
2
3
4
5
6
7
8
9
10
11
crypto = require('crypto')
parse = (data, algorithm, key, iv) ->
decipher = crypto.createDecipheriv(algorithm, key, iv)
decrypted = decipher.update(data, 'base64', 'utf8') # Set input encoding to 'base64' to ask API to base64 decode the input before decryption
decrypted += dechiper.final('utf8')
JSON.parse(decrypted)
Manually Base64 Decoding
1
2
3
4
5
6
7
8
9
10
11
12
13
crypto = require('crypto')
parse = (data, algorithm, key, iv) ->
decipher = crypto.createDecipheriv(algorithm, key, iv)
binary = new Buffer(data,'base64') # Manually Base64 Decode
decrypted = decipher.update(binary, 'binary', 'utf8') # Set input encoding to 'binary'
decrypted += dechiper.final('utf8')
JSON.parse(decrypted)

The previous 2 implementations are very similar except the second one base64 decoded the data manually by using Buffer. Ideally they should be equivalent in behavior. But in fact, they are NOT equivalent!

The previous implementation throws “TypeError: DecipherFinal fail”.
And the reason is that the shortcut way doesn’t ignore the line break, but Buffer does!!! So in the previous implementation, the data is polluted by the line break!

Conclusion

Be careful, when you try to ask the API to base64 decode the data by setting the encoding argument to ‘base64’. It has inconsistent behavior comparing to Buffer class.

I’m not sure whether it is a node.js bug, or it is as is by design. But it is indeed a pitfall that hides so deep. And usually is extremely hard to figure out. Since encrypted binary is hard to human to read, and debugging between 2 languages are also kind of hard!

Pitfall in Nokogiri XPath and Namespace

Nokogiri is a really popular Xml and Html library for Ruby. People loves Nokogiri is not just because it is powerful and fast, the most important is its flexible and convenient.
Nokogiri works perfect in most aspects, but there is a big pitfall when handling the xml namespace!

I met a super weird issue when processing xml returned by Google Data API, and the API returns the following xml document:

API response
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns="http://www.w3.org/2005/Atom" xmlns:media="http://search.yahoo.com/mrss/" xmlns:yt="http://gdata.youtube.com/schemas/2007">
<entry>
<media:group>(...)</media:group>
<yt:position>1</yt:position>
</entry>
<entry>
<media:group>(...)</media:group>
<yt:position>2</yt:position>
</entry>
<entry>
<media:group>(...)</media:group>
<yt:position>3</yt:position>
</entry>
<entry>
<media:group>(...)</media:group>
<yt:position>4</yt:position>
</entry>
</feed>

I instantiated a Nokogiri::XML DOM with the xml document, and then I try to query the DOM with XPath: xml_dom.xpath '//entry':

Query DOM
1
2
xml_dom = Nokogiri::XML Faraday.get api_url
entries = xml_dom.xpath '//entry'

I’m expecting entries is an array with 4 elements, but actually it is empty array. After a few tries, I found the query yields empty array when I introduce the element name in the query.

Try Xpath Queries
1
2
3
4
5
xml_dom.xpath '.' # returns document
xml_dom.xapth '//.' # returns all element nodes
xml_dom.xpath '/feed' # returns empty array
xml_dom.xpath '//entry' # returns empty array
xml_dom.xpath '//media:group', 'media' => 'http://search.yahoo.com/mrss/' # returns 4 the media:group nodes

It is super weird.

After half an hour fighting against the Nokogiri, I begin to realize that it must be related to the namespace.
And I found that there is an attribute applied to the root element of the document: xmlns="http://www.w3.org/2005/Atom", which means all the elements without explicit namespace declaration in the xml dom are under the namespace http://www.w3.org/2005/Atom by default.

And for some reason, the XPath query is namespace sensitive! It requires the full name rather than the local name, which means we should query the DOM with the code: xml_dom.xpath '//atom:entry', 'atom' => 'http://www.w3.org/2005/Atom'.

Fixed XPath Queries
1
2
3
4
5
xml_dom.xpath '.' # returns document
xml_dom.xapth '//.' # returns all element nodes
xml_dom.xpath '/atom:feed', 'atom' => 'http://www.w3.org/2005/Atom' # returns root node
xml_dom.xpath '//atom:entry', 'atom' => 'http://www.w3.org/2005/Atom' # returns 4 entry nodes
xml_dom.xpath '//media:group', 'media' => 'http://search.yahoo.com/mrss/' # returns 4 the media:group nodes

So in a sentence: XPath in Nokogiri doesn’t inherit the default namespace, so when query the DOM with default namespace, we need to explicitly specify the namespace in XPath query. It is really a hidden requirement and is very likely to be ignored by the developers!

So if there is no naming collision issue, it is recommeded to avoid this kind of “silly” issues by removing the namespaces in the DOM. Nokogiri::XML::Document class provides Nokogiri::XML::Document#remove_namespaces! method to achieve this goal.

pitfall when return string in via json in rails

Today we met a weird problem when return a string via json.

Here is the coffee script code:

Front End
1
2
3
4
$.post serverUrl, data, (status) ->
console.log status

And here is our controller:

Backend Action
1
2
3
4
5
6
7
def action
# do some complex logic
render json: "success"
end

Code looks perfect, but we found that the callback is never called! When we check the network traffic, you will found that server does send its response “success”, but the callback is not called!

After spending half an hour to struggle against the jQuery, we finally find the problem!

The reason is that success is not a valid json data! A valid json string should be quoted with “”, or JSON parser will treat it as token, like true or false or nil.

So to fix the problem, we need to change our action code:

Fixed Backend Action
1
2
3
4
5
6
7
def action
# do some complex logic
render json: '"success"'
end

This is really a pitfall, since the wrong code looks so nature!

A pitfall in jQuery form serialization

Today, I was so surprised that I got an empty string when I call the serialize method on a jQuery wrapped form.
The html is written in Haml:

Html
1
2
3
4
5
6
7
%form#graph-option.horizontal-form
%fieldset
%label{ :for=>'start-date'} Start Date
%select#start-date
%label{ :for=>'end-date'} End Date
%select#end-date
%button#submit-option.btn.large-btn

And the script is written in coffee-script:

Script
1
2
3
4
5
6
7
$ ->
$('#submit-option').click ->
option = $('#graph-option').serialize()
$.post '/dashboard/graph', option, (data) ->
renderCharts data

When I execute the script, I got 500 error. And the reason is that the option is empty.
I believe this must be caused by a super silly mistake, so I try to call serialize methods on Twitter Bootstrap website, and I still got empty string!!!!

After half an hour debugging, I just realize that I forgot to assign the name to all the input elements. And according to html specification, the browser uses the name of the elements to identify whom the value belongs to.
So when the name is omitted, the serailizeArray method in jQuery returns an empty array, as a result, the serialize method returns empty string.

According to my experience, it is easy to identify this problem, if the html is in html-like format, such as erb. But it is really hard to identify this issue if the page is written in haml, because in haml, id is used much more often.
To fix this problem, we need to specify the name explicitly for each form element.

Here is the fixed haml code:

Html
1
2
3
4
5
6
7
%form#graph-option.horizontal-form
%fieldset
%label{ :for=>'start-date'} Start Date
%select#start-date{ :name=>'start-date' }
%label{ :for=>'end-date'} End Date
%select#end-date{ :name=>'end-date' }
%button#submit-option.btn.large-btn

Name trap in Rails

I’m a newbie to Rails, and my past few projects are all rails based, including MetaPaas, Recruiting On Rails and current SFP.
I was amazed by the convenience of Rails, and also hurt by its “smartness”.
The power of Rails was described in quite a lot of posts, so I wanna to share some failure experiences.
Actually I have felt into quite a number of pitfalls in Rails, and here is one of the most painful ones.

To explain the problem easier, I just simplify the scenario:
I have a model called “Candidate”, which holds a “Status” to store the status of the candidate, so I have the code like this:

Candidate and Status
1
2
3
4
5
6
7
8
9
10
11
class Candidate < ActiveRecord::Base
has_one status
# other definition
end
class Status < ActiveRecord::Base
belongs_to candidate
# other definition
end

For some reason, I change the relationship between Candidate and Status. It is changed from one-to-one to one-to-many.
So I changed the has_one to has_many:

Candidate and Status
1
2
3
4
5
6
7
8
9
10
11
class Candidate < ActiveRecord::Base
has_many status
# other definition
end
class Status < ActiveRecord::Base
belongs_to candidate
# other definition
end

I thought it is an easy modification, but the app fails to run, even I have done the database migration.
It said rails cannot find a constant named “Statu”!

After my first sight on this error message, I believed it is caused by a typo, I must mistyped “Status” as “Statu”.
So I full-text search the whole project for “Statu”, but I cannot find any.

This error message is quite weird to me, since I have no idea about where the word “Statu” come from!
After spent half an hour on pointless trying, I suddenly noticed that the word “status” is end with “s”, and according to Rails’ convention, rails must think “status” is the plural form of “statu”. So according to the convention again, it try to find a class named “Statu”.
And we should use the plural form noun as name for one-to-many field, since that holds an array rather than a single object.

So after changing status to statuses, the problem solved.

Convention based system is powerful, a lot magic just happened there. But also the magic things are hard to debug when some special case breaks the convention presumption.

Space Pitfall in coffee-script

Coffee Script had fixed quite a lot of pitfalls in Javascript. But on another hand it also introduced some other pitfalls, the most common one is the space.

Space in function declaration

Read the following code:

Show Message:Coffee
1
2
3
4
show = message ->
console.log message
show "space pitfall"

This is a quite simple script, but it failed to run. And if you might also feel confused about the error message: “message is not defined”

What happened to the code? We indeed had declared the message as argument of function show. To reveal the answer, we should analyze the compiled javascript.
Here is the compiled code:

Show Message:JS
1
2
3
4
5
6
7
8
// Generated by CoffeeScript 1.3.1
var show;
show = message(function() {
return console.log(message);
});
showe("space pitfall");

Look the fun declaration, you will see it is not a function declaration as we want but a function call.
The reason is that we omitted the parentheses around the argument and we add a new space between message and ->. So the coffee-script compiler interpret message as a function call with a function as parameter.

Soltuion
To fix this problem, we can remove the space between message and -> to enforce coffee-script compiler interpret them as a whole.

Show Message:Fix
1
2
3
4
show = message->
console.log message
show "space pitfall"

Best Practise
To avoid this pitfall, my suggestion is never omit the parentheses around the arguments, even there is only one argument.
And also including the function call, even coffee-script allow to omit the parentheses. Since you won’t able to chain the method call if you omit the parentheses.
So never omit parentheses, unless you are very certain that there is no any ambiguity and you won’t use method chain.

The space in array index

Since coffee script doesn’t support the negative index. So we should use following code as negative index:

Last Hero:Coffee
1
2
3
heros = ["Egeal Eye", "XMen", "American Captain", "IronMan"]
lastHero = heros[heros -1]
console.log lastHero

This piece of code is also failed to run, and the error message is “property of object is not a function”.
Quite wield right?
Let’s see what is behind the scene, here is the compiled code:

Last Hero:JS
1
2
3
4
5
6
7
8
// Generated by CoffeeScript 1.3.1
var heros, lastHero;
heros = ["Egeal Eyr", "XMen", "American Captain", "IronMan"];
lastHero = heros[heros.length(-1)];
console.log(lastHero);

Same problem, heros.length -1 is interpreted as heros.length(-1) instead of heros.length -1.
To fix this problem, we should write the code in following way:

Last Hero:Fix1
1
2
3
heros = ["Egeal Eye", "XMen", "American Captain", "IronMan"]
lastHero = heros[heros - 1]
console.log lastHero

Or

Last Hero:Fix2
1
2
3
heros = ["Egeal Eye", "XMen", "American Captain", "IronMan"]
lastHero = heros[heros-1]
console.log lastHero

Both solution is try to enforce the compiler divid the component in correct way.

And unfortunately, there is no way to avoid this problem, the only thing you can do is always be aware the spaces in expression.