Today, we found there is a huge pitfall in node.js crypto module! Decipher has potential problem when processing Base64 encoding.
We’re building RESTful web service based on Node.js, which talks to some other services implemented with Ruby.
In ruby, we use the default
Base64 class to handle Base64 encoding.
Base64#encode64 has a very interesting feature:
line break (\n) to output every 60 characters. This format make the output look pretty and be friendly for human reading:
Base64#decode64 class ignores the
line break (\n) when parsing the base64 encoded data, so the
line break won’t pollute the data.
Base64 as one of the 5 standard encodings (
hex). Ideally the data or string can be transcoded between these 4 encodings without data loss.
Buffer class is the simplest way to transcode the data:
encode64 function in node.js won’t add
line break to the output, but the
decode64 function does ignore the
line break when parsing the data. It keeps the consistent behavior with ruby
Base64 class, so we can use this
decode64 function to decode the data from ruby.
base64 is one of the standard encodings, and some of the node.js API does allow set encoding for input and output. So ideally, we can complete the base64 encoding and decoding during processing the data.
It seems Node.js is more convenient comparing to Ruby when dealing with
e.g. We can combine reading file and base64 encoding the content into one operation by setting the encoding to readFileSync API.
It looks like we can always use this trick to avoid manually base64 encoding and decoding when the API has encoding parameter! But actually it is not true! There is a BIG pitfall here!
In our real case, we uses
crypto module to decrypt the the JSON document that encrypted and base64 encoded by Ruby:
The previous 2 implementations are very similar except the second one base64 decoded the data manually by using
Buffer. Ideally they should be equivalent in behavior. But in fact, they are NOT equivalent!
The previous implementation throws “TypeError: DecipherFinal fail”.
And the reason is that the shortcut way doesn’t ignore the
line break, but
Buffer does!!! So in the previous implementation, the data is polluted by the
Be careful, when you try to ask the API to base64 decode the data by setting the encoding argument to ‘base64’. It has inconsistent behavior comparing to
I’m not sure whether it is a node.js bug, or it is as is by design. But it is indeed a pitfall that hides so deep. And usually is extremely hard to figure out. Since encrypted binary is hard to human to read, and debugging between 2 languages are also kind of hard!