In Linux and Unix world, there are 2 common used shell profiles: ~/.bashrc and ~/.bash_profile. These two profiles are usually used to initialize user bash environment, but there still are some slightly differences between them two.
According to bash manual, .bashrc is “interactive-shell startup file”, and .bash_profile is “login-shell startup file”.

What’s the difference between interactive-shell and login-shell

Basically, the login-shell means the shell opened when user log in via console. It could be the shell opened on local computer after you entered correct user name and password, or the shell opened when you ssh to a remote host.
So according to the bash_profile will be loaded only once, that’s right after you logged into a computer, either locally or remotely.

And, on the other hand, the interactive-shell could be more widely used, be seen more often. It is the shell opened after you logged in, such as the shell opened from KDE or Gnome.

Mac Terminal’s Pitfall

According to the manual, the Terminal App on Mac is the typical “interactive-shell”, so theoretically Terminal should load “.bashrc” to initialize the shell environment. But the fact is Terminal doesn’t load the “.bashrc”, instead it load “.bash_profile” for initialization.
So in a word, Mac’s Terminal doesn’t follow the routine strictly. We need to be aware it.

And not all the shell are interactive! If the shell is not interactive, the Terminal App won’t load the profile file to initialize the environment.
A typical non-interactive shell in the shell that TextMate used to run command script, which means in TextMate’s shell, these environment variables, path and even alias you used in you daily life might not be available for TextMate’s command.
And also the most hurt one, the rvm function also won’t be available in TextMate’s command shell, which means if you call rake or rails in TextMate’s command script, you are very possibly got error because it cannot find proper gem or other resources.
So you should always remember to source and run the “.bash_profile” file or setup these values once again.

I was confused about how require function works in node.js for a long time. I found when I require a module, sometimes I can get the object I want, but sometimes, I don’t I just got an empty object, which give an imagination that we cannot export the object by assigning it to exports, but it seems somehow we can export a function by assignment.

Today, I re-read the document again, and I finally make clear that I misunderstood the “require” mechanism and how I did that.

I clearly remember this sentence in the doc

In particular module.exports is the same as the exports object.

So I believed that the exports is just a shortcut alias to module.exports, we can use one instead of another without worrying about the differences between them two.
But this understanding is proved to be wrong. exports and module.exports are different.

Today I found this in the doc:

The exports object is created by the Module system. Sometimes this is not acceptable, many want their module to be an instance of some class. To do this assign the desired export object to module.exports.

So it says that module.exports is different from exports. And it you exports something by assignment, you need to assign it to module.exports.

Let’s try to understand these sentences deeper by code examples.

In the saying

The exports object is created by the Module system.

The word “created by” actually means when node.js try to load a javascript file, before executing any line of code in your file, the module system executes the following code first for you:

1

var exports = module.exports

So the actual interface in node.js’s module system is module object. the actual exported object is module.exports not exports.
And the exports is just a normal variable, and there is not “magic” in it. So if you assign something to it, it is replaced absolutely.

That’s why I failed to get the exported object I want when I assign the it to exports variable.

So to export some variable as a whole, we should always assign it to module.exports.
And at same time, if there is no good excuse, we’d better to keep the convention that exports is the shortcut alias to module.exports. So we should also assign the module.exports to exports.

As a conclusion, to export something in node.js by assignment, we should always follow the following pattern:

1
2
3

exports = module.exports = {
	...
}

Powershell and .net fx provides a dozen of approaches to manipulate hash-object. And it is really easy and convenient to initialize hash-object with values from environment variables, registry or cli arguments.
And Hash-Object can be accessed and built into hierarchy easily, so to use powershell hash-object as deploy configuration is really powerful and convenient.

But in our system, the application uses the ini-like key-value pair plain text as the initial configuration file. So our deploy script need the ability to serialize and deserialize hash-object to and from ini-like config.

So I composed this piece of script.

We found a computer name issue when building our private cloud environment on CloudStack over KVM. We found that KVM doesn’t support to rename new created instance automatically.
As a result, all the instance booted from the same disk image have the exactly same computer name, same administrator password.

So we need to do some manual provision work before user can use the new booted instances.
Administrator password can be changed in several ways, it is not a difficult job. But to rename the computer running Windows is not an easy work. The typical way is to call the WMI interface to rename the computer, which is taken as the most “formal” and “documented” way to rename the computer. But this approach require to reboot the instance, which is what we don’t like.

So we try some “hack” way to solve this problem, we use powershell scripts to hack the registry. By doing this, we can rename the computer without rebooting, and it works fine on our environment.
But since it is a hacking way, it only changed the most common values in registry, which means it won’t modify the rare ones and all kind of cached values. Such as the values cached in the SQL Server or MSMQ service, etc. So there might be some unknown potential issues. Take if on your own risk:

Here is the gist:

This is a script to populate computer hardware configuration in Linux.

1
2
3
4
5
6
7
8
9

#!/bin/bash
# CPU
cat /proc/cpuinfo |  grep -e "model name" -e "cores" -m 2 && cat /proc/cpuinfo | grep -e "physical id"| sort  | uniq -c
# Memory
free -got
# Disk
df -lh -t ext4 --total

Here is a bash script that acts as a TextMate command, which enables blogger to create new post without leaving TextMate.
It depends on “Post” task of the Rakefile in Jekyll Bootstrap.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

cd $TM_PROJECT_DIRECTORY
title=$(CocoaDialog standard-inputbox \
 --title "New Post" \
 --informative-text "Title of the new post:")
[[$(head -n1 <<<"$title") == "2" ]] && exit_discard
title=$(tail -n1 <<<"$title")
output=$(rake post title="$title")
expr="^\(in (.*)\) Creating new post: (.*)$"
if [[$output =~ $expr ]]; then
   	project=${BASH_REMATCH[1]}
	post=${BASH_REMATCH[2]}
	echo "new post file created at $post"
	exit_show_tool_tip
else
   	echo "Error"
	exit_show_tool_tip
fi

I have spent almost a week to fight against the code highlight system in jekyll, but finally failed.

At very beginning, I tried the “legendary” “github flavored markdown”, which supports use “```” to quote and highlight code. But soon, I failed.

Then I tried pygements, but sooner I found the perfectly local rendered code fails to be rendered on github pages, the reason seems caused because the github use a older version of the pygements.

Since I cannot fully control pygements as I expected, so I turned to a javascript version code highlighter, SyntaxHighliter.

Soon, I found SyntaxHighlighter cannot cooperate with jekyll’s markdown compiler well, so I tried several approaches to make them work together. I change part of the SyntaxHighlighter’s implementation and added several new includes liquid template based on Jekyll Bootstrap.
But it still cannot work perfect.

The problem is that jekyll markdown parser cannot distinguish the html snippet perfect, it usually try to parse the code as html.
Then I tried to quote my code with CData tag, but I found the generated html are not consistent. Some of which the “<” and “>” are escaped with “&gt;” and “&lt;”. And some other are not escaped but quoted with “”. It is really hard to deal with so many cases in javascript.

So I gave up. I gave up to use the github to render my pages. So I guess Octopress must be a good option for me.
Since Octopress is based on jekyll but with a lot of new goodies. These new liquid markups are powerful and can fulfill my requirements.

I’ve been working on MetaPaas project for a while. And composing deploy script is one of the my major tasks.
Here are some script snippets I found or I compose, which could be very useful in my daily work.

Download File from Web

This piece of script is useful when the script need to download package from CI server or configuration from configuration server etc.
Especially when wget or curl available.

NOTICE: This piece of script enable you to download the file from web in Powershell without any 3rd party dependences. But its performance is not as good as wget. Wget is still recommended if there are quite a lot of files to be downloaded.

Original Source

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

## Get-WebFile (aka wget for PowerShell)
##############################################################################################################
## Downloads a file or page from the web
## History:
## v3.6 - Add -Passthru switch to output TEXT files
## v3.5 - Add -Quiet switch to turn off the progress reports ...
## v3.4 - Add progress report for files which don't report size
## v3.3 - Add progress report for files which report their size
## v3.2 - Use the pure Stream object because StreamWriter is based on TextWriter:
##        it was messing up binary files, and making mistakes with extended characters in text
## v3.1 - Unwrap the filename when it has quotes around it
## v3   - rewritten completely using HttpWebRequest + HttpWebResponse to figure out the file name, if possible
## v2   - adds a ton of parsing to make the output pretty
##        added measuring the scripts involved in the command, (uses Tokenizer)
##############################################################################################################
function Get-WebFile {
   param(
      $url = (Read-Host "The URL to download"),
      $fileName = $null,
      [switch]$Passthru,
      [switch]$quiet
   )
   $req = [System.Net.HttpWebRequest]::Create($url);
   $res = $req.GetResponse();
   if($fileName -and !(Split-Path $fileName)) {
      $fileName = Join-Path (Get-Location -PSProvider "FileSystem") $fileName
   }
   elseif((!$Passthru -and ($fileName -eq $null)) -or (($fileName -ne $null) -and (Test-Path -PathType "Container" $fileName)))
   {
      [string]$fileName = ([regex]'(?i)filename=(.*)$').Match($res.Headers["Content-Disposition"] ).Groups[1].Value
      $fileName = $fileName.trim("\/""'")
      if(!$fileName) {
         $fileName = $res.ResponseUri.Segments[-1]
         $fileName = $fileName.trim("\/")
         if(!$fileName) {
            $fileName = Read-Host "Please provide a file name"
         }
         $fileName = $fileName.trim("\/")
         if(!([IO.FileInfo]$fileName).Extension) {
            $fileName = $fileName + "." + $res.ContentType.Split(";")[0].Split("/")[1]
         }
      }
      $fileName = Join-Path (Get-Location -PSProvider "FileSystem") $fileName
   }
   if($Passthru) {
      $encoding = [System.Text.Encoding]::GetEncoding($res.CharacterSet )
      [string]$output = ""
   }
   if($res.StatusCode -eq 200) {
      [int]$goal = $res.ContentLength
      $reader = $res.GetResponseStream()
      if($fileName) {
         $writer = new-object System.IO.FileStream $fileName, "Create"
      }
      [byte[]]$buffer = new-object byte[] 4096
      [int]$total = [int]$count = 0
      do
      {
         $count = $reader.Read($buffer, 0, $buffer.Length);
         if($fileName) {
            $writer.Write($buffer, 0, $count);
         }
         if($Passthru){
            $output += $encoding.GetString($buffer,0,$count)
         } elseif(!$quiet) {
            $total += $count
            if($goal -gt 0) {
               Write-Progress "Downloading $url" "Saving $total of $goal" -id 0 -percentComplete (($total/$goal)*100)
            } else {
               Write-Progress "Downloading $url" "Saving $total bytes..." -id 0
            }
         }
      } while ($count -gt 0)
      $reader.Close()
      if($fileName) {
         $writer.Flush()
         $writer.Close()
      }
      if($Passthru){
         $output
      }
   }
   $res.Close();
   if($fileName) {
      ls $fileName
   }
}

Extract File with Windows Explorer

This piece of code enable you to extract zip package without any 3rd party dependencies. It is useful environment provision script, which is usually executed on a Windows without anything being installed.

1
2
3
4
5
6
7
8
9
10
11
12
13
14

Function Extract-Zip ([string] $zipPath, [string]$destination) {
	$progressbar = 4
	$shellApplication = New-Object -com shell.application
	$zipPackage = $shellApplication.NameSpace($zipPath)
	$destinationFolderName = Join-Path $destination "tiger"
	if (Test-Path $destinationFolderName ) {
		Remove-Item $destinationFolderName -Recurse -Force
	}
	New-Item $destinationFolderName -type directory
	$destinationFolder = $shellApplication.NameSpace($destinationFolderName)
	$destinationFolder.CopyHere($zipPackage.Items(), $progressbar)
}

Interpolate value into configuration

This piece of script enable you to replace the place-holders in configuration text with actual values.

1
2
3
4
5
6
7
8
9
10

Function Apply-Config([hashtable]$config) {
	foreach($line in $input) {
		foreach($key in $config.Keys) {
			$line = $line -replace $key, $config[$key]
		}
		$line
	}
}

This piece of script can be used in different scenarios:

Prepare Configuration

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$configuration = @{
	"initialCatalog" = "lion"
	"logInitialCatalog" = "lion_log"
	"webServers" = @("192.168.122.46")
	"dbServer" = "192.168.122.65"
	"SQLServerDataDir" = "C:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA"
	"SQLServerDataFileSize" = "10MB"
	"dataset" = "prod"
	"appPoolUser" = "$($env:COMPUTERNAME)\Lion"
	"appPoolPassword" = "1zhlmcl..oostp"
	"createAppPoolUser" = $true
}

Apply config to in-memory configuration template

1
2
3

$configurationText = $configurationTemplate | Apply-Config($configuration)

Create configuration file according to template

1
2
3

cat $configurationTemplateFile | Apply-Config($configuration) > $configurationFile

Download and apply configuration from configuration server

1
2
3

Get-WebFile -url $configTemplateUrl -Passthru | Apply-Config($configuration) > $configurationFile