Commonality
only inquisitive minds need apply
- Last build:
- Sat, 08 Mar 2008 16:34:13 GMT
- Language:
- en-us
- Feed URL:
- http://www.winterdom.com/weblog/rss.xml
RSS FEED IDEMS: Commonality
- NTFS Data Streams and .NET
Several people have written in the past about accessing Alternate Data Streams in the NTFS file system from .NET code. The reason for this is that accessing streams is not natively supported in .NET. What you don't hear very often is exactly why this is so.
There are two things you might want to do with data streams:
- Manipulate them: create/read/write/delete them
- Enumerate them: list all the alternate data streams for a file
Neither of these operations are not supported by System.IO. To be perfectly honest, I think not supporting (2) is an understandable option; after all, it requires exposing some fairly specific windows APIs that are really only useful with NTFS itself (as far as I know). There's an old MSDN Magazine article on this topic by Stephen Toub, by the way.
But I was always pretty surprised that (1) wasn't natively supported. After all, you don't really need special APIs to do most of that stuff; it's built into the native Win32 APIs that .NET has to call anyway to perform the basic file and I/O operations. Heck, it is so basic that you can create a new alternate stream using "echo" in a cmd prompt!
You can find some mentions out there about this being because complexities introduced by file name aliases and what not, but I'm not sure I buy it. Seriously, number (1) should just work; if it doesn't then something smells wrong. After all, it takes work to get something that you get for free to not work at all!
I think the overall reason this doesn't happen is just that the way paths are handled (and path canonicalization is done) in regards to FileIOPermissions and such, but frankly, I'm not sure why it is such a big deal. There are certainly a bunch of other file system features in NT, which .NET doesn't supports either, but those are a bit more understandable (to me, at least).
Other than that, there's also the fact that System.IO.Path isn't particularly bright about how it handles paths. For example, it is hard to argue that this could be proper behavior:
Fortunately, using alternate data streams isn't so common, so this isn't such a big deal. Still, it is a curious bit.
Sat, 08 Mar 2008 16:34:13 GMT - TortoiseSVN and Putty
Last week, I was trying to connect to an SVN repository over SSH, using TortoiseSVN and Putty. I found a the faq on how to configure everything and checked it to make sure I was doing everything right.
I tried to connect. TortoiseSVN asked for my certificate password. And then, nothing. I could see that indeed TortoiseSVN was connecting to the SSH server; but just sat there doing nothing. I had to kill it several times to abort the connection.
I had already tried connecting using Putty directly, and that seemed to work, so I kept checking my settings, but for the life of me I couldn't figure out what the deal was.
Eventually, I figured out what the deal was:
See that "Don't start a shell or command at all" checkbox? Yep; that should NOT have been checked, as it prevented the server from starting svnserve to handle the SVN connection. Don't know why I had checked it before, but everything worked as soon as I removed it from my Putty profile.
Fri, 07 Mar 2008 13:52:03 GMT - What I don't like about E: Cygwin
My good friend Sam Gentile blogged about his new dev environment for Ruby using the E Text Editor alongside cygwin and Console. I find E an interesting editor because of its TextMate-like functionality, and because pretty much everyone who has used TextMate raves about it (though I haven't used it myself).
As a side note, last time I tried E wasn't very successful. In their defense, I'll say that I've since reached the conclusion that it was probably my virtual machines acting up. VPC2007 sometimes has nasty issues when running on Vista.
However, the thing I don't really like much about E is its dependency on cygwin. I understand the reason for it, but frankly, I just don't like cygwin much. Don't get me wrong, cygwin fills a valuable niche [1], and I've used it in the past, but the truth is that it's much more less needed now than it used to be.
For example, a few years ago it was pretty hard to find native ports of many GNU command line utilities. Now they are widely available (and before that there was unxutils). There are differences of course, but they are very usable for the most part.
Cygwin itself feels pretty clunky. In fact, I don't want a "unix within windows" at all, as that forces to many compromises (many of which aren't really all that needed). If I wanted to, I could always go with something like andLinux, which seems like a better option to me.
Frankly, between PowerShell, the Windows port of GNU utilities and a few other things, I feel right at home on Windows and switch to unix when using my ubuntu machine without much fuzz at all.
Back to Sam's post: he's right on the money about Console; it's pretty sweet. Unlike Sam, however, I dumped my VS environment tab and just configured all the necessary paths in my PowerShell profile. I haven't had a need to switch to a regular CMD.EXE prompt in a few weeks!
[1] There are certainly some cases where cygwin is actually warranted because some specific tools, but, personally, I prefer to just switch to a real unix at that point. It's less painful.
Fri, 07 Mar 2008 13:40:50 GMT - Irony
I've been playing a bit with Irony, an open source .NET compiler construction toolkit created by Roman Ivantsov. My interest in Irony was sparked after watching the video of Roman's presentation at Lang.NET 2008 (I'd link to it, but they are unavailable at the moment).
Currently, I have a little side project where I've been experimenting with the Dynamic Language Runtime, for which I was using GPPG and GPLEX to build the parser and tokenizer. The tools are OK, but it certainly requires quite a bit of manual work, you need to keep regenerating the code, and, frankly, the error messages both tools produce leave a bit to be desired.
I'm aware of ANTLR, but frankly don't want to mess around with the whole java generator thing at this time (already have way too much crap around).
Irony is pretty interesting because all the tokenizer and grammar rules are expressed directly in C# code as a simple object model built from Terminal and NontTerminal objects. The syntax is fairly intuitive because Irony overloads several operators like | and & to make the grammar definition look a lot similar to BNF. Here's a sample of what a simple expression grammar in Irony looks like: http://www.codeplex.com/irony/Wiki/View.aspx?title=Expression%20grammar%20sample
After testing a bit Irony, I'm very encouraged to give it a try on my pet project, so I've been playing around with my grammar (unfortunately broken at this time because of some ambiguities I haven't resolved yet), and its looking very nice overall.
There's also a useful "Grammar Explorer" tool included with the Irony code which can be used to experiment and diagnose a compiled Irony grammar. Unfortunately right now it just has a predefined list of a few test grammars included with the project, but extending it to use others is trivial anyway, so I've been using it to test my grammar before actually committing to it.
Another useful thing in Irony is the concept of TokenFilters, which are objects that can filter the token stream as it is produced by the scanner and can either remove/modify/add new tokens as necessary or provide extra validations. Right now Irony provides two built-in filters, but you can extend it with new ones as necessary:
- CodeOutlineFilter, which can be used in languages where whitespace is significant (indentation and/or newlines).
- BraceMatchingFilter for matching brace characters, for example '()' in scheme/lisp.
Wed, 05 Mar 2008 12:33:17 GMT - The Command History in PowerShell
Sometimes you're executing a series of commands when using the PowerShell console and have to go back and forth repeating commands you've used before. Sometimes, just using the up/down arrow keys is enough (or their close cousins: F7, F8 and F9).
However, sometimes it isn't enough. One of the issues with up/down and friends is that this really isn't implemented by PowerShell (or whatever PowerShell host you're using), but by the Windows Console Subsystem in CRSS.EXE. And sometimes it can also be a drag.
Fortunately, PowerShell has some pretty useful things that I had not played with much before, but they turn out to be pretty useful.
Get-History
PowerShell keeps track of the last N commands you've executed in the "History", where N == 64 by default. You can modify how many commands the history will contain by setting the $MaximumHistoryCount variable in your profile scripts.
Technically speaking, exactly what gets stored here depends on the Host application, but the default ConsoleHost pretty much does what you'd normally expect.
You can check your history using the Get-History command, which by default is aliased as 'ghy', 'h' and 'history'. Go on and give it a try!
One interesting thing is that each entry in the history is given an Id, which you can use later on to refer to that specific entry in the command history.
Repeating a Command
You can repeat (i.e. execute again) an arbitrary entry in the command history by using the Invoke-History command and passing it the Id of the entry to execute again:
Invoke-History 45If you don't provide an Id, then the last entry in the history will be executed again (or you can also pass in a string with the first few characters of the command and the history is searched backwards for the first match).
This is pretty handy also, because Invoke-History is by default aliased as 'r' and 'ihy'. So you can simply repeat the last command using "r<enter>".
One thing to watch out for is that Invoke-History will first write to the console the command it is going to execute, so keep that in mind when looking at the output.
Notice that when you use Invoke-History, the Invoke-History command itself will not go into the command history. However, the command executed as a result of the use of Invoke-History will indeed get appended to the history!
The Last Command
Another interesting thing is that, PowerShell will store the last command executed into the $^ variable. So an easy way to check what the last command executed was is to simply type $^ at the prompt:
tomasr@arcano {~\scripts} $^
clsLet me reiterate something: The $^ variable stores the last command directly executed, not the last command in the history. That means that if you call Invoke-History, then $^ will contain your Invoke-History command, not the actual command executed as a result.
It does provide for some funky stuff. For example, you can repeat the last executed command (yes, even redo an Invoke-History command) using:
tomasr@arcano {~\scripts} &$^
Id CommandLine
-- -----------
89 $ExecutionContext.SessionState
90 $ExecutionContext.SessionState.Host
91 $ExecutionContext.HostA discrepancy
There's another interesting difference between $^ and the history. For example, let's say you execute a compound command like "ls; echo done".
If you immediately check the history, you'll see that the last entry in the list is indeed the entire thing ("ls; echo done"). This means that history really keeps track of entire pipelines executed.
However, if you immediately check the value of $^, you'll notice that it only contains the first part of the pipeline ("ls"); not the entire thing. In fact, $^ only stores the first command in the pipeline executed; so if you execute, say, "ls | sort", still only "ls" is stored; which is certainly a lot less useful (but still interesting).
Add-History
There's also an Add-History command you can use to push stuff into the history without actually executing it first. In theory, this would seem to be aimed primarily at making it possible to persist the history across sessions. All you need to do is save the contents of your history to a file and then read it back and feed it to Add-History.
This is unfortunately less than useful, because it is not done automatically for you. You can partially automate it, but since it requires you to explicitly use a command to close your session, well, little chance of our fingers and brains getting used to it (and even if you do, what happens if you accidentally close the window instead of typing the right command?). Looks like V2 might help here; would be nice.
Have fun!
Thu, 28 Feb 2008 17:00:56 GMT - Filenames Starting with a Dot
In Unix, it's common to name files and folders with names starting with a '.' to make them "hidden". This convention is not really needed in Windows, since there's a file system attribute for that, but still interesting.
See, a few tools ported from Unix that relied on this convention use a different one in Windows, replacing the '.' with a '_'. So, for example, your Vim settings file will be "_vimrc" instead of ".vimrc" and so on. Seems pretty obvious they did this because at some point in time file/folder names could not start with a dot.
Windows 9x probably had this limitation, and probably still lives on in FAT-based file systems (but I don't have one anywhere to test this assertion, nor do I particularly care about it). However, modern Windows and NTFS have no such limitation. Almost.
See, NTFS certainly let's you create a folder named ".vimfiles" or a file named ".txt" for that matter. You can do so programatically using the standard file handling APIs. You can do it from the command line using mkdir/echo.
But you can't do it using Windows Explorer. For example, Vista and WinServer2k8 will complain that "You must type a filename" if you attempt to name a folder/file with a dot as the first character in the name.
Why Explorer explicitly disallows it, I'm not sure, but seems pretty funky. There are probably thousands of applications or utilities out there that might get confused with a file or folder named like that, though.
Mon, 25 Feb 2008 21:25:03 GMT - More Music
Here's some music I've recently gotten my hands on:
- Dokken: From Conception - Live 1981. Dokken playing live at its best. It's also my second Dokken live album, as I already owned their Live from the Sun album.
- Edguy: Hellfire Club. Edguy's always a fun band; several good tracks in this one like "We don't need a Hero", "King of fools" and my favorite, "Navigator".
- Kamelot: Ghost Opera. A solid album, as most of previous Kamelot works.
- Diabolique: Black Sun Collection. This is a double album combining their Wedding the Grotesque and The Black Flower albums, though the second one is far more polished and a lot better overall (I particularly liked songs like "Dark Rivers of the Heart", "Yesmine" and "Play in the dark". I also own their Butterflies album which is one of my favorites around.
- Freedom Call: Eternity. It's an ok album, but clearly not their best.
- Iron Maiden: No Prayer for the Dying. I had a copy of this album many many years ago and lost it, and ran into it a couple of weeks ago at Tower Records and got it. The only reason they're carrying this stuff is because Iron Maiden is giving a concert in Bogota soon, though I won't be attending :-(.
- Winger: IV. Didn't really expect to get this one, given that I was never really a Winger fan, but it actually has a few really good songs, like "Four Leaf Clover".
- David Shankle Group: Ashes to Ashes. Interesting album, though somewhat of a mixed bag. So far I'm liking "The Widow's Peak" the best.
Mon, 25 Feb 2008 21:11:58 GMT - Can't Remove Drive on Vista SP1
I haven't noticed many issues with Vista SP1, despite some stuff I've read around; though overall haven't really noticed any significant performance difference besides file operations using Windows Explorer.
That said, I've noticed one nasty issue: I use an external USB drive that has a bunch of virtual machines, my music and other stuff. I'm usually very careful and always use the "Safely Remove Hardware" icon in the system tray to remove the drive before disconnecting it from the computer or turning it off.
However, ever since installing SP1, this operation fails about 50% of the time, saying the volume is in use and cannot be removed, even though there are no applications open that could be using the drive at all.
The past few times it's happened, I've fired up Process Explorer to try and locate what is holding on to the drive, only to find that the only process that has file handles opened in the drive is the "System" pseudo-process:
No idea which driver is holding on to the handles, but based on the file names, I might guess it has something to do with the Kernel transaction manager and NTFS transactions. Why those handles are being kept there, I have no idea; but once this happens, those don't get released, meaning I have to shut down the drive improperly. Not good.
Sun, 24 Feb 2008 19:42:13 GMT - Ast.ComplexCallHelper()
Something I ran into yesterday: There are two variants to Ast.ComplexCallHelper() in the DLR: One takes a MethodInfo and the other a Delegate as the invocation target.
The MethodInfo variant does not check for the presence of a CodeContext argument, while the Delegate version handles this correctly. Not quite sure why the discrepancy between the two, but sure looks like using the Delegate variant is a better choice for now.
Thu, 21 Feb 2008 21:16:52 GMT - Simple Is Beautiful
Am I the only one that thinks the Windows Server 2008 logon screen, with it's very simple blue background, is so much better looking that Vista's ugly and distracting "aurora" background?
There's just something highly appealing to me about keeping things simple and elegant...
Thu, 21 Feb 2008 13:47:45 GMT - PipelineUtil.CloneMessageContext()
BizTalk Tip of the Day: If you're writing a custom BizTalk Pipeline Component (particularly a receive one), and you're creating a new message from scratch to return from your component, don't forget to copy the original message context into your new one.
And the right way to do this is using the CloneMessageContext() method of the PipelineUtil helper class included with BizTalk. Rolling your own and doing it incorrectly can result in really weird errors later on.
Wed, 20 Feb 2008 19:32:02 GMT - Watch out for ':' in PowerShell
Here's something that tripped me the other day in a Windows PowerShell script. Let's say that you have to variables $user and $machine, and want to print a string with the value of both of them separated by a colon (':').
You can obviously do it using string concatenation:
$user = 'tomasr'
$machine = 'arcano'
write ($user + ':' + $machine)That works, but doesn't PowerShell support that nice string interpolation feature? Why, yes it does! So why don't we just rewrite it like this:
write "$user:$machine"But wait! If you run that last line, all you get is the value of the $machine variable... where did $user go?
The answer is that the colon has a special meaning in PowerShell variable names: It's used to associate the variable with a specific scope or namespace. There are a few scopes defined by PowerShell, such as script and global; so for example a global variable might be named $global:var.
But haven't we also seen that syntax elsewhere? Sure, how about $env:PATH? Same thing; except here env isn't a scope by itself, but a PSDrive. So the part before the ':' can either be either an actual scope defined by PS or a PSDrive. Heck, you can even try something like "${c:autoexec.bat}" and watch the magic happen!
So the problem with our string interpolation is that when PowerShell sees "$user:$machine", it expects to find a scope/psdrive named 'user' and expects a variable name to follow the ':', but '$' isn't valid, and 'user' isn't a scope or a PSDrive either, which confuses PowerShell a bit. But since he's too much of a gentleman he just fixes it up as best as he can and ignores everything up to this point and just dumps $machine without complaining. Nice of him, but it can really trip you up if you don't expect it!
Can we avoid this? Yes, we can, by simply delimiting variable names exactly for PowerShell by using '{}', like this:
write "${user}:$machine"Section 5.8 on page 141 of Bruce Payette's excellent Windows PowerShell in Action covers the topic of variables in PowerShell and gave me the clue to work around this issue.
Wed, 20 Feb 2008 14:30:56 GMT - DLR Notes 5
Here's one interesting issue I ran into while trying a few things out: Conversions.
Given the dynamic nature of the DLR and the languages implemented on top of it, and the statically typed nature of the CLR and most .NET languages, you need to be able to ensure that the right type conversions happen at the right places.
This involves, among other things, ensuring that the right casts and boxing/unboxing operations happen, as well as ensuring that language-specific conversions can be applied (such as, say, converting an arbitrary object to a CLR string through an explicit method call).
The DLR does a bunch of these automatically in a number of places, like variable writes and assignments, call return values and so on. Most of these can be made to follow language-specific behaviors because the conversion expressions are generated through calls to your language ActionBinder implementation (like the RubyBinder class in the case of IronRuby).
For example, many of these scenarios will generate calls to your ActionBinder.ConvertExpression() method. Here's one case I found where this does not happen, making it a bit more tedious:
Today I was debugging some code because of an InvalidCastException getting thrown. Indeed, the problem was that I had a method call that required converting one of the parameters, and it was a set of types I had not tried before. In this case, the necessary conversion was language specific, so I added the rule into my ActionBinder.ConvertExpression() implementation and made sure as well that ActionBinder.CanConvertFrom() handled the case as well.
But it was still not working. After a bit of debugging, I realized that in this particular case, the ActionBinder wasn't getting involved at all for the conversion! What was the deal?
The issue turns out to be that the call was being done through one of my IDynamicObject implementations. In particular, this one handled a Call action by generating a rule target containing a call to the target delegate using the DLR"s Ast.ComplexCallHelper() method.
The problem is that ComplexCallHelper() doesn't really deal with language-specific conversion needs at all. Instead, it relies on Ast.Convert() to convert the actual argument values to the types declared in the target method/delegate signature.
The problem with Ast.Convert() should be obvious by now: All Ast.Convert() and Ast.ConvertHelper() do is insert an explicit Cast expression into the code.
This works great if all that is needed is a box/unbox operation or an upcast/downcast, but not enough in our case, so it is something to watch out for! I still haven't decided yet how to best work around the issue
Tue, 19 Feb 2008 19:43:18 GMT - WinSxS
Size of %WINDIR%\winsxs:
- Clean Windows Server 2008 Enterprise installation: 2.6GB
- Used Windows Vista SP1 installation: 6.52GB
I know that Side by Side installation is important for compatibility and serviceability, but this might just be going a bit overboard....
Fri, 15 Feb 2008 16:36:47 GMT - Vista SP1 Installed
Service Pack 1 for Vista is now available for download from MSDN (I guess the Windows Update rollout will still wait to march or something). For some reason, though, it is a 434MB, 5-language pack.
I downloaded it last night and installed it this morning without too much fuss, except for Windows Live Messenger now keeps triggering Windows Installer on startup and failing (but it works after that, funky).
Haven't really noticed many improvements yet, though moving/copying stuff around in explorer seems quite bit faster (that alone might be worth it).
Fri, 15 Feb 2008 13:11:15 GMT
