Environment Variables

Notice that this post applies only to Windows environment Variables. Until a couple of years ago I had rarely made any use of environment variables other than adding paths to PATH. However, in one of the projects that I have been working on these last years they are used quite a lot (for passing basic data between parent and child processes, for global data rather than using the Registry...) I don't like this way of working, but it's not up to me to change it, and indeed this "env var culture" in this project is due to some external tools also following the same principle. The thing is that because of it, I have learnt a few things about environment variables, and I will share them here. I'm not any sort of expert, and some of the information below (mainly regarding Windows startup) could be inaccurate, so be warned and bear with me.

We all know that there are System Env Vars and User Env Vars. When a user logs on to a machine an Environment Block is created for him, containing a combination of the System and User env vars. If the same variable exists for System and User, the User one will be used, save for PATH, that will be a combination of both. Then, each process runs with an Environment Block associated to it. How is this Environment Block associated to a process? Well, when an application starts another application, via the different Win32 functions: CreateProcess, CreateProcessAsUser... it can either pass it an Environment Block to be used for the process, or pass nothing, in which case the Environment Block of the Parent process will be used. This second case is the most common, it's what happens when you launch a program via Windows Explorer or the command line. So most times your new program will take the (it's a copy, not a reference) environment block of explorer.exe. When you start a program from the command line, unless you have started it with runas or have done some set on it, it will be as starting it from Explorer. I think the explorer.exe in your session is started by winlogon.exe (which in turn was started by wininit.exe) which is running under the System account, so I assume it starts Explorer with a call to CreateProcessAsUser and passing it the environment block for your user. You can read more about the Windows startup process here and here

Windows Services set to run under an account other than System get the environment block of that user, so I assume services.exe (that runs as System) is also creating the process by calling into CreateProcessAsUser with the environment block of that user.

When User1 start a process with runas User2, by default it will use the Environment Block of User2, unless you call it with the /env switch, in which case it will run with the environment block of User1.

The command line:

Apart from setting environment variables via the Windows UI you can also do it in the command line, but this tends to cause confusion. When you use set it's acting on the environment variables of the current process (I.e. cmd.exe). So if you add/change something there it will have an effect on programs that you start from that command line, but not in the ones that you start by any other means (explorer, another cmd). This is like when in .net you use Environment.SetEnvironmentVariable/GetEnvironmentVariable with the EnvironmentVariableTarget.Process parameter.

If you want to set env vars permanently from the command line you have to use setx. Using the /M switch it will set it as a System env var, otherwise as a user env var. So it's like using EnvironmentVariableTarget.Machine or EnvironmentVariableTarget.User in .Net. If you wonder why they use Machine rather than System, it's because environment variables are stored in the Registry, and System ones do it in HKEY_LOCAL_MACHINE hive.

Reactive to change

When you add/change an env var via the UI, new processes started from Explorer will get the new value. This is because when an env var is changed Windows broadcasts a WM_SETTINGCHANGE Windows Message. Any application (like Explorer) that is listening for this kind of message can then update its environment block. I had copypasted on my notes these 2 paragraphs from somewhere:

"To effect these changes without having to log off, broadcast a WM_SETTINGCHANGE message to all windows in the system, so that any interested applications (such as Windows Explorer, Program Manager, Task Manager, Control Panel, and so forth) can perform an update."

"If you're changing the environment through the My Computer -> Properties -> Environment Variables, leaving that dialog broadcasts a global window message that Explorer.exe and a few other processes listen to, informing them that the persistent environment (stored in the registry) has been updated, and those processes proceed to reload their environment from the registry."

Our War

When I travel somewhere I walk like crazy for hours, trying to get as many views of the place as possible. However, some of the best and more everlasting memories come from the times when I finally find a place to sit down and reflect, sometimes not about those remarkable surroundings, but about something inside of mine that they prompt me to analyse. One of those most noteworthy moments was a summer afternoon in Zagreb some years ago. There were some teenagers with their skateboards and they brought me memories of myself as a teenager skateboarder. I suddenly realised that if my teens had happened in Croatia rather than in Asturies, I would have probably ended up forced (or voluntarily, who knows) in the army in the attrocious war that devastated the multiethnic and inspiring (with its alternative to Capitalism and Communism) society that Yugoslavia was. My skateboard would have been replaced by a weapon and I would probably have killed someone or been killed before having my first sexual relation...

That day in Zagreb I was already fully aware (I've been since very long ago, since the times when the ISF sew horror and madness in Algeria) of the dangers that radical islam posed to our civilization and how at risk our model of society was. But I never thought that we would reach to the point of living a war on our soil. The attacks in Paris, and the confirmation of to what extent some of our neighbourhoods are no longer part of our civilization, already drew me into the WE ARE AT WAR mindset, and this week's tragic events in Brusselles have been one more painful reminder. Sure it's a different war, all wars have been different, cause everything is in perpetual change. The horror of the trenches and chemical weapons of the WWI were different from the horrors of city raids and concentration camps in WWII, and they are different from the horrors of salafism, but there's something common, horror, pain and suffering.

I think it's essential that we all understand this, WE ARE AT WAR, and neutrality is not an option. Normal muslims have to make clear that they are on our side, on the side of civilization, and we have to understand that every female piece of shit that we see on the street wearing a full veil, and every male piece of shit that we see on the street wearing the "salafist uniform" is our enemy and as such (enemies in times of war) must be treated. Every embassy or economical interests of those that fund the fundamentalisst perversion (Saudi Arabia, Qatar, Kuwait, Turkey) is a posession of the enemy, and as such (enemies in times of war) must be treated. We need to understand this if we want that our free and multicultural societies survive, WE ARE AT WAR, and we all must take actions. Pressure our governments to apply stricter laws, provide information about anyone preaching hate against "infidels", hack their websites, confront them in forums and streets, and above all, make them feel unease, to the extreme of forcing them to leave our countries and never come back because we don't allow them to live here by their sickening rules and "morals". Thouse that attack our society must be deprived of citizenship.

Those that have committed war crimes or that intended to commit them, must be subjected to war justice, meaning that they must be executed. Oh, yes, sure, many people (I seriously doubt that it's most people) are against death penalty... maybe we can afford that weakness in normal times, but at war times there is no room for weakness, because weakness is destroying us. Salah Abdeslam, the beast that not satisfied with the innocent blood of last November was planning new attacks, must be executed as soon as all the possible information has been obtained from him. The enemy must be annihilated in all senses. To protect beauty and peaceful coexistence, violence will have to be used.

Better Shot Than Arrested

I guess you've read the news, Salah Abdeslam, one of the main responsibles for the Paris carnage of last November has been arrested. Most people consider this like excellent news, I'm not so happy with it. Obviously it's better to have this beast arrested than free on the streets, but the desirable outcome for me would be to have him shot dead during the operation, as it happened in Saint Denis and with the Charlie Hebdo murderers.

Arrested, that means that he will spend a few years in a comfortable French or Belgian prison (those who die trying to cross the Mediterranean scaping from hunger or persecution would be quite happy before such prospect). If he ends up in a French prison it means that he will be fed, dressed and heated with the tax money of among others the families and loved ones of the victims. He will not do any sort of work in prison to pay for his basic needs, the state will pay. He will leverage his stay there to try to redicalize other inmates, so he will continue to harm the Western society that he hates so much but that has given him all the opportunities that many on the other side of the Mediterranean will never have. Oh, yes, he comes from a ghetto (the infamous Molenbeek), he is a "victim" of islamophobia, colonialism and bla, bla, bla... come on, go fuck yourself. Nowadays each time I hear the words Islamophobia and colonialism I feel nausea, because they are used by pseudo-left, pseudo-intellectuals to justify anything and to foster and spread the hatred of our civilization... and you know what?, I proudly continue to affirm that Western European civilization (with all its failures) is the maximum expression of Human civilization, period.

Sure if one of the normal muslims that I know and with which I share time and stories were ever discriminated by its faith I would feel nausea, but I feel that same nausea when someone is accused of islamophobia just for saying that fundamentalist Islam is not compatible with a modern and decent society and that fundamentalist beasts should be expelled of our countries.

The good news about this piece of crap called Salah Abdeslam is that he seems to be a coward. Despite all his hatred, bigotry and fanatism he had no bollocks to blow himself. Maybe he thought that with so many "martyrs" going up to paradise they could be running out of virgins and he could better abuse women by staying here. The thing is that for many radicals he could be seen as a failure and a traitor. Maybe we will be lucky and at some poin Daesh or any other group of beasts will bring up a "fatwa" asking for his execution. So well, maybe some day one "purest form or fundamentalist scum" will chop his head in prison :-D

The horrible news is that he was arrested in his neighbourhood, 500 meters far from his home!!! If this piece of crap felt secure there it's because he felt that many people around the area supported him. How many people knew or suspected that he was there? Shit, sure there are normal people living there, but feels like a sizeable percentage of the population in that neighbourhood should just be stripped of their nationality and expelled.

By the way, I've been to Molenbeek twice, before it got so infamous. The first time (2010) it was by chance, I was strolling around Brussels and as it's close to downtown I ended up there. The second time it was on purpose. You can guess that I am much into social issues and feel terribly attracted by the wonders of integrated multiculturalism and the horrors of non integrated parallel communities, and in 2013 Molenbeek already had a certain reputation as a place to go to take a look. Sure it's not the best area in Brussels, but I did not see dilapidated buildings or hungry kids... I saw schools, Metro stations, city council advertisements... so please, don't use your false stories of poverty and discrimination to justify radicalization, once more it doesn't hold up

You want another horror real story about fundamentalist islam in Europe? This comes from quite a few metro stations far from my flat (but given the "salafist dress code for men" that I can see sometimes right on my street it could come from even closer). One young woman from Bellefontaine (one of the, but not the worst, Toulousain "quartier sensible") has finally made it to the caliphate and is urging others to do so because crossing the Turkish border is getting more difficult (seems like cErdogan has changed his mind and is no longer giving a lift to the new Daesh recruits). To my astonishment, this was her third attempt to reach the "salafist promised land". In the previous occasions she would just be returned to Toulouse, where it's not only that she would not be put in prison, but as her father prevented her from using internet she would try to leverage the free internet connection in an association that tries to keep youngsters away from radicalization!!!.
Indeed, she was rejected access to their premises cause she was trying to brainwash other girls there!!!
Hope she will be forced into marriage (and all sort of sexual abuse) with one after another Daesh beast as they die in combat. The role of a good muslim woman in the caliphate is providing care and happiness to a man, so as soon as her husband dies in combat she has to marry another terrorist ASAP.
Ah, the ice of the cake, in her final travel, her mother joined her!!! Well, assuming that she will be in her forties, hopefully the pedophile Daesh beasts will find her useless and will kick her out of paradise..

From Callback to Await

Let's say that you have a "traditional" asynchronous method that will invoke a callback once it's finished, something like this:

public void Read(string path, Action<string> callback)
  {
   //let's simulate the async read operation
   //this does not block, it will just run the callback in a Threadpool thread
   new Timer((state) => callback("this is the file content"), null, 4000, Timeout.Infinite);
  }

What kind of wrapper around it could we create to use it with the cool await keyword? Without taking into account additional classes provided by the framework, and just using something I were familiar with, I came up with creating a new thread (either directly or through an additional Task, launching the call to the asynchronous method from it and waiting for a signal. This signal will be set from the callback. So we have something like this:

public Task<String> Read2(string path)
  {
   var waitHandle = new ManualResetEvent(false);
   string result = null;
   Action<string> callback = (res) => {
result = res;    
waitHandle.Set();
    
   };
   return Task.Run(() => {
                    //Task.Run will run it in the ThreadPool, so it's better
                    //new Thread(() => this.Read(path, callback)).Start();
                    Task.Run(() => this.Read(path, callback));
                    waitHandle.WaitOne();
        return result;
   });
  }

Nice, but I guessed there had to be a cleaner way to do it. Do a search and you'll find that the TaskCompletionSource class was created just for that. You can rewrite the above like this:

  public Task<String> Read3(string path)
  {
       var t = new TaskCompletionSource<string>();
  
       this.Read(path, s => t.TrySetResult(s));
  
       return t.Task;
  }

Reading the MSDN information about the class brings up an interesting point:

Represents the producer side of a Task unbound to a delegate, providing access to the consumer side through the Task property.

I had not thought about it this way previously. We've heard of this Producer/Consumer couple in other scenarios, like Enumerators, Events... For a Task, it's clear that the consumer is the client that waits on the Task and read a result from it. The Producer (of that Result) usually is the Task itself (the code running in that task), but with the TaskCompletionSource the Task has not code assigned, it's a different "piece" what runs and will eventually produce a result that will be propagated to the Consumers through the Task.

Runtime Code Generation, Methods

The most common occurrence of runtime code generation is not creating new classes like in the previous article, but just creating a "function" (a static method). The idea of expanding one class with new methods or replacing one method implementation for another is not directly doable in C# (well, for the latter maybe you could use some instrumentation and IL library, but the normal way would be to use some dynamic proxy approach).

We have several options for creating new methods. The most immediate one after what we saw in the previous post is just creating a class with that method in it, following any of the 3 approaches used in that post. Once we have created the new Type, we can easily obtain a Delegate pointing to the method using some of the CreateDelegate overloads, let's say:

//use any of the 3 methods "CodeDOM, Roslyn or Mono) from previos post
Type tp = @"
    using System;
    namespace Formatters
             {
              public static class StaticDateFormatter2
              {
                  public static string Format(DateTime dt) { return dt.Year + ""_"" + dt.Month + ""_"" + dt.Day; }
                  //public static string Format(DateTime dt) { return '-'; }
              }
    }",
                "Formatters.StaticDateFormatter2", 
                new List<Assembly>()
                {
                 
                });

Func<string, string> formatter = (Func<string, string>)System.Delegate.CreateDelegate(typeof(Func<string, string>), tp, "Format");

formattedText = formatter("hi");

This approach loads a new assembly in memory to host the new class, and the assembly will remain there until the application is closed. If we want a cleaner approach, creating just a method, without a new class and a new assembly, there are 2 ways to go:

• Dynamic Methods, aka Lightweight code generation. No new assembly is loaded and no new class is loaded, the method is just compiled and kept somewhere in memory. If you create a Delegate from a dynamic method and check for its DefiningType, you'll see it's null:
  myDynamicMethod.CreateDelegate(typeof(Func)).Method.DeclaringType)
  The obvious problem with Dynamic Methods is that you have to use IL. I wrote some IL code many years ago, it's pretty simpler than writing assembly for a real, register based, processor, but anyway it's quite unnatural and error prone for many of us. This project tries to make it a bit more friendly.

• Expression Trees. Honestly, I've never written expression trees myself, but it seems tedious to say the least.

What seems puzzling to me is that Roslyn has not improved anything in this respect. It's useless if what you want is to compile just a function (method) from a C# string and want to avoid the burden of a new assembly being generated and loaded in memory. If you want to spare the loading of new assemblies you still have to resort to the uncomfort of IL or Expression Trees. In this sense, there is library, Sigil acting as a wrapper around ILGenerator that intends to make it easier to write dynamic methods. Years ago Rick Strahl expressed his surprise for not being able to create a Dynamic Method from a C# string. I find it odd that years later the so powerful Roslyn (Microsoft.CodeAnalysis) has not some sort of "DynamicMethodGenerator". Wondering if some "compiler freak" would ever think of doing just something like that, in the recent comments to that post we can find a link to a CodeProject article from one guy that basically did it. I've downloaded the code and the man basically implemented a C# to ILGenerator compiler!

There's another option that spares us having to create the class (in the source code string), we can use the Roslyn Scripting API. You can create a ScriptRunner delegate from a script piece. The third line is to avoid coupling with the technique used to create our method, we better wrap the ScriptRunner in a generic Func delegate:

var script = CSharpScript.Create<string>("var up = '{' + x.ToUpper() + '}'; return up;", globalsType: typeof(Globals));
ScriptRunner runner = script.CreateDelegate();
Func func = (it) => runner(new Globals { x = it }).Result;

But this Roslyn Script still will create a new class and load a new assembly, so it's not any improvement really. I'm pretty surprised by this. I would have expected the scripting api to try to be as lightweight as possible, hence using Lightweight Code Generation (Dynamic Methods), but unfortunately it's not the case.

.Net Memory Limits

We have a .Net 4 application that occasionally will throw an Out Of Memory Exception. Currently the application is compiled as 32 bits, as it uses a 32 bits COM component, so in principle this fact limits the memory space available to the application to 2 GBs. While looking into ways to avoid this issue I've taken some notes on Memory limits in Windows and .Net.

OS limits

The OS splits the virtual address space in 2 parts, one for the OS and one for the Application. We are interested in the latter (it's the application who could need enormous data sets, not the OS)

• For a 32 bits OS this means 2GBs for the OS and 2GBs for the application. There's a flag in the PE (portable executable) header, IMAGE_FILE_LARGE_ADDRESS_AWARE that allows you to change this to 3GBs. So well, for an application managing huge data sets this is for sure a limiting point.
• For a 64 bits OS the theoretical limit (the max addressable memory address) is 18.446.744.073.709.551.616 bytes, that is 16 Exabytes. Windows limits this a bit, you have an updated list here. Physical limits are huge enough to not be a concern for anyone I guess, but the first table, containing the per process limits is more interesting. To summarize:
  For a 64 bits process and IMAGE_FILE_LARGE_ADDRESS_AWARE flag set (it's the default), the limit is 8 TBs, so we are good :-) For 32 bits processes there are good news, if we set IMAGE_FILE_LARGE_ADDRESS_AWARE (by default it's not set) for this binary, we'll be able to use 4 GBs of memory, this can make a difference for some applications. I've tried this with a test applicaton and it really works. The best thing is that you can apply this fix to a problematic application even if you don't have the source code. You can change the IMAGE_FILE_LARGE_ADDRESS_AWARE flag in an existing .exe just using the editbin tool that comes with Visual Studio, just type: EditBin.exe” “myApp.exe” /LARGEADDRESSAWARE. That's all, it will set the falt in the .exe header to 1 and you'll be able to use up to 4GBs of RAM. You can easily see if a binary has this flag on or off by using dumpbin /headers. By the way, another helpful use of editbin is changing the stack size for an application, which could be needed if you are using deep recursions.

.Net Limits

For .Net applications there are additional limits. Until version 4.5 the maximum size of a .Net object (an object needs contiguous memory space) was 2 GBs, with .Net 4.5 and setting gcAllowVeryLargeObjects to true in your config file, you can skip this limit. You, can read more here. Great, so your objects can grow bigger and bigger now, but for the sake of understanding, how can you end up with such monster objects?
Well, obviously you'll need an array, and mainly an array of Value objects. Value objects are embedded in the array, while that for normal objects the array just contains the reference (memory address) to the object, that's just 32 or 64 bits depending on how you've compiled your application, so it would be quite unusual to hit the limit with an array of references. Notice that you can not switch from an Array to a List to skip this issue. A List uses internally an Array for its storage (when it reaches its capacity a new, bigger array is allocated and the data copied to it).

There's something more to consider regarding continuous memory space, fragmentation. .Net divides the memory heap in 2 blocks, the Small Object Heap (in turn divided in 3 blocks, the generation 0, generation 1 and generation 2 used by the GC), and the Large Object Heap (for objects bigger than 85 KBs). When the GC cleans the rubbish in the SOH, it also compacts the memory, but it won't do so for the LOH, as this can be a pretty costly operation. So even if large objects are eventually collected,they'll fragment your memory and can prevent you from finding a new large block in the future, causing an Out of Memory exception when indeed you have plenty of memory available for smaller objects. This is pretty well explained here. Things have changed a bit after that article were written, and now (since .net 4.5.1) the frameworks gives you the possibility of compacting the LOH, by means of a GC setting: GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;.
You can read about it in this excellent post

.