Solidarite avec le Peuple Kurde

This last Saturday an evening of Solidarity with the Kurdish People was held in Toulouse. It aimed also to pay hommage to Ivana Hoffman, the 19 years old AfroGerman revolutionary woman that was killed 1 year ago while fighting against the IslamoFascists beasts of Daesh. Just one year ago another day of solidarity with the Kurdkish struggle was held in Toulouse, but it was set up by different organizations. This time it's been Voie Proletarienne, a French communist organization. For me the "Soiree" has been a complete success.

One year ago, the attendants to the activities were maninly members of the French-Kurdish community (as it happens with the support gatherings and demonstrations that I've seen in Toulouse). This time the place was full of Toulousains of non Kurdish descent, meaning that as in Germany, there is a growing interest and involvement of the population (at least the left leaning population) with the Kurdish cause, regardless of having ethnical links with the Kurdish people. The struggle of the Kurdish people against Daesh and the IslamoNationalist Turkish govenrment is a struggle for basic human rights, and any decent human being should feel touched by it.

The activities consisted of a talk and a concert. Obviously the talk was in French, so I did not expect to understand too much (my French listening skills are still minimum), but anyway I attended to see if there was some reading materials, to put some money in the donation box (destined to provide medicines to the antifascist fighters in Rojava, you can donate online), and to see the atmosphere. As I've said, I was pretty happy to see the large crowd attending (200 people says here), and yes, I found pretty interesting stuff to buy.

I was quite amazed by the fact that they were serving vegetarian food. Of course that's what I expect for acts related to the punk scene, but I don't think it's so common for a communist organization, so my hat off, Mesdames et Messieurs!

The concert was also pretty interesting. First Grup Piya playing traditonal Kurdish music, not what I usually listen at home, but the voice of the singer was mesmerizing. And then Thawra a female left wing rapper from Berlin, who delighted us with a short but very intense and uncompromising set. I pretty much appreciate that she gave an explanation of every song, which quite reminded me of the HxC/Punk scene 20 years ago. The instrumental base of some songs was not much my style, though for other songs it nailed it, and her voice and flow is really, really fresh and special, I loved it. This video will leave you asking for more. I think most of the audience was unaware of the AntiDeutsche, controversial "movement". I'm quite fed up of the Israel/Palestine conflict, over the years both sides have got so full of shit (sionist settlers and jewish supremacist on one side, islamist scum on the other side), that honestly, I quite don't care of what happens in that bloody piece of land. Sure it's selfish and simplistic, but I'm quite more concerned about forgotten conflicts like Western Sahara. Whatever bad and unfair it's the situation in Gaza, living off the diminishing international aid in the driest part of the Sahara desert seems quite worse, and mainly there are no Islamists among the Saharaui population.

Async/Await, the Basis

I've had an odd attitude to the async/await magic added in C# 5.0 some years ago. On one side I was marvelled by all the compiler magic making it possible, but being a unique C# feature I felt more inclined to use ContinueWith, as using Tasks these way felt basically like using Promises in javascript. ES7 will include async/await magic, and Python already includes it, so I felt it was about time to fully jump into the async/await wagon. There are tons of impressive articles and tutorial at any level of detail, but I'll put here my own basic notes, the ones that I hope will be enough to put me up and running again if I spend time without doing any async stuff.

As everything in async/await revolves around Task (and Task<T>) it's important to clear up some concepts. As I said before, we can think of Tasks as javascript Promises, but there is a difference. In javascript we have no control over thread creation as javascript engines are single threaded (at least in the browser, with the exception of Web Workers). I assume under the covers a javascript instruction could do some native code run in separate threads (for example you call a native node module that does some encryption running native code in multiple threads), but you won't have javascript instructions running in parallel. So Promises are just that, a promise of one operation that at some point will finish and will tell you about it, you don't know if maybe under the covers that Promise is blocked waiting for I/O or it's running native threads in parallel. With Tasks the main idea is the same, when the operation is done it will tell you, but you can explicitly create a Task to run code in a separate Thread. That's what happens when you call Task.Run. Indeed you should no longer use "new Thread()", it's the TPL who will do that for you (well, by default it'll use a ThreadPool thread). This reading is pretty good.

A Task represents a promise of work to be completed some time in the future.
There are a couple of options to how a Task gets created:

1. Using Task.Run or Task.Factory.StartNew without any flags - This will queue work on the .NET ThreadPool, as stated.
2. Using Task.Factory.StartNew and specifying the TaskCreationOptions.LongRunning - This will signal the TaskScheduler to use a new thread instead of a ThreadPool thread.
3. When using async-await feature of C#-5, also known as Promise Tasks. When you await on a method returning a Task or a Task, they may not use any threading mechanism, either ThreadPool or new Thread to do their work as they may use a purely asynchronous API, meaning they will continue to execute on the thread which executed them, and will yield control back to calling method once hitting the firstawait keyword.
4. Thanks to Servy for this one. Another options for execution is creating a custom TaskScheduler and passing it to Task.Factory.StartNew. A custom TaskScheduler will give you fine-grained control over the execution of your Task

Of course await does not create new threads on its own or magically turns into asynchronous the method you are calling. That method that you are invoking has to return a Task, and async/await just helps you handle those tasks. As someone nicely put here, The new Async and await keywords allow you to orchestrate concurrency in your applications. They don't actually introduce any concurrency in to your application.

The basis

A method with the await keyword has to be marked as async, otherwise you get this compilation error:
Error CS4032 The 'await' operator can only be used within an async method. Consider marking this method with the 'async' modifier and changing its return type to 'Task<Task<string>>'.

You can call a method marked as async without using await (you'll use then: Task.Wait/Task.Result) but you have to be careful with what you do, as that can be a recipe for a deadlock, unless we are using ConfigureAwait(false) (or we do a ContinueWith in the caller rather than a Wait). You can check here

The basic point to understand how await works is to think of it as if we were calling a ContinueWith on the Task, and passing a lambda to that ContinueWith with the code that comes after the await. If we have several await calls in one method, think of it as chaining continuations. Check this question if you don't believe me. In the end, based on what I've managed to understand from this chapter in that very in depth series, I think it's a bit more complex, you push the lambda, and then the ContinueWith is called later by the Awaiter, but the basic idea is the same.

We can draw a comparison between yield and await. Both of them involve a lot of compiler magic than basically creates a State Machine. We can say that a yield or await keyword suspend the execution of the current method (but they do not suspend the thread). The big difference is that a method suspended via yield is restarted by you (when you call Next in the generated enumerator), while a method suspended by await is "magically" restarted when the Task it's waiting for is complete, you don't have to do anything.

The similarities between yield and await go one step further. Same as we have an IEnumerable and IEnumerator pair, we also have an Awaitable and Awaiter pair. The Awaiter is a bit hidden, most of the time you don't care about it, you just care about the Awaitable (that for the most part is just a Task). The Awaiter manages the Awaitable (checks if it's fully done or there are continuations waiting). In the end is similar to when you use a foreach loop to iterate an IEnumerable, you don't care about the IEnumerator, though it's him who really does the iteration. I like this mention done here about the "duck typing" nature of both patterns

I used to do something like this:

await Task.Run(()=>Thread.Sleep(5000));

you better use await Task.Delay(5000), it's more "academic", and I read somewhere that it uses a Timer internally rather than wasting a ThreadPool thread in this Run-Sleep.

In some later post I intend to talk about Synchronization Contexts

Runtime Code Generation, Classes

I've always found fascinating the idea of generating code at runtime. I think the thing can not get more powerful and easier than in JavaScript eval (the evaled code is executed just as if that had been written in place, with access to the local variables and so on!) For conventional languages this has always been a bit more complicated. Things in C# have changed a bit over the years (I already talked about it a long while ago) and I have partially forgotten some old things and not learnt the new ones, so I thought I would do a fast review of what we have available.

In principle the ideal way to generate code at runtime would be something like JavaScript eval: pass a string of code and get it executed. Notice that given Javascript's nature, if that string is a function declaration or a "class declaration" (pre ES6 style), executing that declaration we'll make it available for later invokation. So you can just run some immediate instructions, or create "code blocks" to be later used (invoked).
In C# there's a clear difference between defining classes and methods, and executing code. So, I will write different posts about creating new classes, creating new "reusable code blocks" (delegates pointing to code created at runtime) and executing "one shot" statements.

Creating classes at runtime
You have 3 options that I know of, but in the end the 3 resort to the same: getting the compiler to generate an assembly from the provided source, loading that Assemby into memory, and obtaining from that Assembly a Type object. Finally you'll create instances of that Type by using Activator.CreateInstance.

• The first and oldest option is using the CodeDOM. This has 2 big disadvantages, first, it launches the compiler (csc.exe) under the covers (as I recently mentioned in this post, I can not understand why the preRoslyn compilers were contained in a binary and could not be used as a library). Because of this, the assembly can't be generated in memory, and it will be written to C:\temp\randomName.dll. Both things are a bit ugly in terms of performance. This reminds me that the first time I read about runtime code generation in C# (sometime in the summer of 2002 if my memory serves me well), they were directly invoking the CSC compiler (Process.Start("csc.exe"...)). The CodeDOM is just layer on top of that.

  private static Type CreateTypeViaCodeDom(string code, string typeName, IEnumerable assemblyNames)
    {
              CodeDomProvider cpd = new CSharpCodeProvider();
              var cp = new CompilerParameters();
              foreach(string assemblyName in assemblyNames)
              {
               cp.ReferencedAssemblies.Add(assemblyName);
              }
              
              cp.GenerateExecutable = false;
              CompilerResults cr = cpd.CompileAssemblyFromSource(cp, code);
     //the assembly gets written to disk (c:\Temp\randomName.dll)
              
              Type tp = cr.CompiledAssembly.GetType(typeName);
              return tp;
          }
  

• The modern way to do it is using Roslyn (Microsoft.CodeAnalysis...). As you know Roslyn is based on the Compiler As a Service idea. The compiler resides in some libraries (so no under the covers invokation of csc.exe), and the Assembly can be generated in memory.

    private static Type CreateTypeViaRoslyn(string code, string typeName, IEnumerable references)
    {
     var assemblyName = Path.GetRandomFileName();
           var syntaxTree = CSharpSyntaxTree.ParseText(code);
   
           List metadataReferences = references.Select(it => (MetadataReference)(MetadataReference.CreateFromFile(it.Location))).ToList();
           var compilation = CSharpCompilation.Create(assemblyName, new[] {syntaxTree}, 
      metadataReferences,
               new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary));
   
   
           using (MemoryStream stream = new MemoryStream())
           {
               var result = compilation.Emit(stream);
   
               if (!result.Success)
               {
                   var failures = result.Diagnostics.Where(diagnostic =>
                       diagnostic.IsWarningAsError ||
                       diagnostic.Severity == DiagnosticSeverity.Error);
   
   
                   var message = ""; // failures.Select(x => $"{x.Id}: {x.GetMessage()}").Join("\n");
                   throw new InvalidOperationException("Compilation failures!\n\n" + message + "\n\nCode:\n\n" + code);
               }
   
               stream.Seek(0, SeekOrigin.Begin);
               Assembly asm = Assembly.Load(stream.ToArray());
               return asm.GetType(typeName);
           }
          }
  

• The third option is using Mono's Compiler as a Service. It has been around for many years before Roslyn, enabling the existence of Mono's REPL. Mono's CaaS does not depend on other Mono specific libraries, so you can run it with Microsoft's Framework, you just need to reference the Mono.CSharp.dll, that can be obtained via nuget. Most samples of use of this CaaS deal with running "one shot" small pieces of code, not with creating new classes for later reuse, and the way I've found to use it for this second purpose is a bit bizarre, but it works, so here it goes:

       private static Type CreateTypeViaMono(string code, string typeName, IEnumerable references)
    {
      var evaluator = new Evaluator(new CompilerContext(
                      new CompilerSettings(),
                      new ConsoleReportPrinter()
              ));
  
              // Make it reference our own assembly so it can use IFoo
              foreach(Assembly assembly in references)
              {
               evaluator.ReferenceAssembly(assembly);
              }
  
              // Feed it some code
              evaluator.Compile(code);
              Assembly asm = ((Type)evaluator.Evaluate("typeof(" + typeName + ");")).Assembly;
              return asm.GetType(typeName);
    }
  

The signatures for method 2 and 3 are the same, for method 1 we have to pass the names of the assemblies to be referenced, rather than the assemblies themselves. If we have an IFormatter interface and we want to create a new Formatter class implementing the interface. We would do something like this:

Type upperCaseFormatterType = CreateTypeViaXXX(@"
             namespace Formatters
             {
              public class UpperCaseFormatter: Formatters.IFormatter
              {
                  public string Format(string s) { return s.ToUpper(); }
              }
    }",
                "Formatters.UpperCaseFormatter", 
                new List()
                {
                 typeof(Formatters.IFormatter).Assembly
                }
    //pass directly "Formatters.dll" for the DOM case
            );
            
            IFormatter formatter = (IFormatter)(Activator.CreateInstance(upperCaseFormatterType));
            Console.WriteLine(formatter.Format("abc"));

Part of this post is based on information and code that I got from this Post, this post, and this question.

FTPS vs SFTP

We've gone through some confusion at work recently. The specifications of something we were developing would use interchangeably (and wrongly) the terms FTPS and SFTP for some data transfers that we had to perform. After a while we realised that these are 2 different things.

FTPS is just what we were thinking about. Your good, old FTP protocol joining forces with the TLS/SSL cryptographic protocol, same as HTTP and HTTPS.

SFTP, less confusingly referred as SSH File Transfer Protocol, has nothing to do with FTP, it's a completely different protocol for transferring files in a secure manner.

The FileZilla Windows cliet application helped a bit in keeping the confusion. I used to think of it as an FTP client, but along with FTP and FTPS it also supports SFTP (notice that this is so for the client, the FileZilla server does not support SFTP). Curl also supports both protocols (along with many more).

The .Net framework supports FTPS out of the box via the FtpWebRequest class, but for SFTP you'll have to use some third party library.

Some Recent Films

These last weeks I've gone to the cinema to watch some films, so I thought I would write some short comments here.

• Carol. Seems like this film has got pretty good reviews, and here in France it's been displayed both by "blockbuster cinemas" and by Art et essai cinemas
. A dramatic "lesbian love story" in the 50's in USA seemed not particularly appealing to me, but well, I felt like going to the cinema, and the spoken English/French subtitles is just what I look for these days. I got what I expected, an entertaining film, but that I will forget in a few weeks and that probably I won't watch again. Of course, Rooney Mara and particularly Cate Blanchett (yes, I love mature beauty) make the film more interesting than it would be otherwise.
• The Hateful Eight. First of all, I'm not at all a Tarantion fan. I can say with no shame at all that for me he's just one more "action films director". I went to watch this film because I felt like going to the cinema and it was an easy choice, for sure it would not delight me, but at least would entertain me. It ended up being right like that. Probably the story is more exciting than average (I like how he made the "racial problem" a constant all over the film, it's pretty interesting bearing in mind that such problem is still present in USA society), but I really miss the presence of some beautiful female character.
• A Second Chance. This is an excellent film. Though the main character is a policeman it's more a drama than a crime film. Harsh and intense, if I had watched it on my laptop I think it would have fallen in that limited category of films that I watch in one go. I can not think of many films which title expresses so well what the film is about. The whole story revolves around a second chance for a child, but it will end up as a second chance for a mother. When it finished I thought that the style reminded me a lot of other films (specially Brothers by a Danish film director that I used to follow until a few years ago. So right, this is the last work by Susanne Bier. I absolutely recommend to watch all her filmography.

I've watched these films in ABC (in its 50th anniversary) and Utopia, the 2 Art et Essai cinemas (cinemas more focused on independent films) here in Toulouse (along with la Cinémathèque). I've watched them in original version with French subtitles (pretty needed for the Danish film or for Tarantino's, cause I barely could undertand a word of the English accents in that film). My spoken French is still null, but have no problems reading ("Je peux lire le français sans problème"). Unfortunately, French films are not screened with subtitles (quite odd, I would expect some more respect for people with hearing disabilities), so there is a good bunch of films that I've had to miss, these are 3 of the main ones that come to mind:

• Tête baissée, a thriller/drama set in Bulgaire and Marseille, maphia, human traffic... seems pretty interesting.
• Les Chevaliers blancs tells the story of the Zoe's Arch charity controversy in Chad.
• Timbuktu revolves around the monstruosities perpetrated by Islamist during the occupation of this ancient city in Mali.