scriptcs

03 Aug 2015

Deprecating Common.Logging - The first step in removing Common.Logging

We decided some time ago to remove Common.Logging as a dependency from scriptcs. In scriptcs 0.15 we’ve taken the first step in this process by introducing our own logging API based on LibLog and deprecating the use of Common.Logging.

Why are we doing this? Common.Logging has been a pain point for some time now for anyone who is using scriptcs in a hosting scenario or building their own NuGet packages based on scriptcs. Package version mismatches/binding redirect tangles, breaking changes in the 2.x version range (either accidental or deliberate) have been plaguing us for a while so we decided it was time to remove the problem once and for all. scriptcs.exe itself also ran into problems when we discovered Common.Logging 2.2 is not compatible with Mono on Linux.

Rather than remove Common.Logging right away, we decided to introduce our new logging API alongside Common.Logging, with Obsolete attributes added to communicate the deprecation. This way, you can upgrade your projects to our 0.15 NuGet packages without any immediately necessary changes.

Unfortunately, as is often the case with such things, the upgrade path isn’t quite so simple. We’ve done our best to ensure backwards API compatibility between our 0.14 and 0.15 packages. This means you should be able to switch out 0.14 binaries for 0.15 binaries for use by your compiled assemblies and everything should continue to work. However, if you are expecting scriptcs to inject a Common.Logging.ILog into your assembly via MEF, this will no longer work, since scriptcs only has the new ScriptCs.Contracts.ILogProvider type registered. Instead, take a dependency on ScriptCs.Contracts.ILogProvider. Also, if your source code refers to Common.Logging.ILog as just ILog with using Common.Logging; and you have using ScriptCs.Contracts;, you will receive a compiler exception because ILog is now ambiguous. This should require a very simple fix to your source code to get things compiling again. We decided to treat this as a non-breaking change, since using statements are syntactic sugar designed to reduce verbosity in source code and are always susceptible to the introduction of a type with the same name as a type in another namespace, whereas backwards compatibility for compiled assemblies, which will always reference types unambiguously, has been preserved.

If you’re hosting or maintaining your own NuGet package and using scriptcs 0.14 or lower, upgrade to 0.15 today and start switching over to our new logging API. We will be removing Common.Logging from scriptcs entirely in a forthcoming release (exactly when is not yet decided) so the sooner you make the switch, the sooner you will immunise yourself against that future change. When we finally remove Common.Logging entirely, scriptcs will have one less (problematic) dependency and will be more lightweight and easier to use for hosting and for building your own scriptcs based NuGet packages.

23 Mar 2015

Script Libraries - Load scripts embedded in NuGet packages

In scriptcs v0.14 we have introduced Script Libraries. This feature allows you to author functionality as a set of csx files, package them up as a NuGet package and then easily reuse that functionality across your scripts.

For those of you familiar with Script Packs this may sound familiar. The biggest downside of Script Packs is that you still need to open up an IDE and deal with projects and interfaces in order to create them. One of the aims of the scriptcs project is to simplify your C# workflow and this is where we believe Script Libraries will shine.

Let’s see this in action!

Let’s assume we have a NuGet package ScriptCs.Calculator. It has been installed into the script folder using the standard scriptcs -install ScriptCs.Calculator option. The NuGet package contains a csx file named CalculatorMain.csx living in the Content\scriptcs folder that contains the following function:

CalculatorMain.csx

public double Add(double a, double b) {
  return a+b;
}

Creating a script as follows:

demo.csx

var calc = new Calculator();
var result = calc.Add(40,2);
Console.WriteLine(result);

And running it will result in the following:

> scriptcs demo.csx
42

Note that we didn’t need to use the Script Pack Require<T> mechanism to bring the Script Library into our script. If the Script Library has been installed, then all you need to do is new it up. Similarly to Require<T> it will automatically inject required usings and imports.

You may also have noticed that the csx file in the NuGet package had a Main suffix. This convention is how we discover the entry script within your NuGet package, and there must be one present. The fact that the csx file was named CalculatorMain.csx, allowed us to new up the Script Libary via the var calc = new Calculator(); line in our script.

Let’s dive a little deeper into the details of how you build a Script Library.

Building a Script Library

Our Script Library is called Calculator, so we need a csx entry script named CalculatorMain.csx.

CalculatorMain.csx

#load "MultiplyAndDivide.csx"
 
private Logger logger = Require<Logger>();
 
public double Add(double a, double b) {
  logger.Info(String.Format("Adding {0} + {1}", a, b));
  return a+b;
}
 
public double Subtract(double a, double b) {
  logger.Info(String.Format("Subtracting {0} - {1}", a, b));
  return a-b;
}

You can see that this script contains basic Add and Subtract functions.

Notice it also pulls in the [scriptcs-logger] (https://github.com/paulbouwer/scriptcs-logger) script pack. It is possible to pull in a traditional Script Pack via the Require<T> mechanism. Note that you cannot use var to declare the variable holding the Script Pack within Script Libraries. You MUST use an explictly typed variable. For the background on why, refer to the Adding a library reference section of the Script Libraries Design Document.

Additional Multiply and Divide functions are pulled in using the #load directive and a MultiplyAndDivide.csx script contained in the Script Library. This is just for illustration in this case, but it allows you to factor your Script Library so it is not a monolithic code file. Here is the contents of that script:

MultiplyAndDivide.csx

public double Multiply(double a, double b) {
  logger.Info(String.Format("Multiplying {0} * {1}", a, b));
  return a*b;
}

public double Divide(double a, double b) {
  logger.Info(String.Format("Dividing {0} / {1}", a, b));
  return a/b;
}

In scriptcs you are required to place all directives (#r, #load, etc) and using statements at the top of your script. To ensure that directives and using statements within your Script Libary do not cause issues, scriptcs will ensure that all of them are parsed, extracted and pre-pended at the top of the generated script output. So there is nothing different you need to do in the csx files included in your Script Library.

Within your script pack you can also contain other classes which the consumer can use.

The final bit of creating the Script Library is to create the NuGet package. This is done, as with any other NuGet package, via a nuspec file. Our NuGet package will be called ScriptCs.Calculator.

ScriptCs.Calculator.nuspec

<?xml version="1.0"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
  <metadata>
    <id>ScriptCs.Calculator</id>
    <version>0.1.0</version>
    <authors>Glenn Block</authors>
    <owners>Glenn Block</owners>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>A calculator Script Library</description>
    <dependencies>
      <dependency id="ScriptCs.Logger.ScriptPack" version="0.1.0" />
    </dependencies>
  </metadata>
  <files>
    <file src="CalculatorMain.csx" target="Content\scriptcs\CalculatorMain.csx" />
    <file src="MultiplyAndDivide.csx" target="Content\scriptcs\MultiplyAndDivide.csx" />
  </files>
</package>

You can see the dependency added for the Script Pack ScriptCs.Logger.ScriptPack and the CalculatorMain.csx and MultiplyAndDivide.csx files added. Note that the csx files must be added into a Content\scriptcs\ target folder.

Create the NuGet package. Congratulations, you now have your first Script Library!

> nuget pack ScriptCs.Calculator.nuspec
Attempting to build package from 'ScriptCs.Calculator.nuspec'.
Successfully created package 'c:\NuGet\ScriptCs.Calculator.0.1.0.nupkg'.

Testing it out

Now that we have a NuGet package ScriptCs.Calculator.0.1.0.nupkg that contains our Script Library Calculator it is time to test it out.

Create a file called demo.csx that will new up the Calculator Script Library and invoke the Add function it exposes.

demo.csx

var calc = new Calculator();
var result = calc.Add(40,2);
Console.WriteLine(result);

Since the NuGet package we’ve just created is not published we’ll use a scriptcs_nuget.config file in our script folder. This allows us to override the system wide NuGet options while running our script. I’ve placed the ScriptCs.Calculator.0.1.0.nupkg NuGet package into the C:\NuGet folder on my machine and added the folder as an additional package source in the scriptcs_nuget.config file.

scriptcs_nuget.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
	<add key="LocalRepo" value="C:\NuGet" />
    <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
  </packageSources>
  <disabledPackageSources />
  <activePackageSource>
    <add key="All" value="(Aggregate source)" />
  </activePackageSource>
</configuration>

Install the ScriptCs.Calculator Script Library. This will install the Script Library and all its dependencies.

> scriptcs -install ScriptCs.Calculator
Installing packages...
Installed: ScriptCs.Calculator
Package installation succeeded.
Saving packages in scriptcs_packages.config...
Creating scriptcs_packages.config...
Added ScriptCs.Calculator (v0.1.0) to scriptcs_packages.config
Added ScriptCs.Contracts (v0.9.0, .NET 4.5) to scriptcs_packages.config
Added ScriptCs.Logger.ScriptPack (v0.1.0, .NET 4.5) to scriptcs_packages.config
Successfully updated scriptcs_packages.config.

Next run the script. Any parameters after the -- are made available in Env.ScriptArgs which is accessible by your scripts, Script Packs and Script Libraries. The -loglevel info will be used by the Logger Script Pack to log details at the appropriate level.

Running demo.csx on Windows

As mentioned earlier, the Calculator class is present because we named the entry point file as CalculatorMain.csx. When scriptcs loads a Script Library, it will automatically put the contents into a wrapper class matching the entry point. This is also advantageous because it removes member conflicts. Two Script Libraries can have the exact same members, but they will not override one another as they are all scoped to their wrappers.

If you look at the scriptcs_packages folder you’ll notice a ScriptLibraries.csx file after you have run the script.

ScriptLibraries.csx

The ScriptLibraries.csx file is built from the Script Libraries present in the scriptcs_packages folder and contains the wrapped libraries. If this file is present, scriptcs will merge this file into the main script file and execute. See the contents below:

public class Calculator : ScriptCs.ScriptLibraryWrapper {
#line 1 "C:\src\test\scriptcs_packages\ScriptCs.Calculator.0.1.0\Content\scriptcs\MultiplyAndDivide.csx"
public double Multiply(double a, double b) {
  logger.Info(String.Format("Multiplying {0} * {1}", a, b));
  return a*b;
}

public double Divide(double a, double b) {
  logger.Info(String.Format("Dividing {0} / {1}", a, b));
  return a/b;
}
 
#line 3 "C:\src\test\scriptcs_packages\ScriptCs.Calculator.0.1.0\Content\scriptcs\CalculatorMain.csx"
private Logger logger = Require<Logger>();
 
public double Add(double a, double b) {
  logger.Info(String.Format("Adding {0} + {1}", a, b));
  return a+b;
}
 
public double Subtract(double a, double b) {
  logger.Info(String.Format("Subtracting {0} - {1}", a, b));
  return a-b;
}
}

This cached version of the Script Libraries is used to ensure that subsequent script executions do not pay the penalty for building this file.

If the scriptcs -install option is invoked to install any new NuGet packages this file will be deleted and automatically rebuilt on the next script execution.

Cross Platform !

Here is the same demo running on Ubuntu 14.10.

Running demo.csx on Ubuntu

If you are testing this demo out using mono then make sure that you have the correct certificate authorities set up. For Ubuntu 14.10 you will have run the following:

> sudo certmgr -ssl -m https://go.microsoft.com
> sudo certmgr -ssl -m https://nugetgallery.blob.core.windows.net
> sudo certmgr -ssl -m https://nuget.org
> mozroots --import --sync

Ensure that you configure your local NuGet repo folder correctly for a non-Windows environment too.

scriptcs_nuget.config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
	<add key="LocalRepo" value="/home/paul/NuGet/" />
    <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
  </packageSources>
  <disabledPackageSources />
  <activePackageSource>
    <add key="All" value="(Aggregate source)" />
  </activePackageSource>
</configuration>

More Details

For some deeper insight into design decisions and implementation details have a look at the Script Libraries Design Document.

With Script Libraries we’re taking the scripting experience another notch. You’ll now be able to easily share your scripts as NuGet packages, so they can be reused without ever having to create a project or touch Visual Studio :-)

We really hope you enjoy the new Script Library experience and look forward to your feedback!

28 Jan 2015

A new file system - Playing better with .NET solutions.

With scriptcs version 0.13 we decided to make some changes to help scriptcs run more smoothly together with existing .NET solutions. A problem which existed until now was the occasional inability to run scriptcs from within the folder of a regular .NET solution due to problematic assemblies in packages or bin folders. These would be assemblies which were not installed in order to be used from scriptcs, but rather from the .NET solution, and cause errors or warnings when running scriptcs. In particular, developers and users of build systems based on scriptcs, e.g. PVC and Bau, were often tripped up by this.

The problem occured because, until now, the file system conventions used by regular .NET and scriptcs overlapped. They both used packages.config, packages/ and nuget.config for NuGet package management and scriptcs searched bin/ for binaries, a folder often used for other purposes in a .NET solution.

Starting with version 0.13, we’ve changed our conventions to isolate our file system artifacts from interference with a regular .NET solution.

.cache            ->   .scriptcs_cache
bin/              ->   scriptcs_bin/
packages/         ->   scriptcs_packages/
nuget.config      ->   scriptcs_nuget.config
packages.config   ->   scriptcs_packages.config

Automatic migration

When you run scriptcs 0.13, it will detect ‘legacy’ file system artifacts (those shared with regular .NET) and automatically migrate them. To ensure anything outside scriptcs keeps functioning (e.g. an existing .NET solution), the existing artifacts are left in place, and copies are made instead. The only artifact which is renamed is .cache (to .scriptcs_cache) since this was never shared with regular .NET, but has still had it’s name changed for consistency.

E.g. if your folder looked like this before running scriptcs 0.13 (you may not have all of these artifacts):

you will see this output when running scriptcs 0.13:

and you your folder will end up looking like this:

If you don’t need the original artifacts, feel free to delete them. It would be very difficult to automatically detect, with certainty, whether or not they’re required so we thought we’d leave it to you to decide.

Performance

Another nice effect of this change is a potential increase in performance. When scriptcs starts up, it scans bin/ and packages/ for modules and script packs. Previously, a large .NET solution may have many assemblies in these folders, not required for script execution, and this would result in a performance hit. With the new file system conventions, only those assemblies needed for script execution are scanned and that should speed things up in many cases.

Hosting

If you are hosting scriptcs in your own application and you want to perform automatic migration, you will need to call scriptServices.FileSystemMigrator.Migrate() at an appropriate place in your application.

Summary

Our hope is that these changes will smooth out any friction you may have running scriptcs in environments shared with regular .NET solutions and projects. In the majority of cases, the upgrade to scriptcs 0.13 should be seamless. Migration to the new conventions is automatic and the only burden left on you is to remove any unwanted ‘legacy’ artifacts. If you do have any problems, please don’t hesitate to let us know by raising an issue.

By the way, the file system changes are not the only changes in scriptcs version 0.13. For full details of what is contained in this release, please see the release notes.

02 Apr 2013

What's Next - Improving the C# scripting experience.

As promised in the previous post we’d like to share with some plans for the near (and more distant future).

There is definitely a ton of exciting stuff ahead!

It’s important to emphasize, that this project is owned by the community, and as such, the roadmap is also shaped by the community! After all, we want to make the C# scripting experience as smooth as possible for everyone.

Here are the major upcoming features.

Mono support

Thanks to the amazing work of Dale Ragan, we are closing in on Mono support. You can follow his fork to see the progress first-hand, and you can join the discussion on Github.

Nuget Package Installation

This has actually just made its way to the scriptcs dev branch. Since nuget.exe does not resolve nested dependencies when restoring packages from a packages.config, we have introduced Nuget installation built-in to scriptcs itself.

To install your NuGet packages along with their dependencies, you can simply run the following command in the same directory as your packages.config file:

 scriptcs -install

If there is anything else you’d like to see in this particular idea, let us now!

Extensibility (Script Packs)

Script Packs provide a way for 3rd parties to plug in their own framework and remove a lot of cruft for developers using them.

The entire extensibility model is designed around three core ideas:

Reference additional assemblies
Importing using statements
Surfacing globally-scoped objects to the script

While the extensibility model for scriptcs is pretty much complete, we are working on creating samples to showcase its usage. We plan to have an ASP.NET Web API sample out there very soon, as well as documentation on how to create your own script packs.

We think script packs will be a great addition for the community and we hope to see packs for all the popular third party frameworks i.e. Nancy, ServiceStack, Caliburn, etc.

Startup performance

Currently we perform a considerable amount of file IO when script gets executed. While for smaller scripts and low number of NuGet packages, it’s mostly not visible, it may become annoying for larger script-based solutions or applications.

We are now working on a set of performance improvements in that area. As always, any feedback is expected - the discussion is going on here.

Exports

We have started laying ground for a project that would allow you to create a Visual Studio solution directlz from your scriptcs script. In other words, you can start working on a script-based project, and if one day you decide it’s time to move to Visual Studio, this tool will facilitate it.

We are also working on allowing an export of your script to an executable (*.exe) which will allow you to share the script with people who wouldn’t have scriptcs installed.

scriptcs clean

If you want to share or move around your scriptcs project, all you really need are *.csx files and packages.config (and perhaps any custom non-NuGet assemblies if you use such).

However Nuget packages and bin folders are known to often weigh tens of megabytes. To improve this, we are introducing a clean command:

        scriptcs -clean

which will clean up the application folder of unnecessary files for you, making it easier to share/move your scriptcs. You can join the discussion on the topic here.

Exception handling/logging

As we are growing more mature, we (with help from @dschenkelman) are working on a better logging/tracing experience for scriptcs, as well as on a good error propagation model.

Lightweight configuration/opts

We are discussing several ways of a possible configuration model for scriptcs. This includes both the configuration of the scriptcs application itself, and the opts file to act as a shortcut to command line commands.

On a side note, we have already embraced PowerArgs for a smooth command line experience.

Unit tests

There have been a few requests about unit testing your C# scripts. We are investigating several potential solutions of allowing you to write and execute unit tests for *.csx files.

Stay tuned!

Tutorials & Distribution

We already have the nightly packages available at MyGet

http://myget.org/gallery/scriptcsnightly

however, we are closing in on a first (pre-release) NuGet drop! As part of that, we are planning on dedicating more time to documentation and tutorials, as these items come up often in Twitter conversations!

Going beyond CLI

We have ScriptCs.Core which can be used to build applications that host scriptcs outside of the command line.

We plan to make sure the API is well-documented, clean, and understandable, as well as, at some point, introduce some samples of consuming ScriptCs.Core.

Octopus Deploy support

Paul Stovell, the creator of Octopus Deploy has just announced that Octopus Deploy 1.5 supports scriptcs!

This is absolutely terrific news, and we’d like to thank Paul (who also contributed to scriptcs source) for being the first to officially embrace our platform!

Further scriptcs adoption

We have some more folks interested in adopting scriptcs in their solutions/tools. We think this is great and are really excited to see how it turns out!

Hopefully many other will follow!

17 Mar 2013

scriptcs - it's not an experiment

Just a little over two weeks ago scriptcs started with Glenn’s blog post. In that post he talked about a lightweight approach to authoring C# programs from outside Visual Studio in a text editor without needing a solution or a project. He posted the first preview of a tool called scriptcs which enabled the workflow leveraging Nuget and Roslyn.

The response from the community was extremely positive!!!

Almost immediately after that scriptcs moved from an experiment to a full-fledged OSS project. It started from a Skype call which resulted in Justin Rusbatch and Filip W joining as coordinators.

Immediately after we created a Github organization, a website, a twitter account, a google group and a jabbr chat.

Flash forward two weeks later. To be honest, things have been going forward at such crazy pace, that it really does feel like it’s been fifty two weeks instead :-)

Therefore, it is perhaps a good moment to look back at what has happened so far.

However, before that, it is necessary to emphasize that the response from the community has been absolutely staggering, and has vastly exceeded any of our expectations! This is also a great testament to how needed a good, lightweight, C# scripting environment is.

Scriptcs, owned by the community

This project has moved far beyond Glenn’s initial experiments. It is now 100% owned by the community. All decisions are made publicly and transparently in our Github issues and you are more than welcome and encouraged to particpate!

As such, I think it’s important to highlight that things have really changed at Microsoft, because not that long ago, it would have been virtually impossible to have a Microsoft person involved in coordinating such a project.

There has really been a tremendous amount of contribution from the community. GitHub has been literally lit up with notifications and Twitter has been beaming with activity too.

As of today, on Github, we have:

5 repos within the organization
17 unique code contributors
226 stars
54 forks
138 issues
71 pull requests

What’s been added

As you can see since the last post we’ve had an incredible amount of energy. Here’s a list of the new features that have been added with links for you to find out more.

Script includes via #load (contributed by @filip_woj)
A sublime text plugin! Better command line parsing and output (contributed by @follesoe)
Proper nuget package binding via directly leveraging nuget.core (contributed by @filip_woj)
Debugging in Visual Studio! (contributed by @dschenkelman)
Pluggable script engines (contributed by @nberardi)
Script Packs, scriptcs extensiblity (contributed by @gblock)
Team city build, nuget packages (contributed by @jrusbactch)
Chocolatey package (contributd by @mrockx)
A ton of samples and a dedicated samples repo (see more below)

Samples

Thanks to a great help from community, we are adding more and more samples (which now have their own dedicate repository too).

At the moment we already have:

Web API self host
ServiceStack self host
Nancy self host (contributed by @thecodejunkie)
Fluent Automation (contributed by @stirno)
SignalR (contributed by @jonhcoder)
RavenDB (contributed by @khalidabuhakmeh)
WPF (with MVVM) (contributed by @khellang)

If you wish to contribute a new sample just head over to the repo! We are always looking for examples, showcasing different uses cases of scriptcs.

Packages feed

We are publishing our nightly builds to our MyGet feed, which is available at

        http://www.myget.org/F/scriptcsnightly/

So you are strongly encouraged to try it out. The initial release is an alpha with version 0.3. This includes:

scriptcs v0.3.0-alpha (Chocolatey package)
ScriptCs.Contracts v0.3.0-alpha (Extension point for script packs)
ScriptCs.Core v0.3.0-alpha (Core service)
ScriptCs.Engine.Roslyn v0.3.0-alpha (Roslyn-based execution engine)

Contributing to it’s success, “you take it”

Scriptcs will only be succesful with the communities help. If you want to jump in and help out, we’ve put up a contribution guideline document which has worked out great, both for the project and everyone wishing to contribute. We also introduced a new ‘you take it’ label for issues that we explicitly WANT you to take!

We also have a project logo, created by Salman Quazi, and that has helped us develop an identity of the project.

Glenn has also been on dotNetRocks podcast to talk to Carl and Richard about scriptcs! Make sure to grab the podcast.

What’s next?

There is a ton of exciting stuff ahead! And that will be covered in our next post.

We want your feedback and your help!

We’d love to hear from you about any use cases that you might have for scriptcs!

To sum it all up - it’s been really crazy two weeks, and exciting times are ahead! We can’t wait to see where this goes!

Official team blog