The Third Annual C# Advent - The journey of porting pretzel to dotnetcore
At the time for registering for the slot I was working on pretzel and tough I want to share my journey how we managed to port it to dotnet core.
First a brief history of pretzel. I migrated from FunnelWeb to Pretzel back in 2016. I wanted to avoid Jekyll at the time, cause there was no WSL back in the day, and it was a pain to use on windows back in the day.
- It is a garage project from Code52
- Originally written in net451
- Support for Liquid
- Support for Razor
- Support for custom plugins
- Uses MEF under the hood
- And a lot of dependencies that needed to be replaced
- Nowin Fast Owin Web server in pure .Net
- RazorEngine RazorEngine - A Templating Engine based on the Razor parser.
- NDesk.Options NDesk.Options is a callback-based program option parser for C#
- DotlessClientOnly This is a project to port the hugely useful Less libary to the .NET world. It give variables, nested rules and operators to CSS.
- System.IO.Abstractions A set of abstractions to help make file system interactions testable.
- ScriptCs ScriptCs.Hosting provides common services necessary for hosting scriptcs in your application.
Before I started working on the project, I thought it would be awesome to have pretzel as a dotnet global tool cause it's a perfect fit for that. It also means I could reduce ceremony on getting pretzel running on azure devops. Currently I use a cake script that basically downloads the latest release from pretzel, unpacks it and then execute some batch commands on it. But that has, of course, some downsides to it. It has a lot of ceremony and moving parts and has therefore multiple points of failure. First download nuget via powershell, restore cake, run the build script that downloads pretzel, afterwards launch pretzel.
With the help of an global tool we can run (if at least dotnetcore2.2 is installed):
dotnet tool install -g Pretzel.Tool pretzel build
Thats it! I doubt it could get any easier!
Another goal was to support jekyll's data files. It was a happy coincidence that SunaCode was asking for that feature, cause my main reason for starting working on pretzel was exactly this feature after I read about using static comments for jekyll to finally replace disqus cause it is hard to justify for privacy and performance reasons for a blog. Since I started my own business starting this year, I now am responsible for that stuff and I really need to care.
We didn't want to force old users to switch to
netcoreapp2.2, so we wanted to support full framework with a reasonable amount of work and aimed to target
net462 cause it's the first version that fully supports
We also aimed for usage compatibility. So from an end user perspective everything needed to be compatible.
I used the term we a lot in the post so far. That's rather rare from my perspective cause I'm only a one man shop (so far). What do I mean with we?
After I prepared my first PR it lit laedit on fire as well. He is a former lead contributor to pretzel, but due lack of time (as this is often the case with side projects) he stopped contributing, but that didn't mean he forgot or abandon the project all together.
We discussed goals, problems, strategies and chances on the project. He did all the code reviews, jumped in when I needed help (esp. for making the build green again on CI). Also he had the awesome idea to create a project on github to make our progress more visible. He also named the project after me, which was a little bit frightening, but also motivating on the other hand. The 1.0 in the name of the project was the most frightening part, nobody really want's to ship 1.0 but I really am proud to be the chosen one.
But the most important thing: WE kept the project on fire, kept good vibes (even if we had sometimes a hard debate on technical implementation details) and had always the goal in front of our inner eye. It is fun to work with such an open minded community, esp. with such a great guy than laedit! Big shout outs to you my friend, it wouldn't be possible without your effort!
The community is the thing that drives any open source project! So jump in and start with it now! Write issues, test things, write docs, do code reviews, write code or write blogs! Every little bit matters a lot. And remember. Be kind to each other! (Hey it's christmas, what would your mom think about you 😲)
Cause there are a lot of dependencies and goals to solve, we decided to tackle our problems in multiple phases.
- Switch to new csproj file to ease multi targeting
- Update all dependencies to the latest version
- Look for all easy replacements for the dependencies that already have
- Lookout for alternatives for those dependencies that are either deprecated or don't provide an easy
- Switch to
- Use the new
System.CommandLine.Experimentalpackage for the deprecated
- Add the data files feature
- After all dependencies are on
- Build and package the
- Somewhere in between provide more docs for usage and plugin authors
Did we follow our plan correctly in that order? No. But a plan helps you keep goals in sight and keep things rather organized
Early on in the project we had some tough decisions to made. We wanted to be compatible with old plugins, but after we started to work on it, we realized we need to make some tradeoffs.
We either could do cross compiling for
netcoreapp2.0 and use different dependencies for
netstandard2.0 and do a lot of
#if DEF compilation, or we force plugin authors to update and recompile their plugins. We choose the second option, cause it's 1.0 anyway and if we choose the first option it would mess up the code a lot. Sometimes it's just better to release old burdens.
I'll not going to cover that in full detail, but you can see the list of PR's but give a higher level of perspective and don't want to bore you with all nitty gritty details.
Converting the project to the new csproj format
dotnet migrate-2017 migrate
After a bit of cleanup it was building and tests were still passing.
The tricky part was here: The new package format does output the artifacts not under
bin/Debug but under
bin/Debug/[TFM]. So getting the build and packaging back ready was a little bit tricky.
Update the packages to the latest version
That one was also easy (except for
System.IO.Abstractions) using the dotnet outdated tool.
dotnet outdated -u
System.IO.Abstractions did make some unit tests fail, so we moved on and fixed that later. I've upgraded to the latest version that made the tests pass manually, and luckily enough that version already supported
netstandard2.0. We decided to upgrade to the latest version later on, cause we were confident enough that everything was working trough our automated and manual tests.
Multi target Pretzel.Logic for net462/netstandard2.0 and replace MEF with System.Composition
netstandard2.0 is somehow compatible with
net462 this was the first real big PR.
It took from 5th September to 17th September, 48 commits and 64 comments through the code review. That was the second largest PR in the journey.
That was one of the points in the journey where we finally decided we need to force plugin authors to recompile. But we didn't just throw a new dependency in, we deeply thought about how we want our plugin architecture and API surface will look like in the future.
System.Composition are somewhat the same conceptional, they are fundamentally different from API perspective. There is a lack of recomposition, metadata is handled differently. There is no built in way to register objects into the container and so on.
But on the other hand I got such a great overview how the project is composed and how it's architecture looks like in detail.
At that point we dropped support for
ScriptCS (for now) cause there is no package provided by the
ScriptCS team that supports
System.Composition. Cause there is no support for
netcoreapp2.0 and we will need to replace it anyway.
Most of the changes are just changing the visibility of members and adding some attributes. At this point nothing in our test suite helped us. Correct configuration of Composition/DI is often not covered by automated unit tests in a project, cause integration tests normally will cover that. That meant a lot of trial and error.
I've never used
System.Composition before. It was quite a learning curve (and a lot of false assumptions I made) but most of the error messages were very helpful (esp. compared to MEF1 & MEF2). We also managed to eliminate some architectural flaws in pretzel. So that was a good start.
Fix warnings and remove used obsolete API's
After upgrading to the latest versions of the packages there were a ton of warnings (I think about 300+). It was crucial to remove them before moving further.
Never let your warnings go wild. You'll miss important warnings if your project has hundreds and hundreds of warnings.
Most of them were related to xUnit and the new analyzer package. We found some bug's in some test cases and also improved the readability of test failures a lot. That helped later on switching to the latest
System.IO.Abstraction package. I was unable to fix the issues cause of bad error messages in the test. Eg.: Expected value was true but actual value was false
The automated fixes by the xunit analyzer helped a lot here.
Add support for data files
After that I was a bit exhausted working on the port. I thought it would be good to start implementing a new feature. This was of course not planned, but loosing the joy on the project wouldn't helped either. At this point SunaCode did his feature request.
I think everyone knows that feeling if you are working to long and to hard on a project you feel worn out.
Implementing the feature wasn't that hard, but varied enough that I kept up the motivation! It was the first feature in the product for quite a while!
That's something I really can take with me for the future. For me, my company and for real life! Get sometimes something fresh and new, that keeps you and your team on track. Don't burn out cause you feel the urge to just get stuff done.
Switch to netstandard2.0 compatible packages
Now we looked what alternatives we have to the existing packages. We picked the easy ones first:
In LOC thats 18. I think that's easy enough.
Those are no drop in replacements by definition, but hey they work just as good as the old dependencies!
What I've learnt here: Development behaves sometimes like water, go the easiest route first.
Replace NDesk.Options with System.CommandLine.Experimental
That was the largest PR. 65 commits, 79 files changed and 117 code review comments. It took from 18th of September till 8th of October to get finally merged.
That's the PR we made almost all decisions how we want out plugin API to look like. And now I finally understand why it takes so long to design nice API's. There was a lot of discussion about what is the most flexible, future proof, but understandable and easy to use API we can imagine for pretzel. It started with a rather simple proposal I thought would be good enough. But laedit pushed me to think harder, try different approaches and we finally got a version we are both happy with.
It also changed a lot in the overall architecture of the product, but I think it was worth the effort. It's easier to maintain, extend and unit test.
Most of the time writing code isn't the most time consuming aspect of development. It's communication, thinking about future consequences and impacts. Thinking and planning for the future is the most time consuming aspect for any real project.
Replace RazorEngine with RazorEngine.NetCore and remove NoWin
That one is one of the down sides of relying on external dependencies.
The maintainer of
RazorEngine.NetCore did a great job in porting the project to dotnet core, but does not seam to care about contributions. So that was a kind of bummer. By so much energy in our ongoing project I was so angry and mad! I really advice people out there: uploading stuff on github and create a nuget package isn't doing open source... But on the other hand I am glad he did anyway cause we decided to fork and created our own package. For like 10 LOC.
NoWin was easy: We already need aspnetcore, so we just used that instead of NoWin.
The heureka moment or when we multi target net462 and netcoreapp2.0
We replaced the last dependency on full framework. Countless hours of debugging and coding/reviewing went into it.
We multi targeted
Some fixing of testcases, mostly about
System.Diagnostics.Process and about resources.
After some manual testing I was wondering: Could that run on Linux?
And I was like. What the actual fuck. I can't even believe it. I'm the first man on earth that ever ran pretzel on linux. 😎
Make an actual global tool
Some adjustments to the project and build script (3 LOC) and we had an global tool!
So the execution journey ends here so far.
Did we release pretzel as a global tool and made the 1.0 happen? Not yet. Are we almost there? Yes!
As you can see on the project there are some goals open we want to tackle before finally releasing 1.0, does that mean we failed? Absolutely NOT. There are a few things open (like for example ScriptCS support) we don't even know if it will land in 1.0.
Was it worth all the effort, tears and blood that flow into the project? Absolutely YES. I learned a lot contributing to the project, a lot on motivation, goals, planning and working in the open with people I never met in person. I worked a lot remotely, but working on open source is completely different. It's such a great feeling to work with people that are really appreciate your work and are passional with a project. Cause every little bit matters a lot.
Hope you had as much joy reading my article as I have working on this awesome open source project. Feel free to jump in! Try out pretzel, give it a star on github. Happy holidays and have a nice remaining C# Advent to anybody out there.
Ps. Again big thanks to Matthew for the slot on his series, keep up the great ideas!