Irmin 0.12 with portable filesystem watching released

Development of the Irmin Git-like data store continues (see here for an introduction). We are releasing Irmin 0.12.0 which brings support for native file-system watchers to greatly improve the performance of watches on the datastore.

Previously, an Irmin application that wanted to use watches would setup file-system scanning/polling by doing:

  let () = Irmin_unix.install_dir_polling_listener 1.0

which would scan the .git/refs directory every second. This worked in practice but was unpredictably latent (if unlucky you might wait for a full second for the watch callbacks to trigger), and disk/CPU intensive as we were scanning the full storage directory every second to detect file changes. In the cases where the store had 1000s of tags, this could easily saturate the CPU. And in case you were wondering, there are increasing number of applications (such as DataKit) that do create thousands of tags regularly, and Canopy that need low latency for interactive development.

In the new 0.12.0 release, you need to use:

   let () = Irmin_unix.set_listen_dir_hook ()

and the Irmin storage will do “the right thing”. If you are on Linux, and have the inotify OPAM package installed, it will use libinotify to get notified by the kernel on every change and re-scan the whole directory. On OSX, if you have the osx-fsevents OPAM package installed, it will do the same thing using the OSX FSEvents.framework. The portable compatibility layer between inotify and fsevents comes via the new irmin-watcher package that has been released recently as well. This may also come in useful for other tools that require portable OCaml access to filesystem hierarchies.

If you are using Irmin, please do let us know how you are getting on via the mailing list and report any bugs on the issue tracker.

»


MirageOS Summer 2016 Hackathon roundup

Our first Cambridge-based MirageOS hackathon took place yesterday - and what a fantastic day it was! The torrential rain may have halted our punting plans, but it didn’t stop progress in the Old Library! Darwin College was a fantastic venue, complete with private islands linked by picturesque wooden bridges and an unwavering wifi connection.

People naturally formed groups to work on similar projects, and we had a handful of brand new users keen to get started with OCaml and Mirage. The major tasks that emerged were:

  • new hypervisor target: the integration of the Solo5 KVM-based hypervisor backend, bringing the number of officially supported targets up to 3 (Xen, Unix and KVM)
  • build system template: establishing a new topkg template for MirageOS libraries, to prepare us for building a unified API documentation bundle that works across all the entire project.
  • CPU portability: improving ARM support via a better base OS image.
  • libraries breadth: hacking on all the things to fill in the blanks, such as btree support for bare-metal Irmin, or a peer-to-peer layer for the DataKit.

We’ll write about all of this in more detail, but for now here are the hackathon notes hot off the press…

Solo5/MirageOS integration (KVM-based backend)

Progress on the Solo5 project has been steaming ahead since January, and this was the perfect opportunity to get everyone together to plan its integration with MirageOS. Dan Williams from IBM Research flew over to join us for the week, and Martin Lucina headed to Cambridge to prepare for the upstreaming of the recent Solo5 work. This included deciding on naming and ownership of the repositories, detailing the relationships between repositories and getting ready to publish the mirage-solo5 packages to OPAM. Mindy Preston, our MirageOS 3.0 release manager, and Anil Madhavapeddy and Thomas Gazagnaire (OPAM minions) were on hand to help plan this smoothly.

See their updates from the day on Canopy and related blog posts:

Onboarding new MirageOS/OCaml users

Our tutorials and onboarding guides really needed a facelift and an update, so Gemma Gordon spent the morning with some of our new users to observe their installation process and tried to pinpoint blockers and areas of misunderstanding. Providing the simple, concise instructions needed in a guide together with alternatives for every possible system and version requirement is a tricky combination to get right, but we made some changes to the installation guide that we hope will help. The next task is to do the same for our other popular tutorials, reconfigure the layout for easy reading and centralise the information as much as possible between the OPAM, MirageOS and OCaml guides. Thank you to Marwan Aljubeh for his insight into this process.

Other industrial users are also steaming ahead with their own MirageOS deployments. Amir Chaudhry spent the hackathon blogging about NFV Platforms with MirageOS unikernels, which details how Ericsson Silicon Valley has been using MirageOS to build lightweight routing kernels.

Packaging

Thomas Gazagnaire was frenetically converting functoria, mirage, mirage-types and mirage-console to use topkg, and the feedback prompted fixes and a new release from Daniel Buenzli.

ARM and Cubieboards

Ian Campbell implemented a (slightly hacky) way to get Alpine Linux onto some Cubieboard2 boxes and provided notes on his process, including how to tailor the base for KVM and Xen respectively.

Meanwhile, Qi Li worked on testing and adapting simple-nat and mirage-nat to provide connectivity control for unikernels on ARM Cubieboards to act as network gateways.

Hannes Mehnert recently published a purely functional ARP package and continued refining it (with code coverage via bisect-ppx) during the hackathon.

MirageOS 3.0 API changes

Our MirageOS release manager, Mindy Preston, was on hand to talk with everyone about their PRs in preparation for the 3.0 release along with some patches for deprecating out of date code. There has been a lot of discussion on the development list. One focus was to address time handling properly in the interfaces: Matthew Gray came up from London to finish up his extensive revision of the CLOCK interface, and Hannes developed a new duration library to handle time unit conversions safely and get rid of the need for floating point handling. We are aiming to minimise the dependency on floating point handling in external interfaces to simplify compilation to very embedded hardware that only has soft floats (particularly for something as ubiquitous as time handling).

Error logging

Thomas Leonard continued with the work he started in Marrakech by updating the error reporting patches (also here) to work with the latest version of MirageOS (which has a different logging system based on Daniel Buenzlis Logs). See the original post for more details.

Ctypes 0.7.0 release

Jeremy released the foreign function interface library Ctypes 0.7.0 which, along with bug fixes, adds the following features:

  • Support for bytecode-only architectures (#410)
  • A new sint type corresponding to a full-range C integer and updated errno support for its use (#411)

See the full changelog online.

P2P key-value store over DataKit

KC Sivaramakrishnan and Philip Dexter took on the challenge of grabbing the Docker DataKit release and started building a distributed key-value store that features flexible JSON synching and merging. Their raw notes are in a Gist – get in touch with them if you want to help hack on the sync system backed by Git.

Developer experience improvements

The OCaml Labs undergraduate interns are spending their summers working on user improvements and CI logs with MirageOS, and used the time at the hackathon to focus on these issues.

Ciaran Lawlor is working on an editor implementation, specifically getting the IOcaml kernel working with the Hydrogen plugin for the Atom editor. This will allow developers to run OCaml code directly in Atom, and eventually interactively build unikernels!

Joel Jakubovic used Angstrom (a fast parser combinator library developed by Spiros Eliopoulos) to ANSI escape codes, usually displayed as colours and styles into HTML for use in viewing CI logs.

Windows Support

Most of the Mirage libraries already work on Windows thanks to lots of work in the wider OCaml community, but other features don’t have full support yet.

Dave Scott from Docker worked on ocaml-wpcap: a ctypes binding to the Windows winpcap.dll which lets OCaml programs send and receive ethernet frames on Windows. The ocaml-wpcap library will hopefully let us run the Mirage TCP/IP stack and all the networking applications too.

David Allsopp continued his OPAM-Windows support by fine-tuning the 80 native Windows OCaml versions - these will hopefully form part of OPAM 2.0. As it turns out, he’s not the only person still interested in being able to run OCaml 3.07…if you are, get in touch!

General Libraries and utilities

Olivier Nicole is working on an implementation of macros in OCaml and started working on the HTML and XML templates using this system. The objective is to have the same behaviour as the Pa_tyxml syntax extension, but in a type-safe and more maintainable way without requiring PPX extensions. This project could be contributed to the development of Ocsigen once implemented.

Nick Betteridge teamed up with Dave Scott to look at using ocaml-btree as a backend for Irmin/xen and spent the day looking at different approaches.

Anil Madhavapeddy built a Docker wrapper for the CI system and spun up a big cluster to run OPAM bulk builds. Several small utilities like jsontee and an immutable log collection server and bulk build scripts will be released in the next few weeks once the builds are running stably, and be re-usable by other OPAM-based projects to use for their own tests.

Christophe Troestler is spending a month at OCaml Labs in Cambridge this summer, and spent the hack day working on implementing a library to allow seamless application switching from HTTP to FastCGI. Christophe has initiated work on a client and server for this protocol using CoHTTP so that it is unikernel-friendly.

»


MirageOS Summer 2016 Hackathon announcement, and talk roundup

As summer starts to shine over an obstinately rainy England, we are organising the second MirageOS hackathon in Cambridge! It will be held on Weds 13th July at the lovely Darwin College from 9am-11pm, with snacks, teas, coffees and a servery lunch provided (thanks to sponsorship from Docker and OCaml Labs).

Anyone is welcome at all skill levels, but we’d appreciate you filling out the Doodle so that we can plan refreshments. We will be working on a variety of projects from improving ARM support, to continuous integration tests, the new Solo5 backend and improving the suite of protocol libraries. If you have something in particular that interests you, please drop a note to the mailing list or check out the full list of Pioneer Projects.

Some other events of note recently:

  • After several years of scribing awesome notes about our development, Amir has handed over the reigns to Enguerrand. Enguerrand joined OCaml Labs as an intern, and has built an IRC-to-Git logging bot which records our meetings over IRC and commits them directly to a repository which is available online. Thanks Amir and Enguerrand for all their hard work on recording the growing amount of development in MirageOS. Gemma Gordon has also joined the project and been coordinating the meetings. The next one is in a few hours, so please join us on #mirage on Freenode IRC at 4pm British time if you would like to participate or are just curious!

  • Our participation in the Outreachy program for 2016 has begun, and the irrepressible Gina Marie Maini (aka wiredsister) has been hacking on syslogd, mentored by Mindy Preston. She has already started blogging (about syslog and OCaml love), as well as podcasting with the stars. Welcome to the crew, Gina!

  • The new Docker for Mac and Docker for Windows products have entered open beta! They use a number of libraries from MirageOS (including most of the network stack) and provide a fast way of getting started with containers and unikernel builds on Mac and Windows. You can find talks about it at the recent JS London meetup and my slides I also spoke at OSCON 2016 about it, but those videos aren’t online yet.

There have also been a number of talks in the past couple of months about MirageOS and its libraries:

  • Dan Williams from IBM Research delivered a paper at USENIX HotCloud 2016 about Unikernel Monitors. This explains the basis of his work on Solo5, which we are currently integrating into MirageOS as a KVM-based boot backend to complement the Xen port. You can also find his talk slides online.
  • Amir Chaudhry has given several talks and demos recently: check out his slides and detailed writeups about GlueCon 2016 and CraftConf 2016 in particular, as they come with instructions on how to reproduce his Mirage/ARM on-stage demonstrations of unikernels.
  • Sean Grove is speaking at Polyconf 2016 next week in Poland. If you are in the region, he would love to meet up with you as well – his talk abstract is below > With libraries like Mirage, js_of_ocaml, & ARM compiler output OCaml apps can operate at such a low level > we don’t even need operating systems on the backend anymore (removing 15 million lines of memory-unsafe code) > - while at the same time, writing UI’s is easier & more reliable than ever before, with lightweight type-checked > code sharing between server, browser clients, & native mobile apps. We’ll look at what’s enabled by new tech > like Unikernels, efficient JS/ARM output, & easy host interop.
»


MirageOS Spring 2016 Hackathon!

Hackathon Trip Reports

We’re looking forward to the next MirageOS hackathon already! We’ve collected some reports from those who were present at our 2016 Spring hackathon to share our excitement! Thanks to the folks who put in the time and effort to organize the event and our wonderful hosts, and a huge thanks to everyone who documented their hackathon experience!

More information is also available at the Canopy site developed and used for information sharing during the hackathon!

Trip Report

by David Kaloper

roof-flash

Last month, the MirageOS community saw its first community-organized, international hackathon. It took place between 11th and 16th March 2016. The venue? Rihad Priscilla, Marrakech, Morocco.

The place turned out to be ideal for a community building exercise. A city bursting with life, scents and colors, a relaxed and friendly hostel with plenty of space, warm and sunny weather – all the elements of a good get-together were there. This is where some 25 hackers from all over the world convened, with various backgrounds and specialties, all sharing an interest in MirageOS.

Not wanting to limit ourselves to breaking only those conventions, we added another layer: the hackathon was set up as a classical anti-conference, with a bare minimum of structure, no pre-defined program, and a strong focus on one-on-one work, teaching, and collaboration.

As this was the first hackathon, this time the focus was on building up the nascent community that already exists around MirageOS. Faces were put to online handles, stories were exchanged, and connections were forged. Meeting in person helped bring a new level of cohesion to the online community around the project, as witnessed by the flurry of online conversations between people that were present, and have continued after the event ended.

One particularly useful (however inglorious) activity proved to be introducing people to the tool chain. Even though the MirageOS website has a documentation section with various documents on the architecture of the project, technical blog posts and a series of examples to get newcomers started, a number of people found it difficult to juggle all the concepts and tools involved. Where is the line dividing ocamlfind from opam and what exactly constitutes an OCaml library? What is the correct way to declare dependencies not covered by the declarative configuration language? When should one use add_to_opam_packages, and when add_to_ocamlfind_libraries? Will the mirage tool take care of installing missing packages so declared?

Although these questions either have answers scattered throughout the docs, or are almost obvious to an experienced MirageOS developer, such getting-started issues proved to be an early obstacle for a number of hackathon participants. While our project documentation certainly could – and will! – be improved with the perspective of a new developer in mind, this was an opportunity to help participants get a more comprehensive overview of the core tooling in an efficient, one-to-one setting. As a result, we saw a number of developers go from trying to get the examples to compile to making their own unikernels within a day, something pretty awesome to witness!

Another fun thread was dogfooding the network stack. Network itself was provided by our venue Priscilla, but we brought our own routers and access points. DHCP on site was served by Charrua, which stood up to the task admirably. We were able to access arbitrary domains on the Internet, almost all of the time!

A group of hackers had a strong web background, and decided to focus their efforts there. Perhaps the most interesting project to come out of this is Canopy. Canopy is best described as the first dynamic offering in the space of static web site generators! It combines Irmin with TyXML, COW, and Mirage HTTP, to create a simple, one-stop solution for putting content on the web. A Canopy unikernel boots, pulls a series of markdown files from a git repository, renders them, and serves them via HTTP. Expect more to come in this space, as Canopy has already proved to be a handy tool to simply put something on the web.

At the same time, the atmosphere was conducive for discussing how OCaml in general, and MirageOS in particular, fits in the web development ecosystem. As a language originally honed in different contexts, it’s the opinion of a number of practicing web developers that the current OCaml ecosystem is not as conducive to supporting their day-to-day work as it could be. These brainstorming sessions led to a writeup which tries to summarize the current state and plot the course forward.

Another group of hackers was more focused on security and privacy technology. MirageOS boasts its own cryptographic core and a TLS stack, providing a solid base for development of cryptographic protocols. We saw coordinated work on improving the cryptographic layer; implementations of a few key-derivation functions (Scrypt and PBKDF); and even a beginning of an IKEv2 implementation.

A further common topic was networking, which is not entirely surprising for a network-centric unikernel platform. Amidst the enthusiasm, hackers in attendance started several projects related to general networking. These include a SWIM membership protocol implementation, the beginnings of telnet for Mirage, SOCKS4 packet handling, DNS wildcard matching, Charrua updates, and more.

In between these threads of activity, people used the time to get general MirageOS work done. This resulted in lots of progress including: making AFL, already supported by OCaml, run against MirageOS unikernels; a comprehensive update of error reporting across the stack; a concentrated push to move away from Camlp4 and adopt PPX; and producing a prototype unikernel displaying rich terminal output via telnet.

Partially motivated by the need to improve the experience of building and running unikernels, a number of hackers worked on improvements to error reporting and logging. Improving the experience when things go wrong will be an important part of helping folks make unikernels with MirageOS.

For more, and less structured, details of what went on, check out the blog some of us kept, or the meeting notes from the few short morning meetings we had.

It seems that when surrounded by like-minded, skilled people, in a pleasant atmosphere, and with absolutely nothing to do, people’s curiosity will reliably kick in. In between lieing in the sun (sunscreen was a hot commodity!), sinking into the midday heat, and talking to other hackers, not a single person failed to learn, practice, or produce something new.

In this way, the first MirageOS hackathon was a resounding success. Friendships were forged, skills shared, and courses plotted. And although the same venue has already been booked for the next year’s event, there is ongoing chit-chat about cutting the downtime in half with a summer edition!

heat


MirageOS hackathon in Marrakech

Text and images by Enguerrand Decorne

Setting up and settling in

The first MirageOS hackathon was held from March 11th-16th 2016, at Priscilla, Queen of the Medina, Marrakech. It successfully gathered around 30 Mirage enthusiasts, some already familiar with the MirageOS ecosystem, and others new to the community. People travelled from Europe and further afield for a week of sun, tajine and hacking.

Main room

Getting to the guesthouse was an adventure, and once there we prepared by quickly setting up a nice internet hotspot then organised groups to head to the souk to meet new arrivals. Soon enough the guest house was filled with people, and various new projects and ideas began to emerge. Having a few books and experienced OCaml developers around helped the OCaml newcomers get stuck in, and it didn’t take long to get their first unikernel or OCaml library up and running. Daily meetings were arranged at noon on the rooftop in order to allow the exchange of project ideas and questions, and we used the hackathon notepad to loosely pair projects and people together. Our DHCP server enabled extensive dogfooding and successfully fulfilled our project-testing needs.

Participants found a wide range of activities to keep themselves occupied during the event: contributing to the MirageOS Pioneer Projects, starting new projects and libraries, improving the MirageOS ecosystem and core components, discussing new ideas… or simply enjoying the sun, delicious tajine, or walking around Marrakech itself. Some expeditions were also (non)organised during the week, allowing sightseeing of the nicest local spots, or negotiating with local stallholders to get the best prices on souvenirs and fresh fruits to enjoy during hard hacking sessions.

Food

My week inside the camel’s nest

A few days before heading up to Marrakech (in a very non-organised fashion, having been offered a hackathon place only two days before!) the idea of writing some kind of notebook using Mirage had been floating around - we wanted to be able to allow people inside the hackathon to exchange ideas, and those not physically present to be kept updated about progress. I decided to write a simple blog unikernel, Canopy which relies on Irmin’s capabilities to synchronise remote git repositiories. By describing new pages in a format similar to Jekyll (and using Markdown) on a git repository, new content pushed there would be pulled to the website and displayed there nicely. This allowed every participant to report on their current projects, and see the content displayed on the notepad after a simple git push.

The project was well received and new ideas started to emerge in order to turn it into a CMS enabling users to easily describe new website with a simple git repository. A huge thank you to Michele for his awesome contributions, as well as everyone involved with answering questions about the Mirage ecosystem along the way. This project also allowed me to dive a little further inside various libraries, report a few issues, discuss features and discover new concepts… A week well spent that I would be glad to repeat at the next MirageOS hackathon :)

Conclusion

Rooftop

This hackathon was a huge success and allowed the MirageOS community to combine sun and high productivity in a crazy yet very relaxing week. We hope (and plan) to see more events like this, so anyone interested in OCaml, Mirage - expert or not - is more than welcome to join us next time!

Cats


MirageOS + OCaml Newcomers

by Alfredo and Sonia

Our experience in Marrakesh was great. We really enjoyed the place, the weather, the food, the people and the atmosphere! I think the setting was a great win, there was lot of open space where you could find a quiet spot for yourself to concentrate while programming, as well as a place with lots of people coding, or a place where you could be talking about anything while enjoying the sun, or just hang out and get lost for a while in the nice Marrakesh’s old city.

We had already learnt some OCaml, but we both are quite new to both OCaml and MirageOS, so we decided to work on a project with low entry barrier so we could get in the loop more easily. Nevertheless we had to invest some time getting more familiar with the specifics of the OCaml environment (libraries, packaging, testing frameworks, etc.). Hannes kindly helped us getting started, showing us a library (ocaml-hkdf) we could use to understand this all better, and from here we could start writing some code. Having most of the authors (Thomas, David, Hannes…) of the libraries we used (nocrypto, cstruct, alcotest, opam…) there with us was also a win. Finally we managed to release a pair of libraries with key derivation functions (ocaml-pbkdf and ocaml-scrypt-kdf), so we are quite happy with the outcome.

The only downside of the hackathon we can think of, if any, is that we didn’t get too deep into the MirageOS specifics (something we are surely willing to fix!), but we wanted to stay focused to keep productive and had enough new things to learn.


Hackathon Projects

by Ximin Luo

Here’s a list of things I did during the hackathon:

  • Read into ocaml-tls and ocaml-otr implementations, as well as David’s “nqsb” TLS paper
  • Talked with David about developing a general pattern for implementing protocols, that allows one to compose components more easily and consistently. He pointed me to many resources that I could learn from and build on top of.
  • Read documents on “Extensible Effects”, “Freer Monads” and “Iteratee pattern” by Oleg Kiselyov.
  • Read documents and source code of the Haskell Pipes library by Gabriel Gonzalez.
  • Sent some PRs to Hannes’ jackline IM client, for better usability under some graphical environments.
  • Showed some people my ocaml-hello “minimal build scripts” example, and my ocaml emacs scripts.
  • Tested the “solo5” system that runs mirageos on kvm as an alternative to xen.

I’m continuing with the following work in my spare time:

  • Read documents and source code of the opam monadlib library with a view to extending this and unifying it with other libraries such as lwt.
  • Using the approach of the Haskel Pipes library to develop a general protocol handler framework. I’m experimenting initially in Haskell but I’d also like to do it in OCaml when the ideas are more solid.

In terms of the event it was great - everything worked out very well, I don’t have any suggestions for improvements :)

»


MirageOS security advisory 00: mirage-net-xen

MirageOS Security Advisory 00 - memory disclosure in mirage-net-xen

  • Module: mirage-net-xen
  • Announced: 2016-05-03
  • Credits: Enguerrand Decorne, Thomas Leonard, Hannes Mehnert, Mindy Preston
  • Affects: mirage-net-xen <1.4.2
  • Corrected: 2016-01-08 1.5.0 release, 2016-05-03 1.4.2 release

For general information regarding MirageOS Security Advisories, please visit https://mirage.io/security.

Hannes published a blog article about the analysis of this issue.

Background

MirageOS is a library operating system using cooperative multitasking, which can be executed as a guest of the Xen hypervisor. Virtual devices, such as a network device, share memory between MirageOS and the hypervisor. MirageOS allocates and grants the hypervisor access to a ringbuffer containing pages to be sent on the network device, and another ringbuffer with pages to be filled with received data. A write on the MirageOS side consists of filling the page with the packet data, submitting a write request to the hypervisor, and awaiting a response from the hypervisor. To correlate the request with the response, a 16bit identifier is used.

Problem Description

Generating this 16bit identifier was not done in a unique manner. When multiple pages share an identifier, and are requested to be transmitted via the wire, the first successful response will mark all pages with this identifier free, even those still waiting to be transmitted. Once marked free, the MirageOS application fills the page for another chunk of data. This leads to corrupted packets being sent, and can lead to disclosure of memory intended for another recipient.

Impact

This issue discloses memory intended for another recipient. All versions before mirage-net-xen 1.4.2 are affected. The receiving side uses a similar mechanism, which may lead to corrupted incoming data (eventually even mutated while being processed).

Version 1.5.0, released on 8th January, already assigns unique identifiers for transmission. Received pages are copied into freshly allocated buffers before passed to the next layer. When 1.5.0 was released, the impact was not clear to us. Version 1.6.1 now additionally ensures that received pages have a unique identifier.

Workaround

No workaround is available.

Solution

The unique identifier is now generated in a unique manner using a monotonic counter.

Transmitting corrupt data and disclosing memory is fixed in versions 1.4.2 and above.

The recommended way to upgrade is: bash opam update opam upgrade mirage-net-xen

Or, explicitly: bash opam upgrade opam reinstall mirage-net-xen=1.4.2

Affected releases have been marked uninstallable in the opam repository.

Correction details

The following list contains the correction revision numbers for each affected branch.

Memory disclosure on transmit:

master: 47de2edfad9c56110d98d0312c1a7e0b9dcc8fbf

1.4: ec9b1046b75cba5ae3473b2d3b223c3d1284489d

Corrupt data while receiving:

master: 0b1e53c0875062a50e2d5823b7da0d8e0a64dc37

1.4: 6daad38af2f0b5c58d6c1fb24252c3eed737ede4

References

mirage-net-xen

You can find the latest version of this advisory online at https://mirage.io/blog/MSA00.

This advisory is signed using OpenPGP, you can verify the signature by downloading our public key from a keyserver (gpg --recv-key 4A732D757C0EDA74), downloading the raw markdown source of this advisory from GitHub and executing gpg --verify 00.md.asc.

»