Date:   Fri, 30 Jun 2017 10:17:17 +0200
From:   Jan Tulak <jtulak@...hat.com>
To:     linux-fsdevel@...r.kernel.org, linux-btrfs@...r.kernel.org,
        linux-ext4@...r.kernel.org, linux-xfs <linux-xfs@...r.kernel.org>
Subject: [proposal] making filesystem tools more machine friendly

AKA filesystem API

Hi guys

Currently, filesystem tools are not made with automation in mind. So
any tool that wants to interact with filesystems (be it for
automation, or to provide a more user-friendly interface) has to
screen scrape everything and cope with changing outputs.

I think it is the time to focus some thoughts on how to make the fs
tools easier to be used by scripts and other tools. Now, to ease you,
the answer to the obvious question "who will do it" is "me". I don't
want to force you into anything, though, so I'm opening this
discussion pretty early with some ideas and I hope to hear from you
what do you think about it before anything is set in the stone. (For
those who visited Vault this year, Justin Mitchell and I had a talk
about this, codename Springfield.)

The following text attempts to identify issues with using
filesystems-related tools in scripts/applications and proposes a
solution to those issues.

Content:
1. A quick introduction
2. Details of the issues
3. Proposed Solutions
4. Conclusion

1. A quick introduction
=================

I discussed this topic with people who are building something around
fs tools. For example, the developer of libblockdev (Vratislav
Podzimek, https://github.com/vpodzime/libblockdev) or system storage
manager (was Lukas Czerner, now it is me,
https://sourceforge.net/projects/storagemanager/), and the listed
issues are a product of experience from working with those tools. The
issues are related mostly to basic operations like mkfs, fsck,
snapshots and resizing. Advanced/debugging tools like xfs_io or xfs_db
are not in the focus, but they might benefit from any possible change
too if included in it.

The main issues of the current state, where all the tools are run with
some flags and options and produce a human readable output:
 * The output format can change, sometimes without an easy way of
detection like a version number.
 * Different output formats for different tools even in a single FS,
thus zero code reuse/sharing.
 * Screenscraping can introduce bugs (a rare inner condition can add a
field into the output and break regular expressions).
 * No (or weak) progress report.
 * Different filesystems have different input formats (flags, options)
even for the most basic capabilities.
 * Thread safety, forking

2. Details of the issues
==================

Let’s look at the issues now: why it is an issue and if we can do some
small change to fix it on its own.


The output format can change, sometimes without an easy way of
detection like a version number. Most filesystems are well behaved,
but still, we don’t know what exactly are people doing with the tools
and even adding a new field can possibly break a script. Keeping a
compatibility with older versions adds another complexity to such
tools.

What can be done about this? The new fields have to be printed somehow
and changing the format of the standard output would break everything.
Making sure that if the input or output changes in any way, it is
always with a detectable difference in the version number is a good
practice, but it doesn’t solve the issue, it only makes hacking around
it easier.

What can really help is to have an alternative output (which can be
turned on when the user wants it), which is easy to parse and which is
resilient to a certain degree of changes because it can express
dynamic items like lists or arrays: JSON, XML...


Different input/output formats for different tools even in a single
FS, thus zero code reuse/sharing: Support for every tool and every
filesystem has to start from a scratch. Nothing can be done about it
without a change in the output format. But if an optional JSON or XML
or something was supported, then instead of creating a parser for
every tool, there could be used just one standard and already a
well-tested library.


Screenscraping can introduce bugs (some rare inner condition can add a
field into the output and break regular expressions): Well, let’s just
look at how many services still can’t even parse and verify an email
address correctly. And we have a lot more complex text… Again, some
easy-to-parse format with existing libraries that would turn the text
into a bunch of variables or an object would help.


No (or weak) progress report: Especially for tools that can run for a
long time, like fsck. Screenscraping everything it throws out and then
deciding whether it is a progress report message (because instead of
“25 %” it says “running operation foo”), a message to tell the user,
or something to just ignore is a lot less comfortable and error prone
than “{level: ‘progress’, stage: 5, msg: ‘operation foo’}”.


Different filesystems have different input formats (flags, options)
even for the most basic capabilities: Similar to “Different
input/output formats for different tools...”. For example, for
labeling a partition, you use mkfs.xfs -L label, but mkfs.vfat -n
label. However, changing this requires getting the same functionality
with a common basic specification to other filesystems too.


Thread safety, forking: The people who work on a library, like
libblockdev, doesn’t like that they have to fork another process over
which they have no control, as they can’t guarantee anything about it.
This can’t be fixed by changing the output format, though, but would
require making a public library providing a similar interface as the
existing fs tools. No detailed access to insides is needed, just a way
how to run mkfs, fsck, snapshots, etc… without spawning another
process and without screenscraping.


3. Proposed Solutions
=================

There are two (complementary) ways how to address the issues: add a
structured, machine-readable format for input/output of the tools
(e.g. JSON, XML, …) and to create a library with the functionality of
the existing tools. Let’s look now at those options. I will focus on
what changes they would require, what would be the price for
maintaining that solution and if there are any drawbacks or additional
advantages.

An optional third option would be to create a system service/daemon,
that would use dbus or some other structured interface to accept jobs
and return results. I think that LVM people are working on something
like this.

The proposed solutions are ordered in regards to their complexity.
Also, they can be seen as follow-ups, because every proposed option
requires big part of the work from the previous one anyway.


3.1. Structured Output
-------------------------------
In other words, what LVM already does with --reportformat
{basic|json}, and lsblk with --json. Possibly, we could also make JSON
input too. That would allow the user to, instead of using all the
flags and options of CLI, make something like
--jsoninput=“{dev:’/dev/abc’, force: true, … }”

Some preliminary notes about the format:
Most likely, this would mean JSON. JSON is currently preferred over
XML because it is easier to read by humans if the need arises, it’s
encoder/parser is lighter and easier to use. Also, other projects like
LVM, multipath or lsblk already uses JSON, so it would be nice to
don’t break the group.

Required implementation changes/expected problems:
In an ideal world, a simple replacement of all prints with a wrapping
function would be enough. However, as far as I know, an overwhelming
majority of the tools has printing functions spread through the code
and prints everything as soon as it knows the specific bit of
information.

Change of the output into a structured format means some refactoring.
Instead of simple printf(), an object, array or structure has to be
created and rather than pure strings, more diagnostically useful
values have to be added into it in place of the current prints. Then,
when it is reasonable, print it out all at once in any desired format.
The “when reasonable” would usually mean at the end, but it could also
print progress if it is a long time running operation.

Because of the kinds of the required changes, the implementation can
be split into two parts: first, clean the code and move all the prints
out from spaghetti to the end. Then, once this is done, add the
structured format.

Maintaining overhead:
Small. By separating the printing from generating the data, we don’t
really care about the output format anywhere except in the printing
function itself, and if a new field or value is added or an old one
removed, then the amount of work is roughly equal to the current
state.

Drawbacks:
Searching for a place in the code where something happens would be
more complicated - instead of simple search for a string that is the
same in the code and in the output, one would search for the line that
adds the data to the message log. This could be simplified by using
__LINE__ macro with debug output. (With JSON, an additional field
would not affect anything, so it would be completely safe.)

Additional advantages:
The refactoring can clean up the code a bit. It is easy to add any
other format in the future. Our own test suite could also benefit from
the structured output.

Comment:
The most bang for the buck option and most of the work done for this
is required also for every other option. In terms of specific
interface (i.e. common JSON elements), we need to identify a common
subset of fields/options that every fs will use and anything else move
into fs-specific extensions.

With regards to the compatibility between different filesystems, the
best way how to specify the format might be a small library that would
take raw data on one side and turn it into JSON string or even print
it (and the reverse, if input supports json too). This way, we would
be sure that there really is a common ground that works the same in
every fs.

Another way how to achieve the compatibility is to make an RFC-like
document. For example: All occurrences of a filesystem identifier MUST
be in a field named 'fsid' which SHOULD contain a UUID formatted
string. I think this is useful even if we end up with the library as a
way to find out the common ground.

3.2. A Library
-------------------
A library implementing basic functions like: mkfs(int argc, char
*argv[]), fsck(), … etc. Once done, binding for other languages like
Python is possible too.

Required implementation changes/expected problems:
If the implementation of this library would follow up after the
changes that add the structured output, then most of the work would be
already done. The most complex issue remaining would probably be that
there can be no exit() call - if anything fails, we have to gracefully
return up the stack and out of the library.

A duplicity of functionality is not an issue because there is none -
the binary tools like mkfs.xfs would become simple wrappers around the
library functions: pass user input to the specified library call, then
print any message created in the library and exit.

Maintaining overhead:
None? The code would be cleaned up and moved, but there wouldn’t be
new things to maintain.

Drawbacks:
I can’t think out any...

Additional advantages:
The refactoring can clean up the code a bit.

Comment:
Useful and nice to have, but doesn’t have to be done ASAP.


3.3. A system service
------------------------------
A system service/daemon, that would use dbus or some other structured
interface to accept jobs and return results.

Required implementation changes/expected problems:
We don’t want the daemon to exit if it can’t access a file, we don’t
want it to do printfs(), … So, at the end, we have to do the
structured output and library and then a lot of work above it.

Maintaining overhead:
All the system services things plus whatever the other two solutions require.

Drawbacks:
The biggest maintaining overhead of all proposed solutions, basically
a new tool/subproject. Using dbus in a third party project is more
work than to just include a library and call one function.

Additional advantages:
It wouldn’t be possible to attempt concurrent modifications of a device.

Comment:
In my opinion, this shouldn’t be our project. Let other people make it
their front if they want something like this and instead, just make it
easier for them by using one of the other solutions.


4. Conclusion
===========

A structured output is something we should aim for. It helps a lot
with the issues and it is the cheapest option. If it goes well, it can
be later on followed by creating a library, although, at this moment,
that seems a premature thing strive for. Creating a daemon is not a
thing any single filesystem should do on its own.

And thank you for reading all this, I look forward to your comments. :-)
Jan

-- 
Jan Tulak
jtulak@...hat.com / jan@...ak.me

Powered by blists - more mailing lists