GSoC 2017 - InterPlanetary File System in Purescript

Posted on August 25, 2017 by Christian Fischer

As a sidetrack from my GSoC2017 project, I’ve started working on a Purescript API to the InterPlanetary File System.

Currently it wraps the `id` and `version` functions, as well as most of the `files` part of the JS API. There are also a couple of helper functions to make usage of the files API more Purescripty.

The library can be found on my GitHub: https://github.com/chfi/purescript-ipfs-api

Basic API

I’ll be omitting the JS FFI code; it’s all basic wrappers over the IPFS API as found here https://github.com/ipfs/js-ipfs-api, and the code can be found on the git repo anyway.

Creating an IPFS instance

There is an IPFS type, corresponding to an instantiated connection to IPFS, and an Effect to keep track of in related functions:

foreign import data IPFS :: Type
foreign import data IPFSEff :: Effect

An instance of IPFS is created using `connect`:

connect :: forall eff. String -> Int -> Eff ( ipfs :: IPFSEff | eff ) IPFS

The String is the host, and the Int the port – this is a point of improvement by e.g. newtyping both, or a record of the whole configuration.

I use some type synonyms to make the type signatures of functions nicer:

type IPFSObject = { path :: String
                  , content :: Buffer
                  }

type AddResult = { path :: String
                 , hash :: String
                 , size :: Int
                 }

`Buffer` is a Node.js Buffer, from the purescript-node-buffers package.

To keep track of paths/hashes, I’ve created a corresponding type, however it currently only supports paths:

data IPFSPath = IPFSPathString String

IPFS.files.cat

This function, given a path or hash, retrieves a file from IPFS, in the form of a Readable Node.js stream. The JS function signature is

ipfs.files.get(ipfsPath, [callback])
// ipfsPath is a path or multihash corresponding to an IPFS file
callback = function(err, res)
// where res is a Readable stream of the file contents

If the callback is omitted, an analogous Promise is returned, which is what I use. The Purescript implementation looks like this:

type FileEff eff = ( ipfs :: IPFSEff | eff )

foreign import catImpl :: ∀ r eff.
                          EffFn2 (FileEff eff)
                          IPFS
                          String
                          (Promise (Readable r (FileEff eff)))

cat ::  r eff.
       IPFS
    -> IPFSPath
    -> Aff (FileEff eff) (Readable r (FileEff eff) )
cat ipfs (IPFSPathString path) = liftEff (runEffFn2 catImpl ipfs path) >>= Promise.toAff

Using toAff from purescript-aff-promises, it’s easy to transform a JS promise into an idiomatic Purescript Aff, which can then be used in any Aff program. Here’s how you would read a file:

path = IPFSPathString "/ipfs/QmVLDAhCY3X9P2uRudKAryuQFPM5zqA3Yij1dY8FpGbL7T/quick-start"

-- Effects omitted for readability
showFile ::  r. Readable r _ -> Aff _ Unit
showFile stream = liftEff $ Stream.onData stream $ log <=< Buffer.toString UTF8

main :: Eff _ Unit
main = do
  ipfs <- IPFS.connect "localhost" 5001
  _ <- launchAff do
    showFile =<< Files.cat ipfs path

IPFS.files.get

This function is currently unimplemented, as the purescript-node-streams package I’m using doesn’t support object-mode streams.

IPFS.files.add

This function adds a file to IPFS. The JS function signature is

ipfs.files.add(data, [callback])
// where data is an array of objects, see IPFSObject above
// and
callback = function(err, res)
// where res is an array of objects, see AddResult above

Like IPFS.files.cat, instead of providing a callback, I transform the Promise into Aff:

foreign import addImpl :: ∀ eff.
                          EffFn2 (FilesEff eff)
                          IPFS
                          (Array IPFSObject)
                          (Promise (Array AddResult))

add ::  eff.
  IPFS
  -> Array IPFSObject
  -> Aff (FilesEff eff) (Array AddResult)
add ipfs objs = liftEff (runEffFn2 addImpl ipfs objs) >>= Promise.toAff

Here’s an example of adding a file, and reading it:

_ <- launchAff do
    buffer <- liftEff $ Buffer.fromString "this is a test file" UTF8
    results <- Files.add ipfs [{ path:"/tmp/testfile.txt"
                              , content: buffer }]

    let hashes :: Array IPFSPath
        hashes = map (IPFSPathString <<< _.hash) results

    traverse_ (showFile <=< Files.cat ipfs) hashes

IPFS.files.createAddStream

Like ipfs.files.add, this function lets us add files. In this case, however, it is done by creating a Writable Node.js stream. Writing data of the same format as used by ipfs.files.add to the stream adds a file to IPFS.

Like ipfs.files.get, this uses object mode streams, which are not supported by the purescript-node-streams package I’m using.

Coroutines

I’ve written a helper function for reading files using coroutines from the purescript-coroutines package. Given an IPFS instance, an encoding (e.g. UTF8) and a path, we get a Producer of Strings:

type CatEffs eff = ( ipfs :: IPFSEff
                   , exception :: EXCEPTION
                   , avar :: AVAR | eff )

catProducer ::  eff.
               IPFS
            -> Encoding
            -> IPFSPath
            -> Aff (CatEffs eff)
                   (Producer String (Aff (CatEffs eff)) Unit)
catProducer ipfs enc (IPFSPathString path) = do
  str <- liftEff (runEffFn2 catImpl ipfs path) >>= Promise.toAff
  pure $ produce' \emit -> do
    onDataString str enc $ emit <<< Left
    onClose str $ emit (Right unit)

Using coroutine transformers, we can easily make the producer more interesting:

type Example = { aString :: String
               , aNumber :: Number }

parser ::  m. Monad m
       => Transformer String (Either String Example) m Unit
parser = transform \str -> do
           json <- jsonParser str

           aString <- maybe (Left "No 'aString' field") Right $
                       json ^? _Object <<< ix "test" <<< _String
           aNumber <- maybe (Left "No 'aNumber field") Right $
                       json ^? _Object <<< ix "other" <<< _Number

           pure { aString, aNumber }


exampleCat ::  eff.
              IPFS
           -> IPFSPath
           -> Aff (CatEffs eff)
                  (Producer (Either String Example) (Aff (CatEffs eff)) Unit)
exampleCat ipfs path = (flip transformProducer parser) <$> catProducer ipfs UTF8 path

Now we have a function that creates producers of parsed JSON objects. This can be used to e.g. parse JSON files from IPFS streams.

Future work

This library is still very young – not only are there plenty of functions left to implement, the existing ones can also be improved! For example:

  • ipfs.files.add should be able to handle both Buffers and Readables
  • IPFSPath needs to be finished, with multihash support
  • ipfs.files.add doesn’t use IPFSPath
  • the ipfs.files.cat coroutine producer should be able to handle whatever file encoding
  • ipfs.files.createAddStream currently throws away all results.

I’ll continue to work on this, but if you find yourself wanting to interface with IPFS from Purescript, contributions are welcome!