mafが便利そう

概要

mafというツールが便利そうだったのでメモ。評価のために必要なめんどくさい処理が簡略化されそうな気がする。実験結果の管理などがヘタなので、mafを使ってちょっとでもうまくなりたい。まだ調べ始めたばかりなので、以降で出てくるコードよりももっとうまい書き方があると思う。

今回は色々とパラメータを変えて学習した分類器を評価する例で進める。

使ってみた

まず、wafとmafとダウンロードする。

$cd /path/to/project/
$wget https://github.com/pfi/maf/raw/master/waf
$wget https://github.com/pfi/maf/raw/master/maf.py
$chmod +x waf

以下の様な wscript を作成。

#!/usr/bin/python
import re
import json
import numpy as np
import maf
import maflib.util

def configure(conf):
    pass

@maflib.util.rule
def jsonize(task):
    """ Calculate accuracy from a format as below:

        Recall[-1]: 0.932965 (21934/23510)
        Prec[-1]: 0.849562 (21934/25818)
        --
        Recall[+1]: 0.478378 (3562/7446)
        Prec[+1]: 0.693266 (3562/5138)
    """
    out = task.parameter
    with open(task.inputs[0].abspath(), 'r') as f:
        num = 0
        num_trues = 0
        for line in f:
            if line.startswith("Prec"):
                sp = line.split()
                nums = re.search("(\d+)/(\d+)", sp[2]).groups()
                num_trues += int(nums[0])
                num += int(nums[1])
        out["accuracy"] = num_trues / float(num)
    with open(task.outputs[0].abspath(), 'w') as f:
        json.dump(out, f)

@maflib.util.rule
def aggregate_by_alg(task):
    out = []
    for i in task.inputs:
        with open(i.abspath(), 'r') as f:
            out.append(json.load(f))
    with open(task.outputs[0].abspath(), 'w') as f:
        json.dump(out, f)

def aggregate_by_param():
    @maflib.util.aggregator
    def body(values, outpath, parameter):
        out = []
        for value in values:
            out.append(value)
        return json.dumps(out)

def build(exp):
    traindata='a1a'
    train = '~/go/src/github.com/tma15/gonline/gonline/gonline train'
    test = '~/go/src/github.com/tma15/gonline/gonline/gonline test'

    NUM_FOLD = 3
    exp(source=traindata,
        target="train dev",
        parameters=[{"fold": i} for i in xrange(NUM_FOLD)],
        rule=maflib.rules.segment_by_line(NUM_FOLD, 'fold'))

    exp(source="train",
        target="model",
        parameters=maflib.util.product({
            "a": ["perceptron", "pa2", "adagrad"],
            "c": np.power(10., np.arange(-12, -5, dtype=np.float64)),
            }),
        rule="%s -a ${a} -m ${TGT[0].abspath()} ${SRC[0].abspath()}" % train)

    exp(source="model dev",
        target="dev_result",
        rule="%s -m ${SRC[0].abspath()} ${SRC[1].abspath()} > ${TGT[0].abspath()}" % test)

    exp(source="dev_result",
            target="accuracy",
            rule=jsonize) ### パラメータごとのaccuracyをjson形式で出力

    exp(source="accuracy",
        target="accuracies_by_param",
        for_each=["a", "c"],
        rule=aggregate_by_param()) ### パラメータ毎にaccuracyを集約する

    exp(source="accuracies_by_param",
        target="avg_acc",
        aggregate_by=["fold"],
        rule=maflib.rules.average) ### パラメータ毎の平均を計算

    exp(source="avg_acc",
            target="for_each_alg", 
            for_each=["a"],
            rule=aggregate_by_alg) ## アルゴリズム毎に集約

    exp(source="for_each_alg",
            target="max_acc",
            aggregate_by = ["fold"],
            rule=maflib.rules.max("accuracy"))

次のように実行。

$ ./waf configure
Setting top to                           : /Users/makino/code/mafexp
Setting out to                           : /Users/makino/code/mafexp/build
'configure' finished successfully (0.004s)
$./waf
Waf: Entering directory `/Users/makino/code/mafexp/build'
[  2/240] 0-train,0-dev: a1a -> build/train/0-train build/dev/0-dev
[  3/240] 1-train,1-dev: a1a -> build/train/1-train build/dev/1-dev
[  3/240] 2-train,2-dev: a1a -> build/train/2-train build/dev/2-dev
[  7/240] 19-model: build/train/1-train -> build/model/19-model
[  7/240] 18-model: build/train/1-train -> build/model/18-model
[  7/240] 22-model: build/train/1-train -> build/model/22-model
...
[240/240] 87-max_acc: build/for_each_alg/87-for_each_alg -> build/max_acc/87-max_acc
Waf: Leaving directory `/Users/makino/code/mafexp/build'
'build' finished successfully (3.450s)

他に

出力のidは、 ./build/.maf_id_table.tsvで対応付けられている。

$cat ./build/.maf_id_table.tsv
0       {'fold': 0}
1       {'fold': 1}
2       {'fold': 2}
3       {'a': 'perceptron', 'fold': 1, 'c': 9.9999999999999998e-13}
4       {'a': 'perceptron', 'fold': 1, 'c': 9.9999999999999994e-12}
...