NLG-evaluation

A toolkit for evaluation of natural language generation (NLG), including BLEU, ROUGE, METEOR, and CIDEr.

Requirements

Make sure the following environment is installed correctly on your machine.

python 2.7+

numpy

If you want to use the METEOR metric, make sure the Java Runtime Environment is configured on your machine.

Usage

Quickstart

You can run evaluation via:

python run_eval.py --hypos output_file --refs reference_file

where output_file is the file that stored the results produced by your system, and reference_file is the file that stored the references. Note that you need to do tokenization before evaluation.

Multiple reference

Evaluation with multiple references is supported. The command is:

python run_eval.py --hypos output_file --refs ref_1 ref_2 ... ref_n

ref_1 ref_2 ... ref_n are n reference files.

Choose metrics

You can choose any metrics you want. By default, all metrics (BLEU, ROUGE, METEOR, and CIDEr) are enabled. If you do not need a metric (e.g., BLEU), the command to disable the metric is:

python run_eval.py --hypos output_file --refs reference_file [-nB | --no_BLEU]

Similarly, you can turn off other metrics:

[-nR | --no_ROUGE]    # for ROUGE
[-nM | --no_METEOR]   # for METEOR
[-nC | --no_CIDEr]    # for CIDEr

Choose ngrams

You can also change the n-gram to obtain the BLEU-n scores:

python run_eval.py --hypos output_file --refs reference_file [-n | --ngram] 3

Then you will get BLEU-n (n <= 3) scores. The default n is 4.

Lowercase

If you want to conduct evaluations in lowercase mode, the command is:

python run_eval.py --hypos output_file --refs reference_file [-lc | --lowercase]