Submission Guidelines

Our library contains a list of algorithms and evolved features for a number of optimization and learning and model fitting problems. You can use this website as a tool to:

• Analyze the strengths and weaknesses of algorithm for a problem across the instance space
• Compare the performance of new algorithms on our library problems and existing algorithms
• Submit and analyze algorithm performances for a new custom problem.
• Download the video tutorials introducing MATILDA and explaining how to create an instance-space from the links below

How to analyse the performance of an algorithm for a library problem

1. From Menu on home page or website header, select Getting Started --> Online Instance Space Analysis.
2. Select "Problem from Our Library".
3. Select a problem from the dropdown list. A list of algorithms available in our library for the given problem will appear. You can also see a list of evolved data-features we used in our publications.
4. Select an algorithm/s to analyze.
5. Select a feature/s to use.
6. Select algorithm performance measurement criteria. For instance, if your performance metric is "number of colors" in graph coloring problem, measurement criteria would be "minimize". On the other hand, this criteria would be "maximize" for a performance metric like "accuracy".
7. Give a meaningful name to your performance metric. For instance, for graph coloring problem in our library, performance metric is "Normalized Number of Colors".
8. You can use subset of dataset for your analysis. For a random subset selection, provide a subset percentage (between 0 and 1). If you want to specify the instances to be used in the analysis, provide a single column csv containing index of instances. Please make sure that the index of instance should match the index in the actual dataset.
9. Provide parameter values for footprint generation (advanced setting) or submit with the default values.

How to analyze the performance of an algorithm for a custom problem

For any problem that is not part of our library, you can submit two or more algorithms and compare their performance across the instance space.

1. From Menu on home page or website header, select Getting Started --> Online Instance Space Analysis.
2. Select "Custom Problem".
3. Upload a csv file containing features' values and algorithms' performances. You need to follow these guidelines to create the csv file.
4. Select the algorithm performance measurement criteria. For instance, if your performance metric is "number of colors" in graph coloring problem, measurement criteria would be "minimize". On the other hand, this criteria would be "maximize" for a metric like "accuracy".
5. Give a meaningful name to your performance metric. For instance, for graph coloring problem in our library, performance metric is "Normalized Number of Colors".
6. You can use subset of dataset for your analysis. For a random subset selection, provide a subset percentage (between 0 and 1). If you want to specify the instances to be used in the analysis, provide a single column csv containing index of instances. Please make sure that the index of instance should match the index in the actual dataset.
7. Provide parameter values for footprint generation (advanced setting) or submit with the default values.

How to Prepare CSV file for New Features and Algorithm Performance

1. The CSV file must contain only 4 types of columns listed below. Column headers should strictly follow the required naming convention.
   1. instances (instance identifier - We expect instance identifier to be of type "String". This column is mandatory)
   2. Source (instance source - This column is optional)
   3. feature_name (The keyword "feature_" concatenated with feature name. For instance, if feature name is "density", header name should be mentioned as "feature_density". If name consists of more than one word, each word should be separated by "_" (spaces are not allowed). You can add one or more features. This column is required either in "Custom Problem" or if you want to add more features in the analysis of a library problem.)
   4. algo_name (The keyword "algo_" concatenated with algorithm name. For instance, if algorithm name is "Greedy", column header should be "algo_greedy". If name consists of more than one word, each word should be separated by "_" (spaces are not allowed). You can add the performance of more than one algorithm in the same csv. This column is required either in "Custom Problem" or if you want to add more algorithms in the analysis of a library problem.)
2. Empty cells, Nan or null values are not accepted. We expect you to handle missing values in your data before submission.
3. Please use this file as a reference.