Evaluating model accuracy is an indispensable lecture in machine learning. To do that, we need a test set comprised of test samples and their ground truths. Whilst standard datasets (e.g., ImageNet) satisfy this requirement, many real world scenarios only provide us with unlabeled test data, rendering common model evaluation methods infeasible. In this talk, I will introduce an important but under-explored problem, automatic model evaluation (AutoEval). Specifically, given a labeled training set and a model, we aim to estimate the model accuracy on unlabeled test datasets. We design an accuracy regression approach based on a meta-dataset: a dataset comprised of datasets generated from the original training set via various image transformations such as rotation, background substitution, etc. Using synthetic meta-dataset and real-world datasets in training and testing, respectively, we obtain reasonable and promising estimates of the model accuracy. 

Join Zoom Meeting