{ "citation": "@article{shankar2019evaluating,\n title={Evaluating Machine Accuracy on ImageNet},\n author={Vaishaal Shankar* and Rebecca Roelofs* and Horia Mania and Alex Fang and Benjamin Recht and Ludwig Schmidt},\n journal={ICML},\n year={2020},\n note={\\url{http://proceedings.mlr.press/v119/shankar20c.html}}\n}\n@article{ImageNetChallenge,\n title={{ImageNet} large scale visual recognition challenge},\n author={Olga Russakovsky and Jia Deng and Hao Su and Jonathan Krause\n and Sanjeev Satheesh and Sean Ma and Zhiheng Huang and Andrej Karpathy and Aditya Khosla and Michael Bernstein and\n Alexander C. Berg and Fei-Fei Li},\n journal={International Journal of Computer Vision},\n year={2015},\n note={\\url{https://arxiv.org/abs/1409.0575}}\n}\n@inproceedings{ImageNet,\n author={Jia Deng and Wei Dong and Richard Socher and Li-Jia Li and Kai Li and Li Fei-Fei},\n booktitle={Conference on Computer Vision and Pattern Recognition (CVPR)},\n title={{ImageNet}: A large-scale hierarchical image database},\n year={2009},\n note={\\url{http://www.image-net.org/papers/imagenet_cvpr09.pdf}}\n}\n@article{vasudevan2022does,\n title={When does dough become a bagel? Analyzing the remaining mistakes on ImageNet},\n author={Vasudevan, Vijay and Caine, Benjamin and Gontijo-Lopes, Raphael and Fridovich-Keil, Sara and Roelofs, Rebecca},\n journal={arXiv preprint arXiv:2205.04596},\n year={2022}\n}", "description": "This dataset contains ILSVRC-2012 (ImageNet) validation images annotated with\nmulti-class labels from\n[\"Evaluating Machine Accuracy on ImageNet\"](http://proceedings.mlr.press/v119/shankar20c/shankar20c.pdf),\nICML, 2020. The multi-class labels were reviewed by a panel of experts\nextensively trained in the intricacies of fine-grained class distinctions in the\nImageNet class hierarchy (see paper for more details).\nCompared to the original labels, these expert-reviewed multi-class labels enable\na more semantically coherent evaluation of accuracy.\n\nVersion 3.0.0 of this dataset contains more corrected labels from\n[\"When does dough become a bagel? Analyzing the remaining mistakes on ImageNet](https://arxiv.org/abs/2205.04596)\nas well as the ImageNet-Major (ImageNet-M) 68-example split under 'imagenet-m'.\n\nOnly 20,000 of the 50,000 ImageNet validation images have multi-label\nannotations. The set of multi-labels was first generated by a testbed of 67\ntrained ImageNet models, and then each individual model prediction was manually\nannotated by the experts as either `correct` (the label is correct for the\nimage),`wrong` (the label is incorrect for the image), or `unclear` (no\nconsensus was reached among the experts).\n\nAdditionally, during annotation, the expert panel identified a set of\n*problematic images*. An image was problematic if it met any of the below\ncriteria:\n\n * The original ImageNet label (top-1 label) was incorrect or unclear\n * Image was a drawing, painting, sketch, cartoon, or computer-rendered\n * Image was excessively edited\n * Image had inappropriate content\n\nThe problematic images are included in this dataset but should be ignored when\ncomputing multi-label accuracy. Additionally, since the initial set of 20,000\nannotations is class-balanced, but the set of problematic images is not, we\nrecommend computing the per-class accuracies and then averaging them. We also\nrecommend counting a prediction as correct if it is marked as correct or unclear\n(i.e., being lenient with the unclear labels).\n\nOne possible way of doing this is with the following NumPy code:\n\n```python\nimport tensorflow_datasets as tfds\n\nds = tfds.load('imagenet2012_multilabel', split='validation')\n\n# We assume that predictions is a dictionary from file_name to a class index between 0 and 999\n\nnum_correct_per_class = {}\nnum_images_per_class = {}\n\nfor example in ds:\n # We ignore all problematic images\n if example[\u2018is_problematic\u2019].numpy():\n continue\n\n # The label of the image in ImageNet\n cur_class = example['original_label'].numpy()\n\n # If we haven't processed this class yet, set the counters to 0\n if cur_class not in num_correct_per_class:\n num_correct_per_class[cur_class] = 0\n assert cur_class not in num_images_per_class\n num_images_per_class[cur_class] = 0\n\n num_images_per_class[cur_class] += 1\n\n # Get the predictions for this image\n cur_pred = predictions[example['file_name'].numpy()]\n\n # We count a prediction as correct if it is marked as correct or unclear\n # (i.e., we are lenient with the unclear labels)\n if cur_pred is in example['correct_multi_labels'].numpy() or cur_pred is in example['unclear_multi_labels'].numpy():\n num_correct_per_class[cur_class] += 1\n\n# Check that we have collected accuracy data for each of the 1,000 classes\nnum_classes = 1000\nassert len(num_correct_per_class) == num_classes\nassert len(num_images_per_class) == num_classes\n\n# Compute the per-class accuracies and then average them\nfinal_avg = 0\nfor cid in range(num_classes):\n assert cid in num_correct_per_class\n assert cid in num_images_per_class\n final_avg += num_correct_per_class[cid] / num_images_per_class[cid]\nfinal_avg /= num_classes\n\n```", "downloadSize": "200417830", "fileFormat": "tfrecord", "location": { "urls": [ "https://github.com/modestyachts/evaluating_machine_accuracy_on_imagenet" ] }, "moduleName": "tensorflow_datasets.image_classification.imagenet2012_multilabel.imagenet2012_multilabel", "name": "imagenet2012_multilabel", "releaseNotes": { "1.0.0": "Initial release.", "2.0.0": "Fixed ILSVRC2012_img_val.tar file.", "3.0.0": "Corrected labels and ImageNet-M split." }, "splits": [ { "filepathTemplate": "{DATASET}-{SPLIT}.{FILEFORMAT}-{SHARD_X_OF_Y}", "name": "validation", "numBytes": "2679448820", "shardLengths": [ "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625", "625" ] }, { "filepathTemplate": "{DATASET}-{SPLIT}.{FILEFORMAT}-{SHARD_X_OF_Y}", "name": "imagenet_m", "numBytes": "8385669", "shardLengths": [ "68" ] } ], "supervisedKeys": { "tuple": { "items": [ { "featureKey": "image" }, { "featureKey": "correct_multi_labels" } ] } }, "version": "3.0.0" }