{"corpus_id":189998981,"paper_sha":"bcfba69c2fadf2efea83be12fda2601f8d4681af","doi":null,"arxiv_id":"1906.07413","pmid":null,"pmcid":null,"mag_id":2952120674,"dblp_id":"conf/nips/CaoWGAM19","acl_id":null,"title":"Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss","year":2019,"publication_date":"2019-06-18","venue":"Neural Information Processing Systems","journal":{"name":null,"pages":"1565-1576","volume":null},"journal_issn":null,"journal_title":null,"publication_types":["JournalArticle"],"pubmed_pub_types":null,"s2_fields_of_study":["Mathematics","Computer Science"],"reference_count":63,"citation_count":1964,"influential_citation_count":401,"is_open_access":false,"arxiv_categories":["cs.LG","cs.CV","stat.ML"],"arxiv_license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","arxiv_journal_ref":null,"mesh_headings":null,"chemicals":null,"comments_corrections":null,"source_flags":1,"s2_open_access_pdf_url":null,"s2_open_access_landing_url":null,"s2_open_access_license":null,"s2_open_access_status":null,"pmc_open_access_pdf_url":null,"pmc_open_access_landing_url":null,"pmc_open_access_license":null,"pmc_open_access_status":null,"unpaywall_open_access_pdf_url":null,"unpaywall_open_access_landing_url":null,"unpaywall_open_access_license":null,"unpaywall_open_access_status":null,"abstract":"Deep learning algorithms can fare poorly when the training dataset suffers from heavy class-imbalance but the testing criterion requires good generalization on less frequent classes. We design two novel methods to improve performance in such scenarios. First, we propose a theoretically-principled label-distribution-aware margin (LDAM) loss motivated by minimizing a margin-based generalization bound. This loss replaces the standard cross-entropy objective during training and can be applied with prior strategies for training with class-imbalance such as re-weighting or re-sampling. Second, we propose a simple, yet effective, training schedule that defers re-weighting until after the initial stage, allowing the model to learn an initial representation while avoiding some of the complications associated with re-weighting or re-sampling. We test our methods on several benchmark vision tasks including the real-world imbalanced dataset iNaturalist 2018. Our experiments show that either of these methods alone can already improve over existing techniques and their combination achieves even better performance gains.","claims":[{"public_id":"cl_24d906a43ff0fe0a9d33261933759dd5","status":"active","text":"A label-distribution-aware margin loss is proposed to improve performance on heavily class-imbalanced training data by targeting less frequent classes.","confidence":0.98,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_24d906a43ff0fe0a9d33261933759dd5"},{"public_id":"cl_6d3fdb523e36d0cd4ce168f6e9643b77","status":"active","text":"Deferring re-weighting until after an initial training stage provides a simple schedule that helps the model learn an initial representation while avoiding some complications of re-weighting or re-sampling.","confidence":0.96,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_6d3fdb523e36d0cd4ce168f6e9643b77"},{"public_id":"cl_45c30ff81851b6472a077d6602a4e9d8","status":"active","text":"Either proposed method alone improves over existing techniques on benchmark vision tasks, and their combination yields even better performance gains on iNaturalist 2018 and other evaluated datasets.","confidence":0.97,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_45c30ff81851b6472a077d6602a4e9d8"},{"public_id":"cl_2e811501a3b532826cfa8d12206e87e3","status":"active","text":"The label-distribution-aware margin loss is motivated by minimizing a margin-based generalization bound and replaces the standard cross-entropy objective during training.","confidence":0.95,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_2e811501a3b532826cfa8d12206e87e3"},{"public_id":"cl_a84d1007dcb23f1282ae8ddbc1bf8bdd","status":"active","text":"The proposed loss can be combined with re-weighting or re-sampling strategies for class-imbalance training.","confidence":0.91,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_a84d1007dcb23f1282ae8ddbc1bf8bdd"}],"concepts":[{"public_id":"co_351defc3aa7cfd5fa659cfd51379e659","status":"active","name":"cross-entropy objective","description":"The standard classification objective used to train many deep learning models.","types":["loss function","objective"],"aliases":["cross-entropy loss"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_351defc3aa7cfd5fa659cfd51379e659"},{"public_id":"co_4209cb9b202e3cf6310712bea1ce8cea","status":"active","name":"re-sampling","description":"A class-imbalance strategy that changes the sampling frequency of training examples.","types":["training strategy"],"aliases":["resampling"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_4209cb9b202e3cf6310712bea1ce8cea"},{"public_id":"co_58a4c004960a3a7a67c9b8b221e4c29e","status":"active","name":"initial representation","description":"The early feature representation learned by a model before later training adjustments.","types":["representation"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_58a4c004960a3a7a67c9b8b221e4c29e"},{"public_id":"co_880accc5779046dd17d725ecdf3f53cd","status":"active","name":"margin-based generalization bound","description":"A theoretical bound relating classification margins to generalization performance.","types":["theoretical bound"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_880accc5779046dd17d725ecdf3f53cd"},{"public_id":"co_8a81de47e4661b4aa7e847f21c30b4bf","status":"active","name":"class-imbalance","description":"A dataset condition in which some classes appear much less often than others.","types":["dataset property","problem setting"],"aliases":["imbalanced datasets"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_8a81de47e4661b4aa7e847f21c30b4bf"},{"public_id":"co_968483ee338ad0363e72430d66495699","status":"active","name":"benchmark vision tasks","description":"Vision classification tasks used as evaluation benchmarks in the experiments.","types":["evaluation task"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_968483ee338ad0363e72430d66495699"},{"public_id":"co_9a7f999354458aa8c9e60614604575b0","status":"active","name":"less frequent classes","description":"Classes that have fewer training examples than the dominant classes in an imbalanced dataset.","types":["class category"],"aliases":["rare classes","minority classes"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_9a7f999354458aa8c9e60614604575b0"},{"public_id":"co_9e4637149014b0e53c431ecadb92a503","status":"active","name":"existing techniques","description":"Previously proposed methods for learning with class-imbalanced data used as comparison baselines.","types":["baseline"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_9e4637149014b0e53c431ecadb92a503"},{"public_id":"co_bddfff7dd1208ad55f8e2d14308a94c5","status":"active","name":"training schedule","description":"An ordered plan for when different training procedures are applied.","types":["training procedure"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_bddfff7dd1208ad55f8e2d14308a94c5"},{"public_id":"co_cbe08819cb0dbcb238b9ecd24d20cf33","status":"active","name":"iNaturalist 2018","description":"A real-world imbalanced vision dataset used for experimental evaluation.","types":["dataset"],"aliases":["iNat 2018"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_cbe08819cb0dbcb238b9ecd24d20cf33"},{"public_id":"co_f70a1daf1102844165706d804d5051de","status":"active","name":"re-weighting","description":"A class-imbalance strategy that changes example or class weights during training.","types":["training strategy"],"aliases":["reweighting"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_f70a1daf1102844165706d804d5051de"},{"public_id":"co_f99b2ffe6f0e7b5526946a535b57ba05","status":"active","name":"label-distribution-aware margin loss","description":"A training loss that adjusts classification margins according to the label distribution.","types":["loss function","method"],"aliases":["LDAM loss","LDAM"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_f99b2ffe6f0e7b5526946a535b57ba05"}],"external_ids":{"DOI":null,"ArXiv":"1906.07413","PubMed":null,"PubMedCentral":null,"MAG":2952120674,"DBLP":"conf/nips/CaoWGAM19","ACL":null},"open_access":{"is_open_access":true,"pdf_url":"https://arxiv.org/pdf/1906.07413","landing_url":"https://arxiv.org/abs/1906.07413","source":"arxiv","pdf_url_source":"derived_arxiv","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","reason":null},"reference_availability":{"status":"available","references_indexed":true,"full_text_available":true,"full_text_source":"arxiv","count_basis":"semantic_scholar_metadata","extraction_status":"not_applicable","reason":null},"source":{"provider":"episteme2","base_corpus":"semantic_scholar_dump","freshness_mode":"unknown","basis":["semantic_scholar_metadata","postgres_metadata"],"limits":["paper metadata is based on indexed upstream scholarly datasets","claims and concepts are available only for extracted papers","absence of claims or concepts means no extracted graph data is available in this response"],"status":"available","degraded":false,"degraded_reasons":[],"diagnostics":{"status":"available","degraded":false,"degraded_reasons":[],"metadata_status":"available","graph_status":"available","abstract_status":"available"},"source_flags":1},"paper_id":634505,"paper_uid":"110978ac-1f05-4106-92de-2ab5b90c3923","canonical_identity":{"paper_id":634505,"paper_uid":"110978ac-1f05-4106-92de-2ab5b90c3923","identity_status":"available","lookup_basis":"semantic_scholar_external_id","compatibility_path":"corpus_id"},"url":"https://sah.borca.ai/papers/189998981"}