[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"doc-detail-151410-en":3,"doc-seo-151410-105":30,"detail-sidebar-cat-0-en-105":91},{"code":4,"msg":5,"data":6},0,"success",{"doc_id":7,"user_id":8,"nickname":9,"user_avatar":10,"doc_module":4,"category_id":11,"category_name":12,"doc_title":13,"doc_description":14,"doc_content":15,"file_id":16,"file_url":17,"file_type":18,"file_size":19,"view_count":4,"is_deleted":4,"is_public":20,"is_downloadable":20,"audit_status":20,"page_count":21,"language":22,"language_code":23,"site_id":24,"html_lang":23,"table_of_contents":25,"faqs":26,"seo_title":27,"seo_description":14,"update_tm":28,"read_time":29},151410,8796095027276,"wps_ap_test_251126_0180","https://avatar.qwps.com/avatar/d3BzX2FwX3Rlc3RfMjUxMTI2XzAxODA=",8,"Research & Report","NLP-Power 2022 - The First Workshop on Efficient Benchmarking in NLP - Proceedings","NLP-Power 2022 presents the workshop proceedings for the First Workshop on Efficient Benchmarking in NLP. The program addresses unresolved methodological concerns in benchmarking and connects benchmark research to computational efficiency, ethical considerations, user preferences, and out-of-domain robustness. The proceedings compile contributions covering efficient model evaluation, transfer learning efficiency estimation, evaluation metrics, robustness and bias assessment, and practical best practices for benchmarking in natural language processing.","NLP-Power 2022  \nThe First Workshop on Efficient Benchmarking in NLP  \nProceedings of the Workshop  \nMay 26, 2022  \nThe NLP-Power organizers gratefully acknowledge the support from the following sponsors.  \nIn cooperation with  \n©2022 Association for Computational Linguistics  \nOrder copies of this and other ACL proceedings from:  \nAssociation for Computational Linguistics (ACL) 209 N. Eighth Street  \nStroudsburg, PA 18360  \nUSA  \nTel: +1-570-476-8006  \nFax: +1-570-476-0860  \n[acl@aclweb.org](acl@aclweb.org)  \nISBN 978-1-955917-47-6  \niii  \nIntroduction  \nNLP Power! is the workshop on efficient benchmarking in NLP.  \nBenchmarking has become a standard practice for evaluating upcoming models against one another and human solvers; there are still many unresolved issues and methodological concerns. The main idea of the workshop is to bring together researchers that work on benchmarks for natural language processing (NLP) and discuss how benchmarking can be improved to account for computational efficiency, ethical considerations, user preferences, and out-of-domain robustness. The workshop proceedings present the collection of research contributions on the computational efficiency of model evaluation, transfer learning efficiency estimation, evaluation metrics, robustness and bias assessment, and general best practices in benchmarking for NLP.  \nThis is the first time we have organized a workshop with this particular scope of interest. Our workshop is hosted by the 60th Annual Meeting of the Association for Computational Linguistics (ACL 2022) . Our program committee consisted of experts from all over the world with years of research experience in the industry and academia. The committee worked hard on every submission and selected 12 research papers to be presented at the workshop in the poster and oral sessions. The workshop program also included one ACL Findings paper. Overall, it resulted in 2 oral presentation sessions, which were intermitted by a poster session, three invited talks, and a round table on the problems of canonic benchmark standards.  \nNLP Power would not be possible without the dedicated intellectual work of the program committee: their peer review and efforts aimed to improve the work have shaped the scientific community, which is now, for the first time, coming forward with a unified workshop mission. We also express our sincere gratitude to the invited speakers: Anna Rumshiski, He He, and Ulises Mejias, for their contribution to the program. We thank the researchers and NLP practitioners for the engagement and responses and hope to continue to provide a platform for fruitful discussions on various topics, ranging from rethinking benchmarking methods to the reproducibility of the leaderboard results.  \nYou can find more details about the workshop on the website: [http://nlp-power.github.io/](http://nlp-power.github.io/) .  \nTatiana Shavrina, Valentin Malykh, Ekaterina Artemova, Vladislav Mikhailov, Laura Weidinger, Oleg Serikov, and Vitaly Protasov  \nOrganizing Committee  \nProgram Chairs  \nTatiana Shavrina, AIRI, SberDevices  \nValentin Malykh, Huawei  \nEkaterina Artemova, HSE University, Huawei  \nVladislav Mikhailov, SberDevices, HSE University  \nOleg Serikov, AIRI, HSE University  \nVitaly Protasov, AIRI  \nProgram Committee  \nSenior Program Committee  \nJ¨urgen Schmidhuber, Swiss AI Lab IDSIA, USI, SUPSI  \nProgram Committee  \nLaura Weidinger, DeepMind  \nLeonid Zhukov, AIRI  \nMikhail Burtsev, AIRI  \nNitish Hemant Joshi, CILVR / ML2  \nRichard Yuanzhe Pang, CILVR / ML2  \nAdaku Uchendu, Penn State University  \nIlya Kuznetsov, TU Darmstadt  \nAnastasia Bonch-Osmolovskaya, HSE University  \nAndrey Kravchenko, Oxford University  \nDaniel Karabekyan, HSE University  \nPreslav Nakov, QCRI  \nSuresh Manandhar, Wiseyak, USA  \nPiotr Pie¸kos, DeepMind  \nOlga Lyashevskaya, Vinogradov IRL RAS, HSE University Arjun Akula, Google Research  \nSecondary Reviewers  \nTatiana Shavrina, AIRI, SberDevices  \nMaria Tikhonova, HSE University, SberDevi","cbCaisaJehyhe5rt","https://ap.wps.com/l/cbCaisaJehyhe5rt","pdf",4123255,1,129,"English","en",105,"# Introduction\n# Table of Contents\n## Raison d' être of the benchmark dataset: A Survey of Current Practices of Benchmark Dataset Sharing Platforms\n## Towards Stronger Adversarial Baselines Through Human-AI Collaboration\n## Benchmarking for Public Health Surveillance tasks on Social Media with a Domain-Specific Pretrained Language Model\n## Why only Micro-F1? Class Weighting of Measures for Relation Classification\n## Automatically Discarding Straplines to Improve Data Quality for Abstractive News Summarization\n## A global analysis of metrics used for measuring performance in natural language processing\n## Beyond Static models and test sets: Benchmarking the potential of pre-trained models across tasks and languages\n## Checking HateCheck: a cross-functional analysis of behaviour-aware learning for hate speech detection\n## Language Invariant Properties in Natural Language Processing\n## DACT-BERT: Differentiable Adaptive Computation Time for an Efficient BERT Inference","[{\"question\":\"What is the main purpose of the NLP-Power 2022 workshop?\",\"answer\":\"To bring together researchers working on NLP benchmarks and discuss how benchmarking can be improved, including computational efficiency, ethical considerations, user preferences, and out-of-domain robustness.\"},{\"question\":\"What kinds of research topics are included in the workshop proceedings?\",\"answer\":\"Contributions cover efficient benchmarking for model evaluation, transfer learning efficiency estimation, evaluation metrics, robustness and bias assessment, and general best practices.\"},{\"question\":\"How many research papers were selected for presentation in the workshop sessions?\",\"answer\":\"The workshop selected 12 research papers for poster and oral sessions, along with one ACL Findings paper and multiple invited activities.\"}]","NLP-Power 2022 - The First Workshop on Efficient Benchmarking in NLP - Proceedings | PDF",1787839458,325,{"code":4,"msg":31,"data":32},"ok",{"site_id":24,"language":23,"slug":33,"title":13,"keywords":34,"description":14,"schema_data":35,"social_meta":86,"head_meta":88,"extra_data":90,"updated_unix":28},"nlp-power-2022-the-first-workshop-on-efficient-benchmarking-in-nlp-proceedings","",{"@graph":36,"@context":85},[37,54,68],{"@type":38,"itemListElement":39},"BreadcrumbList",[40,44,48,51],{"item":41,"name":42,"@type":43,"position":20},"https://docshare.wps.com","Home","ListItem",{"item":45,"name":46,"@type":43,"position":47},"https://docshare.wps.com/document/","Document",2,{"item":49,"name":12,"@type":43,"position":50},"https://docshare.wps.com/document/research-report/",3,{"item":52,"name":13,"@type":43,"position":53},"https://docshare.wps.com/document/nlp-power-2022-the-first-workshop-on-efficient-benchmarking-in-nlp-proceedings/151410/",4,{"url":52,"name":13,"@type":55,"author":56,"headline":13,"publisher":58,"fileFormat":61,"inLanguage":23,"description":14,"dateModified":62,"datePublished":62,"encodingFormat":61,"isAccessibleForFree":63,"interactionStatistic":64},"DigitalDocument",{"name":9,"@type":57},"Person",{"url":41,"name":59,"@type":60},"DocShare","Organization","application/pdf","2026-08-27",true,{"@type":65,"interactionType":66,"userInteractionCount":4},"InteractionCounter",{"@type":67},"ViewAction",{"@type":69,"mainEntity":70},"FAQPage",[71,77,81],{"name":72,"@type":73,"acceptedAnswer":74},"What is the main purpose of the NLP-Power 2022 workshop?","Question",{"text":75,"@type":76},"To bring together researchers working on NLP benchmarks and discuss how benchmarking can be improved, including computational efficiency, ethical considerations, user preferences, and out-of-domain robustness.","Answer",{"name":78,"@type":73,"acceptedAnswer":79},"What kinds of research topics are included in the workshop proceedings?",{"text":80,"@type":76},"Contributions cover efficient benchmarking for model evaluation, transfer learning efficiency estimation, evaluation metrics, robustness and bias assessment, and general best practices.",{"name":82,"@type":73,"acceptedAnswer":83},"How many research papers were selected for presentation in the workshop sessions?",{"text":84,"@type":76},"The workshop selected 12 research papers for poster and oral sessions, along with one ACL Findings paper and multiple invited activities.","https://schema.org",{"og:url":52,"og:type":87,"og:title":13,"og:site_name":59,"og:description":14},"article",{"robots":89,"canonical":52},"index,follow",{"doc_id":7,"site_id":24},{"code":4,"msg":5,"data":92},[93,97,101,105,110,115,120,123,128,131,135],{"id":20,"doc_module":4,"doc_module_name":46,"category_name":94,"show_sort_weight":95,"slug":96},"Story & Novel",90,"story-novel",{"id":47,"doc_module":4,"doc_module_name":46,"category_name":98,"show_sort_weight":99,"slug":100},"Literature",80,"literature",{"id":53,"doc_module":4,"doc_module_name":46,"category_name":102,"show_sort_weight":103,"slug":104},"Exam",70,"exam",{"id":106,"doc_module":4,"doc_module_name":46,"category_name":107,"show_sort_weight":108,"slug":109},5,"Comic",60,"comic",{"id":111,"doc_module":4,"doc_module_name":46,"category_name":112,"show_sort_weight":113,"slug":114},6,"Technology",50,"technology",{"id":116,"doc_module":4,"doc_module_name":46,"category_name":117,"show_sort_weight":118,"slug":119},7,"Healthcare",40,"healthcare",{"id":11,"doc_module":4,"doc_module_name":46,"category_name":12,"show_sort_weight":121,"slug":122},30,"research-report",{"id":124,"doc_module":4,"doc_module_name":46,"category_name":125,"show_sort_weight":126,"slug":127},9,"Religion & Spirituality",20,"religion-spirituality",{"id":126,"doc_module":4,"doc_module_name":46,"category_name":129,"show_sort_weight":126,"slug":130},"World Cup","world-cup",{"id":132,"doc_module":4,"doc_module_name":46,"category_name":133,"show_sort_weight":132,"slug":134},10,"Lifestyle","lifestyle",{"id":136,"doc_module":4,"doc_module_name":46,"category_name":137,"show_sort_weight":106,"slug":138},19,"General","general"]