{"corpus_id":265609417,"paper_sha":"41b7ec333517005b2a63c3f8df362455fa6ef033","doi":"10.48550/arXiv.2312.01939","arxiv_id":"2312.01939","pmid":null,"pmcid":null,"mag_id":null,"dblp_id":"journals/corr/abs-2312-01939","acl_id":null,"title":"Foundations for Transfer in Reinforcement Learning: A Taxonomy of Knowledge Modalities","year":2023,"publication_date":"2023-12-04","venue":"arXiv.org","journal":{"name":"ArXiv","pages":null,"volume":"abs/2312.01939"},"journal_issn":null,"journal_title":null,"publication_types":["JournalArticle"],"pubmed_pub_types":null,"s2_fields_of_study":["Mathematics","Computer Science"],"reference_count":0,"citation_count":9,"influential_citation_count":1,"is_open_access":false,"arxiv_categories":["cs.LG","cs.AI","cs.RO","stat.ML"],"arxiv_license":"http://creativecommons.org/licenses/by-nc-sa/4.0/","arxiv_journal_ref":null,"mesh_headings":null,"chemicals":null,"comments_corrections":null,"source_flags":1,"s2_open_access_pdf_url":null,"s2_open_access_landing_url":null,"s2_open_access_license":null,"s2_open_access_status":null,"pmc_open_access_pdf_url":null,"pmc_open_access_landing_url":null,"pmc_open_access_license":null,"pmc_open_access_status":null,"unpaywall_open_access_pdf_url":null,"unpaywall_open_access_landing_url":null,"unpaywall_open_access_license":null,"unpaywall_open_access_status":null,"abstract":"Contemporary artificial intelligence systems exhibit rapidly growing abilities accompanied by the growth of required resources, expansive datasets and corresponding investments into computing infrastructure. Although earlier successes predominantly focus on constrained settings, recent strides in fundamental research and applications aspire to create increasingly general systems. This evolving landscape presents a dual panorama of opportunities and challenges in refining the generalisation and transfer of knowledge - the extraction from existing sources and adaptation as a comprehensive foundation for tackling new problems. Within the domain of reinforcement learning (RL), the representation of knowledge manifests through various modalities, including dynamics and reward models, value functions, policies, and the original data. This taxonomy systematically targets these modalities and frames its discussion based on their inherent properties and alignment with different objectives and mechanisms for transfer. Where possible, we aim to provide coarse guidance delineating approaches which address requirements such as limiting environment interactions, maximising computational efficiency, and enhancing generalisation across varying axes of change. Finally, we analyse reasons contributing to the prevalence or scarcity of specific forms of transfer, the inherent potential behind pushing these frontiers, and underscore the significance of transitioning from designed to learned transfer.","claims":[{"public_id":"cl_e7aeb9f0b2a9301f14928333a697d1c1","status":"active","text":"A taxonomy of knowledge modalities for transfer in reinforcement learning is provided.","confidence":0.98,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_e7aeb9f0b2a9301f14928333a697d1c1"},{"public_id":"cl_35d7aa7e0af5f58bb9cfaacd6e90b0e6","status":"active","text":"Coarse guidance is given for approaches that limit environment interactions, maximize computational efficiency, and improve generalization across varying axes of change.","confidence":0.9,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_35d7aa7e0af5f58bb9cfaacd6e90b0e6"},{"public_id":"cl_0ec7158c77210d864ad898da98877bf3","status":"active","text":"The analysis highlights reasons for the prevalence or scarcity of specific forms of transfer and emphasizes transitioning from designed transfer to learned transfer.","confidence":0.88,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_0ec7158c77210d864ad898da98877bf3"},{"public_id":"cl_c23549ff0a37098db3e2615db523bf94","status":"active","text":"The discussion is framed by the properties of these modalities and their alignment with objectives and mechanisms for transfer.","confidence":0.92,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_c23549ff0a37098db3e2615db523bf94"},{"public_id":"cl_36507c7904809cebf9b81903c3550dd5","status":"active","text":"The taxonomy organizes reinforcement-learning knowledge into dynamics and reward models, value functions, policies, and original data.","confidence":0.97,"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/claims/cl_36507c7904809cebf9b81903c3550dd5"}],"concepts":[{"public_id":"co_22af8e23f15e793e0fecc4a4f84e31e9","status":"active","name":"knowledge modalities","description":"Different forms in which transferable knowledge is represented in reinforcement learning.","types":["concept"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_22af8e23f15e793e0fecc4a4f84e31e9"},{"public_id":"co_2ceedc4e02a6a86b569bce952c2af1e0","status":"active","name":"specific forms of transfer","description":"Particular kinds of knowledge transfer discussed in the taxonomy and its analysis.","types":["concept"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_2ceedc4e02a6a86b569bce952c2af1e0"},{"public_id":"co_32041069e136d2150de34da4975933e0","status":"active","name":"transfer in reinforcement learning","description":"The process of reusing knowledge from prior reinforcement-learning experience to adapt to new tasks or settings.","types":["research topic"],"aliases":["RL transfer"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_32041069e136d2150de34da4975933e0"},{"public_id":"co_653bb91acb1a10b78fd8d14d24579f64","status":"active","name":"taxonomy of knowledge modalities","description":"A structured classification of knowledge representations used for transfer in reinforcement learning.","types":["taxonomy"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_653bb91acb1a10b78fd8d14d24579f64"},{"public_id":"co_661f2a1595b908c12e2ad1bee92730c7","status":"active","name":"environment interactions","description":"Direct interactions with the environment used to collect experience during reinforcement learning.","types":["process"],"aliases":["environment interaction"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_661f2a1595b908c12e2ad1bee92730c7"},{"public_id":"co_6c19dc36dea36b05a2b61cfaf32cc729","status":"active","name":"objectives and mechanisms for transfer","description":"The goals and procedural means used to move knowledge from one setting to another.","types":["framework"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_6c19dc36dea36b05a2b61cfaf32cc729"},{"public_id":"co_7300d3c5a25eb5d55d509f274ff54241","status":"active","name":"computational efficiency","description":"The amount of computation required to learn or apply a reinforcement-learning approach.","types":["performance criterion"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_7300d3c5a25eb5d55d509f274ff54241"},{"public_id":"co_8726c5a6bf465789f748b17027456634","status":"active","name":"generalisation across varying axes of change","description":"Robustness or adaptability to differences such as task, environment, or other changing factors.","types":["property"],"aliases":["generalization across varying axes of change"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_8726c5a6bf465789f748b17027456634"},{"public_id":"co_9f80220c2a88f8495017ba9f237ed13e","status":"active","name":"learned transfer","description":"Transfer mechanisms that are acquired through learning from data or experience.","types":["approach"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_9f80220c2a88f8495017ba9f237ed13e"},{"public_id":"co_c022e3dd052d63dc7c116091bfd446b7","status":"active","name":"policies","description":"Decision rules that map states or observations to actions in reinforcement learning.","types":["method"],"aliases":["policy"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_c022e3dd052d63dc7c116091bfd446b7"},{"public_id":"co_c7efb5ab9bb32d48399ae30b0c7ac30c","status":"active","name":"value functions","description":"Functions that estimate the expected return associated with states or actions in reinforcement learning.","types":["function"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_c7efb5ab9bb32d48399ae30b0c7ac30c"},{"public_id":"co_df7cdf969d13e71c030dc4446f8a76c1","status":"active","name":"designed transfer","description":"Transfer mechanisms that are manually specified or engineered rather than learned.","types":["approach"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_df7cdf969d13e71c030dc4446f8a76c1"},{"public_id":"co_dfb1da9b8159f2f12e0496e96fdbca07","status":"active","name":"original data","description":"The source experience or raw interaction data from which transfer-relevant knowledge can be extracted.","types":["data"],"aliases":[],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_dfb1da9b8159f2f12e0496e96fdbca07"},{"public_id":"co_e2ce685f5b6ae5344d49599098b9d0d2","status":"active","name":"dynamics and reward models","description":"Models that represent environment transitions and reward structure in reinforcement learning.","types":["model"],"aliases":["dynamics models","reward models"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_e2ce685f5b6ae5344d49599098b9d0d2"},{"public_id":"co_eb3f1204e02333102284beb5a44c92f7","status":"active","name":"reinforcement learning","description":"A learning paradigm in which an agent learns by interacting with an environment and receiving rewards.","types":["machine learning paradigm"],"aliases":["RL"],"contributors":[{"id":1,"public_id":"12632b8b5f","public_label":"Anonymous (12632b8b5f)","roles":["extraction"],"url":"https://sah.borca.ai/u/12632b8b5f"}],"url":"https://sah.borca.ai/concepts/co_eb3f1204e02333102284beb5a44c92f7"}],"external_ids":{"DOI":"10.48550/arXiv.2312.01939","ArXiv":"2312.01939","PubMed":null,"PubMedCentral":null,"MAG":null,"DBLP":"journals/corr/abs-2312-01939","ACL":null},"open_access":{"is_open_access":true,"pdf_url":"https://arxiv.org/pdf/2312.01939","landing_url":"https://arxiv.org/abs/2312.01939","source":"arxiv","pdf_url_source":"derived_arxiv","license":"http://creativecommons.org/licenses/by-nc-sa/4.0/","reason":null},"reference_availability":{"status":"not_analyzed","references_indexed":false,"full_text_available":true,"full_text_source":"arxiv","count_basis":"semantic_scholar_metadata","extraction_status":"not_indexed","reason":"semantic_scholar_graph_empty_full_text_available"},"source":{"provider":"episteme2","base_corpus":"semantic_scholar_dump","freshness_mode":"unknown","basis":["semantic_scholar_metadata","postgres_metadata"],"limits":["paper metadata is based on indexed upstream scholarly datasets","claims and concepts are available only for extracted papers","absence of claims or concepts means no extracted graph data is available in this response"],"status":"available","degraded":false,"degraded_reasons":[],"diagnostics":{"status":"available","degraded":false,"degraded_reasons":[],"metadata_status":"available","graph_status":"available","abstract_status":"available"},"source_flags":1},"paper_id":636450,"paper_uid":"b8c4d912-2670-42f5-8ea6-bdaff2d0da47","canonical_identity":{"paper_id":636450,"paper_uid":"b8c4d912-2670-42f5-8ea6-bdaff2d0da47","identity_status":"available","lookup_basis":"semantic_scholar_external_id","compatibility_path":"corpus_id"},"url":"https://sah.borca.ai/papers/265609417"}