{"corpus_id":214435307,"paper_sha":"e4d79861bff8f14a0d8abd9b415424253fd9d992","doi":"10.1016/j.rse.2019.111599","arxiv_id":null,"pmid":null,"pmcid":null,"mag_id":2996041315,"dblp_id":null,"acl_id":null,"title":"Soybean yield prediction from UAV using multimodal data fusion and deep learning","year":2020,"publication_date":"2020-02-01","venue":"","journal":{"name":"Remote Sensing of Environment","pages":"111599","volume":"237"},"journal_issn":null,"journal_title":null,"publication_types":[],"pubmed_pub_types":null,"s2_fields_of_study":["Agricultural and Food Sciences","Computer Science","Engineering"],"reference_count":176,"citation_count":780,"influential_citation_count":28,"is_open_access":true,"arxiv_categories":null,"arxiv_license":null,"arxiv_journal_ref":null,"mesh_headings":null,"chemicals":null,"comments_corrections":null,"source_flags":1,"s2_open_access_pdf_url":"http://manuscript.elsevier.com/S0034425719306194/pdf/S0034425719306194.pdf","s2_open_access_landing_url":"https://www.semanticscholar.org/paper/e4d79861bff8f14a0d8abd9b415424253fd9d992","s2_open_access_license":"publisher-specific-oa","s2_open_access_status":"BRONZE","pmc_open_access_pdf_url":null,"pmc_open_access_landing_url":null,"pmc_open_access_license":null,"pmc_open_access_status":null,"unpaywall_open_access_pdf_url":null,"unpaywall_open_access_landing_url":null,"unpaywall_open_access_license":null,"unpaywall_open_access_status":null,"abstract":"Abstract Preharvest crop yield prediction is critical for grain policy making and food security. Early estimation of yield at field or plot scale also contributes to high-throughput plant phenotyping and precision agriculture. New developments in Unmanned Aerial Vehicle (UAV) platforms and sensor technology facilitate cost-effective data collection through simultaneous multi-sensor/multimodal data collection at very high spatial and spectral resolutions. The objective of this study is to evaluate the power of UAV-based multimodal data fusion using RGB, multispectral and thermal sensors to estimate soybean (Glycine max) grain yield within the framework of Deep Neural Network (DNN). RGB, multispectral, and thermal images were collected using a low-cost multi-sensory UAV from a test site in Columbia, Missouri, USA. Multimodal information, such as canopy spectral, structure, thermal and texture features, was extracted and combined to predict crop grain yield using Partial Least Squares Regression (PLSR), Random Forest Regression (RFR), Support Vector Regression (SVR), input-level feature fusion based DNN (DNN-F1) and intermediate-level feature fusion based DNN (DNN-F2). The results can be summarized in three messages: (1) multimodal data fusion improves the yield prediction accuracy and is more adaptable to spatial variations; (2) DNN-based models improve yield prediction model accuracy: the highest accuracy was obtained by DNN-F2 with an R2 of 0.720 and a relative root mean square error (RMSE%) of 15.9%; (3) DNN-based models were less prone to saturation effects, and exhibited more adaptive performance in predicting grain yields across the Dwight, Pana and AG3432 soybean genotypes in our study. Furthermore, DNN-based models demonstrated consistent performance over space with less spatial dependency and variations. This study indicates that multimodal data fusion using low-cost UAV within a DNN framework can provide a relatively accurate and robust estimation of crop yield, and deliver valuable insight for high-throughput phenotyping and crop field management with high spatial precision.","claims":[{"public_id":"cl_ff30617440cdfc502facacb00a0d695c","status":"active","text":"DNN-based models are less prone to saturation effects and show more adaptive yield prediction performance across the Dwight, Pana, and AG3432 soybean genotypes than the other tested models.","confidence":0.85,"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/claims/cl_ff30617440cdfc502facacb00a0d695c"},{"public_id":"cl_e50b2b5c70dfefad70a40ada7f213579","status":"active","text":"DNN-based models demonstrate more consistent soybean grain yield prediction performance over space, with less spatial dependency and variation than the other tested models.","confidence":0.8,"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/claims/cl_e50b2b5c70dfefad70a40ada7f213579"},{"public_id":"cl_99e61211d3633a5aa99021eef03f3f4f","status":"active","text":"DNN-based models improve soybean grain yield prediction accuracy over other model types, with the intermediate-level fusion model DNN-F2 achieving the highest accuracy (R2 = 0.720, relative RMSE = 15.9%).","confidence":0.92,"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/claims/cl_99e61211d3633a5aa99021eef03f3f4f"},{"public_id":"cl_6c002c55a57cda5d03202362bd9b9b0c","status":"active","text":"UAV-based multimodal data fusion combining RGB, multispectral, and thermal sensor data improves soybean grain yield prediction accuracy and is more adaptable to spatial variation than single-modality data.","confidence":0.9,"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/claims/cl_6c002c55a57cda5d03202362bd9b9b0c"}],"concepts":[{"public_id":"co_0c0a947aad25582fe4bdd14594ccaed6","status":"active","name":"saturation effects","description":"A phenomenon in yield prediction models where prediction accuracy plateaus or degrades at high yield values.","types":["phenomenon"],"aliases":[],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_0c0a947aad25582fe4bdd14594ccaed6"},{"public_id":"co_140b655f6cbe50b8c4f42f8226bf550b","status":"active","name":"DNN-based models","description":"Collective term for the DNN-F1 and DNN-F2 deep neural network yield prediction models evaluated in this study.","types":["model class","method"],"aliases":[],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_140b655f6cbe50b8c4f42f8226bf550b"},{"public_id":"co_2136dbd400fc0f942337fd88106e3206","status":"active","name":"RGB images","description":"Visible-light color images collected by the UAV and used to derive canopy structure and texture features.","types":["data modality"],"aliases":["RGB imagery"],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_2136dbd400fc0f942337fd88106e3206"},{"public_id":"co_2a8b4ec9c1c818b7e1be088eebae8c32","status":"active","name":"multispectral images","description":"Images captured across multiple spectral bands by the UAV, used to derive canopy spectral features.","types":["data modality"],"aliases":["multispectral imagery"],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_2a8b4ec9c1c818b7e1be088eebae8c32"},{"public_id":"co_4465801f617fac8aea2db491c84931be","status":"active","name":"multimodal data fusion","description":"The combination of spectral, structural, thermal, and texture features derived from multiple UAV sensor types into a single yield prediction model.","types":["method"],"aliases":[],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_4465801f617fac8aea2db491c84931be"},{"public_id":"co_5356c6f0643de9036b1f6cfbd7c61fec","status":"active","name":"soybean grain yield","description":"The harvested seed yield of soybean (Glycine max) plants, the outcome variable predicted from UAV sensor data in this study.","types":["outcome","measurement"],"aliases":["grain yield","crop yield"],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_5356c6f0643de9036b1f6cfbd7c61fec"},{"public_id":"co_6019b1cd8ce917429e09479a57dc16bd","status":"active","name":"Random Forest Regression (RFR)","description":"An ensemble tree-based regression method used as one of the baseline models for predicting soybean grain yield.","types":["method","baseline"],"aliases":["RFR"],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_6019b1cd8ce917429e09479a57dc16bd"},{"public_id":"co_80ec0b8454ce007deb9bdf5fdc57a7e4","status":"active","name":"Partial Least Squares Regression (PLSR)","description":"A linear regression method used as one of the baseline models for predicting soybean grain yield from extracted features.","types":["method","baseline"],"aliases":["PLSR"],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_80ec0b8454ce007deb9bdf5fdc57a7e4"},{"public_id":"co_90b59316371788c5faf0771cfc395316","status":"active","name":"DNN-F2","description":"A deep neural network yield prediction model that fuses multimodal features at an intermediate network layer rather than at the input.","types":["method"],"aliases":["intermediate-level feature fusion based DNN"],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_90b59316371788c5faf0771cfc395316"},{"public_id":"co_c26907a6e60f22217ff9e801316983f9","status":"active","name":"Unmanned Aerial Vehicle (UAV)","description":"A low-cost aerial drone platform used to collect RGB, multispectral, and thermal imagery over the soybean field.","types":["platform","technology"],"aliases":["UAV"],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_c26907a6e60f22217ff9e801316983f9"},{"public_id":"co_d54273f050ccd2c82020672395a49230","status":"active","name":"soybean genotypes","description":"The Dwight, Pana, and AG3432 soybean varieties across which yield prediction model performance was evaluated in this study.","types":["study factor","biological entity"],"aliases":[],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_d54273f050ccd2c82020672395a49230"},{"public_id":"co_df32111c98347bf3bca40c0c8bc6cddf","status":"active","name":"Support Vector Regression (SVR)","description":"A kernel-based regression method used as one of the baseline models for predicting soybean grain yield.","types":["method","baseline"],"aliases":["SVR"],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_df32111c98347bf3bca40c0c8bc6cddf"},{"public_id":"co_e74cc3f956d272c0beebde4fffb24b9c","status":"active","name":"thermal images","description":"Infrared images captured by the UAV, used to derive canopy thermal features.","types":["data modality"],"aliases":["thermal imagery"],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_e74cc3f956d272c0beebde4fffb24b9c"},{"public_id":"co_fef02c4d8886efc929b2389763c1b150","status":"active","name":"DNN-F1","description":"A deep neural network yield prediction model that fuses multimodal features at the input level.","types":["method"],"aliases":["input-level feature fusion based DNN"],"contributors":[{"id":17,"public_id":"322360f1c1","public_label":"Killer Whale (322360f1c1)","roles":["extraction"],"url":"https://sah.borca.ai/u/322360f1c1"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"}],"url":"https://sah.borca.ai/concepts/co_fef02c4d8886efc929b2389763c1b150"}],"external_ids":{"DOI":"10.1016/j.rse.2019.111599","ArXiv":null,"PubMed":null,"PubMedCentral":null,"MAG":2996041315,"DBLP":null,"ACL":null},"open_access":{"is_open_access":true,"pdf_url":"http://manuscript.elsevier.com/S0034425719306194/pdf/S0034425719306194.pdf","landing_url":"https://www.semanticscholar.org/paper/e4d79861bff8f14a0d8abd9b415424253fd9d992","source":"semantic_scholar","pdf_url_source":"semantic_scholar_open_access_pdf","license":"publisher-specific-oa","status":"BRONZE","reason":null},"reference_availability":{"status":"available","references_indexed":true,"full_text_available":false,"full_text_source":null,"count_basis":"semantic_scholar_metadata","extraction_status":"not_applicable","reason":null},"source":{"provider":"episteme2","base_corpus":"semantic_scholar_dump","freshness_mode":"unknown","basis":["semantic_scholar_metadata","postgres_metadata"],"limits":["paper metadata is based on indexed upstream scholarly datasets","claims and concepts are available only for extracted papers","absence of claims or concepts means no extracted graph data is available in this response"],"status":"available","degraded":false,"degraded_reasons":[],"diagnostics":{"status":"available","degraded":false,"degraded_reasons":[],"metadata_status":"available","graph_status":"available","abstract_status":"available"},"source_flags":1},"paper_id":632233,"paper_uid":"0c2337e5-96ed-4ee5-bd27-c330b162d2a7","canonical_identity":{"paper_id":632233,"paper_uid":"0c2337e5-96ed-4ee5-bd27-c330b162d2a7","identity_status":"available","lookup_basis":"semantic_scholar_external_id","compatibility_path":"corpus_id"},"url":"https://sah.borca.ai/papers/214435307"}