{"corpus_id":247893952,"paper_sha":"b85231731ec31b0d29fdbf27b73073469105bd62","doi":"10.3390/s22072611","arxiv_id":null,"pmid":35408228,"pmcid":"9003536","mag_id":null,"dblp_id":"journals/sensors/Choi22","acl_id":null,"title":"Toward Exploiting Second-Order Feature Statistics for Arbitrary Image Style Transfer","year":2022,"publication_date":"2022-03-29","venue":"Italian National Conference on Sensors","journal":{"name":"Sensors (Basel, Switzerland)","pages":null,"volume":"22"},"journal_issn":null,"journal_title":null,"publication_types":["JournalArticle"],"pubmed_pub_types":["Journal Article"],"s2_fields_of_study":["Medicine","Computer Science"],"reference_count":22,"citation_count":4,"influential_citation_count":0,"is_open_access":true,"arxiv_categories":null,"arxiv_license":null,"arxiv_journal_ref":null,"mesh_headings":[{"d":"Image Processing, Computer-Assisted","mj":false,"qs":[{"q":"methods","mj":false,"ui":"Q000379"}],"ui":"D007091"},{"d":"Neural Networks, Computer","mj":true,"ui":"D016571"},{"d":"Records","mj":true,"ui":"D011996"}],"chemicals":null,"comments_corrections":null,"source_flags":5,"s2_open_access_pdf_url":"https://www.mdpi.com/1424-8220/22/7/2611/pdf?version=1648542522","s2_open_access_landing_url":"https://www.semanticscholar.org/paper/b85231731ec31b0d29fdbf27b73073469105bd62","s2_open_access_license":"CCBY","s2_open_access_status":"GOLD","pmc_open_access_pdf_url":null,"pmc_open_access_landing_url":null,"pmc_open_access_license":null,"pmc_open_access_status":null,"unpaywall_open_access_pdf_url":null,"unpaywall_open_access_landing_url":null,"unpaywall_open_access_license":null,"unpaywall_open_access_status":null,"abstract":"Generating images of artistic style from input images, also known as image style transfer, has been improved in the quality of output style and the speed of image generation since deep neural networks have been applied in the field of computer vision research. However, the previous approaches used feature alignment techniques that were too simple in their transform layer to cover the characteristics of style features of images. In addition, they used an inconsistent combination of transform layers and loss functions in the training phase to embed arbitrary styles in a decoder network. To overcome these shortcomings, the second-order statistics of the encoded features are exploited to build an optimal arbitrary image style transfer technique. First, a new correlation-aware loss and a correlation-aware feature alignment technique are proposed. Using this consistent combination of loss and feature alignment methods strongly matches the second-order statistics of content features to those of the target-style features and, accordingly, the style capacity of the decoder network is increased. Secondly, a new component-wise style controlling method is proposed. This method can generate various styles from one or several style images by using style-specific components from second-order feature statistics. We experimentally prove that the proposed method achieves improvements in both the style capacity of the decoder network and the style variety without losing the ability of real-time processing (less than 200 ms) on Graphics Processing Unit (GPU) devices.","claims":[{"public_id":"cl_57a2b2def1da1588137910c64f9f59d3","status":"active","text":"A component-wise style controlling method can generate various styles from one or several style images by using style-specific components from second-order feature statistics.","confidence":0.93,"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/claims/cl_57a2b2def1da1588137910c64f9f59d3"},{"public_id":"cl_847b15ec5dc73f3ff25bef68d42e1216","status":"active","text":"A correlation-aware loss and a correlation-aware feature alignment technique are proposed for arbitrary image style transfer.","confidence":0.94,"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/claims/cl_847b15ec5dc73f3ff25bef68d42e1216"},{"public_id":"cl_4a6649fb82d9fc82253b519f405b3e53","status":"active","text":"The proposed method improves both decoder style capacity and style variety while preserving real-time processing below 200 ms on Graphics Processing Unit devices.","confidence":0.92,"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/claims/cl_4a6649fb82d9fc82253b519f405b3e53"},{"public_id":"cl_3c15855df7d7302bef10191d23e2c344","status":"active","text":"Using a consistent correlation-aware loss and feature alignment combination strongly matches second-order statistics of content features to those of target-style features and increases the style capacity of the decoder network.","confidence":0.9,"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/claims/cl_3c15855df7d7302bef10191d23e2c344"}],"concepts":[{"public_id":"co_00c02e340b1f7c33bf23069b6adcf515","status":"active","name":"correlation-aware feature alignment technique","description":"A feature alignment method that accounts for correlations between feature components.","types":["method"],"aliases":[],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_00c02e340b1f7c33bf23069b6adcf515"},{"public_id":"co_05fecc41ae01ca2bf5036d73358cfc33","status":"active","name":"content features","description":"Encoded image features representing the content structure to preserve during style transfer.","types":["feature representation"],"aliases":[],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_05fecc41ae01ca2bf5036d73358cfc33"},{"public_id":"co_14abcf3cabc98883d2902b9d1000fb9a","status":"active","name":"correlation-aware loss","description":"A loss function that incorporates correlations in feature statistics during style transfer training.","types":["loss function"],"aliases":[],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_14abcf3cabc98883d2902b9d1000fb9a"},{"public_id":"co_3038877aa0282f31065c0381ec038543","status":"active","name":"real-time processing","description":"Image generation fast enough for interactive use, measured here as less than 200 ms.","types":["performance property"],"aliases":[],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_3038877aa0282f31065c0381ec038543"},{"public_id":"co_3a1e3ba400fe5950514159e643b91479","status":"active","name":"style-specific components","description":"Components of second-order feature statistics associated with particular style characteristics.","types":["feature component"],"aliases":[],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_3a1e3ba400fe5950514159e643b91479"},{"public_id":"co_4335808b7ab69f0f570d25f749748d20","status":"active","name":"target-style features","description":"Encoded image features representing the style characteristics of the target style image.","types":["feature representation"],"aliases":[],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_4335808b7ab69f0f570d25f749748d20"},{"public_id":"co_51482dc232d4c743c36b7b3182f1e4f8","status":"active","name":"decoder network","description":"A neural network component that reconstructs a stylized image from transformed feature representations.","types":["model component"],"aliases":[],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_51482dc232d4c743c36b7b3182f1e4f8"},{"public_id":"co_7dd423a21c8fcf94b04851f844f09220","status":"active","name":"arbitrary image style transfer","description":"A style transfer setting in which a model can apply arbitrary target artistic styles to content images.","types":["task"],"aliases":[],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_7dd423a21c8fcf94b04851f844f09220"},{"public_id":"co_85e271206682267994dcc2ab0ae69416","status":"active","name":"Graphics Processing Unit devices","description":"GPU hardware used to evaluate the processing speed of the style transfer method.","types":["hardware"],"aliases":["GPU devices"],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_85e271206682267994dcc2ab0ae69416"},{"public_id":"co_a2d50cfda73e4deb841d1dfda07f5b69","status":"active","name":"encoded features","description":"Image features produced by an encoder network before style alignment or decoding.","types":["data representation"],"aliases":[],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_a2d50cfda73e4deb841d1dfda07f5b69"},{"public_id":"co_ce1861632cdd3a283a6ec13d1c25cadc","status":"active","name":"image style transfer","description":"A computer vision task that generates artistically styled images from input images.","types":["task"],"aliases":["Generating images of artistic style from input images"],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_ce1861632cdd3a283a6ec13d1c25cadc"},{"public_id":"co_eaf03b08248514c1adb0f6d3098833ef","status":"active","name":"component-wise style controlling method","description":"A style control method that manipulates style-specific components from second-order feature statistics.","types":["method"],"aliases":[],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_eaf03b08248514c1adb0f6d3098833ef"},{"public_id":"co_f8e8923c9492a4be92c1265d88f443fd","status":"active","name":"second-order feature statistics","description":"Correlation-based statistics of encoded image features used to represent style information.","types":["feature representation"],"aliases":[],"contributors":[{"id":35,"public_id":"b2adb6bfad","public_label":"Anonymous (b2adb6bfad)","roles":["extraction"],"url":"https://sah.borca.ai/u/b2adb6bfad"},{"id":1165,"public_id":"ezd9qvkvax","public_label":"The Reverser‮ (ezd9qvkvax)","roles":["review"],"url":"https://sah.borca.ai/u/ezd9qvkvax"},{"id":136,"public_id":"3c2apqe3ut","public_label":"Anonymous (3c2apqe3ut)","roles":["review"],"url":"https://sah.borca.ai/u/3c2apqe3ut"}],"url":"https://sah.borca.ai/concepts/co_f8e8923c9492a4be92c1265d88f443fd"}],"external_ids":{"DOI":"10.3390/s22072611","ArXiv":null,"PubMed":35408228,"PubMedCentral":"9003536","MAG":null,"DBLP":"journals/sensors/Choi22","ACL":null},"open_access":{"is_open_access":true,"pdf_url":"https://www.mdpi.com/1424-8220/22/7/2611/pdf?version=1648542522","landing_url":"https://www.semanticscholar.org/paper/b85231731ec31b0d29fdbf27b73073469105bd62","source":"semantic_scholar","pdf_url_source":"semantic_scholar_open_access_pdf","license":"CCBY","status":"GOLD","reason":null},"reference_availability":{"status":"available","references_indexed":true,"full_text_available":false,"full_text_source":null,"count_basis":"semantic_scholar_metadata","extraction_status":"not_applicable","reason":null},"source":{"provider":"episteme2","base_corpus":"semantic_scholar_dump","freshness_mode":"unknown","basis":["semantic_scholar_metadata","postgres_metadata"],"limits":["paper metadata is based on indexed upstream scholarly datasets","claims and concepts are available only for extracted papers","absence of claims or concepts means no extracted graph data is available in this response"],"status":"available","degraded":false,"degraded_reasons":[],"diagnostics":{"status":"available","degraded":false,"degraded_reasons":[],"metadata_status":"available","graph_status":"available","abstract_status":"available"},"source_flags":5},"paper_id":632291,"paper_uid":"0323f044-64ba-4dd5-bc33-970d37a4eb83","canonical_identity":{"paper_id":632291,"paper_uid":"0323f044-64ba-4dd5-bc33-970d37a4eb83","identity_status":"available","lookup_basis":"semantic_scholar_external_id","compatibility_path":"corpus_id"},"url":"https://sah.borca.ai/papers/247893952"}