arXiv subjects

Ichiro Ide

Publications and source records attributed to Ichiro Ide.

21 records · Page 2Linked to original sources

arXiv2023

IPA-CLIP: Integrating Phonetic Priors into Vision and Language Pretraining

Recently, large-scale Vision and Language (V\&L) pretraining has become the standard backbone of many multimedia systems. While it has shown remarkable performance even in unseen situations, it often performs in ways not intuitive to humans. Particularly, they usually do not consider the pronunciation of the input, which humans would utilize to understand language, especially when it comes to unknown words. Thus, this paper inserts phonetic prior into Contrastive Language-Image Pretraining (CLIP), one of the V\&L pretrained models, to make it consider the pronunciation similarity among its pronunciation inputs. To achieve this, we first propose a phoneme embedding that utilizes the phoneme relationships provided by the International Phonetic Alphabet (IPA) chart as a phonetic prior. Next, by distilling the frozen CLIP text encoder, we train a pronunciation encoder employing the IPA-based embedding. The proposed model named IPA-CLIP comprises this pronunciation encoder and the original CLIP encoders (image and text). Quantitative evaluation reveals that the phoneme distribution on the embedding space represents phonetic relationships more accurately when using the proposed phoneme embedding. Furthermore, in some multimodal retrieval tasks, we confirm that the proposed pronunciation encoder enhances the performance of the text encoder and that the pronunciation encoder handles nonsense words in a more phonetic manner than the text encoder. Finally, qualitative evaluation verifies the correlation between the pronunciation encoder and human perception regarding pronunciation similarity.

cs.MM↗

arXiv2022

A Novel Approach for Pill-Prescription Matching with GNN Assistance and Contrastive Learning

Medication mistaking is one of the risks that can result in unpredictable consequences for patients. To mitigate this risk, we develop an automatic system that correctly identifies pill-prescription from mobile images. Specifically, we define a so-called pill-prescription matching task, which attempts to match the images of the pills taken with the pills' names in the prescription. We then propose PIMA, a novel approach using Graph Neural Network (GNN) and contrastive learning to address the targeted problem. In particular, GNN is used to learn the spatial correlation between the text boxes in the prescription and thereby highlight the text boxes carrying the pill names. In addition, contrastive learning is employed to facilitate the modeling of cross-modal similarity between textual representations of pill names and visual representations of pill images. We conducted extensive experiments and demonstrated that PIMA outperforms baseline models on a real-world dataset of pill and prescription images that we constructed. Specifically, PIMA improves the accuracy from 19.09% to 46.95% compared to other baselines. We believe our work can open up new opportunities to build new clinical applications and improve medication safety and patient care.

cs.CV↗

arXiv2020

Off-resonant coherent electron transport over three nanometers in multi-heme protein bioelectronic junctions

Multi-heme cytochromes (MHC) are fascinating proteins used by bacterial organisms to shuttle electrons within and between their cells. When placed in a solid state electronic junction, they support temperature-independent currents over several nanometers that are three orders of magnitude higher compared to other redox proteins of comparable size. To gain microscopic insight into their astonishingly high conductivities, we present herein the first current-voltage calculations of its kind, for a MHC sandwiched between two Au(111) electrodes, complemented by photo-emission spectroscopy experiments. We find that conduction proceeds via off-resonant coherent tunneling mediated by a large number of protein valence-band orbitals that are strongly delocalized over heme and protein residues, effectively "gating" the current between the two electrodes. This picture is profoundly different from the dominant electron hopping mechanism supported by the same protein in aqueous solution. Our results imply that current output in MHC junctions could be even further increased in the resonant regime, e.g. by application of a gate voltage, making these proteins extremely interesting for next-generation bionanoelectronic devices.

physics.bio-ph↗