Multi-modal Sensing Co-creation Consortium Industry-academia co-creation platform Joint research promotion program OPERA


Toyohashi University of Technology
Graduate Schools of Engineering

Professor, Kazuaki Sawada

 Robots and sensors are selected as among the core technologies inclusive to the Japan Revitalization Strategy. Sensors that keep us safe and secure play a major role in the IoT/AI era.

 Various causal relationships can be observed to cause changes in the environment, such as from natural disasters and diseases. A variety of sensors are needed to obtain diverse information. We should be able to develop a universal sensor based on advanced integrated semiconductor technology, capable of being installed in many places. For example, to understand the cellular level behavior in the human body, it is effective to visualize and analyze the correlation among several parameters simultaneously. We should study not only a single phenomenon of cellular ion channel activity but also other physical and chemical phenomena coinciding (ex. gases released from cells, mechanical changes, action potentials) in real-time visualization with microscopic position and time information.

  Coordinating sensor technology and big data analysis enable us to understand the causal relationships of various events. We will establish a "multimodal sensor technology" that integrates sensors capable of detecting multiple types of information. Multimodal sensors can enable the visualization of changes in various signals for which the correlation remains unclear, however, with higher spatial and temporal resolution. Our goal is to achieve a significant increase in the "data" value to support the IoT/AI era.

 In this program, we aim to establish fundamental technologies to improve the spatial and temporal resolution of the world's first "Ion Image Sensor" technology that visualize the chemical phenomena, developed by Toyohashi University of Technology. Also, we plan to achieve three types of multimodal sensors: multimodal gas component sensor, multimodal physical and chemical sensor, and multimodal chemical and biosensor.
 In the feasibility study phase, we completed the sensor's basic element structure design with a micron-level spatial resolution and sub-millisecond temporal resolution. We also developed a sensitive membrane formation technology for detecting various physical and chemical phenomena such as gases, pressures, and biomaterials. In the full-scale deployment phase, we will apply multimodal sensing technology to environmental studies, agriculture, medical and drug discovery, healthcare, and human-machine harmony (Robotics) and produce various multimodal sensor prototypes. We intend to deploy our results broadly across society.

 Through the industry-academia co-creation platform, we collaborate with participating companies in the technological development and social deployment of sensors. We aim to revitalize the semiconductor industry that supports super-smart society and create core industries to advance information technology in the medical, biotechnology, and chemical industries.


 The world's first CMOS/CCD ion image sensor was developed in 1997. The principle was to convert changes in the electric charge on the sensor surface into changes in the semiconductor's potential, featuring such as cumulative operations. Since then, we have been developing technologies to enhance ion detection sensitivity, to raise the number of pixels in ion image sensors, to increase density, and to downsize the sensor.

 We have established the Multimodal Bio-Imager Study Group to cooperate with the private sector, and public research institutes and have been working towards the social deployment of CMOS/CCD ion image sensors. In the early stage of development, we only achieved an 8-pixel density sensor. The current ion image sensor technology has achieved a 65,000-pixel (256x256) density in practical applications.
 A CMOS/CCD type ion image sensor can work as a highly sensitive potential detector array. We apply this characteristic to detect light, gas molecules, pressure, and biological substances by developing a sensitive membrane that reacts to various physical and chemical phenomena. Furthermore, our semiconductor manufacturing technology enables us to form multiple sensitive membranes that have led to the creation of the multimodal sensing technology that can visualize physical and chemical information at the micron level as proposed in this program.