Self-supervised video processing with self-calibration on an analogue computing platform based on a selector-less memristor array

Self-supervised video processing with self-calibration on an analogue computing platform based on a selector-less memristor array

Play all audios:

Loading...

ABSTRACT Memristor-based platforms could be used to create compact and energy-efficient artificial intelligence (AI) edge-computing systems due to their parallel computation ability in the


analogue domain. However, systems based on memristor arrays face challenges implementing real-time AI algorithms with fully on-device learning due to reliability issues, such as low yield,


poor uniformity and endurance problems. Here we report an analogue computing platform based on a selector-less analogue memristor array. We use interfacial-type titanium oxide memristors


with a gradual oxygen distribution that exhibit high reliability, high linearity, forming-free attribute and self-rectification. Our platform—which consists of a selector-less


(one-memristor) 1 K (32 × 32) crossbar array, peripheral circuitry and digital controller—can run AI algorithms in the analogue domain by self-calibration without compensation operations or


pretraining. We illustrate the capabilities of the system with real-time video foreground and background separation, achieving an average peak signal-to-noise ratio of 30.49 dB and a


structural similarity index measure of 0.81; these values are similar to those of simulations for the ideal case. Access through your institution Buy or subscribe This is a preview of


subscription content, access via your institution ACCESS OPTIONS Access through your institution Access Nature and 54 other Nature Portfolio journals Get Nature+, our best-value


online-access subscription $29.99 / 30 days cancel any time Learn more Subscribe to this journal Receive 12 digital issues and online access to articles $119.00 per year only $9.92 per issue


Learn more Buy this article * Purchase on SpringerLink * Instant access to full article PDF Buy now Prices may be subject to local taxes which are calculated during checkout ADDITIONAL


ACCESS OPTIONS: * Log in * Learn about institutional subscriptions * Read our FAQs * Contact customer support SIMILAR CONTENT BEING VIEWED BY OTHERS FULLY INTEGRATED MULTI-MODE


OPTOELECTRONIC MEMRISTOR ARRAY FOR DIVERSIFIED IN-SENSOR COMPUTING Article 08 November 2024 A MEMRISTOR-BASED ANALOGUE RESERVOIR COMPUTING SYSTEM FOR REAL-TIME AND POWER-EFFICIENT SIGNAL


PROCESSING Article 26 September 2022 HYBRID ARCHITECTURE BASED ON TWO-DIMENSIONAL MEMRISTOR CROSSBAR ARRAY AND CMOS INTEGRATED CIRCUIT FOR EDGE COMPUTING Article Open access 21 January 2022


DATA AVAILABILITY The data that support the plots within this article and other findings of this study are available from the corresponding author upon reasonable request. CODE AVAILABILITY


The code used for the simulation is available from the corresponding author upon reasonable request. REFERENCES * Verbraeken, J. et al. A survey on distributed machine learning. _ACM Comput.


Surv._ 53, 30 (2020). MATH  Google Scholar  * Masanet, E., Shehabi, A., Lei, N., Smith, S. & Koomey, J. Recalibrating global data center energy-use estimates. _Science_ 367, 984–986


(2020). Article  Google Scholar  * Yu, W. et al. A survey on the edge computing for the Internet of Things. _IEEE Access_ 6, 6900–6919 (2018). Article  MATH  Google Scholar  * Li, E., Zeng,


L. K., Zhou, Z. & Chen, X. Edge AI: on-demand accelerating deep neural network inference via edge computing. _IEEE Trans. Wirel. Commun._ 19, 447–457 (2020). Article  MATH  Google


Scholar  * Take it to the edge. _Nat. Electron_. 2, 1 (2019). * Satyanarayanan, M., Bahl, P., Cáceres, R. & Davies, N. The case for VM-based cloudlets in mobile computing. _IEEE


Pervasive Comput._ 8, 14–23 (2009). Article  Google Scholar  * Cao, K. Y., Liu, Y. F., Meng, G. J. & Sun, Q. M. An overview on edge computing research. _IEEE Access_ 8, 85714–85728


(2020). Article  MATH  Google Scholar  * Murshed, M. G. S. et al. Machine learning at the network edge: a survey. _ACM Comput. Surv._ 54, 170 (2021). MATH  Google Scholar  * Yao, P., Gao, B.


& Wu, H. Transforming edge hardware with in situ learning features. _Nat. Rev. Electr. Eng._ 1, 141–142 (2024). Article  MATH  Google Scholar  * Zhang, W. Q. et al. Neuro-inspired


computing chips. _Nat. Electron._ 3, 371–382 (2020). Article  MATH  Google Scholar  * Song, M. K. et al. Recent advances and future prospects for memristive materials, devices, and systems.


_ACS Nano_ 17, 11994–12039 (2023). Article  MATH  Google Scholar  * Wang, Z. R. et al. Reinforcement learning with analogue memristor arrays. _Nat. Electron._ 2, 115–124 (2019). Article 


MATH  Google Scholar  * Liu, Q. et al. _Proc. 2020 IEEE International Solid-State Circuits Conference (ISSCC)_ (IEEE, 2020). * Hung, J. M. et al. A four-megabit compute-in-memory macro with


eight-bit precision based on CMOS and resistive random-access memory for AI edge devices. _Nat. Electron._ 4, 921–930 (2021). Article  MATH  Google Scholar  * Kim, H., Mahmoodi, M. R., Nili,


H. & Strukov, D. B. 4K-memristor analog-grade passive crossbar circuit. _Nat. Commun._ 12, 5198 (2021). Article  Google Scholar  * Choi, S. et al. SiGe epitaxial memory for neuromorphic


computing with reproducible high performance based on engineered dislocations. _Nat. Mater._ 17, 335–340 (2018). Article  MATH  Google Scholar  * Wu, W. et al. _Proc. 2018 IEEE Symposium on


VLSI Technology_ (IEEE, 2018). * Li, C. et al. Efficient and self-adaptive in-situ learning in multilayer memristor neural networks. _Nat. Commun._ 9, 2385 (2018). Article  MATH  Google


Scholar  * Park, S. O. et al. Linear conductance update improvement of CMOS-compatible second-order memristors for fast and energy-efficient training of a neural network using a memristor


crossbar array. _Nanoscale Horiz._ 8, 1366–1376 (2023). Article  MATH  Google Scholar  * Rao, M. Y. et al. Thousands of conductance levels in memristors integrated on CMOS. _Nature_ 615,


823–829 (2023). Article  MATH  Google Scholar  * Zhang, W. B. et al. Edge learning using a fully integrated neuro-inspired memristor chip. _Science_ 381, 1205–1211 (2023). Article  MATH 


Google Scholar  * Yao, P. et al. Fully hardware-implemented memristor convolutional neural network. _Nature_ 577, 641–646 (2020). Article  MATH  Google Scholar  * Wan, W. E. et al. A


compute-in-memory chip based on resistive random-access memory. _Nature_ 608, 504–512 (2022). Article  MATH  Google Scholar  * Alibart, F., Gao, L. G., Hoskins, B. D. & Strukov, D. B.


High precision tuning of state for memristive devices by adaptable variation-tolerant algorithm. _Nanotechnology_ 23, 075201 (2012). Article  Google Scholar  * Milo, V. et al. Accurate


program/verify schemes of resistive switching memory (RRAM) for in-memory neural network circuits. _IEEE Trans. Electron Devices_ 68, 3832–3837 (2021). Article  MATH  Google Scholar  * Yao,


P. et al. Face classification using electronic synapses. _Nat. Commun._ 8, 15199 (2017). Article  Google Scholar  * Yan, B. N., Yang, J. H., Wu, Q., Chen, Y. R. & Li, H. _Proc. 2017


IEEE/ACM International Conference on Computer-Aided Design (ICCAD)_ (IEEE, 2017). * Liu, X. X. et al. Harmonica: a framework of heterogeneous computing systems with memristor-based


neuromorphic computing accelerators. _IEEE Trans. Circuits Syst. I Regul. Pap._ 63, 617–628 (2016). Article  MATH  Google Scholar  * Xia, Q. F. & Yang, J. J. Memristive crossbar arrays


for brain-inspired computing. _Nat. Mater._ 18, 309–323 (2019). Article  MATH  Google Scholar  * Ielmini, D. & Pedretti, G. Device and circuit architectures for in-memory computing.


_Adv. Intell. Syst._ 2, 2000040 (2020). Article  MATH  Google Scholar  * Park, S. O., Jeong, H., Park, J., Bae, J. & Choi, S. Experimental demonstration of highly reliable dynamic


memristor for artificial neuron and neuromorphic computing. _Nat. Commun._ 13, 2888 (2022). Article  MATH  Google Scholar  * Kim, K. M. et al. Low-power, self-rectifying, and forming-free


memristor with an asymmetric programing voltage for a high-sensity crossbar application. _Nano Lett._ 16, 6724–6732 (2016). Article  MATH  Google Scholar  * Li, C. et al. Three-dimensional


crossbar arrays of self-rectifying Si/SiO2/Si memristors. _Nat. Commun._ 8, 15666 (2017). Article  MATH  Google Scholar  * Candès, E. J., Li, X., Ma, Y. & Wright, J. Robust principal


component analysis? _J. ACM_ 58, 11 (2011). Article  MathSciNet  MATH  Google Scholar  * Han, S., Cho, E.-S., Park, I., Shin, K. & Yoon, Y.-G. Efficient neural network approximation of


robust PCA for automated analysis of calcium imaging data. In _Proc. Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2021: 24th International Conference_ Part VII 24 (eds


de Bruijne, M. et al.) 595–604 (Springer, 2021). * Chen, P. et al. _Proc. 2023 IEEE International Solid-State Circuits Conference (ISSCC)_ (IEEE, 2023). * Yang, Y., Nagarajaiah, S. &


Ni, Y. Q. Data compression of very large‐scale structural seismic and typhoon responses by low‐rank representation with matrix reshape. _Struct. Control Health Monit._ 22, 1119–1131 (2015).


Article  MATH  Google Scholar  * Ohno, T. et al. Short-term plasticity and long-term potentiation mimicked in single inorganic synapses. _Nat. Mater._ 10, 591–595 (2011). Article  MATH 


Google Scholar  * Wang, Z. et al. Memristors with diffusive dynamics as synaptic emulators for neuromorphic computing. _Nat. Mater._ 16, 101–108 (2017). Article  MATH  Google Scholar  * Wan,


J. et al. Efficient implementation of synaptic learning rules for neuromorphic computing based on plasma-treated ZnO nanowire memristors. _J. Phys. D_ 53, 055303 (2019). Article  MATH 


Google Scholar  * Grossi, A. et al. Impact of intercell and intracell variability on forming and switching parameters in RRAM arrays. _IEEE Trans. Electron Devices_ 62, 2502–2509 (2015).


Article  MATH  Google Scholar  * Abbas, Y. et al. Compliance-free, digital SET and analog RESET synaptic characteristics of sub-tantalum oxide based neuromorphic device. _Sci. Rep._ 8, 1228


(2018). Article  MATH  Google Scholar  * Siegel, S. et al. Trade‐off between data retention and switching speed in resistive switching ReRAM devices. _Adv. Electron. Mater._ 7, 2000815


(2021). Article  MATH  Google Scholar  * Ambrogio, S. et al. Equivalent-accuracy accelerated neural-network training using analogue memory. _Nature_ 558, 60–67 (2018). Article  Google


Scholar  * Li, C. et al. Analogue signal and image processing with large memristor crossbars. _Nat. Electron._ 1, 52–59 (2018). Article  MATH  Google Scholar  * Cai, F. X. et al. A fully


integrated reprogrammable memristor-CMOS system for efficient multiply-accumulate operations. _Nat. Electron._ 2, 290–299 (2019). Article  MATH  Google Scholar  * Wu, H., Judd, P., Zhang,


X., Isaev, M. & Micikevicius, P. Integer quantization for deep learning inference: principles and empirical evaluation. Preprint at https://arxiv.org/abs/2004.09602 (2020). * Cho, J.,


Han, S., Cho, E.-S., Shin, K. & Yoon, Y.-G. _Proc. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ (IEEE, 2023). Download references ACKNOWLEDGEMENTS This work


was supported by the National Research Foundation (NRF) funded by the Korean government (MSIT) (grant nos. RS-2024-00401234 to H.J., S.-O.P., J.B., T.J., Y.C., S.S., T.P. and S.C.;


2022M3I7A2078273 to H.J., S.-O.P., J.B., T.J., Y.C., S.S., H.-J.J., S.P., T.P. and S.C.; 2022M3F3A2A01072851 to H.J., S.-O.P., J.B., T.J., Y.C., S.S., H.-J.J., S.P., T.P., J.O., J.P., D.J.,


I.K. and S.C.; RS-2023-00209473 to S.H. and Y.-G.Y.; and 2020R1C1C1007464 to H.J., S.-O.P., T.R.K., J.B., T.J., Y.C., S.S., H.-J.J., S.P., T.P. and S.C.) and Institute of Information &


Communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (grant no. RS-2023-00216370 to K.K. and K.-H.K.). AUTHOR INFORMATION Author notes *


These authors contributed equally: Hakcheon Jeong, Seungjae Han. AUTHORS AND AFFILIATIONS * School of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST),


Daejeon, Republic of Korea Hakcheon Jeong, Seungjae Han, See-On Park, Tae Ryong Kim, Jongmin Bae, Taehwan Jang, Yoonho Cho, Seokho Seo, Hyun-Jun Jeong, Seungwoo Park, Taehoon Park, Young-Gyu


Yoon & Shinhyun Choi * Graduate School of Convergence Science and Technology, Seoul National University, Seoul, Republic of Korea Juyoung Oh * Department of Electrical and Computer


Engineering, Sungkyunkwan University, Suwon, Republic of Korea Jeongwoo Park * Electronics and Telecommunications Research Institute (ETRI), Daejeon, Republic of Korea Kwangwon Koh & 


Kang-Ho Kim * Department of Intelligence and Information, Seoul National University, Seoul, Republic of Korea Dongsuk Jeon * Inter-university Semiconductor Research Center, Seoul National


University, Seoul, Republic of Korea Dongsuk Jeon * Department of Radiological Science, Yonsei University, Wonju, Republic of Korea Inyong Kwon Authors * Hakcheon Jeong View author


publications You can also search for this author inPubMed Google Scholar * Seungjae Han View author publications You can also search for this author inPubMed Google Scholar * See-On Park


View author publications You can also search for this author inPubMed Google Scholar * Tae Ryong Kim View author publications You can also search for this author inPubMed Google Scholar *


Jongmin Bae View author publications You can also search for this author inPubMed Google Scholar * Taehwan Jang View author publications You can also search for this author inPubMed Google


Scholar * Yoonho Cho View author publications You can also search for this author inPubMed Google Scholar * Seokho Seo View author publications You can also search for this author inPubMed 


Google Scholar * Hyun-Jun Jeong View author publications You can also search for this author inPubMed Google Scholar * Seungwoo Park View author publications You can also search for this


author inPubMed Google Scholar * Taehoon Park View author publications You can also search for this author inPubMed Google Scholar * Juyoung Oh View author publications You can also search


for this author inPubMed Google Scholar * Jeongwoo Park View author publications You can also search for this author inPubMed Google Scholar * Kwangwon Koh View author publications You can


also search for this author inPubMed Google Scholar * Kang-Ho Kim View author publications You can also search for this author inPubMed Google Scholar * Dongsuk Jeon View author publications


You can also search for this author inPubMed Google Scholar * Inyong Kwon View author publications You can also search for this author inPubMed Google Scholar * Young-Gyu Yoon View author


publications You can also search for this author inPubMed Google Scholar * Shinhyun Choi View author publications You can also search for this author inPubMed Google Scholar CONTRIBUTIONS


H.J., S.H., S.-O.P., Y.-G.Y. and S.C. conceived this work. H.J., S.H., Y.-G.Y. and S.C. designed the experiments and overall simulation. H.J., S.-O.P., H.-J.J. and T.J. designed and


fabricated the memristor array. H.J., S.-O.P., J.B., S.S. and T.P. performed material analysis. H.J., J.B. and Y.C. conducted electrical measurement of the device. H.J., T.R.K., S.P., J.O.,


J.P., D.J. and I.K. designed the analogue computing unit. S.H. designed the video processing. H.J. and S.H. designed the real-time platform. H.J., S.H., K.K. and K.-H.K. conducted and


improved the video processing implementation. H.J., S.H., S.-O.P., Y.-G.Y. and S.C. prepared the manuscript. All authors contributed to the discussion and analysis of the results regarding


the manuscript. Y.-G.Y. and S.C. supervised the study. CORRESPONDING AUTHORS Correspondence to Young-Gyu Yoon or Shinhyun Choi. ETHICS DECLARATIONS COMPETING INTERESTS The authors declare no


competing interests. PEER REVIEW PEER REVIEW INFORMATION _Nature Electronics_ thanks Muhammad Khan, Kyusang Lee and Guangdong Zhou for their contribution to the peer review of this work.


ADDITIONAL INFORMATION PUBLISHER’S NOTE Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. SUPPLEMENTARY INFORMATION


SUPPLEMENTARY INFORMATION Supplementary Figs. 1–31, Discussion and Tables 1 and 2. SUPPLEMENTARY VIDEO 1 Real-time video foreground and background separation using the developed platform


with 1 K highly reliable selector-less memristor array. RIGHTS AND PERMISSIONS Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under


a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such


publishing agreement and applicable law. Reprints and permissions ABOUT THIS ARTICLE CITE THIS ARTICLE Jeong, H., Han, S., Park, SO. _et al._ Self-supervised video processing with


self-calibration on an analogue computing platform based on a selector-less memristor array. _Nat Electron_ 8, 168–178 (2025). https://doi.org/10.1038/s41928-024-01318-6 Download citation *


Received: 15 February 2024 * Accepted: 20 November 2024 * Published: 08 January 2025 * Issue Date: February 2025 * DOI: https://doi.org/10.1038/s41928-024-01318-6 SHARE THIS ARTICLE Anyone


you share the following link with will be able to read this content: Get shareable link Sorry, a shareable link is not currently available for this article. Copy to clipboard Provided by the


Springer Nature SharedIt content-sharing initiative