Distributed Data Engineering Frameworks for Real-Time Healthcare Record Integration
Abstract
Real-time distributed data engineering enables the concurrent ingestion, processing, and provisioning of records from multiple sources throughout both the internal and external distributed systems of healthcare organizations. A widely dispersed set of assets is harnessed: Electronic Health Records (EHRs) must be translated into a common set of clinical concept identifiers; external systems providing timely information such as medication prescribing and dispense events, clinical pathology results, and respiratory disease hazard alerts must be interfaced; and continually streaming internal sensor data are made available for clinical decision support.
Central to real-time data integration in data engineering are the principles of consistency, time versus throughput balance, and fault tolerance within a regulated domain. The low-latency ingestion and streaming processing patterns that underpin these solutions are described in terms of their event-based schema, on-the-fly schema evolution capabilities, and the inherent need for backpressure. Particularly for Electronic Health Records (EHRs), the key model elements are defined in terms of patients and their medical contacts with the institution, the observable clinical facts collected over time, and the medications associated with those facts, with a strong focus on recognition and value assignment through standardized terminologies such as SNOMED CT.
Article Information
Journal |
International Journal of Science, Research and Technology |
|---|---|
Volume (Issue) |
Vol. 3 No. 6 (2020): International Journal of Science, Research and Technology (IJSRAT) |
DOI |
|
Pages |
4659-4673 |
Published |
December 5, 2020 |
| Copyright |
All rights reserved |
Open Access |
This work is licensed under a Creative Commons Attribution 4.0 International License. |
How to Cite |
Dileep Valiki (%2020). Distributed Data Engineering Frameworks for Real-Time Healthcare Record Integration. International Journal of Science, Research and Technology , Vol. 3 No. 6 (2020): International Journal of Science, Research and Technology (IJSRAT) , pp. 4659-4673. https://doi.org/10.15662/IJSRAT.2020.0306002 |
References
[2] Amistapuram, K. Energy-Efficient System Design for High-Volume Insurance Applications in Cloud-Native Environments. International Journal of Innovative Research in Electrical, Electronics, Instrumentation and Control Engineering (IJIREEICE), DOI, 10.
[3] Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A distributed messaging system for log processing. Proceedings of the NetDB Workshop.
[4] Stonebraker, M., Çetintemel, U., & Zdonik, S. (2005). The 8 requirements of real-time stream processing. ACM SIGMOD Record, 34(4), 42–47.
[5] Carbone, P., Katsifodimos, A., Ewen, S., et al. (2015). Apache Flink: Stream and batch processing in a single engine. IEEE Data Engineering Bulletin, 38(4), 28–38.
[6] Inala, R. Designing Scalable Technology Architectures for Customer Data in Group Insurance and Investment Platforms.
[7] Kleppmann, M. (2017). Designing data-intensive applications. O’Reilly Media.
[8] Newman, S. (2015). Building microservices. O’Reilly Media.
[9] Pahl, C. (2015). Containerization and the PaaS cloud. IEEE Cloud Computing, 2(3), 24–31.
[10] Varri, D. B. S. (2020). Automated Vulnerability Detection and Remediation Framework for Enterprise Databases. Available at SSRN 5774865.
[11] Fielding, R. T. (2000). Architectural styles and the design of network-based software architectures. Doctoral dissertation, University of California.
[12] Dean, J., & Ghemawat, S. (2008). MapReduce: Simplified data processing on large clusters. Communications of the ACM, 51(1), 107–113.
[13] Tanenbaum, A. S., & Van Steen, M. (2017). Distributed systems (2nd ed.). Pearson.
[14] Bass, L., Clements, P., & Kazman, R. (2013). Software architecture in practice (3rd ed.). Addison-Wesley.
[15] Gottimukkala, V. R. R. (2020). Energy-Efficient Design Patterns for Large-Scale Banking Applications Deployed on AWS Cloud. power, 9(12).
[16] Fowler, M. (2018). Patterns of enterprise application architecture. Addison-Wesley.
[17] Bender, D., & Sartipi, K. (2013). HL7 FHIR: An agile and RESTful approach to healthcare information exchange. Proceedings of CBMS, 326–331.
[18] Davuluri, P. N. (2020). Improving Data Quality and Lineage in Regulated Financial Data Platforms. Finance and Economics, 1(1), 1-14.
[19] Lehne, M., et al. (2019). The use of FHIR in digital health. Studies in Health Technology and Informatics, 267, 52–58.
[20] Rongali, S. K. (2020). Predictive Modeling and Machine Learning Frameworks for Early Disease Detection in Healthcare Data Systems. Current Research in Public Health, 1(1), 1-15.
[21] Jensen, P. B., Jensen, L. J., & Brunak, S. (2012). Mining electronic health records. Nature Reviews Genetics, 13(6), 395–405.
[22] Shickel, B., Tighe, P. J., Bihorac, A., & Rashidi, P. (2018). Deep EHR. IEEE Journal of Biomedical and Health Informatics, 22(5), 1589–1604.
[23] Singireddy, S., & Adusupalli, B. (2019). Cloud Security Challenges in Modernizing Insurance Operations with Multi-Tenant Architectures. International Journal of Engineering and Computer Science, 8, 12.
[24] Esteva, A., et al. (2019). A guide to deep learning in healthcare. Nature Medicine, 25(1), 24–29.
[25] Raghupathi, W., & Raghupathi, V. (2014). Big data analytics in healthcare. Health Information Science and Systems, 2, 3.
[26] Sittig, D. F., & Singh, H. (2016). A socio-technical approach to EHR-related safety hazards. Journal of the American Medical Informatics Association, 23(4), 641–647.
[27] Vadisetty, R., Polamarasetti, A., Guntupalli, R., Rongali, S. K., Raghunath, V., Jyothi, V. K., & Kudithipudi, K. (2020). Generative AI for Cloud Infrastructure Automation. International Journal of Artificial Intelligence, Data Science, and Machine Learning, 1(3), 15-20.
[28] Wilkowska, W., & Ziefle, M. (2012). Privacy and data security in e-health. Health Policy and Technology, 1(3), 119–131.
[29] Abouelmehdi, K., Beni-Hessane, A., & Khaloufi, H. (2018). Big healthcare data security. Journal of Big Data, 5, 1.
[30] Inala, R. (2020). Building Foundational Data Products for Financial Services: A MDM-Based Approach to Customer, and Product Data Integration. Universal Journal of Finance and Economics, 1(1), 1-18.
[31] Berwick, D. M., Nolan, T. W., & Whittington, J. (2008). The triple aim. Health Affairs, 27(3), 759–769.
[32] Porter, M. E. (2010). What is value in health care? New England Journal of Medicine, 363(26), 2477–2481.
[33] Pope, G. C., et al. (2004). Risk adjustment of Medicare capitation payments using the CMS-HCC model. Health Care Financing Review, 25(4), 119–141.
[34] Rose, S. (2016). A machine learning framework for plan payment risk adjustment. Health Services Research, 51(6), 2358–2374.
[35] Braverman, A., et al. (2020). Deep learning for Medicare risk adjustment. Health Services Research, 55(1), 68–77.
[36] Adusupalli, B., Pandiri, L., & Singireddy, S. (2019). DevOps Enablement in Legacy Insurance Infrastructure for Agile Policy and Claims Deployment. risk, 7(12).
[38] Breunig, M. M., et al. (2000). LOF. Proceedings of SIGMOD, 93–104.
[39] Liu, F. T., et al. (2008). Isolation forest. Proceedings of ICDM, 413–422.
[40] Kingma, D. P., & Welling, M. (2014). Auto-encoding variational Bayes. ICLR.
[41] Goodfellow, I., et al. (2014). Generative adversarial nets. NeurIPS.
[42] Kipf, T. N., & Welling, M. (2017). Graph convolutional networks. ICLR.
[43] Hamilton, W., Ying, Z., & Leskovec, J. (2017). GraphSAGE. NeurIPS.
[44] Segireddy, A. R. (2020). Cloud Migration Strategies for High-Volume Financial Messaging Systems.
[45] Wu, Z., et al. (2020). A comprehensive survey on graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 32(1), 4–24.
[46] Grover, A., & Leskovec, J. (2016). node2vec. Proceedings of KDD, 855–864.
[47] Perozzi, B., et al. (2014). DeepWalk. Proceedings of KDD, 701–710.
[48] Paszke, A., et al. (2019). PyTorch. Advances in Neural Information Processing Systems, 8024–8035.
[49] Pedregosa, F., et al. (2011). Scikit-learn. Journal of Machine Learning Research, 12, 2825–2830.
[50] Akhtar, N., & Mian, A. (2018). Adversarial attacks. IEEE Access, 6, 14410–14430.
[51] Finlayson, S. G., et al. (2019). Adversarial attacks on medical machine learning. Science, 363(6433), 1287–1289.
[52] Ribeiro, M. T., et al. (2016). Why should I trust you? Proceedings of KDD, 1135–1144.
[53] Lundberg, S. M., & Lee, S.-I. (2017). SHAP. Advances in Neural Information Processing Systems, 4765–4774.
[54] Rudin, C. (2019). Stop explaining black box machine learning models. Nature Machine Intelligence, 1, 206–215.
[55] Molnar, C. (2019). Interpretable machine learning. Lulu.
[56] Pandiri, L., Singireddy, S., & Adusupalli, B. (2020). Digital Transformation of Underwriting Processes through Automation and Data Integration. Global Research Development (GRD) ISSN, 2455-5703.
[57] Johnson, A. E. W., et al. (2020). MIMIC-IV. Scientific Data, 7, 412.
[58] Haux, R. (2010). Medical informatics: Past, present, future. International Journal of Medical Informatics, 79(9), 599–610.
[59] Topol, E. (2019). Deep medicine. Basic Books.
[60] Beam, A. L., & Kohane, I. S. (2018). Translating artificial intelligence into clinical care. New England Journal of Medicine, 379(23), 2165–2167.
[61] Obermeyer, Z., & Emanuel, E. J. (2016). Predicting the future. New England Journal of Medicine, 375(13), 1216–1219.
[62] Wiens, J., et al. (2019). Do no harm. Nature Medicine, 25(9), 1337–1340.
[63] World Health Organization. (2016). Global diffusion of eHealth. WHO.
[64] OECD. (2019). Health in the 21st century. OECD Publishing.
[65] European Commission. (2020). European data strategy.
[66] Kuo, A. M. H. (2011). Cloud computing in healthcare. Journal of Medical Internet Research, 13(3), e67.
[67] Buyya, R., et al. (2010). Cloud computing and emerging IT platforms. Wiley.
[68] Amor, D., et al. (2020). Data governance in healthcare. Health Informatics Journal, 26(2), 1167–1181.
[69] Mans, R., et al. (2008). Application of process mining in healthcare. Information Systems, 33(4–5), 389–406.
[70] Rojas, E., et al. (2016). Process mining in healthcare. Journal of Biomedical Informatics, 61, 224–236.
[71] van der Aalst, W. (2016). Process mining (2nd ed.). Springer.
[72] Dumas, M., et al. (2018). Fundamentals of business process management (2nd ed.). Springer.