Intelligent Data Tiering and Automated Lifecycle Management for Enterprise Storage and Scalable Real-Time Analytics
Abstract
Enterprise data platforms need to consider the speed of access to frequently accessed data with the cost, capacity, longevity and latency of storage resources. Fixed placement policies and manually maintained retention schedule may drive up storage costs, introduce processing overhead, as well as postpone the analytical loads as volumes of data and access patterns evolve. In this paper, a conceptual framework of intelligent data tiering and automated lifecycle management that integrates metadata, access telemetry, business criticality, retention requirements and workload service-level objectives will be presented. Responses Data is profiled at the object, table, partition, or dataset level; estimates both access demand and lifecycle-state, chooses the right storage tier, and manages migration, retention, archival and deletion using policy-controlled processes. Caching, hot-tier protection, event-driven policy updates, and active query safeguards ensure a real-time analytics (data wording) journey. The paper evaluates the framework analytically based on decisions quality, latency, costs awareness, scalability, governance and recoverability. It does not purport to achieve measured performance gains; empirical validation on enterprise traces of representative size is yet to be demonstrated
Article Information
Journal |
International Journal of Science, Research and Technology |
|---|---|
Volume (Issue) |
Vol. 8 No. 1 (2025): International Journal of Science, Research and Technology (IJSRAT) |
DOI |
|
Pages |
13545-13554 |
Published |
February 22, 2025 |
| Copyright |
All rights reserved |
Open Access |
This work is licensed under a Creative Commons Attribution 4.0 International License. |
How to Cite |
Dr.R.Sugumar (%2025). Intelligent Data Tiering and Automated Lifecycle Management for Enterprise Storage and Scalable Real-Time Analytics. International Journal of Science, Research and Technology , Vol. 8 No. 1 (2025): International Journal of Science, Research and Technology (IJSRAT) , pp. 13545-13554. https://doi.org/10.15662/IJSRAT.2025.0801007 |
References
[2] Zaharia, M., Xin, R. S., Wendell, P., Das, T., Armbrust, M., Dave, A., Meng, X., Rosen, J., Venkataraman, S., Franklin, M. J., Ghodsi, A., Gonzalez, J., Shenker, S., & Stoica, I. (2016). Apache Spark: A unified engine for big data processing. Communications of the ACM, 59(11), 56–65. https://doi.org/10.1145/2934664
[3] Isard, M., Budiu, M., Yu, Y., Birrell, A., & Fetterly, D. (2007). Dryad: Distributed data-parallel programs from sequential building blocks. ACM SIGOPS Operating Systems Review, 41(3), 59–72. https://doi.org/10.1145/1272998.1273005
[4] Ghemawat, S., Gobioff, H., & Leung, S.-T. (2003). The Google file system. In Proceedings of the 19th ACM Symposium on Operating Systems Principles (pp. 29–43). ACM. https://doi.org/10.1145/945445.945450
[5] Chang, F., Dean, J., Ghemawat, S., Hsieh, W. C., Wallach, D. A., Burrows, M., Chandra, T., Fikes, A., & Gruber, R. E. (2008). Bigtable: A distributed storage system for structured data. ACM Transactions on Computer Systems, 26(2), Article 4. https://doi.org/10.1145/1365815.1365816
[6] Selvarajan, K. (2023). Optimizing large-scale data lakehouse architectures for performance and cost efficiency. International Journal of Research and Applied Innovations, 6(6), 10093–10101.
[7] Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., & Stoica, I. (2013). Discretized streams: Fault-tolerant streaming computation at scale. In Proceedings of the 24th ACM Symposium on Operating Systems Principles (pp. 423–438). ACM. https://doi.org/10.1145/2517349.2522737
[8] Akidau, T., Bradshaw, R., Chambers, C., Chernyak, S., Fernández-Moctezuma, R. J., Lax, R., McVeety, S., Mills, D., Perry, F., Schmidt, E., & Whittle, S. (2015). The dataflow model: A practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing. Proceedings of the VLDB Endowment, 8(12), 1792–1803. https://doi.org/10.14778/2824032.2824076
[9] Lakshman, A., & Malik, P. (2010). Cassandra: A decentralized structured storage system. ACM SIGOPS Operating Systems Review, 44(2), 35–40. https://doi.org/10.1145/1773912.1773922
[10] Stonebraker, M., Madden, S., Abadi, D. J., Harizopoulos, S., Hachem, N., & Helland, P. (2007). The end of an architectural era (it’s time for a complete rewrite). In Proceedings of the 33rd International Conference on Very Large Data Bases (pp. 1150–1160). VLDB Endowment.
[11] Pavlo, A., Paulson, E., Rasin, A., Abadi, D. J., DeWitt, D. J., Madden, S., & Stonebraker, M. (2009). A comparison of approaches to large-scale data analysis. In Proceedings of the ACM SIGMOD International Conference on Management of Data (pp. 165–178). ACM. https://doi.org/10.1145/1559845.1559865
[12] Agarwal, S., Kandula, S., Bruno, N., Wu, M.-C., Stoica, I., & Zhou, J. (2012). Re-optimizing data-parallel computing. In Proceedings of the USENIX Symposium on Networked Systems Design and Implementation (pp. 281–294). USENIX Association.
[13] Mell, P., & Grance, T. (2011). The NIST definition of cloud computing (NIST Special Publication 800-145). National Institute of Standards and Technology. https://doi.org/10.6028/NIST.SP.800-145
[14] National Institute of Standards and Technology. (2020). Security and privacy controls for information systems and organizations (NIST Special Publication 800-53, Rev. 5). U.S. Department of Commerce. https://doi.org/10.6028/NIST.SP.800-53r5
[15] International Organization for Standardization. (2016). Information and documentation—Records management—Part 1: Concepts and principles (ISO Standard No. 15489-1:2016). ISO.