Hi, I'm Senior Software Engineer, Team Leader and Solutions Architect with broad experience in the field of computer science. My expertise includes, but is not limited to:
I am available for ad-hoc consulting, presenting various topics to your employees, delivering functional PoCs, and implementing functional products.
My expertise in both Big Data technologies and security allows me to design and build Big Data processing systems that are both efficient and secure, helping companies improve their data management and analysis capabilities while also protecting their sensitive information. Whether it's implementing data lineage tools to track metadata or hardening security practices, I can help your company succeed in an increasingly data-driven world.
I possess a strong background in developing and managing ETL pipelines. I have expertise in various data processing technologies and have worked with various scheduling & orchestrating systems (be it for ML or not) to ensure efficient and reliable execution of these pipelines. I am also skilled with many NoSQL databases, streaming platforms, relational databases and distributed systems in general.
I have experience working with various table and file formats to optimize data storage and retrieval. Pretty much every company needs to solve problems such as schema evolution (with easy access to data with obsolete schemas) or to store and load the data efficiently to save costs. You might also make use of my ability to implement data lineage and integrating it with the metadata store.
I am also familiar with various data lakehouse platforms. These can bring you the benefit of efficient querying and analysis of data stored in various places, effectively consolidating the data to one place.
Are you stuck with long-running Spark jobs that are actually pretty expensive on your budget? I can optimize Big Data processes for you to solve common issues such as data skew or bottlenecks in Spark job stages.
Or you are having difficulty with obsolete Hadoop / Spark versions because of an dependency hell? I can also help you with that. For example, I can containerize your ETL setup, so it can run within orchestration platform of your choice (maybe Kubernetes) to reach maximal efficiency and bring in cost saving auto-scalability, while at the same time, leave Hadoop and use pure Spark cluster.
How do you let developers test their changes in dev environment? You give them raw production data? You see, a developer doesn't need to have real-world scenario in his testing data. He would be okay with data that are valid, while at the same time anonymized. Which also brings you an advantage of one less thing to care about when it comes to GDPR. Now, I can do that.
Do you have any software components that are facing the real world? Have they been tested for exploitable threats? And how about continuous security tests? Something that is secure today doesn't have to be secure tomorrow. Maybe new threat was discovered, or somebody changed configuration of the orchestration platform a little and suddenly, you have an intruder that encrypted all of your data and demands a payout. I don't want to scare you, but those are well known cases.
Once you let me analyze your setup, you get an extensive security report. I give you a security score based on my review of your cloud setup, product architecture designs, exploitable vulnerabilities and more. I will also review the development processes and the technologies that are used to run your programs. Comprehensive list of action items is included along with their severity.
I am experienced in orchestrating systems using technologies such as Kubernetes, Azure and Openshift, ensuring that your systems are running smoothly and efficiently.
I bring in the best practices from the containerized world not limited to only CI/CD or development security and environments stability. I can help you debug networking problems, assure the right access policies are set up throughout your company, help you set up third-party operators or simply implement and deploy a new one according to your needs.
Maybe you need to introduce new operator/CRD? It might not exist yet, so I can also write you one in Go and provide a Helm chart along with it. It will take good care of your application.
I have expertise in setting up stable monitoring environments, including defining and gathering metrics, creating alerting rules and establishing SLAs and SLOs. This helps you to keep track of your systems and ensure that they are operating at optimal performance.
It can also help you track your users activity and satisfaction as a side effect, further gaining more happy clients. You can use the solution to also inform the public about current outages and let your team focus on resolving the problem at the same time.
(Not only) as a solutions architect, I am proactive in proposing and implementing the best solutions for new projects. I am a fast learner, have a strong ability to understand the needs of a project and find the most appropriate and effective approach to meet those needs. Overall, I have a wealth of experience and expertise not only in the field of Big Data, but also in the ever expanding world of microservices and relational databases, monitoring, security, but also raw hardware and low-level programming thanks to the college I've studied in (Electrical Engineering, Automation). I am dedicated to continue learning and growing in this dynamic and rapidly-evolving field of software engineering world.
Wide range of portfolio allows me to blend all my skills together. For example, I make sure that data are well protected and systems providing them are secure while at the same time my experience with Big Data allows me to extract valuable insights from large datasets reliably and cost-effectively thanks to my expertise in microservices and monitoring tools.
With me, you get all the perks, at once.
If we were to talk only about Big Data, completely ignoring my other skills, I can still improve your data management and analysis capabilities by creating ETL pipelines to extract, transform, and load data from various sources into a structured format that can be easily queried and analyzed.
For example I would incorporate Flink processes with the integration of data lineage using Marquez to track the movement and transformation of the data throughout Kafka up to the consumer. It would be simple and efficient, but robust Big Data processing system.
In the end, you are able to analyze and gain insights from your data more effectively, potentially leading to improved decision-making and better business outcomes.