Data Engineering Reddit - Worst Data Engineering Mistake youve seen? : r/dataengineering.

New comments cannot be posted and votes cannot …. The stuff you mentioned, Spark, Hadoop, streaming, etc. Worked as a data analyst from 2010 to 2014. In Big Data, functional-relational mapping is better than object relational mapping:. I am see in the UK that in general. Maybe you’re rebuilding a car or perhaps you love your car but there’s a problem with the existing eng. Create frameworks, not pipelines. Lambda - A very cheap way to run short scripts in Python (or other languages), and have them trigger in response to either events you specify or on a schedule, without having to configure servers. palm springs weather radar Max run time of 15 minutes, limited storage but sufficient for a lot. I've become the technical lead on my team. Typically it is a data engineer who is either 10+ YoE, vast experience outside data engineering (general software, security, infra, cloud) or just a proven track record of being a highly productive problem solver. In industry it's better IME, working ~37hrs/wk and usually taking lunch currently. Are you looking for an effective way to boost traffic to your website? Look no further than Reddit. dtc u0073 chevrolet Data science is more than the tools you use. Data Science from Scratch by Joel Grus. Data Science is the most desired skill set. New comments cannot be posted and votes cannot be cast. (You may be able to do OMSCS and get a masters in CS as well. It seems I’ll be working with customer engagement data, and providing visualizations and metrics to Amazon site sellers. craigslist general help wanted I always wanted my manager to understand that while creating a new etl …. AWS Managed Airflow = Airflow but like 3 versions behind. Note: We run a small warehouse size on Snowlfake. Type 3: Data platform engineer. I wrote a series on Reddit called From a Beginner to Beginners documenting my entire programming journey. DE is tightly coupled with distributed systems. The problem is it's harder to start as DevOps than as Data Engineer (at least the Junior DE vacancies I see outnumber the Junior DevOps). As our project grows, we've seen first-hand how difficult it is for others to contribute to open-source: from setting up the development environment, understanding the codebase, drafting a PR, etc. Note that datasets can be unbounded streams (ie a stream of incoming data). Use of the Internet and networking is essential for advancing research in science, medicine. This is a recurring thread that happens quarterly and was created to help increase transparency around salary and compensation for Data Engineering. At a big tech company typically the first round is technical and tests you on SQL and/or python. Anyway, I think you are going the right path. that people are mentioning, it’s a very good idea to run your own database server locally, put some data in it and query it. Complete learning path for data engineer with best books, best courses and best free resources for every subject in the path. Data Engineering is a means to get to Data Science, which is means to get to Decisions. Type 2 is recently (or not recently) named as an analytics engineer. Because it will help you understand data from the source side. Quite honestly, the experience I gained there was much more overrated than what I expected the role to be at the time. But when you hear about data engineering, immediate change in attitude. Most people love talking about themselves. I've seen some SQL questions on a few reddit subs where what they are trying to do is a. If it’s a data engineering role that you want, try and highlight more of the data related tasks from that full stack position. In that case, I would also say that you're doing Data Engineering. After going through it, I felt like course does cover a lot of great info specifically for GCP. If the company that you're interviewing for has no difference between a data engineer and a software engineer then it's better to prepare for a typical software engineer interview. Find a team/company that have diverse data roles so you can focus on data “engineering”. The course is broken up into five sections, Data Modeling, Cloud Data Warehouses, Data Lake with Spark, Data Pipelines with Airflow, and a capstone project. Data Science and Data Engineering in the PH. How data engineering is defined and what are the roles and responsibilities of a Data Engineer. Are you looking for a new engine for your vehicle? Whether you’re replacing an old engine or upgrading to a more powerful one, finding the perfect engine for your vehicle can be a. The only really appealing paths are from SWE if you like data/backend, and from BI/ETL/database engineering because that area has relatively low salaries (and people who like data/backend). I also see each time more DE roles evolving to reach some maturity in terms of software development, whereas much of the evolution come from backend best practices. Junior data engineer £25k - £35k Experienced data engineer £30k - £45k Lead / Principal data engineer £45k - £60k. You could easily become a DE from there because as an AE you’ll run into all sorts of DE problems. you break i fix york pa The skills will topple a cheap cert. The data is then processed in real-time and stored to the data lake periodically (every two minutes). Tools like ChatGPT, or their open-source equivalents, have made unstructured data, such as PDFs and DOCX files, exploitable on a grand scale. Then you add the infrastructure (k8s, cloud, etc. Data engineering is such a broad subject. Trigonometry is used by engineers, medical services technicians, mathematicians, data entry specialists, loggers, statisticians, actuaries, drafters, chemists, economists, physicis. You should pick a cloud environment, automate a data pull from an API, and create an analytics layer out if it, and throw in on your resume. Many BI roles are shifting towards hybrid roles where also engineering is an aspect especially in small teams. Data engineering isn’t just spinning up some etl process and boom your pipeline is done lol, even at companies with very mature and complex data architectures and internal tooling it takes a lot of engineers to maintain those tools and …. But its a tool, the underlying concepts of data lineage, data quality. Also other many components for example Apache projects are also developed with java and scala. DS skills aren’t necessary for data engineering, but to be fair, a unicorn hire for data teams is a senior data engineer who is also a senior data scientist. Be it Azure, gcp AWS or databricks cloud, they provide managed cluster platforms. On Reddit, people shared supposed past-life memories. The only additional topic was security related, which is both Zero Trust Architectures and Policy Based Access. Related Data engineering Engineering Computer science Sciences Applied science Information & communications technology Formal science Science Technology forward back r/InternationalDev A forum to discuss matters relating to International Development, encompassing themes such as poverty, education, global health, conflict, gender equality. Here’s everything you need to know about crate engines s. I have started to implement very simple project, small automation, cli apps etc. "rcc transfer agreements" Preparing more can make me confident, I reckon. Scaling the engineering or product teams are a challenge too. Only tables with the sensitive value are locked down from general business use. It emphasizes the valid and efficient collection, storage, management, and processing of datasets to support computation and data driven systems important to data science and data analytics functions. It can help to improve the quality and reliability of your data, make it easier to collaborate with others, and reduce the time and effort required to build and maintain your data pipelines. Analytics Intern under a research team in university: • To support on labelling and processing of various industry-related data • To organise various datasets to specified requirements for use as input to predictive machine learning • To contribute and feedback on insights regarding accuracy, completeness and organisation of datasets. Attention to detail is very important. They have a Data Science, and Engineering track (and you can do more than one) but not specifically Data Engineering, and they have real life projects to practice skills which you can check out before signing up. My favorites are Arrow, Airflow, Hudi, Druid, Iceberg, Flink, NiFi, Cassandra. Here’s my career progression, to show you how I made it from $40k to $287k in 7 years: Salary: $40k. How to Become a Data Engineer in 2023: 5 Steps for Career Success. The best guys generally move to the US or are in good companies. As long as you have a few years of experience, you shouldn't have too much trouble finding something. Talend would generate the Java code and submit it to a spark cluster, and they have SCD tools that would generate spark compatibile code. “ML” and “AI” is 98% data engineering. Related Data engineering Engineering Computer science Sciences Applied science Information & communications technology Formal science Science Technology forward back r/MLQuestions A place for beginners to ask stupid questions and for experts to help them! /r/Machine learning is a great subreddit, but it is for interesting articles and news. Right from data acquisition to delivering a modeling data base or to a data pool, data engineering skills are best evaluated on effort made to understand and cleanse missing records why and how they were cleaned. Maybe run a remote jupyter notebook server. Chances for BEng Electronics and Data Engineering AY2023/24. Keep it to 1 page unless you have more than a decade of direct experience in high level roles that NEEDS to be. News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL, distributed systems, streaming, batch, Big Data, and workflow engines. Unexpectedly, I find myself fending off a hostile takeover from a leader on the engineering team, who is declaring that data engineering needs to be moved in with the engineering org. As a Senior Data Engineer I have hired new grads for titled Data Engineering positions, but it is less common. reddit's new API changes kill third party apps that offer accessibility features, mod tools. And there is huge potential for data to improve things (identity bottlenecks, wrong decisions) or even predict things and eventually automatically control them. Additionally, AI such as co-pilots and no-code environments make this domain even more competitive. I also thought whats special about it some time ago. Pure functions are functions that take inputs and always return the same outputs without any “side effects”. A lot of csp data engineering/etl services are built on open source foundation such as GCP cloud composer or AWS mwaa. General unemployment rate is still 3. During the search, I realised that maybe the data science field is kinda saturated, so I want to know if data engineering is a good career choice. CSCareerQuestions protests in solidarity with the developers who made third party reddit apps. Data Engineering your bread and better is sql and etl scripts. In the EU, I would risk to say that 99% of the positions like Data Engineer and alike, English will be enough. For people who want to get into Data without fighting tooth and nail for a CS spot in Etam lol. Hi all, I am data engineer with more than one year of experience. Our data engineering team is in a different department, but their manager is also great to work with, so I don't think culture is a huge factor here. Wᴇʟᴄᴏᴍᴇ ᴛᴏ ʀ/SGExᴀᴍs – the largest community on reddit discussing. After 6 month I got another interview and I got asked more complex questions, one python problem (no complex until I had to reduce the time complexity. Your end goal is to get a job as a data engineer and you're going to do that with an awesome personal project. python doesn't enforce these concepts fully so it will be useful to pick up java. al harameen city codes pdf You will need extensive git experience, devops, computer networking. Now I have/want to deal with AWS. With millions of active users and countless communities, Reddit offers a uni. 2017 - DS is not enough, Machine Learning is the most desired skill. From zero to job-ready in 5 months. Docker is key for packaging environments for data pipelines that are runnable and tested for both local development and production deployment. With the increasing number of cyber threats and data breaches, it is essentia. value of old pepsi machine IMO, this is almost always the correct answer to these types of questions. The hate comes from the fact that the modern data stack lowered the barrier to entry in the data engineering field, and as a result you ended up with Analysts/Data Scientists/BI people building poorly designed data models with its help (aided by misleading marketing from some of the MDS companies). Data Engineering as fallback once the LLM hype dies down? I am facing quite a lot of anxiety about the DS field right now. WallStreetBets founder Jaime Rogozinski says social-media giant Reddit ousted him as moderator to take control of the meme-stock forum. When I joined the company my first rotation was in what I thought could be described as a data engineering role, despite not officially having that title. EC2 - A virtual server where you can run code. sounds like a challenging and interesting. Good for small to medium sized workloads. In the Bay Area, a decent data engineer makes almost the same as a software engineer (back-end/front-end), full-stack makes a bit more. Related Data engineering Engineering Computer science Sciences Applied science Information & communications technology Formal science Science Technology forward back r/CompTIA From the "looking to get certified," to conversations/questions from current students, to certified and working professionals - this subreddit is dedicated to CompTIA. LeetCode easy/medium and focusing on string and dictionary problems seems to be 80% of the Python questions I've seen. Here is some specific advice I gave on programming and getting real progress on goals. This could be an insanely good opportunity to grow. All you need for basic data engineering is the ability to source data, manipulate data, store data and automate the process. Redshift = Expensive Greenplum. Many companies (outside of FAANG) which want to become data driven will just hire data scientists believing that they can solve every data problem because it's the only data profession they heard about in the news and social media. reddit's new API changes kill third party apps that offer. Data scientists build statistical and predictive models. Solution: make a mouse clicker script or the like. 95% of data engineering is done in Java. LinkedIn - There are definitely some people worth following (Zach Wilson, Seattle Data Guy, etc. Rust does have a lot of momentum and a great community, but it is somewhat of a chicken and egg problem. Why don't you check out Edureka's Data Engineering course. Podes verlo cómo que el DataEngineer es la versión. That said, it ultimately depends on what role you’re targeting. dbt offers integration with lots of other services either its enterprise or not and also its open source. However, I chose Edureka's Data Engineering Masters program because it offers hands-on learning with real-time projects and excellent instructor support. Data Engineering is becoming as integral as accounting or marketing. Most big data tools are developed around JVM languages for a reason. If you have at work Hadoop or cloud environment - it's the best. Tech Stack — Python, Spark, Airflow, API Services. Did you make this move in India and internally to your old company or you switched to a different company. Will Data science be replaced by AI. At least for the next 5 - 10 years. 4% which means still a job applicant’s market (healthy economy is usually 3-5%). Like for example NBA Analytics, connect ka to API then webscrape, data cleansing and prep, tapos load mo sa SQL gawan mo na din ng data modeling and warehouse tapos data visualization. Long story short, I run a data team at a 100 person company that includes Data Science and Data engineering. These are user-facing and so may support multiple redundant sets of data with different partitioning schemes, aggregate tables, etc. /r/Statistics is going dark from June 12-14th as an act of protest against Reddit's treatment. Handles dependencies, tests, documentation all in a declarative manner. I'm looking for advice on 1) how likely is someone to get involved in any entry level data science or associate data science position with an unrelated mechanical engineering degree, after a few months of learning Python, Ruby, or R and learning a bit about algorithms in general (my plans for the next few months, as well as applying to jobs in. Our team was frustrated with Lucid chart and Word so we built 2 tools to use internally: - ER Diagram: https://dbdiagram. Data Engineer needs to provision compute (AWS EC2, EMR, Lambda) to move data and the provision data stores to store that data, e. Cover letters are your opportunity to show your intangibles and your personality- and why you are a good culture fit for an organization (while resumes demonstrate fitness for non-soft skills). Cloud storage - I'm more familiar with AWS so things like S3, Glacier, etc. Related Data engineering Engineering Computer science Sciences Applied science Information & communications technology Formal science Science Technology forward back r/bashonubuntuonwindows This is the Windows Subsystem for Linux (WSL, WSL2, WSLg) Subreddit where you can get help installing, running or using the Linux on Windows …. Crate engines are a great way to get your car running again, but there are a few things you should know before you buy one. The basics should be the same anyway) Learn Data Structures. orange county jury duty excuse It's by far the most common target for non-CS people trying to break into the tech world because it's perceived as easier to self-teach compared to conventional software engineering. There’s more to life than what meets the eye. This supports you anywhere and makes you understand the tools you use better and solve the problems with data easier. The majority of my work has been designing a data product with Python used for marketing segmentation/QA and writing/running ETL pipelines with Python or PySpark + …. If your company outsources to India, chances are the guys working on it are . Because the tooling and ecosystem has become more mature, more companies are integrating BI / DS into their company strategy (e. Building a Data Engineering Project in 20 Minutes. And like OP os concerned about, adds tons of failure points and unnecessary architecture. I learned SQL by diving into SQL projects someone else did, and modifying them. Java is the best language to learn object orientation since it uses every principle such as variable types, encapsulation, polymorphism and design patterns that are useful to not just software developers but data engineers. In today’s digital age, online security has become a top concern for individuals and businesses alike. However, for many data engineering projects, the benefits of using dbt are clear. Even though it's more expensive, it seems like a much better deal to me than Data Engineer Academy. Data Engineering definitely can seem more boring and less analytical. Typically, you will be working with Big Data, compiling reports, and sending them to data scientists for study in this capacity. - get the team's info that you are or will be or aspire to work with: Data engg for analysts and BI teams or Data Engg for AI/ML teams. Data engineering is the practice of designing and building systems for collecting, storing, and analyzing data at scale. If you want to understand better trendy concepts: Modern Data Stack, Data Mesh, Analytics Engineering you can start by reading those articles. A DE is not a business analyst. News & discussion on Data Engineering topics, including but not . If you just google Data Engineer Jobs in Linkedin and set the location for the EU, you will find over 33k jobs. To summarize: create a supportive environment for continuous improvement and development. Not sure how important Japanese ability is for data jobs, but it isn't important for many SWE roles. BlackBerry said Monday that it wasn't aware of "any material, undisclosed corporate developments" that could rationally fuel its rally. Using the ibm cloud for the interactive components was ok. And yeah, this are just some of the best practices for data quality engineering. A data engineer manages the data sets themselves and develops pipelines to move data from operational databases into analytical databases. The target audience for this is not those "breaking into data engineering. Data engineering is more popular than DS. Choosing between Data Engineering and Data Science Offer. Given a choice between converting a systems engineer or a data scientist to a data engineer, I’d take the systems engineer 10 times out of 10. There is a lot info online and frankly it feels a bit overwhelming to select the top tools that are being used or the "right" skillset, therefore was hoping that a bootcamp will help me. But, the software engineering that data engineering exposes you to is not the same as other careers. Your start and end dates look backwards for the full stack role. But deciphering what the data means to the business seems like an overreach. q13 news live Data engineering is a practical/applied field that draws from fundamental computer science concepts. Get all the skills and knowledge you need to become a data engineer. With python you can handle transforms not possible with SQL: very complex logic, leveraging third-party libraries (ex: convert all possible IPv6 formats to a single consistent format). As for work experience, I started working during my second year of Bachelor's as a Software. This is valuable experience even if you pivot to the data science role in the future. Engineering Excellence: Dive into the world of Data Engineering and discover how it structures the data ecosystem for optimal storage, processing, and retrieval. I'd bet AWS has a similar cred and I know GCP does. The former is too hard to scale as the data engineers end up needing to understand every domain at the company. This is most informative infographic about data engineering. Understanding at a basic level how and why joins work lends itself to general knowledge of how data is stored and why it’s separated into different logical entities. We use Macs for our workstations because company policy is Windows or Mac for ease of fleet management and IT support. Is the allure of data science compared to data engineering a function of discovering answers and solving problems compared to engineering a system someone else will use? More importantly however, the behavior of reddit leadership in implementing these changes has been reprehensible. You can run almost all the same “cloud” services on prem using open source tools. I would stick with CS --> DE (2 to 3 years) --> slowly move into Cybersecurity. Jump to The founder of WallStreetBets is sui. You are literally 90% of the way there. Reddit is a popular social media platform that has gained immense popularity over the years. Doing Data Science by Cathy O'Neil. Due to advances in off-the-shelf SaaS products, the entry level DE simply isn't needed anymore, as analysts can do most of it after an initial set up by an infrastructure engineer or software engineer masquerading as a DE temporarily. Greetings everyone, I am a Data Engineer with approximately three to four years of experience in this domain. - get their tools info - master those tools and master the most used DevOps tools. It's still a personal preference I believe. what happened to gabriella premus This is a Fakespot Reviews Analysis bot. Ang point lang is makita nila may initiative to learn ka on your own and may experience ka na developing data pipeline. Also, some data processing is too complicated for SQL or some simple Python code. Data engineering has many specific roles depending on the business but is ultimately to build data pipelines to process data for downstream use cases (machine learning etc) CSCareerQuestions protests in solidarity with the developers who made third party reddit apps. Store that data in a database and model it. So an industrial engineer typically will not be exposed to this side of things unless you were part of IT organization of several years. You can find the latest post with all of the links here. I am wondering if it is even okay for a data engineer to be deciphering meaning from the data. Valheim; Genshin Impact; Minecraft; News & discussion on Data Engineering topics, including but not limited to: data pipelines, databases, data formats, storage, data modeling, data governance, cleansing, NoSQL. limestone chapel The speed sensor is a crucial component. I would say that you'd be pretty hard pressed to avoid mathematics in data engineering. I also create data modelling, develop and define metrics, ETL documentation, and do a lot of data. reddit's new API changes kill third party apps that offer accessibility features, mod. Company 1 Title: Software Programmer. For issues where an ID is a primary/foreign key like an SSN, An approach I've seen is to use a "linkage spine". In my experience it's been SQL (usually pretty easy), algorithms and data structures, and system design. fortnite r34 A good understanding of how databases work will make your life as a Data engineer much much easier. With millions of users and a vast variety of communities, Reddit has emerged as o. For inside London it can be very variable, if you’re an excellent DE also with excellent communication & networking skills you can get very good salaries. To engage with some new technologies, you should try a project like sspaeti’s 20 minute data engineering project. They should have a few years of SQL and Python experience. where is monkey kaka from The results of each user is shared within 8-10 secs after the data criteria is shared. The high supply has made salaries for DS lower than DE (this is in UK btw). AWS EC2 - cloud server, compute power and how to make use of it. UCSandiego has some courses on distributed computing systems (Big Data specialization). It is a broad field with applications in just about every industry. From my research it appears that Azure is easier to work with since it’s GUI based and is heavy on T-SQL. I avoid using python to do data transformations unless it's necessary to load into the DW, like for xlsx files. Related Data engineering Engineering Computer science Sciences Applied science Information & communications technology Formal science Science Technology forward back r/uwo A subreddit for students, faculty, staff, and alumni at Western University in London, Ontario, Canada. As a data engineer I did: I'd say 60-70% of the job was Data Engineering, though. One powerful tool that can help. Others may prefer the stability and higher income of a software engineering career. Its a neat mix of software development, devops, and data science. Check for tech blog posts and see if they're talking about data engineering concepts or projects at all. AWS data engineering certifications. The way I am thinking myself is that I have two possible paths, (1) go towards a more architect role in a consulting company, meaning more of a technical sales role including tech selection, drawing archtecture blueprints and not …. Alternatively, if you have experience in software development and database design, you might consider a career in data engineering. Now, really, it’s a data science degree. DEs also are closer to Data Scientists as well. Data Engineering covers a wide berth of skillsets. pick a cloud and figure out the main components needed for DE work. easiest entry level would be help desk. There is nothing preventing a no code GUI being built around it. So yeah, I would say your job goes a bit beyond Data Analysis. A data scientists is looking to make a new discovery a data engineer is focused on making things functional. The three have one data engineer certification. The official Python community for Reddit! Stay up to date with the latest news, packages, and. IMO, if you already have Python in your tool set, Java probably won't add a ton of value compared to other tools unless your end goal is to move into software engineering. Layoffs in tech were bigger than other sectors but it’s still not a bad market, there’s just not frenzied capital waiting to be spent. module test 1 In data science, if you want something static typed, Java or even rust is a better choice than Go. If you follow this, you are 100% going to burn out. I've moved to Data Architect from Data Engineering, and envision more opportunities in Data Engineering than regular software development. It's most powerful feature is the ability to write dynamic SQL in templates. Then you get “real” feedback from a real database on whether your query does what you intended it to do. Pipelines, platform, infra, BI, analysis, Databases, ML, Cloud, AI a DE can be involved in all these, with a focus on the ETL pipeline. A lot of data engineering is python/sql, which transfers nicely across platforms. lyndie dupuis new husband The way I am thinking myself is that I have two possible paths, (1) go towards a more architect role in a consulting company, meaning more of a technical sales role including tech selection, drawing archtecture blueprints and not participating so much in the actual coding/implementation. Check out Quantium, Serbian and Experian too. reddit's new API changes kill third party apps that offer accessibility features, mod tools, and other features not. I was dead set on building a Kappa architecture where everything lives in either Redis, Kafka, or Kinesis and then I learned the basics of how to build data lakes and data warehouses. DS/MLE are the sexiest because they produce product defining technologies and/or ridiculous cost savings / business insight at scale. It’s so weird to me because as I was learning data over the past like 8 years, everywhere I go it was like “Kimball method/star schema is a mature approach to data warehousing that is widely accepted as an industry standard”, but then when I got into consulting it was like everyone just did one-off reports and would say things like “it. The market for experienced DS professionals is still good, but the …. Data Scientist vs Data Engineer Salary: According to a review by glassdoor, you may make up to $137,000 per year as a data scientist. Python is a great language for transforming data. Engineering is all about efficiency, and what could be more efficient than learning a course online in a way that fits your lifestyle? Some courses are more expensive than others,. Data engineering is not only about reading some data, applying some transformer and dump it somewhere else. owner financed homes in nc With SQL joins between tables and aggregations are much less work. Kafka is a solution for large-scale problems that tends to get used for small-scale problems. "Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing" by Reuven Lax, Slava Chernyak, and Tyler Akidau. The principles of software engineering are applied in the field of DE and a backend developer can easily switch to data engineering and vice versa, Now there are some cases where switching fields in software development can be a little bit annoying and difficult, for example an android developer would find it a bit. You can have a weekly 30-minute one-on-one sessions for discussing challenges and sharing knowledge wrt leadership. works great with heavy loads, good std lib, enough 3rd party libs for. I transitioned from a data analyst to a data engineer, and the most important things for me where acquiring technical skills and finding the right organization that fostered continuous learning and opportunities. If your dataset is in a relational database than you could just use sql for the processing. 70-80 percent of his time is spent on cleaning the data itself. optimized for devops & modern agile: support automated testing & deployment & version control. Scaling data engineering teams with UI tools are pretty linear. Check out some sites like Japan-dev and Tokyo-dev. If you like to play factorio or have neat pipelines delivering properly data, enjoy watching little processes running smoothly and passing tests, that’s probably a job for you. Surely there must be a better way you can unify your code base to more easily add new data sources; that's the entertaining part for me. Adding data engineering where you can pull data from the back end, load it up into some database and let them build reports around it can be a godsend. varo bank holiday schedule Well the big picture idea is python is slow so write the stuff that needs to be fast in C/C++. The 2 to 3 in DE will give you that broad systems understanding on APIs, DB's, on premise vs cloud, vnets, physical networks, coding, etc. A data engineer uses the systems to automate data pipelines, treating the data as their primary asset/product. Other PII data can be restricted on a needs basis.