data engineer
Database infrastructure specialist
A data engineer builds and maintains the systems that collect, store, and process large amounts of data.
Where data scientists analyze data to extract insights, data engineers create the infrastructure that makes such analysis possible—pipelines, databases, and processing frameworks.
We need a data engineer to rebuild our ETL pipeline before the analysts can work with this data.
Long-distance fluid transport tube
- extractionAct of removing
- transformationThorough change in form
- loadingAdding weight or cargo
- batch processingRunning queued jobs automatically
streaming · dataflow · scheduling · ELT · Luigi
Large nonvenomous squeezer
- SQLDatabase manipulation language
- scriptingWriting executable commands
- GitDistributed version control system
- version controlTracks changes to files
debugging · APIs · unit testing · code review · notebooks · refactoring · functions · automation
Database manipulation language
- NoSQLSchema-free database systems
- schemasConceptual organizing structures
- indexingSystematic organization for retrieval
- normalizationAdjusting to a standard
relational · MySQL · Snowflake · Redshift · tables · queries · joins · partitioning
Czech-Jewish literary surname
- DockerApplication containerization platform
- containersObjects for holding things
- orchestrationScoring for instrumental ensemble
- HadoopOpen-source cluster framework
clusters · microservices · deployment · monitoring · logging · Prometheus · reliability · fault tolerance
Cloud computing platform
- AzureMicrosoft cloud computing service
- GCPGoogle's cloud computing platform
- serverlessCloud-managed compute model
- cloud storageOnline data preservation service
Lambda · EMR · Athena · cloud-native · auto-scaling · multi-cloud
Descriptive information about content
- complianceAdhering to requirements
- GDPREU data protection law
- access controlRestricting resource availability selectively
- documentationWritten proof or instructions
Great Expectations · testing · monitoring · alerting · audit trails
pipeline
Long-distance fluid transport tube
- extractionAct of removing
- transformationThorough change in form
- loadingAdding weight or cargo
batch processing · streaming · dataflow · scheduling · ELT · Luigi
Python
Large nonvenomous squeezer
- SQLDatabase manipulation language
- scriptingWriting executable commands
- GitDistributed version control system
version control · debugging · APIs · unit testing · code review · notebooks · refactoring · functions · automation · shell scripting
SQL
Database manipulation language
- NoSQLSchema-free database systems
- schemasConceptual organizing structures
- indexingSystematic organization for retrieval
normalization · relational · MySQL · Snowflake · Redshift · tables · queries · joins · partitioning · ACID
Kafka
Czech-Jewish literary surname
- DockerApplication containerization platform
- containersObjects for holding things
- orchestrationScoring for instrumental ensemble
Hadoop · clusters · microservices · deployment · monitoring · logging · Prometheus · reliability · fault tolerance · scalability
AWS
Cloud computing platform
- AzureMicrosoft cloud computing service
- GCPGoogle's cloud computing platform
- serverlessCloud-managed compute model
cloud storage · Lambda · EMR · Athena · cloud-native · auto-scaling · multi-cloud
metadata
Descriptive information about content
- complianceAdhering to requirements
- GDPREU data protection law
- access controlRestricting resource availability selectively
documentation · Great Expectations · testing · monitoring · alerting · audit trails
- engineers
- tech jobsprogrammer, developer, coder, software developers, IT support
- tech workersprogrammers, programmer, developers, software developer, coder
- motorsport rolesdriver, racing driver, pit crew, mechanic, crew chief
- math careersmathematician, statistician, math teacher, actuary, biostatistician
- recordkeepersarchivist, chronicler, registrar, recorder, scribe
data engineerdata analyst
Studies information for insights
business analyst · end user · data scientist · decision maker
data engineerUI designer
Creates digital interface layouts
web designer · mobile developer · graphic designer · product designer
data, engineer, engineering, engineered, engineers, reengineering, reengineer