Skip to content

YB-nt/eCommerceETL

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

32 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

E-Commerce Data ETL

python으로 만들어준 log,user,item,rating 데이터를 사용해서 ETL을 진행하는 프로젝트

제목 없음-2023-09-15-1742

Getting Started


docker-compose up -d

Option

Edit .env file

./envset.sh

Airflow

KakaoTalk_20230915_181304240

  • localhost:8080

Dag Graph

KakaoTalk_20230915_181231258

Spark

KakaoTalk_20230915_181115284
  • Product recommendation system using cosine similarity
  • localhost:7077
  • localhost:8081

Postgres

  • localhost:5432

Metabase

  • Setting custom Data visualization

KakaoTalk_20230915_092626468

  • localhost:3000

추가 예정

  • AWS s3 연결 (airflow s3hook) 사용해서 데이터 생성 및 저장
  • 생성데이터 늘려주고, AWS EMR 으로 데이터 처리
  • log 데이터 형식 변경 & 처리방식 변경

About

ECOMMERCE DATA ETL + RECOMMEND

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published