Skip to content

[Volume-9] 상품 랭킹 기능 구현 - #396

Merged
kimsungho93 merged 35 commits into
loopers-labs:kimsungho93from
kimsungho93:volume-9
Jul 19, 2026
Merged

[Volume-9] 상품 랭킹 기능 구현#396
kimsungho93 merged 35 commits into
loopers-labs:kimsungho93from
kimsungho93:volume-9

Conversation

@kimsungho93

Copy link
Copy Markdown
Contributor

🧭 Context & Decision

문제 정의

  • 현재 동작/제약:

    • 상품 조회, 좋아요, 결제 완료 이벤트를 조합한 일간 인기 상품 랭킹이 필요했습니다.
    • 랭킹은 자주 조회되므로 요청마다 MySQL에서 집계하고 정렬하기에는 비용이 큽니다.
    • Redis에 최종 점수만 저장하면 장애나 가중치 변경 시 랭킹을 다시 만들기 어렵습니다.
  • 문제(또는 리스크):

    • Kafka 이벤트가 재전달되면 ZINCRBY 점수가 중복 가산될 수 있습니다.
    • 일간 랭킹을 매일 0점에서 시작하면 자정 직후 콜드 스타트가 발생합니다.
    • Redis 데이터가 유실되거나 오염됐을 때 당일 랭킹을 복구할 기준이 필요합니다.
    • 이벤트를 건별로 저장하면 MySQL과 Redis의 I/O가 증가합니다.
  • 성공 기준(완료 정의):

    • 조회, 좋아요, 주문 지표를 가중치에 따라 합산해 일간 랭킹을 제공합니다.
    • 중복 이벤트가 발생해도 점수는 한 번만 반영되어야 합니다.
    • 전날 랭킹의 10%를 이월해 콜드 스타트를 완화합니다.
    • Redis 랭킹을 삭제해도 MySQL의 시간별 지표로 다시 만들 수 있어야 합니다.
    • 재생성 중 비어 있거나 일부만 생성된 랭킹이 노출되지 않아야 합니다.

전체 구조

flowchart LR
    EVENT["상품 조회 · 좋아요 · 결제 완료"] --> KAFKA["Kafka<br/>catalog-events"]

    KAFKA --> RC["Ranking Consumer<br/>중복 확인 · 점수 계산"]
    KAFKA --> MC["Metrics Consumer<br/>상품 · 시간 단위 사전 집계"]

    RC --> REDIS[("Redis ZSET<br/>조회용 최종 점수")]
    MC --> MYSQL[("MySQL<br/>product_metric_hourly<br/>가중치 적용 전 지표")]

    REDIS --> API["Ranking API"]
    API --> USER["사용자"]

    MYSQL -.->|"장애 복구 · 가중치 변경"| JOB["Ranking Rebuild Job"]
    JOB -.->|"임시 키 완성 후 원자적 교체"| REDIS
Loading

선택지와 결정

대안 방식 장점 한계 판단
A. Redis 단독 ZSET에 최종 점수만 저장 구조가 단순하고 조회와 반영이 빠름 장애 복구와 가중치 재계산이 어려움 제외
B. MySQL 단독 조회 시 지표를 합산하고 정렬 영속성과 재계산 가능성 확보 조회마다 집계와 정렬 비용이 발생 제외
C. 하이브리드 Redis는 조회, MySQL은 재계산 재료 보관 빠른 조회와 복구 가능성을 함께 확보 저장소와 복구 경로가 늘어나 복잡도 증가 선택
  • 최종 결정:
    • Redis에는 사용자에게 제공할 일간 랭킹의 최종 점수를 저장합니다.
    • 점수는 조회 수 × 0.1 + 좋아요 증감 × 0.2 + (주문 금액 / 10,000) × 0.7로 계산합니다.
    • MySQL product_metric_hourly에는 가중치 적용 전 시간별 지표를 저장합니다.
    • Kafka poll에서 받은 이벤트는 상품과 시간 단위로 합산한 뒤 JDBC Batch로 반영합니다.
    • Redis에서는 Lua Script로 중복 확인, 점수 증가, TTL 설정을 원자적으로 처리합니다.
    • 전날 랭킹의 10%를 오늘 랭킹에 이월합니다.

랭킹 재생성 흐름

sequenceDiagram
    participant API as 랭킹 API
    participant LIVE as 현재 Redis 랭킹
    participant JOB as Rebuild Job
    participant DB as MySQL 시간별 지표
    participant TEMP as Redis 임시 랭킹

    API->>LIVE: 기존 랭킹 조회
    JOB->>DB: 대상 날짜의 시간별 지표 조회
    DB-->>JOB: 가중치 적용 전 집계값
    JOB->>TEMP: 점수 재계산
    JOB->>TEMP: 전날 랭킹 10% 반영
    JOB->>TEMP: 생성 결과 검증

    alt 검증 성공
        JOB->>LIVE: 임시 키를 원자적으로 교체
        API->>LIVE: 완성된 새 랭킹 조회
    else 검증 실패
        Note over LIVE: 기존 랭킹 유지
    end
Loading
  • 트레이드오프:

    • Redis 단독 구조보다 Kafka Consumer, 시간별 지표, 중복 처리 기록과 배치 작업이 추가됩니다.
    • 시간별 집계는 현재 랭킹 공식에는 충분하지만 원본 이벤트의 모든 정보를 보존하지는 않습니다.
    • 현재 가중치로는 재생성할 수 있지만, 가중치 버전이 없어 과거에 노출됐던 결과를 그대로 재현할 수는 없습니다.
    • Carry-over는 콜드 스타트를 완화하지만 전날의 인기가 오늘 랭킹에 영향을 줍니다.
  • 추후 개선 여지(있다면):

    • Kafka offset 또는 watermark를 이용해 재생성 대상 이벤트의 반영 완료 여부를 검증합니다.
    • 가중치 버전을 저장해 과거 공식과 현재 공식을 구분합니다.
    • 순 방문자나 사용자별 중복 제거가 필요해지면 원본 이벤트 보존을 검토합니다.
    • 상품 수가 커지면 Redis 메모리, Top-N 유지, 조회 결과 캐싱을 측정합니다.

🤔 고민한 점 / 막혔던 부분

  • product_metric_hourly를 SOT라고 부를 수 있는지 고민했습니다. 현재 랭킹을 재현하기에는 충분하지만 원본 정보 전체를 보존하지 않으므로, 시스템 전체의 SOT보다는 현재 유스케이스의 복구 기준 데이터 또는 Materialized View에 가깝다고 판단했습니다.

  • Kafka의 at-least-once 전달과 ZINCRBY는 자연스럽게 멱등하지 않습니다. Redis에서는 Lua Script와 handled event 집합을 사용하고, MySQL에서는 event_handled 기록과 지표 반영을 같은 트랜잭션으로 묶었습니다.

  • Kafka의 배치 크기 3,000은 이벤트가 모두 모일 때까지 기다린다는 의미가 아닙니다. 한 번의 poll에서 가져올 수 있는 최대 개수이며, 실제로 가져온 레코드는 개수와 관계없이 즉시 처리합니다.

  • 재생성 전에 기존 Redis 키를 삭제하면 사용자에게 빈 랭킹이 노출될 수 있습니다. 임시 키에서 새 랭킹을 완성하고 검증한 뒤 원자적으로 교체하도록 설계했습니다.

🙋 기타

  • 날짜와 페이지 조건을 받는 랭킹 목록 API를 추가했습니다.
  • 상품 상세 응답에 현재 순위를 추가했으며, 랭킹에 없는 상품은 null을 반환합니다.
  • 중복 이벤트, Carry-over 재실행, 랭킹 재생성과 원자적 교체를 검증하는 테스트를 추가했습니다.
  • 3,000건 이벤트 부하 시나리오와 Grafana 모니터링 환경을 추가했습니다.

@coderabbitai

coderabbitai Bot commented Jul 17, 2026

Copy link
Copy Markdown

Important

Review skipped

Too many files!

This PR contains 138 files, which is 38 over the limit of 100.

To get a review, narrow the scope:
• coderabbit review --type committed # exclude uncommitted changes
• coderabbit review --dir # limit to a subdirectory
• coderabbit review --base # compare against a closer base

Upgrade to a paid plan to raise the limit.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 86c85789-32af-4489-9fd7-2592a3d75a10

📥 Commits

Reviewing files that changed from the base of the PR and between 4dc460e and cee7356.

⛔ Files ignored due to path filters (2)
  • docker/grafana/screenshots/catalog-metrics-load-3000-live-processing.png is excluded by !**/*.png, !**/*.png and included by **
  • docker/grafana/screenshots/catalog-metrics-load-3000-live-top.png is excluded by !**/*.png, !**/*.png and included by **
📒 Files selected for processing (138)
  • apps/commerce-api/build.gradle.kts
  • apps/commerce-api/src/main/java/com/loopers/catalog/application/CatalogEventMessage.java
  • apps/commerce-api/src/main/java/com/loopers/catalog/application/CatalogEventPayload.java
  • apps/commerce-api/src/main/java/com/loopers/catalog/application/CatalogEventType.java
  • apps/commerce-api/src/main/java/com/loopers/catalog/application/listener/OrderPaidCatalogEventOutboxListener.java
  • apps/commerce-api/src/main/java/com/loopers/config/security/SecurityConfig.java
  • apps/commerce-api/src/main/java/com/loopers/config/time/TimeConfig.java
  • apps/commerce-api/src/main/java/com/loopers/payment/application/PaymentFacade.java
  • apps/commerce-api/src/main/java/com/loopers/payment/application/PaymentRecoveryResultHandler.java
  • apps/commerce-api/src/main/java/com/loopers/payment/application/event/OrderPaidEvent.java
  • apps/commerce-api/src/main/java/com/loopers/payment/application/event/OrderPaidItemSnapshot.java
  • apps/commerce-api/src/main/java/com/loopers/payment/application/event/OrderPaymentEventPublisher.java
  • apps/commerce-api/src/main/java/com/loopers/product/application/ProductDetailMetrics.java
  • apps/commerce-api/src/main/java/com/loopers/product/application/ProductDetailResult.java
  • apps/commerce-api/src/main/java/com/loopers/product/application/ProductFacade.java
  • apps/commerce-api/src/main/java/com/loopers/product/application/ProductListQuery.java
  • apps/commerce-api/src/main/java/com/loopers/product/infrastructure/CachedProductListQuery.java
  • apps/commerce-api/src/main/java/com/loopers/product/infrastructure/ProductListQueryDsl.java
  • apps/commerce-api/src/main/java/com/loopers/product/interfaces/api/ProductV1Controller.java
  • apps/commerce-api/src/main/java/com/loopers/product/interfaces/api/ProductV1Dto.java
  • apps/commerce-api/src/main/java/com/loopers/ranking/application/RankingEntries.java
  • apps/commerce-api/src/main/java/com/loopers/ranking/application/RankingFacade.java
  • apps/commerce-api/src/main/java/com/loopers/ranking/application/RankingItemInfo.java
  • apps/commerce-api/src/main/java/com/loopers/ranking/application/RankingMetrics.java
  • apps/commerce-api/src/main/java/com/loopers/ranking/application/RankingPosition.java
  • apps/commerce-api/src/main/java/com/loopers/ranking/application/RankingQuery.java
  • apps/commerce-api/src/main/java/com/loopers/ranking/application/RankingService.java
  • apps/commerce-api/src/main/java/com/loopers/ranking/infrastructure/RedisRankingQuery.java
  • apps/commerce-api/src/main/java/com/loopers/ranking/interfaces/api/RankingV1Controller.java
  • apps/commerce-api/src/main/java/com/loopers/ranking/interfaces/api/RankingV1Dto.java
  • apps/commerce-api/src/test/java/com/loopers/catalog/application/listener/OrderPaidCatalogEventOutboxListenerTest.java
  • apps/commerce-api/src/test/java/com/loopers/catalog/infrastructure/CatalogEventOutboxWriterImplTest.java
  • apps/commerce-api/src/test/java/com/loopers/payment/application/PaymentFacadeTest.java
  • apps/commerce-api/src/test/java/com/loopers/payment/application/PaymentRecoveryResultHandlerTest.java
  • apps/commerce-api/src/test/java/com/loopers/payment/application/event/OrderPaymentEventPublisherTest.java
  • apps/commerce-api/src/test/java/com/loopers/product/application/ProductFacadeTest.java
  • apps/commerce-api/src/test/java/com/loopers/product/infrastructure/ProductListQueryDslIntegrationTest.java
  • apps/commerce-api/src/test/java/com/loopers/product/interfaces/api/ProductV1ApiE2ETest.java
  • apps/commerce-api/src/test/java/com/loopers/ranking/application/RankingFacadeTest.java
  • apps/commerce-api/src/test/java/com/loopers/ranking/application/RankingMetricsTest.java
  • apps/commerce-api/src/test/java/com/loopers/ranking/application/RankingServiceTest.java
  • apps/commerce-api/src/test/java/com/loopers/ranking/infrastructure/RedisRankingQueryIntegrationTest.java
  • apps/commerce-api/src/test/java/com/loopers/ranking/interfaces/api/RankingV1ApiE2ETest.java
  • apps/commerce-api/src/test/java/com/loopers/ranking/interfaces/api/RankingV1ApiFailureE2ETest.java
  • apps/commerce-batch/build.gradle.kts
  • apps/commerce-batch/src/main/java/com/loopers/batch/job/ranking/RankingCarryOverJobConfig.java
  • apps/commerce-batch/src/main/java/com/loopers/batch/job/ranking/RankingCarryOverJobParameterValidator.java
  • apps/commerce-batch/src/main/java/com/loopers/batch/job/ranking/RankingCarryOverJobParameters.java
  • apps/commerce-batch/src/main/java/com/loopers/batch/job/ranking/RankingRebuildJobConfig.java
  • apps/commerce-batch/src/main/java/com/loopers/batch/job/ranking/RankingRebuildJobParameterValidator.java
  • apps/commerce-batch/src/main/java/com/loopers/batch/job/ranking/RankingRebuildJobParameters.java
  • apps/commerce-batch/src/main/java/com/loopers/batch/job/ranking/step/RankingCarryOverTasklet.java
  • apps/commerce-batch/src/main/java/com/loopers/batch/job/ranking/step/RankingRebuildTasklet.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/application/RankingCarryOverRepository.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/application/RankingCarryOverService.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/application/RankingColdStartProperties.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/application/RankingRebuildMetric.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/application/RankingRebuildMetricRepository.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/application/RankingRebuildRepository.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/application/RankingRebuildScore.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/application/RankingRebuildService.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/infrastructure/JdbcRankingRebuildMetricRepository.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/infrastructure/RankingCarryOverConfig.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/infrastructure/RedisRankingCarryOverRepository.java
  • apps/commerce-batch/src/main/java/com/loopers/ranking/infrastructure/RedisRankingRebuildRepository.java
  • apps/commerce-batch/src/main/resources/application.yml
  • apps/commerce-batch/src/test/java/com/loopers/CommerceBatchApplicationTest.java
  • apps/commerce-batch/src/test/java/com/loopers/batch/job/ranking/RankingCarryOverJobParametersTest.java
  • apps/commerce-batch/src/test/java/com/loopers/batch/job/ranking/RankingRebuildJobParametersTest.java
  • apps/commerce-batch/src/test/java/com/loopers/batch/job/ranking/step/RankingRebuildTaskletTest.java
  • apps/commerce-batch/src/test/java/com/loopers/job/ranking/RankingCarryOverJobConcurrencyE2ETest.java
  • apps/commerce-batch/src/test/java/com/loopers/job/ranking/RankingCarryOverJobE2ETest.java
  • apps/commerce-batch/src/test/java/com/loopers/job/ranking/RankingRebuildJobE2ETest.java
  • apps/commerce-batch/src/test/java/com/loopers/ranking/application/RankingCarryOverServiceTest.java
  • apps/commerce-batch/src/test/java/com/loopers/ranking/application/RankingColdStartPropertiesTest.java
  • apps/commerce-batch/src/test/java/com/loopers/ranking/application/RankingRebuildServiceTest.java
  • apps/commerce-batch/src/test/java/com/loopers/ranking/infrastructure/JdbcRankingRebuildMetricRepositoryIntegrationTest.java
  • apps/commerce-batch/src/test/java/com/loopers/ranking/infrastructure/RedisRankingCarryOverRepositoryIntegrationTest.java
  • apps/commerce-batch/src/test/java/com/loopers/ranking/infrastructure/RedisRankingRebuildRepositoryIntegrationTest.java
  • apps/commerce-streamer/build.gradle.kts
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/application/CatalogEventPayload.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/application/CatalogEventType.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/application/CatalogMetricsMetrics.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/application/ProductMetricDelta.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/application/ProductMetricEventCommand.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/application/ProductMetricEventHandler.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/application/ProductMetricHourlyDelta.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/application/ProductMetricHourlyRepository.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/application/ProductMetricsRepository.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/application/ProductOrderEventData.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/infrastructure/JdbcProductMetricHourlyRepository.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/infrastructure/JdbcProductMetricsRepository.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/infrastructure/ProductMetricHourly.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/infrastructure/ProductMetricHourlyId.java
  • apps/commerce-streamer/src/main/java/com/loopers/metrics/interfaces/consumer/CatalogMetricsConsumer.java
  • apps/commerce-streamer/src/main/java/com/loopers/ranking/application/RankingScoreBatch.java
  • apps/commerce-streamer/src/main/java/com/loopers/ranking/application/RankingScoreDelta.java
  • apps/commerce-streamer/src/main/java/com/loopers/ranking/application/RankingScoreEventHandler.java
  • apps/commerce-streamer/src/main/java/com/loopers/ranking/application/RankingScoreRepository.java
  • apps/commerce-streamer/src/main/java/com/loopers/ranking/infrastructure/RankingClockConfig.java
  • apps/commerce-streamer/src/main/java/com/loopers/ranking/infrastructure/RedisRankingScoreRepository.java
  • apps/commerce-streamer/src/main/java/com/loopers/ranking/interfaces/consumer/CatalogRankingConsumer.java
  • apps/commerce-streamer/src/main/resources/application.yml
  • apps/commerce-streamer/src/test/java/com/loopers/metrics/application/CatalogMetricsMetricsTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/metrics/application/ProductMetricDeltaTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/metrics/application/ProductMetricEventHandlerIntegrationTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/metrics/application/ProductMetricEventHandlerTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/metrics/application/ProductMetricHourlyDeltaTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/metrics/application/ProductOrderEventDataTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/metrics/infrastructure/MetricsJdbcRepositoryIntegrationTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/metrics/infrastructure/ProductMetricHourlyJdbcRepositoryIntegrationTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/metrics/interfaces/consumer/CatalogMetricsConsumerTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/ranking/application/RankingScoreEventHandlerTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/ranking/infrastructure/RedisRankingScoreRepositoryIntegrationTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/ranking/interfaces/consumer/CatalogRankingConsumerTest.java
  • apps/commerce-streamer/src/test/java/com/loopers/ranking/interfaces/consumer/CatalogRankingKafkaE2ETest.java
  • docker/grafana/README.md
  • docker/grafana/catalog-metrics-performance-test.md
  • docker/grafana/dashboards/catalog-metrics-batch.json
  • docker/grafana/prometheus.yml
  • docker/grafana/provisioning/dashboards/dashboard.yml
  • docker/grafana/provisioning/datasources/datasource.yml
  • docker/monitoring-compose.yml
  • modules/ranking/build.gradle.kts
  • modules/ranking/src/main/java/com/loopers/ranking/RankingExpirationPolicy.java
  • modules/ranking/src/main/java/com/loopers/ranking/RankingRedisKey.java
  • modules/ranking/src/main/java/com/loopers/ranking/RankingScorePolicy.java
  • modules/ranking/src/main/java/com/loopers/ranking/RankingWindow.java
  • modules/ranking/src/main/java/com/loopers/ranking/config/RankingScoreConfig.java
  • modules/ranking/src/main/java/com/loopers/ranking/config/RankingScoreProperties.java
  • modules/ranking/src/main/resources/ranking.yml
  • modules/ranking/src/test/java/com/loopers/ranking/RankingExpirationPolicyTest.java
  • modules/ranking/src/test/java/com/loopers/ranking/RankingRedisKeyTest.java
  • modules/ranking/src/test/java/com/loopers/ranking/RankingScorePolicyTest.java
  • modules/ranking/src/test/java/com/loopers/ranking/RankingWindowTest.java
  • scripts/catalog-metrics-load.ps1
  • scripts/catalog-ranking-e2e.ps1
  • settings.gradle.kts

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Review on demand using usage pricing

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@kimsungho93
kimsungho93 merged commit 1b881af into loopers-labs:kimsungho93 Jul 19, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant